This is a follow up to my original post:

I've written a class that will start from a random Wikipedia page, then choose the first link in the main body, and then navigate following the links until it finds the Philosophy page. When I run the testCrawler() method, it crawls starting from 20 pages and then plots the lengths of all of the paths. This generally works but I just want to confirm that the code looks clean/intuitive.

I implemented the vast majority of the suggestions and came up with the following code. I just wanted to make sure that everything looks fine stylistically.

import sys
from urlparse import urljoin

import requests
from lxml.html import fromstring
from bs4 import BeautifulSoup,NavigableString, Tag
import matplotlib.pyplot as plt
import scipy
import scipy.stats


class Crawler:
    """ Class used to crawl wikipedia pages starting from a random article.
    def __init__(self):
        self.base_url = "https://en.wikipedia.org"
        self.NUM_PAGES_TO_CRAWL = 500

    def get_valid_link(self, curr_response):
        """Takes an html response and returns the first link in the main body of the article.
        curr_root = BeautifulSoup(curr_response.text,"lxml")
        first = curr_root.select_one("#mw-content-text") # locate main body
        if not first:
            return None
        par = first.find_all("p",recursive = False,limit = 10)
        heading = curr_root.select_one("#firstHeading").text
        heading = reformat_string('(',heading)
        first_paragraph_found = False
        head_tokens = tokenize(heading)

        # Find which paragraph has the first link
        i = 0
        for i in range(len(par)):
            if par[i].b is not None:
                bold = ""
                for string in par[i].find_all("b"):
                    bold += " " + string.text
                bold = reformat_string('(', bold)
                bold_tokens = tokenize(bold)
                heading_match = check_name_match(head_tokens,bold_tokens)
                if heading_match:
                    first_paragraph_found = True
                if heading_match and par[i].a:
            if par[i].a is not None:
                anchor = par[i].a.text
                if anchor:
                    anchor = reformat_string('(', anchor)
                    a_tokens = tokenize(anchor)
                    heading_match = check_name_match(head_tokens,a_tokens)
                    if heading_match:
            if first_paragraph_found and par[i].a:
            i += 1

        # if none of the paragraphs have a link and article contains only a list
        if i >= len(par)-1 and first_paragraph_found:
            u_list = first.find_all('ul')
                return u_list[0].li.a['href']
            except (IndexError, AttributeError,TypeError):
                return None
        elif i >= len(par)-1:# Reached article with no main body
            return None

        main_body_idx = i
        stack = []
        # Find the first link before or after parentheses 
        for child in par[main_body_idx].children:
            if isinstance(child,NavigableString):
                if "(" in child:
                if ")" in child:
                    except IndexError: # html malformed
                        return None

            if isinstance(child, Tag) and child.name == "a" and not stack:
                link = child['href']        
                link = reformat_string('#',link)
                    return str(link)
                except KeyError: # Reached article with no main body
                    return None

    def crawl_to_philosophy(self, start_url,session):
        """Follow the path of each url until the philosophy page is reached and return the path.
        link_path = []
        # Get first link
            init_response = session.get(start_url)
        except requests.exceptions.RequestException as e: # bad link
            return None

        init_link = self.get_valid_link(init_response)
        if not init_link:
            return None
        link_path.append(urljoin(self.base_url, init_link))

        # Follow path of links until the philosophy page is reached
        i = 0
        while True:
            if "philosophy" in  link_path[i].lower():
                curr_response = session.get(link_path[i])
            except requests.exceptions.RequestException as e: # bad link
                return None 

            curr_link = self.get_valid_link(curr_response)
            if not curr_link or "redlink" in curr_link:
                return None
            new_link = urljoin(self.base_url, curr_link)
            for i in range(len(link_path)):
                if new_link in link_path[i] : # loop found
                    return None
            i += 1
        return link_path

    def find_paths_to_philosophy(self,url):
        """Find paths starting from 500 links.
        i = 0
        crawl_list = []
        with requests.Session() as s:
            while i < self.NUM_PAGES_TO_CRAWL:
                path = self.crawl_to_philosophy(url,s)
                if path is not None:
                    i += 1

def plot_lengths(lens):
    """Plot the distribution of path lengths.
    freq = {}
    max_len = 0

    for length in lens:
        max_len = max(length,max_len)
        if length in freq:
            freq[length] += 1
            freq[length] = 1

    max_freq = max(freq, key=freq.get)
    bins = range(0, max_len + 1, 2)
    plt.hist(lens,bins,histtype = 'bar',rwidth = 0.8)
    plt.ylabel('Path Lengths')
    plt.title('Distribution of path lengths')
    dist_names = ['gamma', 'beta', 'rayleigh', 'norm', 'pareto']

    for dist_name in dist_names:
        dist = getattr(scipy.stats, dist_name)
        param = dist.fit(lens)
        pdf_fitted = dist.pdf(bins, *param[:-2], loc=param[-2], scale=param[-1]) * max_len
        plt.plot(pdf_fitted, label=dist_name)

    plt.legend(loc='upper right')

# Utility functions used by Crawler class

def reformat_string(char, word):
    """Remove passed in char from a string and convert its characters to lowercase.
    word = word.lower()
    char_idx = word.find(char)
    if char_idx != -1:
        return word[:char_idx]
    return word

def check_name_match(heading, string):
    """Determine whether or not any part of the article heading is in the string and vice versa.
    for i in range(len(string)):
        for j in range(len(heading)):
            if heading[j] in string[i] or string[i] in heading[j]:
                return True
    return False

def tokenize(word):
    """Split the passed in 'word' on space characters and return a list of tokens.
    tokens = []
    curr_word = ""
    for i in range(len(word)):
        if word[i] == " " and i == len(word)-1:
            tokens.append(word.strip(" "))
            return tokens
        curr_word += word[i]
        if word[i] == " " :
            curr_word = ""
        if i == len(word)-1:
            return tokens

if __name__ == "__main__":
    url = "https://en.wikipedia.org/wiki/Special:Random"
    crawler = Crawler()
  • 4
    \$\begingroup\$ All crawlers should obey the robots.txt protocol. Wikipedia's version is here file. You are going to piss somebody off otherwise. \$\endgroup\$ Apr 3, 2017 at 4:25

1 Answer 1



  1. Docstrings, should follow PEP257, so basically your docstrings should say:

Take an html response....

instead of:

Takes an html response

Also, you should think about using one line dosctrings, I don't see much of reasons why do you do the multiline in your code.


Two spaces should be used for inline comments, so this:

except KeyError: # Reached article with no main body

should become:

except KeyError:  # Reached article with no main body
  1. Catching exceptions:

If you are not going to work with an exception that you don't have to store in in the variable, e.g:

except requests.exceptions.RequestException as e:  # bad link
    return None

So just catch it like this:

except requests.exceptions.RequestException:
    return None

There are also other issues, like missing spaces, newlines, so I suggest you install flake8 and run it against your project to find out all the issues, I also suggest you install plugins such as flake8-import-order and flake8-docstrings

Code improvements

Class Crawler

def __init__(self):
    self.base_url = "https://en.wikipedia.org"
    self.NUM_PAGES_TO_CRAWL = 500

Those two attributes are not related to the instance, well at least, they don't depend on the instance, so it makes sense to move it out of __init__ and make them class variables/constants.

Removing those 2 variables from __init__, will also make __init__ itself not needed, which indicates that your class now is just a namespace and you don't need a class anymore, since there is no state in your instances.

The method get_valid_link is way too complex, you've already split it into smaller pieces by newlines and comments, but it's better if you define separate methods for that.


The function check_name_match can be replaced by set.intersection. See the documentation for sets.

Function tokenize, as far as I understand it, can just be replaced by str.split, the only difference is that your tokenize will also add trailing space to each of your tokens, which I think is not needed.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.