Wiki Link Mapper — A webcrawler which crawls links recursively and prints them into a graph

Question

I would like some suggestions on how to make it better and also if any features that i should add to this Script. I am using beautifulsoup,requests for scraping and networkx for plotting the graph.

https://github.com/HardikM1999/WikiLinkMapper

import requests
from bs4 import BeautifulSoup
import sys
import networkx as nx 
import matplotlib.pyplot as plt

#Taking Arguments
#Starting string
start = sys.argv[1]

#No of repetitions
rep = int(sys.argv[2])
tot_list = []

#Creating graph
G = nx.Graph()
G.add_node(start)

#recur function to scrape links
def recur(url,depth,tot_list,parent,rep):
    if(depth<=rep):
        page = requests.get('https://www.wikipedia.org/wiki/' + url)
        #parsing the page
        soup = BeautifulSoup(page.text,'html.parser')
        cnt = 0
        #list for particular depth
        depth_list = []
        for para_tag in soup.select('p'):
            for anchor_tag in para_tag.select('a'):
                if cnt>rep:
                    break
                #getting the link
                check_string = anchor_tag['href']
                if check_string.startswith('#cite') == False and check_string.startswith('/wiki/Help') == False and check_string.startswith('/wiki///en:') == False and check_string.startswith('/wiki/Wikipedia') == False :
                    cnt = cnt + 1
                    depth_list.append(check_string[6:])
            if cnt>rep:
                break
        for link in depth_list:
            tot_list.append((link,parent))  
            recur(link,depth+1,tot_list,link,rep)
    return tot_list


tot_list = recur(start,0,tot_list,start,rep)
node_sizes = []
node_sizes.append(100*len(start))

#Adding tot_list to the graph
for a,b in tot_list:
    G.add_node(a)
    G.add_edge(a,b)
    node_sizes.append(100*len(a))

#Drawing the Graph
nx.draw(G,node_color = 'orange',node_size = node_sizes,with_labels = True)
plt.draw()
plt.show()

Answer 1

Wikipedia contains circular references. Currently you follow them around until your max depth is reached. Instead, keep a set of all sites visited so far.

You also don't need this function to be recursive, you can make it iterative like this:

from collections import deque

def build_link_graph(start, max_pages=10):
    queue = deque([start])
    visited = set()
    i = 0
    while queue and i < max_pages:
        next_url = queue.popleft()
        urls = get_links(next_url)
        i += 1
        # make sure not to visit any page twice
        queue.extend(url for url in urls if url not in visited)  
        visited.update(urls)
        yield next_url, urls

The actual parsing can also be simplified using the CSS selector p > a, meaning all a tags which are contained in a p tag. I would also use a requests.Session to keep the connection alive and use the lxml parser, both of which slightly speed up the scraping.

def filter_links(links):
    blacklist = ['#cite', '/wiki/Help', '/wiki///en:', '/wiki/Wikipedia']                                   
    return [link for link in links                    
            if not any(link.startswith(prefix) for prefix in blacklist)]

def get_links(url):
    page = SESSION.get(url)
    soup = BeautifulSoup(page.text, "lxml")
    links = [a["href"]
             for a in soup.select("p > a")
             if "selflink" not in a.get("class", [])]
    return [BASE_URL + link for link in filter_links(links)]

With these functions, building the graph is rather easy. Note that you might want to choose a directed graph, since links are usually unidirectional. Also, networkx automatically adds missing nodes when adding edges, so no need to do that.

import sys

if __name__ == "__main__":
    BASE_URL = "'https://en.wikipedia.org'"
    if len(sys.argv) == 2:
        start = sys.argv[1]
    else:
        start = BASE_URL + '/wiki/Earthquake_engineering'
    SESSION = requests.Session()
    g = nx.DiGraph()
    for url, urls in build_link_graph(start):
        graph.add_edges_from((url, link) for link in urls)

I added a if __name__ == "__main__": guard to allow importing from this script from another script without running the scraping

---

Note that if you increase max_pages too much, you will run into a too many requests.response. Currently there is no code to handle that, but you might add some. The easiest way is to just time.sleep some time and try again afterwards. There are also other ways, some of which may not be in compliance with Wikipedia's ToS:

# robots.txt for Wikipedia and friends
# Please note: There are a lot of pages on this site, and there are
# some misbehaved spiders out there that go _way_ too fast. If you're
# irresponsible, your access to the site may be blocked.

There may be other ways to build this graph than scraping. Wikipedia supplies databases to download which should contain everything you need: https://en.wikipedia.org/wiki/Wikipedia:Database_download Note that it is 58 GB when uncompressed, though.