# Wiki Link Mapper — A webcrawler which crawls links recursively and prints them into a graph

I would like some suggestions on how to make it better and also if any features that i should add to this Script. I am using beautifulsoup,requests for scraping and networkx for plotting the graph.

import requests
from bs4 import BeautifulSoup
import sys
import networkx as nx
import matplotlib.pyplot as plt

#Taking Arguments
#Starting string
start = sys.argv[1]

#No of repetitions
rep = int(sys.argv[2])
tot_list = []

#Creating graph
G = nx.Graph()

def recur(url,depth,tot_list,parent,rep):
if(depth<=rep):
page = requests.get('https://www.wikipedia.org/wiki/' + url)
#parsing the page
soup = BeautifulSoup(page.text,'html.parser')
cnt = 0
#list for particular depth
depth_list = []
for para_tag in soup.select('p'):
for anchor_tag in para_tag.select('a'):
if cnt>rep:
break
check_string = anchor_tag['href']
if check_string.startswith('#cite') == False and check_string.startswith('/wiki/Help') == False and check_string.startswith('/wiki///en:') == False and check_string.startswith('/wiki/Wikipedia') == False :
cnt = cnt + 1
depth_list.append(check_string[6:])
if cnt>rep:
break

tot_list = recur(start,0,tot_list,start,rep)
node_sizes = []
node_sizes.append(100*len(start))

for a,b in tot_list:
node_sizes.append(100*len(a))

#Drawing the Graph
nx.draw(G,node_color = 'orange',node_size = node_sizes,with_labels = True)
plt.draw()
plt.show()


Wikipedia contains circular references. Currently you follow them around until your max depth is reached. Instead, keep a set of all sites visited so far.

You also don't need this function to be recursive, you can make it iterative like this:

from collections import deque

queue = deque([start])
visited = set()
i = 0
while queue and i < max_pages:
next_url = queue.popleft()
i += 1
# make sure not to visit any page twice
queue.extend(url for url in urls if url not in visited)
visited.update(urls)
yield next_url, urls


The actual parsing can also be simplified using the CSS selector p > a, meaning all a tags which are contained in a p tag. I would also use a requests.Session to keep the connection alive and use the lxml parser, both of which slightly speed up the scraping.

def filter_links(links):
blacklist = ['#cite', '/wiki/Help', '/wiki///en:', '/wiki/Wikipedia']
if not any(link.startswith(prefix) for prefix in blacklist)]

page = SESSION.get(url)
soup = BeautifulSoup(page.text, "lxml")
for a in soup.select("p > a")
if "selflink" not in a.get("class", [])]


With these functions, building the graph is rather easy. Note that you might want to choose a directed graph, since links are usually unidirectional. Also, networkx automatically adds missing nodes when adding edges, so no need to do that.

import sys

if __name__ == "__main__":
BASE_URL = "'https://en.wikipedia.org'"
if len(sys.argv) == 2:
start = sys.argv[1]
else:
start = BASE_URL + '/wiki/Earthquake_engineering'
SESSION = requests.Session()
g = nx.DiGraph()


I added a if __name__ == "__main__": guard to allow importing from this script from another script without running the scraping

Note that if you increase max_pages too much, you will run into a too many requests.response. Currently there is no code to handle that, but you might add some. The easiest way is to just time.sleep some time and try again afterwards. There are also other ways, some of which may not be in compliance with Wikipedia's ToS:

# robots.txt for Wikipedia and friends
# Please note: There are a lot of pages on this site, and there are
# some misbehaved spiders out there that go _way_ too fast. If you're