Reach certain Wiki page through random links

Question

I just read about Wiki game on Wikipedia and wanted to implement it as a fun project for myself using python.

Task:To reach Mathematics Wikipedia page using wikipedia hyperlinks randomly.

Using Beautiful Soup documentation,I was able to create following crude code which is quite slow for my purpose.

import random
import requests
from bs4 import BeautifulSoup

r = requests.get("https://en.wikipedia.org/wiki/Main_Page")
soup = BeautifulSoup(r.content) 
a = [link.get('href') for link in soup.find_all('a')]
del a[0:6]
while(soup.title.text != 'Mathematics - Wikipedia'):
    a = [link.get('href') for link in soup.find_all('a')]
    del a[0:6]

    if (a == []):
        add = "/wiki/Special:Random"
    else: 
        add = random.choice(a)
        if(add == None):
            add = random.choice(a)

    url = "https://en.wikipedia.org/"+add
    r = requests.get(url)
    soup = BeautifulSoup(r.content)
    print(soup.title.text)

First,I would like to reduce the useless links that this program goes through such as About Wiki,user login,etc. I knew I can simply use if-else statements but wanted to know if there is another simpler or faster implementation for it in beautiful soup or something else.

Secondly,Any suggestions to improve code would be welcome including additional reading resources.

Answer 1

Wikipedia has an API available. You can make calls to e.g. "https://en.wikipedia.org/w/api.php?action=query&prop=links&titles={title}&pllimit=500" and get back a list of titles the Wikipedia page title links to, without all the internal links (but with non-existing pages and things like categories).

Incidentally, there is even a python package available that makes using it a lot easier, Wikipedia-API. With this your code would become:

import requests
from bs4 import BeautifulSoup
import wikipediaapi
import random
from itertools import count

WIKI = wikipediaapi.Wikipedia('en')

def random_page():
    r = requests.get("https://en.wikipedia.org/wiki/Special:Random")
    soup = BeautifulSoup(r.content)
    page = WIKI.page(soup.title.text[:-12])
    assert page.exists()
    return page

BLACKLISTED = ["Wikipedia:", "Category:", "Template:", "Template talk:", "User:",
               "User talk:", "Module:", "Help:", "File:", "Portal:"]
def blacklisted(title):
    return any(title.startswith(x) for x in BLACKLISTED)

def random_walk(target_title):
    page = random_page()
    for i in count():
        print(i, page)
        if page.title == target_title:
            return i
        links = list(page.links.values())
        # print(f"{len(links)} links")
        if not links:
            return None
        page = random.choice(links)
        while blacklisted(page.title) or not page.exists():
            page = random.choice(links)


if __name__ == "__main__":
    print(random_walk('Mathematics'))

This still needs to blacklist some pages (see BLACKLISTED constant, the content of which I found by trial-and-error) and I'm not quite happy about the trial and error way of getting a random page, but filtering for existing pages needs to fetch them all, which is quite slow.

In any case, this should be a bit faster than actually getting the whole page and parsing it yourself.

I also put the code into function and guarded the calling under a if __name__ == "__main__": to allow reusing of the code. In doing this I took the liberty of adding a counter and returning it in case the target page is found (12000 pages and counting...).

---

Another question is if this is the best/most fun way to play this game. The way I know it, the goal is to reach a target page as fast as possible. In that case you will want to do a breadth-first search of all links instead (depth-first would fail because there are loops, meaning you could get lost forever). Although I have to admit, it is fun watching the bot and seeing where it wanders...

A good compromise might be checking if any of the links on the current page is to Mathematics, and only randomly jumping to the next page if not.

---

Some generic comments about your code:

• You don't need parenthesis around your conditions, and they are discouraged unless needed to make the condition span multiple lines.
• Empty lists are falsey, so you can replace if (a == []): with if not a:.
• Your code only tries twice to get a random page, which will work most of the time but will fail eventually. Instead use a while loop that continues indefinitely.