The HTML page shows list of a friend network of a person (each Name has anchor <a> tag w. link to list of friend network). Since the page has a timer, I've written Python code to scrape the mth position (friend) of the nth count (page) by traversing through the cycle: (m->n->m->n....). And it works!

import urllib.request, urllib.parse, urllib.error
from bs4 import BeautifulSoup
import ssl

# Ignore SSL certificate errors
ctx = ssl.create_default_context()
ctx.check_hostname = False
ctx.verify_mode = ssl.CERT_NONE

url = input('Enter URL: ')
position = int(input('Enter position: '))  #Name/link Traverse 
count = int(input('Enter count: ')) #Page Traverse    
print("Retrieving:", url)
for c in range(count):      #returns range of indices
    html = urllib.request.urlopen(url, context=ctx).read() #opening URL
    soup = BeautifulSoup(html, 'html.parser')
    link=a_tags[position-1].get('href', None) #url = href(key) value pair
    content=a_tags[position-1].contents     #name=a_tag.contents
    print("Retrieving:", url)


Enter URL: http://py4e-data.dr-chuck.net/known_by_Kory.html
Enter position: 1
Enter count: 10


Retrieving: http://py4e-data.dr-chuck.net/known_by_Kory.html
Retrieving: http://py4e-data.dr-chuck.net/known_by_Shaurya.html
Retrieving: http://py4e-data.dr-chuck.net/known_by_Raigen.html
Retrieving: http://py4e-data.dr-chuck.net/known_by_Dougal.html
Retrieving: http://py4e-data.dr-chuck.net/known_by_Aonghus.html
Retrieving: http://py4e-data.dr-chuck.net/known_by_Daryn.html
Retrieving: http://py4e-data.dr-chuck.net/known_by_Pauline.html
Retrieving: http://py4e-data.dr-chuck.net/known_by_Laia.html
Retrieving: http://py4e-data.dr-chuck.net/known_by_Iagan.html
Retrieving: http://py4e-data.dr-chuck.net/known_by_Leanna.html
Retrieving: http://py4e-data.dr-chuck.net/known_by_Malakhy.html


  1. Is there a better way to approach this? (libraries, workarounds to delay the timer)

  2. My goals is to make an exhaustive 'list' of friends of all unique Names here; I don't want any code, just suggestions and approaches will do.

  • 2
    \$\begingroup\$ What do you mean with "the page has a timer"? Once you have retrieved the page content, the website cannot magically change the string in memory (unlike when visiting the page with a browser, where dynamic elements can exist). \$\endgroup\$
    – Graipher
    Mar 21, 2019 at 8:00
  • 1
    \$\begingroup\$ I mean that the page has a timer when opened in a browser; is there a better way to approach this problem, or should I just proceed like how I've done so far. Just looking for a specific library that can open each link and scrap content from the 'n' pages for me, without me looping through. \$\endgroup\$
    – Sumax
    Mar 21, 2019 at 8:37
  • \$\begingroup\$ Welcome to Code Review! Please don't comment comments asking for clarification or additional information: edit your post. \$\endgroup\$
    – greybeard
    Mar 21, 2019 at 16:16

1 Answer 1


Currently you only get one new name per page request. However, the page contains all people known by that person. So I would implement a queue of yet to be visited people and a set of people already visited. If there are no more people left to visit, you have found all people (assuming there are no disjoint sets of people and that this is your actual goal).

In addition, using requests is usually a bit more user-friendly than using urllib directly.

import requests
from bs4 import BeautifulSoup, SoupStrainer
import re

STRAINER = SoupStrainer("a")

def get_name(url):
    match = re.match(r"http://py4e-data.dr-chuck.net/known_by_(.*).html", url)
    if match is not None:
        return match.groups()[0]

def find_all_people(start_url):
    with requests.Session() as session:
        queue = set([start_url])
        visited = set()
        while queue:
            url = queue.pop()
            print(len(visited), "/", len(visited) + len(queue), url)
            response = session.get(url)
            soup = BeautifulSoup(response.text, "lxml", parse_only=STRAINER)
                         for a in soup.select("a")
                         if a["href"] not in visited)
        return list(map(get_name, visited))

if __name__ == "__main__":
    url = "http://py4e-data.dr-chuck.net/known_by_Kory.html"
    people = find_all_people(url)

This uses a request.Session to keep the connection alive, speeding it up a tiny bit. It also has a if __name__ == "__main__": guard to allow importing from this script from another script without the code running, a bs4.SoupStrainer to only parse the parts of the page needed and it uses the faster lxml parser.

It still takes quite some time to find all people. Finding that there are probably 5754 people takes only a few seconds.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.