Data retrieval from Dynamic HTML page with time-out (Web scraping w. Python)

Question

The HTML page shows list of a friend network of a person (each Name has anchor <a> tag w. link to list of friend network). Since the page has a timer, I've written Python code to scrape the mth position (friend) of the nth count (page) by traversing through the cycle: (m->n->m->n....). And it works!

import urllib.request, urllib.parse, urllib.error
from bs4 import BeautifulSoup
import ssl

# Ignore SSL certificate errors
ctx = ssl.create_default_context()
ctx.check_hostname = False
ctx.verify_mode = ssl.CERT_NONE

url = input('Enter URL: ')
position = int(input('Enter position: '))  #Name/link Traverse 
count = int(input('Enter count: ')) #Page Traverse    
print("Retrieving:", url)
for c in range(count):      #returns range of indices
    html = urllib.request.urlopen(url, context=ctx).read() #opening URL
    soup = BeautifulSoup(html, 'html.parser')
    a_tags=soup('a')
    link=a_tags[position-1].get('href', None) #url = href(key) value pair
    content=a_tags[position-1].contents     #name=a_tag.contents
    url=link
    print("Retrieving:", url)

Input:

Enter URL: http://py4e-data.dr-chuck.net/known_by_Kory.html
Enter position: 1
Enter count: 10

Output:

Retrieving: http://py4e-data.dr-chuck.net/known_by_Kory.html
Retrieving: http://py4e-data.dr-chuck.net/known_by_Shaurya.html
Retrieving: http://py4e-data.dr-chuck.net/known_by_Raigen.html
Retrieving: http://py4e-data.dr-chuck.net/known_by_Dougal.html
Retrieving: http://py4e-data.dr-chuck.net/known_by_Aonghus.html
Retrieving: http://py4e-data.dr-chuck.net/known_by_Daryn.html
Retrieving: http://py4e-data.dr-chuck.net/known_by_Pauline.html
Retrieving: http://py4e-data.dr-chuck.net/known_by_Laia.html
Retrieving: http://py4e-data.dr-chuck.net/known_by_Iagan.html
Retrieving: http://py4e-data.dr-chuck.net/known_by_Leanna.html
Retrieving: http://py4e-data.dr-chuck.net/known_by_Malakhy.html

Questions:

1. Is there a better way to approach this? (libraries, workarounds to delay the timer)

2. My goals is to make an exhaustive 'list' of friends of all unique Names here; I don't want any code, just suggestions and approaches will do.

Answer 1

Currently you only get one new name per page request. However, the page contains all people known by that person. So I would implement a queue of yet to be visited people and a set of people already visited. If there are no more people left to visit, you have found all people (assuming there are no disjoint sets of people and that this is your actual goal).

In addition, using requests is usually a bit more user-friendly than using urllib directly.

import requests
from bs4 import BeautifulSoup, SoupStrainer
import re

STRAINER = SoupStrainer("a")

def get_name(url):
    match = re.match(r"http://py4e-data.dr-chuck.net/known_by_(.*).html", url)
    if match is not None:
        return match.groups()[0]

def find_all_people(start_url):
    with requests.Session() as session:
        queue = set([start_url])
        visited = set()
        while queue:
            url = queue.pop()
            visited.add(url)
            print(len(visited), "/", len(visited) + len(queue), url)
            response = session.get(url)
            soup = BeautifulSoup(response.text, "lxml", parse_only=STRAINER)
            queue.update(a["href"]
                         for a in soup.select("a")
                         if a["href"] not in visited)
        return list(map(get_name, visited))

if __name__ == "__main__":
    url = "http://py4e-data.dr-chuck.net/known_by_Kory.html"
    people = find_all_people(url)
    print(len(people))

This uses a request.Session to keep the connection alive, speeding it up a tiny bit. It also has a if __name__ == "__main__": guard to allow importing from this script from another script without the code running, a bs4.SoupStrainer to only parse the parts of the page needed and it uses the faster lxml parser.

It still takes quite some time to find all people. Finding that there are probably 5754 people takes only a few seconds.