# Data retrieval from Dynamic HTML page with time-out (Web scraping w. Python)

The HTML page shows list of a friend network of a person (each Name has anchor <a> tag w. link to list of friend network). Since the page has a timer, I've written a py code to scrap the mth position (friend) of the nth count (page) by traversing through the cycle: (m->n->m->n....). And it works!

import urllib.request, urllib.parse, urllib.error
from bs4 import BeautifulSoup
import ssl

# Ignore SSL certificate errors
ctx = ssl.create_default_context()
ctx.check_hostname = False
ctx.verify_mode = ssl.CERT_NONE

url = input('Enter URL: ')
position = int(input('Enter position: '))  #Name/link Traverse
count = int(input('Enter count: ')) #Page Traverse
print("Retrieving:", url)
for c in range(count):      #returns range of indices
html = urllib.request.urlopen(url, context=ctx).read() #opening URL
soup = BeautifulSoup(html, 'html.parser')
a_tags=soup('a')
link=a_tags[position-1].get('href', None) #url = href(key) value pair
content=a_tags[position-1].contents     #name=a_tag.contents
print("Retrieving:", url)


Input:

Enter URL: http://py4e-data.dr-chuck.net/known_by_Kory.html
Enter position: 1
Enter count: 10


Output:

Retrieving: http://py4e-data.dr-chuck.net/known_by_Kory.html
Retrieving: http://py4e-data.dr-chuck.net/known_by_Shaurya.html
Retrieving: http://py4e-data.dr-chuck.net/known_by_Raigen.html
Retrieving: http://py4e-data.dr-chuck.net/known_by_Dougal.html
Retrieving: http://py4e-data.dr-chuck.net/known_by_Aonghus.html
Retrieving: http://py4e-data.dr-chuck.net/known_by_Daryn.html
Retrieving: http://py4e-data.dr-chuck.net/known_by_Pauline.html
Retrieving: http://py4e-data.dr-chuck.net/known_by_Laia.html
Retrieving: http://py4e-data.dr-chuck.net/known_by_Iagan.html
Retrieving: http://py4e-data.dr-chuck.net/known_by_Leanna.html
Retrieving: http://py4e-data.dr-chuck.net/known_by_Malakhy.html


Questions:

1. Is there a better way to approach this? (libraries, workarounds to delay the timer)

2. My goals is to make an exhaustive 'list' of friends of all unique Names here; I don't want any code, just suggestions and approaches will do.

• What do you mean with "the page has a timer"? Once you have retrieved the page content, the website cannot magically change the string in memory (unlike when visiting the page with a browser, where dynamic elements can exist). – Graipher Mar 21 at 8:00
• I mean that the page has a timer when opened in a browser; is there a better way to approach this problem, or should I just proceed like how I've done so far. Just looking for a specific library that can open each link and scrap content from the 'n' pages for me, without me looping through. – Sumanth Lazarus Mar 21 at 8:37

Currently you only get one new name per page request. However, the page contains all people known by that person. So I would implement a queue of yet to be visited people and a set of people already visited. If there are no more people left to visit, you have found all people (assuming there are no disjoint sets of people and that this is your actual goal).

In addition, using requests is usually a bit more user-friendly than using urllib directly.

import requests
from bs4 import BeautifulSoup, SoupStrainer
import re

STRAINER = SoupStrainer("a")

def get_name(url):
match = re.match(r"http://py4e-data.dr-chuck.net/known_by_(.*).html", url)
if match is not None:
return match.groups()[0]

def find_all_people(start_url):
with requests.Session() as session:
queue = set([start_url])
visited = set()
while queue:
url = queue.pop()
print(len(visited), "/", len(visited) + len(queue), url)
response = session.get(url)
soup = BeautifulSoup(response.text, "lxml", parse_only=STRAINER)
queue.update(a["href"]
for a in soup.select("a")
if a["href"] not in visited)
return list(map(get_name, visited))

if __name__ == "__main__":
url = "http://py4e-data.dr-chuck.net/known_by_Kory.html"
people = find_all_people(url)
print(len(people))


This uses a request.Session to keep the connection alive, speeding it up a tiny bit. It also has a if __name__ == "__main__": guard to allow importing from this script from another script without the code running, a bs4.SoupStrainer to only parse the parts of the page needed and it uses the faster lxml parser.

It still takes quite some time to find all people. Finding that there are probably 5754 people takes only a few seconds.