instead of manually looking for a tag a desired position and handling
pos increment in the loop, I think you could just simply get the value by index:
curpos = tags[int(pos_1) - 1].get('href', None)
count = count - 1 could be simplified as
count -= 1
- follow the PEP8
lower_case_with_underscores variable naming guideline
- what if you prefix the variable names containing user-defined values with
input_ (see below)?
and, I think a "for" loop with a negative step would be a simpler solution that the while loop here:
for count in range(int(Count_) + 1, 1, -1):
- Or, to bring it one step further, what if we apply a generally easier to follow recursive flow instead of the iterative approach you currently have. The base condition for the recursion could be the input count reaching 1. And, we'll improve on DRY with that function as well.
you don't have to call
.read() on the result of
BeautifulSoup also accepts file-like objects:
soup = BeautifulSoup(ur.urlopen(url), "html.parser")
lxml may help drastically improve HTML-parsing performance
- instead of using
urllib(), you could switch to
requests and re-use a session which would help avoid an overhead of re-establishing network connection to the host on every request
- you could use
SoupStrainer to let
BeautifulSoup parse only the
- you should also account for relative links and use
urljoin() to combine base urls and relative links
The code with the above and other improvements applied:
from urllib.parse import urljoin
from bs4 import BeautifulSoup, SoupStrainer
only_links = SoupStrainer('a')
def follow_link(session, url, position, count):
"""Follows a link at a given "position" "count" number of times."""
if count <= 1:
print('Retrieving: ', url)
response = session.get(url)
soup = BeautifulSoup(response.content, "lxml", parse_only=only_links)
links = soup('a')
next_url = links[position - 1].get('href', None)
return follow_link(session, urljoin(url, next_url), position, count - 1)
if __name__ == '__main__':
input_url = str(input('Enter URL- '))
input_count = int(input('Enter count: '))
input_position = int(input('Enter position: '))
with requests.Session() as session:
follow_link(session, input_url, input_position, input_count)
- what if there is no link at the desired position available on some page?
- if we once get a link which links itself, this code in this state would get stuck on this page alone until the
count is exchausted
The goal is to have cleaner code, with a #comment on each line (unless extremely basic) so as to improve my annotation habits.
Comments on each line could overall decrease readability of the code and they are, in essence, extra information and weight you need to make sure stay up to date with the actual code. Self-documented code is something you should strive to achieve, using comments as an extra measure used to explain the "why" some decisions were made and not the "how".