I'm working through a scraping function where pages of results lead to product pages. I've added a default maximum number of results pages, and pages per set of results, to prevent a simple mistake of crawling the whole site.
Here's what I have so far. Does the way I'm implementing the maximums with the for
loops make sense? Is there a more "pythonic" way? I'm coming at this from a completely learning perspective.
def my_crawler(url, max_pages = 1, max_items = 1):
for page_number in range(1, max_pages + 1):
source_code = requests.get(url + str(page_number)).text
products = SoupStrainer(class_ = 'productTags')
soup = BeautifulSoup(source_code, 'html.parser', parse_only=products)
for item_number, a in enumerate(soup.find_all('a')):
print(str(item_number) + ': ' + a['href'])
if item_number == max_items - 1: break
my_crawler('http://www.thesite.com/productResults.aspx?&No=')