Simple Python Recursive Web Scraper

Question

I tried to make a simple recursive web scraper using Python. My idea was to grab all the links, titles and tag names.

Website: https://lifebridgecapital.com/podcast/

Course of Action:

Grab all the tags links from the Website.

tag_words_links(Website) --> [https://lifebridgecapital.com/tag/multifamily/][2]

My script fetches all the links, tag names and titles from those links which tag_words_links returned. Some of these pages have pagination and some don't, so I used an if condition to catch those pages which contain class="page-numbers".

By looking at the code, anyone can see clearly there is a lot of repetition going on in there, therefore I'd like to keep it DRY. Any suggestions and ideas are much appreciated.

Here is the code:

from requests_html import HTMLSession
import csv
import time


def tag_words_links(url):
    global _session
    _request = _session.get(url)
    tags = _request.html.find('a.tag-cloud-link')
    links = []
    for link in tags:
        links.append(link.find('a', first=True).attrs['href'])
    
    return links

def parse_tag_links(link):
    global _session
    _request = _session.get(link)
    article_links = _request.html.find('h3 a')
    tag_names = [tag.text for tag in _request.html.find('div.infinite-page-caption')]
    articles = [article.find('a', first=True).attrs['href'] for article in article_links]
    titles = [title.text for title in _request.html.find('h3.gdlr-core-blog-title')]
    if 'class="page-numbers"' in _request.text:
        next_page = _request.html.find('a.page-numbers')
        url = {url.find('a', first=True).attrs['href'] for url in next_page}
        for page in url:
            next_page_request = _session.get(page)
            article_links = next_page_request.html.find('h3 a')
            for article in article_links:
                articles.append(article.find('a', first=True).attrs['href'])
            for title in article_links:
                titles.append([title.text for title in title.find('h3.gdlr-core-blog-title')])
            for tags in article_links:
                tag_names.append([tags for tags in tags.find('div.infinite-page-caption')])

    scraped_data = {
        'Title': titles,
        'Tag_Name': tag_names,
        'Link': articles
    }


    return scraped_data


if __name__ == '__main__':
    data = []
    _session = HTMLSession()
    url = 'https://lifebridgecapital.com/podcast/'
    links = tag_words_links(url)
    for link in links:
        data.append(parse_tag_links(link))
        time.sleep(2)

    with open('life-bridge-capital-tags.csv', 'w', newline='', encoding='utf-8') as csv_file:
        writer = csv.DictWriter(csv_file, fieldnames=data[0].keys())
        writer.writeheader()
        for row in data:
            writer.writerow(row)
    

Answer 1

First: if you only cared about the episode links, there's no reason to scrape. They have an RSS feed that is not only machine-readable and structured (unlike the HTML), it's structured better than the resulting CSV you've produced, and will in all likelihood be more reliable and fast to retrieve, surviving web markup changes since RSS has a standardized schema. If there's no particular reason for you to use CSV, the entire exercise is reduced to downloading and saving the above XML - a single curl or wget call.

However, my original read of your question was wrong - you care mostly about the tags and nothing else. Good news - you still shouldn't scrape. This is a WordPress site with an accessible API, so just get https://lifebridgecapital.com/wp-json/wp/v2/tags?per_page=100 . There's even highly-comprehensive, built-in documentation here: https://lifebridgecapital.com/wp-json/

Answer 2

Here are the changes I made:

1. removed tag_word_links function and replaced it with a visual block. It just added extra code and indirection.

2. passed in session explicitly as argument instead of referencing it as a global (globals should generally be avoided).

3. Used upper case for URL as it is a constant.

4. renamed the parse_tag_links to get_data as it is more general than the name suggested.

5. Used list comprehensions more consistently. Not only are they more elegant and readable, but code is less interspersed, and easier to follow, e.g. the data = [] declaration long before building the data list is no longer needed.

6. more descriptive and consistent variable names.

7. corrected incorrect use of request with page in variable naming (page is what the response to the request returns and not the request itself)

8. avoid looping through article_links 3 times (and using different variable names each time for the same) and do it in one pass.

9. avoid unnecessary nested for loops; instead once the article_links list is created, the dependant lists can be built in a separate top level for loop.

10. Avoid underscores for variables. Underscores in python are generally used for class methods that are not part of a class's API. Specifically calling a certain local variable private doesn't really make any sense (even less if you're using the variable as a global!)

    from requests_html import HTMLSession
    import csv
    import time
    
    
    
    def get_data(session, link):
        main_page = session.get(link)
        article_links = main_page.html.find('h3 a')
    
        if 'class="page-numbers"' in main_page.text:
            next_page_numbers = main_page.html.find('a.page-numbers')
            next_page_links = [number.find('a', first=True).attrs['href'] for number in next_page_numbers]
            for link in next_page_links:
                next_page = session.get(link)
                article_links += next_page.html.find('h3 a')
    
        articles, titles, tag_names = [], [], []
        for article in article_links:
            articles.append(article)
            titles.extend([title.text for title in article.find('h3.gdlr-core-blog-title')])
            tag_names.extend([tag.text for tag in article.find('div.infinite-page-caption')])
    
        time.sleep(2)
        return {
            'titles': titles,
            'tag_names': tag_names,
            'links': articles
        }
    
    
    if __name__ == '__main__':
        session = HTMLSession()
        URL = 'https://lifebridgecapital.com/podcast/'
    
        request = session.get(URL)
        tags = request.html.find('a.tag-cloud-link')
        links = [link.find('a', first=True).attrs['href'] for link in tags]
    
        data = [get_data(session, link) for link in links]
    
        with open('life-bridge-capital-tags.csv', 'w', newline='', encoding='utf-8') as csv_file:
            writer = csv.DictWriter(csv_file, fieldnames=data[0].keys())
            writer.writeheader()
            for row in data:
                writer.writerow(row)