This code scrapes post links (among other information) from a table on a forum. While the current code works, I would like to know if there is a better/simpler way of writing it (maybe not as many for-loops?) and with the assumption that additional functionality will be added, such as:

  • Currently only scrapes page 1, but eventually scrape pages 2-134 too.
  • Scrape each individual post (e.g., each text post, which user posted it, date/time it was posted, etc.)
  • Store data to dataframe/csv instead of printing it

Here is the current code:

import requests
from bs4 import BeautifulSoup

# Set Soup
url = 'http://www.cornerofberkshireandfairfax.ca/forum/investment-ideas'
headers = {'User-Agent': 'Mozilla/5.0'}
page = requests.get(url, headers=headers)
soup = BeautifulSoup(page.content, 'html.parser')

# Set table and rows
table = soup.find_all("table")[0]
rows = table.find_all("tr")

# Scrape each row of the table
for row in rows:

    # Set table columns and classes
    subject_column = row.find_all("td", class_="subject windowbg2")
    stats_column = row.find_all("td", class_="stats windowbg")
    last_post_column = row.find_all("td", class_="lastpost windowbg2")

    # Scrape company name and first page
    for company in subject_column:
        company = company.find_all("a")[0]
        page_one = company['href']
        print("Company: " + company.text)
        print("- Page One: " + page_one)

    # Scrape username and profile link
    for started_by_user in subject_column:
        started_by_user = started_by_user.find_all("a")[1]
        user_profile = started_by_user['href']
        print("- Started by User: " + started_by_user.text)
        print("- User profile:" + user_profile)

    # Scrape number of views and replies
    for stats in stats_column:
        stats = stats.get_text().split()
        replies = stats[0]
        views = stats[2]
        print("- Replies: " + replies)
        print("- Views: " + views)

    # Scrape link for last post
    for last_post in last_post_column:
        last_post = last_post.find_all("a")[0]
        last_post = last_post['href']
        print("- Last Post: " + last_post)

  • \$\begingroup\$ Welcome to Code Review! I removed the pandas tag since you are not using pandas anywhere in the code at the moment. \$\endgroup\$ – AlexV Jun 26 at 19:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.