I'm scraping a table and am trying to get nested td tags down the tbody tree but the code seems kind of verbose. Is there a more Pythonic way to do this?

def get_top_subreddits(url):

    r = urllib.request.urlopen(url).read()
    soup = BeautifulSoup(r, "lxml")

    body = soup.find_all('tbody')

    top_subreddits = []
    for i in body:
        trs = i.find_all('tr')
        for tr in trs:
            tds = tr.find_all('td')
            for td in tds:
                texts = td.get_text()
                if '/r/' in texts:
    return top_subreddits
  • \$\begingroup\$ What does the HTML look like? Are there any <td> that appear outside a <tr> in a <tbody>? \$\endgroup\$ – 200_success Sep 19 '17 at 4:47

Yes, there is a more concise way to do it - using CSS selectors and a list comprehension:

top_subreddits = [
    for td in soup.select("tbody tr td")
    if '/r/' in td.text

tbody tr td would locate all td elements under tr elements which are under tbody.

I don't really like getting texts of td elements twice here, there is possibly a way to filter the desired information directly. E.g. if you were up to the subreddit links, we could've applied the /r/ check inside a selector:

top_subreddits = [
    for a in soup.select('tbody tr td a[href^="/r/"]')

^= here means "starts with".

|improve this answer|||||

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.