Skip to main content
added 6 characters in body
Source Link
Kate
  • 7.6k
  • 8
  • 21
stock_table = soup.find("tbody", {"class":"tv-data-table__tbody"})
rows = tablestock_table.findAll(lambda tag: tag.name=='tr')
for row in rows:
    columns = row.find_all('td')
    symbol_tag = row.find("a", {"class": "tv-screener__symbol"})
    if symbol_tag is None:
        symbol = "Not found"
    else:
        symbol = symbol_tag.get_text().strip()

    company_tag = row.find("span", {"class": "tv-screener__description"})
    if company_tag is None:
        company_name = "Not found"
    else:
        company_name = company_tag.get_text().strip()

    print(f"symbol: {symbol}, company name: {company_name}")
stock_table = soup.find("tbody", {"class":"tv-data-table__tbody"})
rows = table.findAll(lambda tag: tag.name=='tr')
for row in rows:
    columns = row.find_all('td')
    symbol_tag = row.find("a", {"class": "tv-screener__symbol"})
    if symbol_tag is None:
        symbol = "Not found"
    else:
        symbol = symbol_tag.get_text().strip()

    company_tag = row.find("span", {"class": "tv-screener__description"})
    if company_tag is None:
        company_name = "Not found"
    else:
        company_name = company_tag.get_text().strip()

    print(f"symbol: {symbol}, company name: {company_name}")
stock_table = soup.find("tbody", {"class":"tv-data-table__tbody"})
rows = stock_table.findAll(lambda tag: tag.name=='tr')
for row in rows:
    symbol_tag = row.find("a", {"class": "tv-screener__symbol"})
    if symbol_tag is None:
        symbol = "Not found"
    else:
        symbol = symbol_tag.get_text().strip()

    company_tag = row.find("span", {"class": "tv-screener__description"})
    if company_tag is None:
        company_name = "Not found"
    else:
        company_name = company_tag.get_text().strip()

    print(f"symbol: {symbol}, company name: {company_name}")
Source Link
Kate
  • 7.6k
  • 8
  • 21

I am bit short on time so I apologize for not doing a more comprehensive review of your code. But I think one area where you can improve is utilization of BeautifulSoup.

The selection method is clusmy, and you are addressing tags that are of no use to you. You can go straight to the DOM elements that matter to you and ignore the rest.

The only trick is finding the right selectors for the page. Here is some code to parse the main table:

stock_table = soup.find("tbody", {"class":"tv-data-table__tbody"})
rows = table.findAll(lambda tag: tag.name=='tr')
for row in rows:
    columns = row.find_all('td')
    symbol_tag = row.find("a", {"class": "tv-screener__symbol"})
    if symbol_tag is None:
        symbol = "Not found"
    else:
        symbol = symbol_tag.get_text().strip()

    company_tag = row.find("span", {"class": "tv-screener__description"})
    if company_tag is None:
        company_name = "Not found"
    else:
        company_name = company_tag.get_text().strip()

    print(f"symbol: {symbol}, company name: {company_name}")

Output:

symbol: MSFT, company name: Microsoft Corp.
symbol: AAPL, company name: Apple Inc
symbol: AMZN, company name: AMAZON COM INC
symbol: GOOG, company name: Alphabet Inc (Google) Class C
symbol: GOOGL, company name: Alphabet Inc (Google) Class A
symbol: BABA, company name: Alibaba Group Holdings Ltd.
symbol: FB, company name: FACEBOOK INC
symbol: BRK.A, company name: BERKSHIRE HATHAWAY INC
...

I think you can easily complete the rest. Note that in this code I am skipping the headers because I selected tbody instead of table. Otherwise the first row would return None upon find, but I am handling the case as you can see.

What would be good is handle exceptions, and also if a tag is not found don't ignore the error but investigate and fix your code to make it more reliable. The HTML of that page will certainly change at some point and you should watch out for changes.

Since you use both find and find_all, keep in mind that they behave differently:

If find_all() can’t find anything, it returns an empty list. If find() can’t find anything, it returns None

Source: BS4 doc

find should be used when you are expecting to find only one matching element, not find_all.

FYI Pandas can also load HTML tables, just this line of code will give you something:

pandas.read_html(url)
[                                           Unnamed: 0  Unnamed: 1 Unnamed: 2  Unnamed: 3  Unnamed: 4 Unnamed: 5 Unnamed: 6 Unnamed: 7 Unnamed: 8 Unnamed: 9            Unnamed: 10
0                               MSFT  Microsoft Corp.      174.78      0.73%        1.26  Strong Buy     7.936M  1328.701B      29.21       5.80  144000.00    Technology Services
1                                     AAPL  Apple Inc      280.77      1.69%        4.67         Buy     8.914M  1229.641B      21.20      12.75  137000.00  Electronic Technology
2                                AMZN  AMAZON COM INC     2409.76      1.96%       46.27         Buy     1.602M  1202.053B     101.14      23.48  798000.00           Retail Trade
3                 GOOG  Alphabet Inc (Google) Class C     1286.47      1.84%       23.26  Strong Buy   343.776K   884.984B      24.73      49.61  118899.00    Technology Services
4                GOOGL  Alphabet Inc (Google) Class A     1281.35      1.82%       22.94  Strong Buy   479.905K   880.654B      24.65      49.61  118899.00    Technology Services
..                                                ...         ...        ...         ...         ...        ...        ...        ...        ...        ...                    ...
95  BDXA  BECTON DICKINSON & CO DEP SHS REPSTG 1/2...       63.21      0.32%        0.20  Strong Buy    25.530K    72.338B      22.20       2.76   70093.00      Health Technology
96                                  SHOP  SHOPIFY INC      621.56     -0.80%       -5.00         Buy     1.448M    72.324B          —      -1.11          —           Retail Trade
97                               MO  ALTRIA GROUP INC       38.59      2.06%        0.78        Sell     1.394M    71.761B          —      -0.70    7300.00  Consumer Non-Durables
98                        VRTX  VERTEX PHARMACEUTICAL      276.21      2.54%        6.84  Strong Buy   371.397K    71.657B      58.33       4.58    3000.00      Health Technology
99  RDS.A  ROYAL DUTCH SHELL ADR EA REP 2 CL'A' EU...       35.89      2.95%        1.03         Buy     2.025M    71.269B       8.44       3.93          —        Energy Minerals

[100 rows x 11 columns]]

But since some cleanup is required (parsing a & span tags) you might want to stick with BS (personally I would).