I am bit short on time so I apologize for not doing a more comprehensive review of your code. But I think one area where you can improve is utilization of BeautifulSoup.
The selection method is clusmy, and you are addressing tags that are of no use to you. You can go straight to the DOM elements that matter to you and ignore the rest.
The only trick is finding the right selectors for the page. Here is some code to parse the main table:
stock_table = soup.find("tbody", {"class":"tv-data-table__tbody"})
rows = table.findAll(lambda tag: tag.name=='tr')
for row in rows:
columns = row.find_all('td')
symbol_tag = row.find("a", {"class": "tv-screener__symbol"})
if symbol_tag is None:
symbol = "Not found"
else:
symbol = symbol_tag.get_text().strip()
company_tag = row.find("span", {"class": "tv-screener__description"})
if company_tag is None:
company_name = "Not found"
else:
company_name = company_tag.get_text().strip()
print(f"symbol: {symbol}, company name: {company_name}")
Output:
symbol: MSFT, company name: Microsoft Corp.
symbol: AAPL, company name: Apple Inc
symbol: AMZN, company name: AMAZON COM INC
symbol: GOOG, company name: Alphabet Inc (Google) Class C
symbol: GOOGL, company name: Alphabet Inc (Google) Class A
symbol: BABA, company name: Alibaba Group Holdings Ltd.
symbol: FB, company name: FACEBOOK INC
symbol: BRK.A, company name: BERKSHIRE HATHAWAY INC
...
I think you can easily complete the rest. Note that in this code I am skipping the headers because I selected tbody
instead of table
. Otherwise the first row would return None
upon find
, but I am handling the case as you can see.
What would be good is handle exceptions, and also if a tag is not found don't ignore the error but investigate and fix your code to make it more reliable. The HTML of that page will certainly change at some point and you should watch out for changes.
Since you use both find
and find_all
, keep in mind that they behave differently:
If find_all()
can’t find anything, it returns an empty list. If find()
can’t find anything, it returns None
Source: BS4 doc
find
should be used when you are expecting to find only one matching element, not find_all
.
FYI Pandas can also load HTML tables, just this line of code will give you something:
pandas.read_html(url)
[ Unnamed: 0 Unnamed: 1 Unnamed: 2 Unnamed: 3 Unnamed: 4 Unnamed: 5 Unnamed: 6 Unnamed: 7 Unnamed: 8 Unnamed: 9 Unnamed: 10
0 MSFT Microsoft Corp. 174.78 0.73% 1.26 Strong Buy 7.936M 1328.701B 29.21 5.80 144000.00 Technology Services
1 AAPL Apple Inc 280.77 1.69% 4.67 Buy 8.914M 1229.641B 21.20 12.75 137000.00 Electronic Technology
2 AMZN AMAZON COM INC 2409.76 1.96% 46.27 Buy 1.602M 1202.053B 101.14 23.48 798000.00 Retail Trade
3 GOOG Alphabet Inc (Google) Class C 1286.47 1.84% 23.26 Strong Buy 343.776K 884.984B 24.73 49.61 118899.00 Technology Services
4 GOOGL Alphabet Inc (Google) Class A 1281.35 1.82% 22.94 Strong Buy 479.905K 880.654B 24.65 49.61 118899.00 Technology Services
.. ... ... ... ... ... ... ... ... ... ... ...
95 BDXA BECTON DICKINSON & CO DEP SHS REPSTG 1/2... 63.21 0.32% 0.20 Strong Buy 25.530K 72.338B 22.20 2.76 70093.00 Health Technology
96 SHOP SHOPIFY INC 621.56 -0.80% -5.00 Buy 1.448M 72.324B — -1.11 — Retail Trade
97 MO ALTRIA GROUP INC 38.59 2.06% 0.78 Sell 1.394M 71.761B — -0.70 7300.00 Consumer Non-Durables
98 VRTX VERTEX PHARMACEUTICAL 276.21 2.54% 6.84 Strong Buy 371.397K 71.657B 58.33 4.58 3000.00 Health Technology
99 RDS.A ROYAL DUTCH SHELL ADR EA REP 2 CL'A' EU... 35.89 2.95% 1.03 Buy 2.025M 71.269B 8.44 3.93 — Energy Minerals
[100 rows x 11 columns]]
But since some cleanup is required (parsing a & span tags) you might want to stick with BS (personally I would).