3
\$\begingroup\$

This is my first Web Scraping project in which I am retrieving the current stock information from here. This program works as expected, but I would certainly think someone with more experience with the language and web scraping could improve it.

#Imports
from bs4 import BeautifulSoup
from colorama import Fore as F
from time import sleep
import requests
import webbrowser
import pandas
import functools
import subprocess
from os import system
import geoip2.database
#Uses Maxmind GeoLite2-City Database for IP Location

#Compatible with most *nix systems only.  Please leave feedback if compatability for Windows is wanted.
#Should I make a function to check internet connection or just let an error arise?
#Beginning of program messages
print("""
 \033[32m /$$$$$$ 
 /$$__  $$
| $$  \__/
|  $$$$$$ \033[34m_____             ______         
 \033[32m\____  $$\033[34m__  /________________  /_________
 \033[32m/$$  \ $$\033[34m_  __/  __ \_  __ \_  //_/_  ___/
\033[32m|  $$$$$$/\033[34m/ /_ / /_/ /  / / /  ,<  _(__  )
 \033[32m\______/ \033[34m\__/ \____//_/ /_//_/|_| /____/

    """)
print(F.BLUE + "[!]Enlarge window as much as possible for easier observations" + F.RESET)
sleep(2)

#subprocess.run("clear")
#Variables
stock_chart = {"Value": False, "Data": False}
#Functions
def internet_test():
    proc = subprocess.Popen("ping google.com",
                            stdout=subprocess.PIPE,
                            stderr=subprocess.PIPE,
                            shell=True,
                            universal_newlines=True)
    if proc.returncode == 0:
        return True
    return False
def display(df):
    formatters = {}
    for li in list(df.columns):
        max = df[li].str.len().max()
        form = "{{:<{}s}}".format(max)
        formatters[li] = functools.partial(str.format, form)
    print(F.LIGHTGREEN_EX + df.to_string(formatters=formatters,
                                         index=False,
                                         justify="left"))


def search_df(search_str: str, df: pandas.DataFrame) -> pandas.DataFrame:
    results = pandas.concat([df[df["Symbol"].str.contains(search_str.upper())], df[df["Company"].str.contains(search_str,case=False)]])
    return results



#Function for fetching stocks, returns pandas.DataFrame object containing stock info
#Stocks pulled from https://www.tradingview.com/markets/stocks-usa/market-movers-large-cap
def stocks():
    #Set pandas options
    pandas.set_option("display.max_rows", 1000)
    pandas.set_option("display.max_columns", 1000)
    pandas.set_option("display.width", 1000)

    headers = {"User-Agent": "Mozilla/5.0 (Windows NT 6.3; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko)"
               " Chrome/80.0.3987.149 Safari/537.36"}

    #Make Request to site
    site = requests.get("https://www.tradingview.com/markets/stocks-usa/market-movers-large-cap", headers)

    #BeautifulSoup Object
    soup = BeautifulSoup(site.content, "html.parser")

    #Process to go achieve a list of stocks !!!SUGGESTIONS FOR EFICIENCY!!!
    html = list(soup.children)[3]
    body = list(html.children)[3]
    div = list(body.children)[5]
    div2 = list(div.children)[9]
    div3 = list(div2.children)[1]
    div4 = list(div3.children)[3]
    div5 = list(div4.children)[1]
    div6 = list(div5.children)[3]
    div7 = list(div6.children)[3]
    div8 = list(div7.children)[1]
    table = list(div8.children)[1]
    tbody = list(table.children)[3]
    stocks = tbody.find_all("tr")
    chart = {"Symbol": [], "Company": [], "Price Per Share": [], "Change(%)": [], "Change(Points)": []}

    #Find each component of stock and put it into a chart
    for stock in stocks:
        symbol = list(stock.find("td").find("div").find("div"))[1].get_text()
        name = stock.find("td").find("div").find("div").find("span").get_text().strip()
        last_price = "$" + stock.find_all("td")[1].get_text()
        change_percent = stock.find_all("td")[2].get_text()
        change_points = stock.find_all("td")[3].get_text()
        chart["Symbol"].append(symbol)
        chart["Company"].append(name)
        chart["Price Per Share"].append(last_price)
        chart["Change(%)"].append(change_percent)
        chart["Change(Points)"].append(change_points)

    panda_chart = pandas.DataFrame(chart)
    return panda_chart


def ip_info(ip):
    print(F.YELLOW + "[!]IP information is approximate.  Please use IPv6 for more accurate results.")
    try:
        reader = geoip2.database.Reader("GeoLite2-City.mmdb")
        print(F.GREEN + "[√]Database Loaded")
    except FileNotFoundError:
        print(F.RED + "[!]Could not open database; Exiting application")
        exit(1)
    #subprocess.run("clear")
    response = reader.city(ip)
    print(F.LIGHTBLUE_EX + """
    ISO Code: {iso}
    Country Name: {country}
    State: {state}
    City: {city}
    Postal Code: {post}
    Latitude: {lat}
    Longitude: {long}
    Network: {net}""".format(iso=response.country.iso_code, country=response.country.name,
                             state=response.subdivisions.most_specific.name, city=response.city.name,
                             post=response.postal.code, lat=response.location.latitude, long=response.location.longitude,
                             net=response.traits.network))
    print("\n\nEnter \"q\" to go back to menu or \"op\" to open predicted location in Google Maps.", end="\n\n\n\n\n\n")
    while True:
        inp = input()
        if inp == "q":
            break
        elif inp == "op":
            webbrowser.open(f"https://www.google.com/maps/search/{response.location.latitude},{response.location.longitude}", new=0)
            break

#Main
def main():
    try:
        global stock_chart
        internet = internet_test()
        print("""\033[33mOptions:

          \033[94m[1] - Display a chart of popular stocks
          [2] - Search a chart of popular stocks
          [3] - Locate an Internet Protocol (IP) Address
        """)
        while True:
            choice = input(F.YELLOW + "Enter Option Number[1-3]> " + F.WHITE)
            if choice in ["1", "2", "3"]:
                break
            print(F.RED + "[!]Option invalid")
        if choice in ["1", "2"]:
            if not stock_chart["Value"]:
                stock_chart["Value"] = True
                stock_chart["Data"] = stocks()
            if choice == "1":
                display(stock_chart["Data"])
            else:
                search = input(F.LIGHTBLUE_EX + "Enter name to search for> ")
                display(search_df(search, stock_chart["Data"]))
                sleep(1)
        else:
            ip_addr = input(F.GREEN + "Enter an Internet Protocol (IP) Address[IPv4 or IPv6]> ")
            try:
                ip_info(ip_addr)
            except ValueError:
                print(F.RED + "IP Address invalid")
                sleep(1)
        main()
    except KeyboardInterrupt:
        print(F.RED + "[!]Exiting..." + F.RESET)



if __name__ == "__main__":
    main()
\$\endgroup\$
3
\$\begingroup\$

I am bit short on time so I apologize for not doing a more comprehensive review of your code. But I think one area where you can improve is utilization of BeautifulSoup.

The selection method is clusmy, and you are addressing tags that are of no use to you. You can go straight to the DOM elements that matter to you and ignore the rest.

The only trick is finding the right selectors for the page. Here is some code to parse the main table:

stock_table = soup.find("tbody", {"class":"tv-data-table__tbody"})
rows = stock_table.findAll(lambda tag: tag.name=='tr')
for row in rows:
    symbol_tag = row.find("a", {"class": "tv-screener__symbol"})
    if symbol_tag is None:
        symbol = "Not found"
    else:
        symbol = symbol_tag.get_text().strip()

    company_tag = row.find("span", {"class": "tv-screener__description"})
    if company_tag is None:
        company_name = "Not found"
    else:
        company_name = company_tag.get_text().strip()

    print(f"symbol: {symbol}, company name: {company_name}")

Output:

symbol: MSFT, company name: Microsoft Corp.
symbol: AAPL, company name: Apple Inc
symbol: AMZN, company name: AMAZON COM INC
symbol: GOOG, company name: Alphabet Inc (Google) Class C
symbol: GOOGL, company name: Alphabet Inc (Google) Class A
symbol: BABA, company name: Alibaba Group Holdings Ltd.
symbol: FB, company name: FACEBOOK INC
symbol: BRK.A, company name: BERKSHIRE HATHAWAY INC
...

I think you can easily complete the rest. Note that in this code I am skipping the headers because I selected tbody instead of table. Otherwise the first row would return None upon find, but I am handling the case as you can see.

What would be good is handle exceptions, and also if a tag is not found don't ignore the error but investigate and fix your code to make it more reliable. The HTML of that page will certainly change at some point and you should watch out for changes.

Since you use both find and find_all, keep in mind that they behave differently:

If find_all() can’t find anything, it returns an empty list. If find() can’t find anything, it returns None

Source: BS4 doc

find should be used when you are expecting to find only one matching element, not find_all.

FYI Pandas can also load HTML tables, just this line of code will give you something:

pandas.read_html(url)
[                                           Unnamed: 0  Unnamed: 1 Unnamed: 2  Unnamed: 3  Unnamed: 4 Unnamed: 5 Unnamed: 6 Unnamed: 7 Unnamed: 8 Unnamed: 9            Unnamed: 10
0                               MSFT  Microsoft Corp.      174.78      0.73%        1.26  Strong Buy     7.936M  1328.701B      29.21       5.80  144000.00    Technology Services
1                                     AAPL  Apple Inc      280.77      1.69%        4.67         Buy     8.914M  1229.641B      21.20      12.75  137000.00  Electronic Technology
2                                AMZN  AMAZON COM INC     2409.76      1.96%       46.27         Buy     1.602M  1202.053B     101.14      23.48  798000.00           Retail Trade
3                 GOOG  Alphabet Inc (Google) Class C     1286.47      1.84%       23.26  Strong Buy   343.776K   884.984B      24.73      49.61  118899.00    Technology Services
4                GOOGL  Alphabet Inc (Google) Class A     1281.35      1.82%       22.94  Strong Buy   479.905K   880.654B      24.65      49.61  118899.00    Technology Services
..                                                ...         ...        ...         ...         ...        ...        ...        ...        ...        ...                    ...
95  BDXA  BECTON DICKINSON & CO DEP SHS REPSTG 1/2...       63.21      0.32%        0.20  Strong Buy    25.530K    72.338B      22.20       2.76   70093.00      Health Technology
96                                  SHOP  SHOPIFY INC      621.56     -0.80%       -5.00         Buy     1.448M    72.324B          —      -1.11          —           Retail Trade
97                               MO  ALTRIA GROUP INC       38.59      2.06%        0.78        Sell     1.394M    71.761B          —      -0.70    7300.00  Consumer Non-Durables
98                        VRTX  VERTEX PHARMACEUTICAL      276.21      2.54%        6.84  Strong Buy   371.397K    71.657B      58.33       4.58    3000.00      Health Technology
99  RDS.A  ROYAL DUTCH SHELL ADR EA REP 2 CL'A' EU...       35.89      2.95%        1.03         Buy     2.025M    71.269B       8.44       3.93          —        Energy Minerals

[100 rows x 11 columns]]

But since some cleanup is required (parsing a & span tags) you might want to stick with BS (personally I would).

| improve this answer | |
\$\endgroup\$
  • \$\begingroup\$ Thank you for the BS help as it was troubling for me to find examples that I could use \$\endgroup\$ – unkn0wn.dev May 8 at 19:00
  • \$\begingroup\$ Also what is the lambda function doing in `stock_table.findAlll \$\endgroup\$ – unkn0wn.dev May 8 at 19:34
  • \$\begingroup\$ If you are not familiar with lambda functions in Python here is an intro: Lambda functions. Think of it as a shortcut method when you want to write an inline function without fully defining it using def. I could have written findAll('tr') simply. On this page you will find some examples of findAll used in conjunction with lambda, that will better show how lambda can be used to write succinct code. \$\endgroup\$ – Anonymous May 8 at 20:47
  • \$\begingroup\$ Alright thank you for the references, I was thinking that findAll('tr') would work, but I wasn't for sure. \$\endgroup\$ – unkn0wn.dev May 9 at 1:53
2
\$\begingroup\$

Treating users like adults

This is personal opinion, but

[!]Enlarge window as much as possible for easier observations

is the kind of thing that I'm sure users can figure out, and is better left unstated.

Artificial hangs

This kind of sleep:

sleep(2)

after a prompt is not what I consider good user interface design. It's halfway between 'press any key to continue', which can be useful if the user needs to pause and look at stuff; and to simply not sleep and get on with life. In this case I think the latter is more appropriate.

Internet test

It doesn't appear that this is actually used for anything. Why is it here? You say

I forgot to incorporate the internet test into the main function

but I'm not convinced that it should be incorporated, or exist at all. The standard thing for a script like this is to assume that the internet is accessible, and if network calls fail, error or retry as appropriate.

Even if you did want to test within the program that the network is available, there is a better thing to do - try a connection to tradingview.com. It's what you actually care about.

Expression complexity

    results = pandas.concat([df[df["Symbol"].str.contains(search_str.upper())], df[df["Company"].str.contains(search_str,case=False)]])

should be broken up onto multiple lines.

Iterating over a list

Why is this:

for li in list(df.columns):

cast to a list? You can probably just iterate over columns directly.

Shadowing

In this:

    max = df[li].str.len().max()

do not name a variable max, since there is already a built-in with the same name.

Element selection

It's very doubtful that this:

html = list(soup.children)[3]
body = list(html.children)[3]
div = list(body.children)[5]

(etc.) is the best way to select these elements. Go back through the webpage and identify, based on the attributes of the elements and the structure of the DOM, the most specific and simple way to identify what you need. For example, the collection of tr for the main table can be accessed via the CSS selector

#js-screener-container tbody > tr

This, and this alone, should be enough to select all of the tr that you're interested in if you pass it to soup.select.

You'll want to similarly reduce your other selected elements to use more meaningful paths through the DOM.

String interpolation

It can simplify your format call here; note the leading f:

f"""
    ISO Code: {response.country.iso_code}
    Country Name: {response.country.name}
    etc
"""

Set membership

if choice in ["1", "2", "3"]

can be

if choice in {"1", "2", "3"}

It's technically higher-performance although you certainly won't see a difference. Also it captures your meaning better: "If the choice is in this set of things, where the order doesn't matter".

IP?

What is this program actually doing, other than looking up stocks? Why is there an ip_info feature? This seems like it has absolutely nothing to do with stocks and should be a separate script.

| improve this answer | |
\$\endgroup\$
  • \$\begingroup\$ Alright thank you for the feedback and I forgot to incorporate the internet test into the main function. Other than that is the internet test logic correct? \$\endgroup\$ – unkn0wn.dev Apr 23 at 1:25
  • \$\begingroup\$ Also could you suggest a module to use for the "press any key to continue" function because most I find require the user to be a system administrator. \$\endgroup\$ – unkn0wn.dev Apr 23 at 1:28
  • \$\begingroup\$ The IP thing was just for fun to make it locate IP's as well. This wasn't a really big or serious project, it was just practicing web scraping and it was fun to implement the IP locating function. \$\endgroup\$ – unkn0wn.dev May 8 at 19:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.