Just starting out trying to do some python for data analysis. I am a total beginner and have figure out how to brute force somethings, but I know it is inefficient but don't know how to do it without messing up what i have.
I have multiple webpages to scrape and then store data in a data frame. The code is identical for all of the pages. How do I set it up as a routine instead just repeating the same code over and over again.
As an example the two urls are: https://etfdb.com/etf/IWD/ https://etfdb.com/etf/IWF/
The html is identical so the web scraping working exactly the same for both.
Once scraped, I want them in a single data frame.
The below works, but is me taking the least sophisticated approach since I know so little. The actual code is likely not pretty but it works.
Any help appreciated on how this should be improved.
import bs4
from urllib.request import urlopen as uReq
from bs4 import BeautifulSoup as soup
import pandas as pd
import numpy as np
from IPython.display import display
iwd_url = 'https://etfdb.com/etf/IWD/'
uClient = uReq(iwd_url)
page_html = uClient.read()
uClient.close()
page_soup = soup(page_html, "html.parser")
#Isolate header to get name and symbol
h1 = page_soup.h1
#Isolate stock symbol
title = h1.findAll("span",{"class":"label-primary"})
titlet = title[0].text
#print(titlet)
#strip space and line break
strip1 = h1.text.strip()
#strip stock symbol
strip2 = strip1.strip(titlet)
#strip remaining line break
strip3 = strip2.strip()
#print(strip3)
IWD = page_soup.findAll("table",{"class":"chart base-table"})[1]
#Create lists to fill
sectordata=[]
sectorname=[]
sectorweight=[]
for row in IWD.findAll("td"):
sectordata.append(row.text)
#list created
#Assign every other value to proper list to get 2 columns
sectorname = sectordata[::2]
sectorweight = sectordata[1::2]
#Insert name/symbol for clarification/validation
sectorweight.insert(0,titlet)
sectorname.insert(0,strip3)
# create empty data frame in pandas
df = pd.DataFrame()
#Add the first column to the empty dataframe.
df['Sector'] = sectorname
#Now add the second column.
df['Weight'] = sectorweight
##display(df)
### NEXT
iwf_url = 'https://etfdb.com/etf/IWF/'
uClient = uReq(iwf_url)
page_html = uClient.read()
uClient.close()
page_soup = soup(page_html, "html.parser")
#Isolate header to get name and symbol
h1 = page_soup.h1
#Isolate stock symbol
title = h1.findAll("span",{"class":"label-primary"})
titlet = title[0].text
#print(titlet)
#strip space and line break
strip1 = h1.text.strip()
#strip stock symbol
strip2 = strip1.strip(titlet)
#strip remaining line break
strip3 = strip2.strip()
#print(strip3)
IWD = page_soup.findAll("table",{"class":"chart base-table"})[1]
#Create lists to fill
sectordata=[]
sectorname=[]
sectorweight=[]
for row in IWD.findAll("td"):
sectordata.append(row.text)
#list created
#Assign every other value to proper list to get 2 columns
sectorname = sectordata[::2]
sectorweight = sectordata[1::2]
#Insert name/symbol for clarification/validation
sectorweight.insert(0,titlet)
sectorname.insert(0,strip3)
# create empty data frame in pandas
df2 = pd.DataFrame()
#Add the first column to the empty dataframe.
df2['Sector'] = sectorname
#Now add the second column.
df2['Weight'] = sectorweight
#display(df2)
results = df.merge(df2, on = "Sector")
results.columns = ['Sector', 'IWD', 'IWF']
display(results)
Like I said, this works, but it isn't automated and its ham-handed way of getting there. Please help me to get better!