This code uses the
Split function to extract specific information from the following website: https://www.webscraper.io/test-sites/tables.
The required information are the four tables visible on the page with headers
"#", "First Name","Last Name","Username". I am extracting the information within these into 4 dataframes.
I use the
requests library to make the
GET request, and split the response text on
"table table-bordered" to generate my individual table chunks.
There is a fair amount of annoying fiddly indexing to get just the info I want, but the tutorial I am following requires the use of the
Split function, and not something far more logical, to my mind, like Beautiful Soup, where I could just apply CSS selectors, for example, and grab what I want. The latter method would be less fragile as well.
I have written a function,
GetTable, to parse the required information from each chunk and return a dataframe. There is a difference between the
Split delimiter for table 1 versus 2-4.
There isn't an awful lot of code but I would appreciate any pointers on improving the code I have written.
I am running this from Spyder 3.2.8 with Python 3.6.
def GetTable(tableChunk): split1 = tableChunk.split('tbody') split2 = split1.split('<table') values =  aList = split2.split('>\n\t\t\t\t<') if len(aList) !=1: for item in aList[1:]: values.append(item.split('</').split('d>')) else: aList = split2.split('</td') for item in aList[:-1]: values.append(item.split('td>')) headers = ["#", "First Name", "Last Name", "User Name"] numberOfColumns = len(headers) numberOfRows = int((len(values) / numberOfColumns)) df = pd.DataFrame(np.array(values).reshape( numberOfRows, numberOfColumns ) , columns = headers) return df import requests as req import pandas as pd import numpy as np url = "http://webscraper.io/test-sites/tables" response = req.get(url) htmlText = response.text tableChunks = htmlText.split('table table-bordered') for tableChunk in tableChunks[1:]: print(GetTable(tableChunk)) print('\n')