Using the Wikipedia API
As mentioned in another answer, Wikipedia provides an HTTP API for fetching article content, and you can use it to get the content in much cleaner formats than HTML. Often this is much better (for example, it's a much better choice for this little project I wrote that extracts the first sentence of a Wikipedia article).
However, in your case, you have to parse tables anyway. Whether you're parsing them from HTML or from the Wikipedia API's "wikitext" format, I don't think there's much of a difference. So I consider this subjective.
Using requests, not urllib
Never use urllib, unless you're in an environment where for some reason you can't install external libraries. The requests
library is preferred for fetching HTML.
In your case, to get the HTML, the code would be:
html = requests.get(url).text
with an import requests
up-top. For your simple example, this isn't actually any easier, but it's just a general best-practice with Python programming to always use requests
, not urllib
. It's the preferred library.
What's with all those with blocks?
You don't need three with
blocks. When I glance at this code, three with
blocks make it seem like the code is doing something much more complicated than it actually is - it makes it seem like maybe you're writing multiple CSVs, which you aren't. Just use one, it works the same:
with open('data.csv', 'w',encoding='UTF-8', newline='') as f:
fields = ['Title', 'Year']
writer = csv.writer(f, delimiter=',')
writer.writerow(fields)
for tr in My_table.find_all('tr')[2:]: # [2:] is to skip empty and header
tds = tr.find_all('td')
try:
title = tds[0].text.replace('\n','')
except:
title = ""
try:
year = tds[2].text.replace('\n','')
except:
year = ""
writer.writerow([title, year])
for tr in My_second_table.find_all('tr')[2:]: # [2:] is to skip empty and header
tds = tr.find_all('td')
row = "{}, {}".format(tds[0].text.replace('\n',''), tds[2].text.replace('\n',''))
writer.writerow(row.split(','))
Are those two tables really that different?
You have two for loops, each one processing a table and writing it to the CSV. The body of the two for loops is different, so at a glance it looks like maybe the two tables have a different format or something... but they don't. You can copy paste the body of the first for loop into the second and it works the same:
for tr in My_table.find_all('tr')[2:]: # [2:] is to skip empty and header
tds = tr.find_all('td')
try:
title = tds[0].text.replace('\n','')
except:
title = ""
try:
year = tds[2].text.replace('\n','')
except:
year = ""
writer.writerow([title, year])
for tr in My_second_table.find_all('tr')[2:]: # [2:] is to skip empty and header
tds = tr.find_all('td')
try:
title = tds[0].text.replace('\n','')
except:
title = ""
try:
year = tds[2].text.replace('\n','')
except:
year = ""
With the above code, the only difference to the resulting CSV is that there aren't spaces after the commas, which I assume is not really important to you. Now that we've established the code doesn't need to be different in the two loops, we can just do this:
table_rows = My_table.find_all('tr')[2:] + My_second_table.find_all('tr')[2:]
for tr in table_rows:
tds = tr.find_all('td')
try:
title = tds[0].text.replace('\n','')
except:
title = ""
try:
year = tds[2].text.replace('\n','')
except:
year = ""
writer.writerow([title, year])
Only one for loop. Much easier to understand!
Parsing strings by hand
So that second loop doesn't need to be there at all, but let's look at the code inside of it anyway:
row = "{}, {}".format(tds[0].text.replace('\n',''), tds[2].text.replace('\n',''))
writer.writerow(row.split(','))
Um... you just concatenated two strings together with a comma, just to call split
and split them apart at the comma at the very next line. I'm sure now it's pointed out to you you can see this is pointless, but I want to pull you up on one other thing in these two lines of code.
You are essentially trying to parse data by hand with that row.split
, which is always dangerous. This is an important and general lesson about programming. What if the name of the chip had a comma in it? Then row
would contain more commas than just the one that you put in there, and your call to writerow
would end up inserting more than two columns!
Never parse data by hand unless you absolutely have to, and never write data in formats like CSV or JSON by hand unless you absolutely have to. Always use a library, because there are always pathological edge cases like a comma in the chip name that you won't think of and which will break your code. The libraries, if they're been around for a while, have had those bugs ironed out. With these two lines:
row = "{}, {}".format(tds[0].text.replace('\n',''), tds[2].text.replace('\n',''))
writer.writerow(row.split(','))
you are attempting to split a table row into its two columns yourself, by hand, which is why you made a mistake (just like anyone would). Whereas in the first loop, the code which does this splitting are the two lines:
title = tds[0].text.replace('\n','')
year = tds[2].text.replace('\n','')
Here you are relying on BeautifulSoup to have split the columns cleanly into tds[0]
and tds[2]
, which is much safer and is why this code is much better.
Mixing of input parsing and output generating
The code that parses the HTML is mixed together with the code that generates the CSV. This is poor breaking of a problem down into sub-problems. The code that writes the CSV should just be thinking in terms of titles and years, it shouldn't have to know that they come from HTML, and the code that parses the HTML should just be solving the problem of extracting the titles and years, it should have no idea that that data is going to be written to a CSV. In other words, I want that for loop that writes the CSV to look like this:
for (title, year) in rows:
writer.writerow([title, year])
We can do this by rewriting the with block like this:
with open('data.csv', 'w',encoding='UTF-8', newline='') as f:
fields = ['Title', 'Year']
writer = csv.writer(f, delimiter=',')
writer.writerow(fields)
table_rows = My_table.find_all('tr')[2:] + My_second_table.find_all('tr')[2:]
parsed_rows = []
for tr in table_rows:
tds = tr.find_all('td')
try:
title = tds[0].text.replace('\n','')
except:
title = ""
try:
year = tds[2].text.replace('\n','')
except:
year = ""
parsed_rows.append((title, year))
for (title, year) in parsed_rows:
writer.writerow([title, year])
Factoring into functions
To make the code more readable and really separate the HTML stuff from the CSV stuff, we can break the script into functions. Here's my complete script.
import requests
from bs4 import BeautifulSoup
import csv
urls = ['https://en.wikipedia.org/wiki/Transistor_count']
data = []
def get_rows(html):
soup = BeautifulSoup(html,'html.parser')
My_table = soup.find('table',{'class':'wikitable sortable'})
My_second_table = My_table.find_next_sibling('table')
table_rows = My_table.find_all('tr')[2:] + My_second_table.find_all('tr')[2:]
parsed_rows = []
for tr in table_rows:
tds = tr.find_all('td')
try:
title = tds[0].text.replace('\n','')
except:
title = ""
try:
year = tds[2].text.replace('\n','')
except:
year = ""
parsed_rows.append((title, year))
return parsed_rows
for url in urls:
html = requests.get(url).text
parsed_rows = get_rows(html)
with open('data.csv', 'w',encoding='UTF-8', newline='') as f:
fields = ['Title', 'Year']
writer = csv.writer(f, delimiter=',')
writer.writerow(fields)
for (title, year) in parsed_rows:
writer.writerow([title, year])
What would be even better would be for get_rows
to be a generator rather than a regular function, but that's advanced Python programming. This is fine for now.