I started practicing in web-scraping few days ago. I made this code to extract data from a wikipedia page. There are several tables that classify mountains based on their height. However there is a problem with the size of the matrices. Some of them contain 5 columns while others 4. So I made this algorithm to extract all the names and the attributes of the mountains into separate lists. My approach was to create a length list that contains the number of <td> within the <tr> tags. The algorithm finds which table contains four columns and fills the column in excess (in the case of 5 columns) with NONE. However, I believe that there is a more efficient and more pythonic way to do it especially in the part where I use the find.next() function repedetly. Any suggestions are welcomed.

import requests
from bs4 import BeautifulSoup
import pandas as pd



all_tables=soup.find_all("table",{"class":["sortable", "plainrowheaders"]})



for table in range(len(all_tables)):


  for row in all_tables[table].find_all("tr"):

            if lengths[table]==5:
| improve this question | | | | |
  • 1
    \$\begingroup\$ Is the code working as expected? \$\endgroup\$ – Sᴀᴍ Onᴇᴌᴀ Jun 25 '18 at 23:17
  • \$\begingroup\$ Yes, totally. However i want to find -out a better way to scrape tables rather than using find_next() all the time. \$\endgroup\$ – brain_dead_cow Jun 25 '18 at 23:18
  • 3
    \$\begingroup\$ Alright; By the way Welcome to Code Review. Hopefully you receive good answers! \$\endgroup\$ – Sᴀᴍ Onᴇᴌᴀ Jun 25 '18 at 23:19
  • 1
    \$\begingroup\$ Please do not update the code in your question to incorporate feedback from answers, doing so goes against the Question + Answer style of Code Review. This is not a forum where you should keep the most updated version in your question. Please see what you may and may not do after receiving answers. \$\endgroup\$ – Sᴀᴍ Onᴇᴌᴀ Jun 26 '18 at 14:59

You're just looping on the rows, but not on the cells:

 for row in all_tables[table].find_all("tr"):

Rather than using multiple find_next("td") one after the other, add another loop using row.find_all('td') and append each row and cell to a 2D array.

Manipulating a 2D array is much easier and will make your code look much cleaner than row.find("td").find_next("td").find_next("td").

Good luck!

Those questions contain some answers that might interest you:

To be more specific, this code snippet from @shaktimaan:

data = []
table = soup.find('table', attrs={'class':'lineItemsTable'})
table_body = table.find('tbody')

rows = table_body.find_all('tr')
for row in rows:
    cols = row.find_all('td')
    cols = [ele.text.strip() for ele in cols]
    data.append([ele for ele in cols if ele])
| improve this answer | | | | |
  • \$\begingroup\$ Thank you for your reply. Because I am new in scraping and in python generally what I understand is that you mean to replace the (try) part of my code with this loop? I did it by the way but data list is empty. \$\endgroup\$ – brain_dead_cow Jun 26 '18 at 11:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.