3
\$\begingroup\$

I'm new to python, and after doing a few tutorials, some about scraping, I been trying some simple scrapping on my own. Using beautifulsoup I manage to get data from webpages where everything has labels, but without them I'm doing a poor job.

I'm trying to get the dollar exchange rate from: http://www.bancochile.cl/cgi-bin/cgi_mone?pagina=inversiones/mon_tasa/cgi_mone

Bank

The value I'm after is highlighted in yellow

After a lot of trial and error, I manage to get the dollar exchange rate, but I think there has to be a better way.

import requests
from bs4 import BeautifulSoup

page = requests.get("http://www.bancochile.cl/cgi-bin /cgi_mone?pagina=inversiones/mon_tasa/cgi_mone")
soup = BeautifulSoup(page.content, 'html.parser')

tables = soup.find_all("table")
dollar = tables[4].find_all("td")

print(dollar[5].string)

Is there a better, or more correct way to do this? Also, I'm not sure if the problem is in the way I coded, or in not being able to better understand the html structure, to navigate to the information in a more efficient way.

\$\endgroup\$
3
\$\begingroup\$

The markup is definitely not easy to parse because of the nested table elements with no meaningful attributes. But, you are right that relying on relative index of a table and the desired cell being the 6th in the table is quite a fragile strategy.

Instead, let's use the row title as our "anchor". Then, we'll get the following cell via the .find_next_sibling():

DESIRED_MONEDAS = "DOLAR USA"

label = soup.find(lambda tag: tag.name == "td" and tag.get_text(strip=True) == DESIRED_MONEDAS)
value = label.find_next_sibling("td").get_text(strip=True)
print(value)
\$\endgroup\$
  • \$\begingroup\$ Thank you @alecxe, this is exactly what I was looking for. I'm still working out the details of how your code works, but is great study material to improve my understanding of scraping. \$\endgroup\$ – Pablo Mar 31 '17 at 17:15
  • \$\begingroup\$ @Pablo sure, glad to help. The important thing to understand that you don't have to "traverse" the HTML tree top to bottom, you can go sideways too. BeautifulSoup's API allows you to explore the tree, find elements in so many different ways, really great library. \$\endgroup\$ – alecxe Mar 31 '17 at 17:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.