I am scraping an online shop page, trying to get the price mentioned in that page. In the following block the price is mentioned:
<span id="hs18Price" itemprop="price" title="New Baby Care Pack">
<span class='WebRupee'>₹</span> 999</span>
I am using Beautiful Soup to get this tag and using a regular expression to get the price:
# -*- coding: utf8 -*-
import re
import requests
from bs4 import BeautifulSoup
hs18_test_urls = ['http://www.homeshop18.com/diary-wimpy-kid-hard-luck/author:jeff-kinney/isbn:9780141350677/books/juvenile-fiction/product:30926027/cid:10065/?it_category=HP&it_action=BO-R01001&it_label=HP-R01001-140121094603-30926027-PR-BO-RM-OT-RT01_BooksFlat40PercentOff-RL02-160120&it_value=0',
'http://www.homeshop18.com/apple-ipad-air-wi-fi-16gb-space-grey/computers-tablets/tablets/product:31228967/cid:16327/',
'http://www.homeshop18.com/reebok-men-black-yellow-sandals-j97184/footwear/men/product:30795219/cid:15067/',
'http://www.homeshop18.com/american-swan-women-shirt-pink/clothing/women/product:31225645/cid:15021/',
'http://www.homeshop18.com/diva-fashion-art-silk-saree-parrot-green/clothing/women/product:31514557/cid:15011/?it_category=hs18bot&it_action=recentlySoldProducts&it_label=31225645&it_value=1']
hs18_expected_test_results = [u'210', u'35900', u'1499', u'479', u'1199']
def get_homeshop18_product_meta(url):
reg = ur'^ ₹? (\d+)'
response = requests.get(url)
if response.status_code == requests.codes.ok:
soup = BeautifulSoup(response.text)
product_name = soup.find('meta', {'property' : 'og:title'})['content']
product_url = soup.find('meta', {'property' : 'og:url'})['content']
product_img_url = soup.find('meta', {'property': 'og:image'})['content']
product_price_tag_element = soup.find('span', {'id': 'hs18Price', 'itemprop': 'price'})
product_price_match = re.match(reg, product_price_tag_element.text)
if product_price_match:
product_price = product_price_match.group(1)
else:
product_price = None
return (product_url, product_name, product_img_url, product_price)
return None
if __name__ == '__main__':
assert(map(lambda x: x[3], map(get_homeshop18_product_meta, hs18_test_urls)) == hs18_expected_test_results)
Questions:
- With beautiful soup I can get contents within that span tag. Is there any way to just get the content of the outer span and ignoring whatever is within the inner span tag? i.e. it should give me only
210
. - If the above is not possible, then is there any further improvements you suggest with regards to
re
or the code in general? I am using a unicode character(
\U20B9
) in my script. Should I be doing that? I couldn't getre
to match even after sendingre.UNICODE
flag.reg = r'^ \U20B9? (\d+)'
product_price_match = re.match(reg, product_price_tag_element.text, re.UNICODE)
For the reason mentioned in #3, I am using
-*- coding: utf8 -*-
at the start. Is that okay? Is this the pythonic way? or any suggested way?
\u20B9
instead of\U20B9
(lowercase for 4 instead of 8 nybbles). If that still doesn't work, I'd consider either dropping the raw string prefix, or splitting the string to avoid the literal₹
. \$\endgroup\$u'\u20B9'
\$\endgroup\$