13
\$\begingroup\$

I am scraping an online shop page, trying to get the price mentioned in that page. In the following block the price is mentioned:

<span id="hs18Price" itemprop="price" title="New Baby Care Pack">
<span class='WebRupee'>&#x20B9;</span>&nbsp;999</span>

I am using Beautiful Soup to get this tag and using a regular expression to get the price:

# -*- coding: utf8 -*-

import re
import requests
from bs4 import BeautifulSoup

hs18_test_urls = ['http://www.homeshop18.com/diary-wimpy-kid-hard-luck/author:jeff-kinney/isbn:9780141350677/books/juvenile-fiction/product:30926027/cid:10065/?it_category=HP&it_action=BO-R01001&it_label=HP-R01001-140121094603-30926027-PR-BO-RM-OT-RT01_BooksFlat40PercentOff-RL02-160120&it_value=0',
            'http://www.homeshop18.com/apple-ipad-air-wi-fi-16gb-space-grey/computers-tablets/tablets/product:31228967/cid:16327/',
            'http://www.homeshop18.com/reebok-men-black-yellow-sandals-j97184/footwear/men/product:30795219/cid:15067/',
            'http://www.homeshop18.com/american-swan-women-shirt-pink/clothing/women/product:31225645/cid:15021/',
            'http://www.homeshop18.com/diva-fashion-art-silk-saree-parrot-green/clothing/women/product:31514557/cid:15011/?it_category=hs18bot&it_action=recentlySoldProducts&it_label=31225645&it_value=1']

hs18_expected_test_results = [u'210', u'35900', u'1499', u'479', u'1199']

def get_homeshop18_product_meta(url):
    reg = ur'^ ₹? (\d+)'
    response = requests.get(url)
    if response.status_code == requests.codes.ok:
        soup = BeautifulSoup(response.text)
        product_name = soup.find('meta', {'property' : 'og:title'})['content']
        product_url = soup.find('meta', {'property' : 'og:url'})['content']
        product_img_url = soup.find('meta', {'property': 'og:image'})['content']
        product_price_tag_element = soup.find('span', {'id': 'hs18Price', 'itemprop': 'price'})
        product_price_match = re.match(reg, product_price_tag_element.text)
        if product_price_match:
            product_price = product_price_match.group(1)
        else:
            product_price = None
        return (product_url, product_name, product_img_url, product_price)
    return None

if __name__ == '__main__':
    assert(map(lambda x: x[3], map(get_homeshop18_product_meta, hs18_test_urls)) == hs18_expected_test_results)

Questions:

  1. With beautiful soup I can get contents within that span tag. Is there any way to just get the content of the outer span and ignoring whatever is within the inner span tag? i.e. it should give me only &nbsp; 210.
  2. If the above is not possible, then is there any further improvements you suggest with regards to re or the code in general?
  3. I am using a unicode character(\U20B9) in my script. Should I be doing that? I couldn't get re to match even after sending re.UNICODE flag.

    reg = r'^ \U20B9? (\d+)'
    

    product_price_match = re.match(reg, product_price_tag_element.text, re.UNICODE)
    
  4. For the reason mentioned in #3, I am using -*- coding: utf8 -*- at the start. Is that okay? Is this the pythonic way? or any suggested way?

\$\endgroup\$
  • \$\begingroup\$ I think you need \u20B9 instead of \U20B9 (lowercase for 4 instead of 8 nybbles). If that still doesn't work, I'd consider either dropping the raw string prefix, or splitting the string to avoid the literal . \$\endgroup\$ – Michael Urman Feb 3 '14 at 3:31
  • \$\begingroup\$ ^yes! it matches to u'\u20B9' \$\endgroup\$ – avi Feb 3 '14 at 16:26
7
\$\begingroup\$

As it happens, there are positive answers to each of your questions:

  1. with Beautiful Soup you can remove the WebRupee span with replace_with() entirely....

    webrupee_element = soup.find('span', {'class': 'WebRupee'})
    webrupee_element.replace_with('')
    

    ... then, when you get the text value of the product_price_tag_element.text it will not have the symbol.

    EDIT: Of course, it would be faster/better to do:

    for wr in product_price_tag_element.find('span'):
        wr.replace_with('')
    
  2. Your regex is not matching the value properly because the &nbsp; character may not be mapping directly to the regular ' ' character in your regex. You should use the 'whitespace' escape-sequence \s in your regex instead of ' ', like reg = ur'^\s*₹?\s*(\d+)\s*'

  3. Unicode string literals in python should be escaped with either the \u or \U escape, but they are different:

    Specific code points can be written using the \u escape sequence, which is followed by four hex digits giving the code point. The \U escape sequence is similar, but expects 8 hex digits, not 4.

    in your case you should use lower-case \u

  4. You can remove the utf-8/unicode declaration if you encode the values in the more-pythonic way of unicode escapes.

\$\endgroup\$
  • \$\begingroup\$ any reason using for loop? Since there is only one span tag, why not this, or just to be safe?: product_price_tag_element.find('span').replace_with('') price = product_price_tag_element.text.strip() \$\endgroup\$ – avi Feb 3 '14 at 17:07
  • \$\begingroup\$ it is just a way to make testing for the actual span a bit easier/safer. If there is no child WebRupee span the loop is still safe. Actually, I would even go so far as to recommend product_price_tag_element.children \$\endgroup\$ – rolfl Feb 3 '14 at 17:08
  • \$\begingroup\$ makes sense. however product_price_tag_element.children also contains the text along with span as child. So I cannot actually do replace_with('') that would be replacing my text also. \$\endgroup\$ – avi Feb 3 '14 at 17:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.