This code simply returns a small section of HTML code and then gets rid of all tags except for break tags.

It seems inefficient because you cannot search and replace with a beautiful soup object as you can with a Python string, so I was forced to switch it back and forth from a beautiful soup object to a string several times so I could use string functions and beautiful soup functions. It seems that there must be a simpler way to do this without switching back and forth between soup objects and strings.

def whiteScrape(webaddress):
     url = (webaddress)
     ourUrl = opener.open(url).read()

     soup = BeautifulSoup(ourUrl)
     soup = soup.find("div", { "class" : "addy" })
     #below code will delete tags except /br
     soup = str(soup)
     soup = soup.replace('<br/>' , '^')
     soup = BeautifulSoup(soup)
     soup = (soup.get_text())
     soup = str(soup)
     soup=soup.replace('^' , '<br/>')

     return soup

The original HTML is something like:

<div class = "addy"> <p> blah blah <br/> blahblah <br/> blah </p></div>

The output should be:

blah blah <br/> blahblah <br/> blah

By PEP 8, whiteScrape() should be renamed to white_scrape().

Variable names should be purposeful. Therefore, it's a bad idea to continually redefine soup. If someone were to ask you to explain what the variable soup contained, you would have a hard time explaining.

It seems like what you want to do is to stringify the children of div(s) in question. One way to do that would be:

def white_scrape(url):
    page = opener.open(url).read()
    soup = BeautifulSoup(page)
    addr = soup.find('div', { 'class': 'addy' })
    return ''.join(str(child) for child in addr.children)

Note that this is not exactly equivalent to your original code. For example, if an address contains an element, such as 123 1<sup>st</sup> St., this solution would preserve the <sup> tag and its contents, whereas your original code would discard the tag but keep its contents and strip the tag.

  • \$\begingroup\$ Thanks for your answer. I'll try to remember to use the underscore as a separator in the future, wasn't aware of that. I think the second part of your answer is leading me in the right direction however, it still returns some unwanted tags. The output is <p> blah <br/> blah <br/> blah </p> . The original HTML was just a div containing that <p> and its contents. I don't want to output the <p> tags. Obviously I could figure some workaround but I feel like I must be missing something in BeautifulSoup. I would think that i could output child of child but I can't get that to work as of yet. \$\endgroup\$
    – ElioRubens
    Aug 30 '14 at 18:04
  • \$\begingroup\$ Why don't you edit your question to state what HTML input you want to be able to handle, and what you expect the output to look like? \$\endgroup\$
    – user50399
    Aug 30 '14 at 18:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.