Extracting book information from a web page, with possible missing attributes

Question

I'm just starting to code and I found a solution to first check if a div class exist before to try to get_text (and avoid an error if it doesn't). But the code doesn't looks really nice repeating the same parameters.

The code is working but if there is something I could learn, it would be better.

headers = {'User-Agent': 'Mozilla/5.0'}
response = requests.get(url_book, headers=headers)
soup = BeautifulSoup(response.text, 'html.parser')

book_info = soup.find_all('div', {'class': 'jr_pgContainer jr_itemDetail hReview-aggregate'})

for content in book_info:
    title = content.find('span', {'class': 'fn'}).get_text()
    if content.find('div', {'class': 'fieldRow jr_autorestr'}):
        author = content.find('div', {'class': 'fieldRow jr_autorestr'}).get_text().replace("Autore","")
    else:
        author = 'None'
    if content.find('div', {'class': 'fieldRow jr_editore'}):
        editor = content.find('div', {'class': 'fieldRow jr_editore'}).get_text().replace("Casa editrice","")
    else:
        editor = 'None'
    if content.find('div', {'class': 'fieldRow jr_narrstrcontinente'}):
        classification = content.find('div', {'class': 'fieldRow jr_narrstrcontinente'}).get_text().replace("Classificazione","")
    else:
        classification = 'None'
    if content.find('div', {'class': 'contentFulltext'}):
        description =  content.find('div', {'class': 'contentFulltext'}).get_text().replace("\t","").replace("\n","")
    else:
        description = 'None'
    if content.find('h4', {'class': 'jr_reviewTitle'}):
        title_review = content.find('h4', {'class': 'jr_reviewTitle'}).get_text()
    else:
        title_review = 'None'
    if content.find('div', {'class': 'jr_reviewComment'}):
        review = content.find('div', {'class': 'jr_reviewComment'}).get_text().replace("\t","").replace("\n\n","")
    else:
        review = 'None'

#CREATE AND PRINT DICTIONARY
book_detail = {'title: ': title, 'author: ': author, 'editor: ': editor, 'classification: ': classification, 'description: ': description, 'revew title: ': title_review, 'review: ': review }

for keys,values in book_detail.items():
    print(keys)
    print(values)
    print("************************************")

Answer 1

The obvious thing to do when you have repeated functionality is to extract it into a function. In this case, it looks like the parameters for the function are the content itself, the element type to find, the classname, and a list of text to replace.

You can also tidy things up by only doing the find once.

def find_text(content, element, classname, *replacements):
    item = content.find(element, {'class': classname})
    if item:
        text = item.get_text()
        for replacement in replacements:
            text = text.replace(replacement, "")
        return text
    else:
        return 'None'

Now your main loop looks like:

for content in book_info:
    title = find_text('span', 'fn')
    author = find_text('div', 'fieldRow jr_autorestr', "Autore")
    classification = find_text('div', 'fieldRow jr_narrstrcontinente', "Classificazione")
    ... etc ....