Scrape company data from Google search

Question

I have written this code for crawling and scraping search results from Google. But it's super slow when I run it.

What it does: read a pandas column, search values in Google and take the first result link then extract the siret number in link Ex.

https://www.infogreffe.fr/entreprise-societe/380760702-entreprise-de-decoration-et-d-innovation-sarl-340591B001420000.html the siret is 380760702. In every iteration I save this number in found_results.

To do that, there are three functions:

• rechercher_google() to get the html
• parse_results() to parse the url where the digit is
• recherche_google() to wrap to these two functions above.

Could it be less greedy in complexity or be better?

# Function pour recupere la html
USER_AGENT = {'User-Agent':'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/61.0.3163.100 Safari/537.36'}

def rechercher_google(search_term, number_results, language_code):
    '''
    input: Nom de l'entreprise à siretiser, le nombre de page et la langue
    output: La page resultat dde google en html stocker au format raw html
    '''
    assert isinstance(search_term, str), 'Il doit etre en string'
    assert isinstance(number_results, int), 'DOit etre en chiffre'
    escaped_search_term = search_term.replace(' ', '+')

    g_url = 'https://www.google.fr/search?q={}&num={}&hl={}'.format(escaped_search_term, number_results, language_code)
    response = requests.get(g_url, headers = USER_AGENT)
    response.raise_for_status()

    return response.text, search_term

# function pour recuperer ce qu'on a besoin dans la page
def parse_results(html, keyword):
    '''
    Input: la sortie de la fortion fetch_results()
    output: le nom de l'entreprise et le siret
    '''    
    soup = bs(html, 'html.parser')

    found_results = []
    rank = 1
    result_block = soup.find_all('div', attrs={'class': 'g'}) # g indique q'on est bien dans un resultat normal
    for result in result_block:
        link = result.find('a', href=True)
        title = result.find('h3')
        if link and title:
            link = link['href']
            title = title.get_text()
            if link != '#':
                try:
                    found_results.append({'siret':link.split('/')[4].split('-')[0]})
                except Exception as list_error:
                    print('error in fund_results {}'.format(list_error))

                rank += 1
    return found_results


# wrap up, le tout dans un global function qui appelle les deux premiere.
def scrape_google(search_term, number_results, language_code):
    '''
    input: il mange les deux premier functions
    output: une liste de dict des nom d'entreprise avec leur siret et club
    '''
    try:
        html, keyword = rechercher_google(search_term, number_results, language_code)
        results = parse_results(html, keyword)
        return results
    except AssertionError:
        raise Exception("Probleme d'argument")
    except requests.HTTPError:
        raise Exception(" :/ Google a bloqué le ip")
    except requests.RequestException:
        raise Exception(" :/ Probleme de connexion/proxy")

To run it

# Siret à partir de google
start_time = time.time()
data = []
for keyword in liste_partenaire.Partenaire:
    keyword = keyword+ ' ' + 'infogreffe Montpellier'
    name.append(keyword)
    try:
        results = scrape_google(keyword, 1, "fr")
        for result in results:
            data.append(results[0]['siret'])
    except Exception as e:
        print(' :( Error in results:: [ {} ]'. format(e))
    finally:
        time.sleep(11)
print("--- %s seconds d'éxecution ---" % (time.time() - start_time)) # 48 minutes
# save

Answer 1

Your escaping for the search term is not sufficient. Other characters than spaces might be in the user entered string, which also need to be escaped. Luckily, requests can do it all for you:

def rechercher_google(search_term, number_results, language_code):
    '''
    input: Nom de l'entreprise à siretiser, le nombre de page et la langue
    output: La page resultat dde google en html stocker au format raw html
    '''
    assert isinstance(search_term, str), 'Il doit etre en string'
    assert isinstance(number_results, int), 'DOit etre en chiffre'

    g_url = 'https://www.google.fr/search'
    params = {"q": search_term, "num": number_results, "hl": language_code}
    response = requests.get(g_url, params=params, headers=USER_AGENT)
    response.raise_for_status()
    return response.text, search_term

Note that I also followed Python's official style-guide, PEP8, and did not use spaces around the = for keyword arguments.

But the obvious improvement is not to parse the Google search results at all, but directly query the website you are interested in. If I understand it correctly, what you actually want to parse is:

url = "https://www.infogreffe.fr/recherche-entreprise-dirigeants/resultats-entreprise-dirigeants.html"
params = {"ga_cat": "globale", "ga_q"=search_term}
response = requests.get(url, params=params, header=USER_AGENT)

That website is not particularly fast either (some pages take 5s to load). But they will probably have worse bot protection than Google, which is a bit evil, but if your scraping is evil, you should not be doing it in the first place. Always read the terms & conditions of websites you want to scrape, so at least you know what you are doing. The Wikipedia page Search engine scraping has quite a lot of information about what Google is doing to prevent you from scraping their search result page too aggressively. In the same vein, you could try optimizing the wait time between the queries, which is currently 11 seconds. With some trial & error you might find a shorter time that also works.