2
\$\begingroup\$

I have written this code for crawling and scraping search results from Google. But it's super slow when I run it.

What it does: read a pandas column, search values in Google and take the first result link then extract the siret number in link Ex.

https://www.infogreffe.fr/entreprise-societe/380760702-entreprise-de-decoration-et-d-innovation-sarl-340591B001420000.html the siret is 380760702. In every iteration I save this number in found_results.

To do that, there are three functions:

  • rechercher_google() to get the html
  • parse_results() to parse the url where the digit is
  • recherche_google() to wrap to these two functions above.

Could it be less greedy in complexity or be better?

# Function pour recupere la html
USER_AGENT = {'User-Agent':'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/61.0.3163.100 Safari/537.36'}

def rechercher_google(search_term, number_results, language_code):
    '''
    input: Nom de l'entreprise à siretiser, le nombre de page et la langue
    output: La page resultat dde google en html stocker au format raw html
    '''
    assert isinstance(search_term, str), 'Il doit etre en string'
    assert isinstance(number_results, int), 'DOit etre en chiffre'
    escaped_search_term = search_term.replace(' ', '+')

    g_url = 'https://www.google.fr/search?q={}&num={}&hl={}'.format(escaped_search_term, number_results, language_code)
    response = requests.get(g_url, headers = USER_AGENT)
    response.raise_for_status()

    return response.text, search_term

# function pour recuperer ce qu'on a besoin dans la page
def parse_results(html, keyword):
    '''
    Input: la sortie de la fortion fetch_results()
    output: le nom de l'entreprise et le siret
    '''    
    soup = bs(html, 'html.parser')

    found_results = []
    rank = 1
    result_block = soup.find_all('div', attrs={'class': 'g'}) # g indique q'on est bien dans un resultat normal
    for result in result_block:
        link = result.find('a', href=True)
        title = result.find('h3')
        if link and title:
            link = link['href']
            title = title.get_text()
            if link != '#':
                try:
                    found_results.append({'siret':link.split('/')[4].split('-')[0]})
                except Exception as list_error:
                    print('error in fund_results {}'.format(list_error))

                rank += 1
    return found_results


# wrap up, le tout dans un global function qui appelle les deux premiere.
def scrape_google(search_term, number_results, language_code):
    '''
    input: il mange les deux premier functions
    output: une liste de dict des nom d'entreprise avec leur siret et club
    '''
    try:
        html, keyword = rechercher_google(search_term, number_results, language_code)
        results = parse_results(html, keyword)
        return results
    except AssertionError:
        raise Exception("Probleme d'argument")
    except requests.HTTPError:
        raise Exception(" :/ Google a bloqué le ip")
    except requests.RequestException:
        raise Exception(" :/ Probleme de connexion/proxy")

To run it

# Siret à partir de google
start_time = time.time()
data = []
for keyword in liste_partenaire.Partenaire:
    keyword = keyword+ ' ' + 'infogreffe Montpellier'
    name.append(keyword)
    try:
        results = scrape_google(keyword, 1, "fr")
        for result in results:
            data.append(results[0]['siret'])
    except Exception as e:
        print(' :( Error in results:: [ {} ]'. format(e))
    finally:
        time.sleep(11)
print("--- %s seconds d'éxecution ---" % (time.time() - start_time)) # 48 minutes
# save

\$\endgroup\$
  • 1
    \$\begingroup\$ Welcome to Code Review! Seems like you have missed the hint to "State the task that your code accomplishes. Make your title distinctive." in the question template. Please also have a look at our How to Ask page to get a few more hints how to write an interesting title for your question. \$\endgroup\$ – AlexV Nov 21 '19 at 11:27
  • \$\begingroup\$ Ok thanks let le review it. \$\endgroup\$ – abdoulsn Nov 21 '19 at 11:40
  • 2
    \$\begingroup\$ You said it's super slow, then why are you waiting for 11 seconds on each iteration with time.sleep(11) ?! \$\endgroup\$ – RomanPerekhrest Nov 21 '19 at 11:40
  • \$\begingroup\$ This is to avoid google bot, can I make it 5s or less? \$\endgroup\$ – abdoulsn Nov 21 '19 at 12:35
  • 1
    \$\begingroup\$ I have rephrased your question and your title to the best of my knowledge to make it more readable. Since I don't speak French, there might be errors. Please have a look at the rephrased version, and especially check "to grape a siret", which I could not understand. \$\endgroup\$ – AlexV Nov 21 '19 at 12:36
3
\$\begingroup\$

Your escaping for the search term is not sufficient. Other characters than spaces might be in the user entered string, which also need to be escaped. Luckily, requests can do it all for you:

def rechercher_google(search_term, number_results, language_code):
    '''
    input: Nom de l'entreprise à siretiser, le nombre de page et la langue
    output: La page resultat dde google en html stocker au format raw html
    '''
    assert isinstance(search_term, str), 'Il doit etre en string'
    assert isinstance(number_results, int), 'DOit etre en chiffre'

    g_url = 'https://www.google.fr/search'
    params = {"q": search_term, "num": number_results, "hl": language_code}
    response = requests.get(g_url, params=params, headers=USER_AGENT)
    response.raise_for_status()
    return response.text, search_term

Note that I also followed Python's official style-guide, PEP8, and did not use spaces around the = for keyword arguments.

But the obvious improvement is not to parse the Google search results at all, but directly query the website you are interested in. If I understand it correctly, what you actually want to parse is:

url = "https://www.infogreffe.fr/recherche-entreprise-dirigeants/resultats-entreprise-dirigeants.html"
params = {"ga_cat": "globale", "ga_q"=search_term}
response = requests.get(url, params=params, header=USER_AGENT)

That website is not particularly fast either (some pages take 5s to load). But they will probably have worse bot protection than Google, which is a bit evil, but if your scraping is evil, you should not be doing it in the first place. Always read the terms & conditions of websites you want to scrape, so at least you know what you are doing. The Wikipedia page Search engine scraping has quite a lot of information about what Google is doing to prevent you from scraping their search result page too aggressively. In the same vein, you could try optimizing the wait time between the queries, which is currently 11 seconds. With some trial & error you might find a shorter time that also works.

|improve this answer|||||
\$\endgroup\$
  • \$\begingroup\$ Thanks lot I'll change this. \$\endgroup\$ – abdoulsn Nov 21 '19 at 13:33
  • 2
    \$\begingroup\$ @abdoulsn: Glad it helped. But please don't update the code in the question. If you feel like you need more review, either un-accept my answer and wait for more answers, or ask a new question with updated code. Have a look at our help center for more information on what to do after you have received an answer. \$\endgroup\$ – Graipher Nov 21 '19 at 13:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.