Number of Google search results over a period of time, saved to database

Question

I am writing a Python script that scrapes data from Google search results and stores it in a database. I couldn't find any Google API for this, so I am just sending a HTTP GET request on Google's main site (and also Google News site). Then, using Beautiful Soup, I extract the number of search results found. I store this number in a mongo database using pymongo.

I want to collect the number of search results for specific keywords for next few months and then visualize the data. I would want something like this in the end:

Here is my code and I will run this following script four times a day using cron:

#! /bin/python

import re
import datetime

import pymongo
from pymongo import MongoClient
import requests
from bs4 import BeautifulSoup

REGEX = r'About (.*) results'
keywords = ['Barack Obama', 'Gandhi', 'Putin']

def number_of_search_results(key):
    def extract_results_stat(url):
        headers = { 
            'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64; rv:12.0) Gecko/20100101 Firefox/21.0'
        }
        search_results = requests.get(url, headers=headers, allow_redirects=True)
        soup = BeautifulSoup(search_results.text)
        result_stats = soup.find(id='resultStats')
        m = re.match(REGEX, result_stats.text)
        # print m.group(1)
        return int(m.group(1).replace(',',''))

    google_main_url = 'https://www.google.co.in/search?q=' + key
    google_news_url = 'https://www.google.co.in/search?hl=en&gl=in&tbm=nws&authuser=0&q=' + key
    return (extract_results_stat(google_main_url), extract_results_stat(google_news_url))

if __name__ == '__main__':
    conn = MongoClient()
    db = conn['search_results']
    current_time = datetime.datetime.utcnow()
    for key in keywords:
        google_main, google_news = number_of_search_results(key) 
        # print key, google_main, google_news
        db.search_results.insert({'time': current_time, 'name': key, 'google_main': google_main, 'google_news': google_news})

1. I am new to databases, so is my mongo schema good for my project?
2. Any general improvements?

Answer 1

If you just need the number of search results, you can surely use what you're using now and grep the total result number, otherwise you may use Google Search API, or find an open source scrawling tool like Scrappy.

Answer 2

1. As MongoDB doesn't enforce schema, and you are storing rows in schema, there shouldn't be any issue.

   But there are follow up questions (It seems I don't have enough points to comment). Is there any particular reason why you selected MongoClient? You would be performing suits better for relational DB or key value stores.

2. If it saves time and helps, check Google Trends.