Update Google Sheet with Python

Question

Below is a code to update a google sheet using the database values. The script is working fine but it is rather slow because of the sleep function which I have to put because of the "Quota exceeding error" below. If I remove the sleep time it would work much faster but after a certain number of rows get updated it will throw the below error.

I have to update three columns namely A,B & C of which the value start row is from 4th row onwards. So basically I need advise/recommendations to improve this code on the below points:

1. To make the query execution faster without getting the below error.
2. Is there a way to generalise this script, means without hardcoding the spreadsheet name (DB details I can get in a generalised way and read from config, for testing purpose only hardcoded the values)

gspread.exceptions.APIError: {   "error": {
    "code": 429,
    "message": "Insufficient tokens for quota 'ReadGroup' and limit 'USER-100s' of service 'sheets.googleapis.com' for consumer 'project_number:*******'.",
    "status": "RESOURCE_EXHAUSTED",
    "details": [
      {
        "@type": "type.googleapis.com/google.rpc.Help",
        "links": [
          {
            "description": "Google developer console API key",
            "url": "https://console.developers.google.com/project/***/apiui/credential"
          }
        ]
      }
    ]   } }

Python Code:

import psycopg2
import gspread
from oauth2client.service_account import ServiceAccountCredentials
from time import sleep
import datetime

def update_sheet(sheet, table, rangeStart='A', rangeEnd='C'):
 for index, row in enumerate(table):
    range = '{start}{i}:{end}{i}'.format(start=rangeStart, end=rangeEnd, i=index+4)
    cell_list = sheet.range(range)
    for i, cell in enumerate(cell_list):
      start_time = datetime.datetime.now()
      cell.value = row[i]
      sheet.update_cells(cell_list)
      end_time = datetime.datetime.now()
      if (end_time - start_time).total_seconds() < 1:
         sleep(1.01 - (end_time - start_time).total_seconds())


cnx_psql = psycopg2.connect(host="xxx.xxx.xxx.xx", database="postgres", user="postgres",password="******", port="5432")
print('DB connected')
psql_cursor = cnx_psql.cursor()

METADATA_QUERY = '''select product_id,CAST(low_stock_date as TEXT) low_stock_date,sku from test.low_stock_date;'''
psql_cursor.execute(METADATA_QUERY)
results = psql_cursor.fetchall()
cell_values = (results)

scope =  ['https://spreadsheets.google.com/feeds', 'https://www.googleapis.com/auth/drive']
creds = ServiceAccountCredentials.from_json_keyfile_name('/Users/lins/Documents/GS_secret/secret_key.json',scope)
client = gspread.authorize(creds)
sheet = client.open_by_url('https://docs.google.com/spreadsheets/d/1****zQ/edit#gid=0').sheet1

#Function Call
update_sheet(sheet, cell_values)

psql_curso.close()
cnx_psql.close()

Answer 1

The update_cells call can update multiple cells at once. So you should calculate all new values locally and push the changes in one update. This should use only one token, instead of one per cell (untested code):

def update_sheet(sheet, table, start='A', end='C'):
    to_update = []
    for i, row in enumerate(table):
        cells = sheet.range(f'{start}{i+4}:{end}{i+4}')
        for cell, value in zip(cells, row):
            cell.value = value
        to_update.extend(cells)
    sheet.update_cells(to_update)

I also changed the names and indentation to follow Python's official style-guide, PEP8, and used an f-string for easier string formatting.

This still has the problem that it needs to get each row from the sheet, each of which does an API call. Instead you can get each column as a range and update one column at a time, which saves calls if your table has less columns than rows. For this we need to transpose the table first, though:

def update_sheet(sheet, table, columns="ABC", header=4):
    to_update = []
    table = list(zip(*table))    # transpose the table
    for col_name, col in zip(columns, table):   # iterating over columns now
        r = f"{col_name}{header}:{col_name}{len(col)+header}"  # changed range
        print(r)  # for debugging
        cells = sheet.range(r)
        for cell, value in zip(cells, col):
            cell.value = value
        to_update.extend(cells)
    sheet.update_cells(to_update)

With this it is no problem to use e.g. this table of size 2500 x 3:

import numpy as np

...
table = list(map(list, np.arange(2500*3).reshape(-1, 3).astype(str)))
update_sheet(sheet, table)