Skip to main content
1 of 4
Zach
  • 31
  • 2

How can I make this Python code faster through vectorization?

I have a list of accounts, accounts that's passed to a function query_builder which combines a set of variables and produces a query string for consumption by requests. Is this a situation where vecotrization would help? I've been reading about speeding up python loops and this comes up a lot, but I don't quite see how it would help in my case since it's not a math function problem. Unless there's a method to use matrix multiplication to combine the strings to create query's out of numpy arrays.

Example:

import pandas as pd
import urllib.parse

def query_builder(account_id):
    url = "https://example.com/example/vb/query/data/json?json="
    id = "TOM"
    account = account_id
    user = "[email protected]"
    scoping = locals()
    params = {
        key: eval(key, scoping)
        for key in [
            "id",
            "account",
            "user",
        ]
    }
    query = url + urllib.parse.quote(json.dumps(params))
    return(query)

# Generate the numbers for Account List
def fun(start, end, step):
    num = np.linspace(start, end, (end - start) * int(1 / step) + 1).tolist()
    return [round(i, 2) for i in num]

acct_list = fun(1, 100000, 1)

accounts_table = {'account_id': [], 'query': []}

for i in acct_list:
    query = query_builder(i)
    accounts_table["account_id"].append(i)
    accounts_table["query"].append(query)

df = pd.DataFrame.from_dict(accounts_table)

Right now this takes about 6 minutes to run with 10mil accounts. I've attempted to use a lambda function within the accounts_table dictionary but that did little to speed up the process.

Zach
  • 31
  • 2