I have a list of accounts, accounts that's passed to a function query_builder which combines a set of variables and produces a query string for consumption by requests. Is this a situation where vecotrization would help? I've been reading about speeding up python loops and this comes up a lot, but I don't quite see how it would help in my case since it's not a math function problem. Unless there's a method to use matrix multiplication to combine the strings to create query's out of numpy arrays.
Example:
import pandas as pd
import urllib.parse
import numpy as np
def query_builder(account_id):
url = "https://example.com/example/vb/query/data/json?json="
id = "TOM"
account = account_id
user = "[email protected]"
scoping = locals()
params = {
key: eval(key, scoping)
for key in [
"id",
"account",
"user",
]
}
query = url + urllib.parse.quote(json.dumps(params))
return(query)
# Generate the numbers for Account List
def fun(start, end, step):
num = np.linspace(start, end, (end - start) * int(1 / step) + 1).tolist()
return [round(i, 2) for i in num]
acct_list = fun(1, 100000, 1)
accounts_table = {'account_id': [], 'query': []}
for i in acct_list:
query = query_builder(i)
accounts_table["account_id"].append(i)
accounts_table["query"].append(query)
df = pd.DataFrame.from_dict(accounts_table)
Right now this takes about 6 minutes to run with 10mil accounts. I've attempted to use a lambda function within the accounts_table dictionary but that did little to speed up the process.
