Is there a way to vectorize the following string concatenation technique to enhance overall performance?

Question

I have a list of accounts, accounts that's passed to a function query_builder which combines a set of variables and produces a query string for consumption by requests. Is this a situation where vecotrization would help? I've been reading about speeding up python loops and this comes up a lot, but I don't quite see how it would help in my case since it's not a math function problem. Unless there's a method to use matrix multiplication to combine the strings to create query's out of numpy arrays.

Example:

import pandas as pd
import urllib.parse
import numpy as np

def query_builder(account_id):
    url = "https://example.com/example/vb/query/data/json?json="
    id = "TOM"
    account = account_id
    user = "[email protected]"
    scoping = locals()
    params = {
        key: eval(key, scoping)
        for key in [
            "id",
            "account",
            "user",
        ]
    }
    query = url + urllib.parse.quote(json.dumps(params))
    return(query)

# Generate the numbers for Account List
def fun(start, end, step):
    num = np.linspace(start, end, (end - start) * int(1 / step) + 1).tolist()
    return [round(i, 2) for i in num]

acct_list = fun(1, 100000, 1)

accounts_table = {'account_id': [], 'query': []}

for i in acct_list:
    query = query_builder(i)
    accounts_table["account_id"].append(i)
    accounts_table["query"].append(query)

df = pd.DataFrame.from_dict(accounts_table)

Right now this takes about 6 minutes to run with 10mil accounts. I've attempted to use a lambda function within the accounts_table dictionary but that did little to speed up the process.

Answer 1

eval is evil

    params = {
        key: eval(key, scoping)
        for key in [
            "id",
            "account",
            "user",
        ]
    }

Ugghhh! No, no, 1e3 times no. Due to its security exposure, eval() is almost always the wrong tool to reach for. In Python, or in any other language.

Instead of the eval() clause, you wanted locals()[key].

But this would have been far more sensible, and much faster:

    params = {
        "id": "TOM",
        "account": account_id,
        "user": "[email protected]",
    }

Computing a dict of two dozen locals doesn't take a lot of time, but do it a million times and it takes perceptibly long, about one second. No need to do that busy work.

meaningful name

Independent of whether we're having fun now, this is a terrible throwaway name:

def fun( ... ):

If you don't want to name a FUNction, python already has a lambda keyword for making anonymous ones.

The comment makes it pretty clear that you meant to write def get_account_ids().

I was a little surprised to see that we're not coercing account IDs to int.

large integers

fun(1, 100000, 1)

You have an opportunity to make that a little more legible for human readers:

fun(1, 100_000, 1)

backend query

You're presumably doing this work on the client, assembling ten million query strings, because you plan to issue ten million queries.

Don't do that.

If the backend is storing accounts in an RDBMS, you'd be much better off designing an API endpoint which accepts a few hundred account IDs at a time. That lets us ask the DB about many account results at once, which we send to the client as a giant response. It could be a single-step enormous query, or the steps could be (1.) populate temp table with IDs, followed by (2.) do a JOIN. Batching things also lets TCP get out of slow start and open the congestion window, for better throughput.