2
\$\begingroup\$

I have a list of accounts, accounts that's passed to a function query_builder which combines a set of variables and produces a query string for consumption by requests. Is this a situation where vecotrization would help? I've been reading about speeding up python loops and this comes up a lot, but I don't quite see how it would help in my case since it's not a math function problem. Unless there's a method to use matrix multiplication to combine the strings to create query's out of numpy arrays.

Example:

import pandas as pd
import urllib.parse
import numpy as np

def query_builder(account_id):
    url = "https://example.com/example/vb/query/data/json?json="
    id = "TOM"
    account = account_id
    user = "[email protected]"
    scoping = locals()
    params = {
        key: eval(key, scoping)
        for key in [
            "id",
            "account",
            "user",
        ]
    }
    query = url + urllib.parse.quote(json.dumps(params))
    return(query)

# Generate the numbers for Account List
def fun(start, end, step):
    num = np.linspace(start, end, (end - start) * int(1 / step) + 1).tolist()
    return [round(i, 2) for i in num]

acct_list = fun(1, 100000, 1)

accounts_table = {'account_id': [], 'query': []}

for i in acct_list:
    query = query_builder(i)
    accounts_table["account_id"].append(i)
    accounts_table["query"].append(query)

df = pd.DataFrame.from_dict(accounts_table)

Right now this takes about 6 minutes to run with 10mil accounts. I've attempted to use a lambda function within the accounts_table dictionary but that did little to speed up the process.

\$\endgroup\$
2
  • 1
    \$\begingroup\$ Welcome to Code Review! The current question title, which states your concerns about the code, is too general to be useful here. Please edit to the site standard, which is for the title to simply state the task accomplished by the code. Please see How to get the best value out of Code Review: Asking Questions for guidance on writing good question titles. \$\endgroup\$ Dec 23, 2022 at 12:19
  • 2
    \$\begingroup\$ Your stated goal is enhancing performance, reducing the time spent. Yet you didn't tell us where the code is spending its time. Update the question to include cProfile output. We could guess, but it's better to know. Your builder calls quote & dump 1e7 times. Consider calling them just once on the final result. There's at least two ways to call the DataFrame() ctor. Rather than building a pair of accounts_table lists, consider building a list of 1e7 two-element dicts. Prefer literal_eval to eval \$\endgroup\$
    – J_H
    Dec 23, 2022 at 15:31

1 Answer 1

2
\$\begingroup\$

eval is evil

    params = {
        key: eval(key, scoping)
        for key in [
            "id",
            "account",
            "user",
        ]
    }

Ugghhh! No, no, 1e3 times no. Due to its security exposure, eval() is almost always the wrong tool to reach for. In Python, or in any other language.

Instead of the eval() clause, you wanted locals()[key].

But this would have been far more sensible, and much faster:

    params = {
        "id": "TOM",
        "account": account_id,
        "user": "[email protected]",
    }

Computing a dict of two dozen locals doesn't take a lot of time, but do it a million times and it takes perceptibly long, about one second. No need to do that busy work.

meaningful name

Independent of whether we're having fun now, this is a terrible throwaway name:

def fun( ... ):

If you don't want to name a FUNction, python already has a lambda keyword for making anonymous ones.

The comment makes it pretty clear that you meant to write def get_account_ids().

I was a little surprised to see that we're not coercing account IDs to int.

large integers

fun(1, 100000, 1)

You have an opportunity to make that a little more legible for human readers:

fun(1, 100_000, 1)

backend query

You're presumably doing this work on the client, assembling ten million query strings, because you plan to issue ten million queries.

Don't do that.

If the backend is storing accounts in an RDBMS, you'd be much better off designing an API endpoint which accepts a few hundred account IDs at a time. That lets us ask the DB about many account results at once, which we send to the client as a giant response. It could be a single-step enormous query, or the steps could be (1.) populate temp table with IDs, followed by (2.) do a JOIN. Batching things also lets TCP get out of slow start and open the congestion window, for better throughput.

\$\endgroup\$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.