0
\$\begingroup\$

I have the below code that seems to be taking a while to run over time.

info_participants = info['participants']

for participant in info_participants:
    info_gamestatistics_table = flatdict.FlatDict(participant, delimiter='_')
    info_gamestatistics_table = dict(info_gamestatistics_table)
    info_gamestatistics_df = info_gamestatistics_df.append(info_gamestatistics_table, ignore_index=True)

info participants is a list of json objects.

I'm flattening a json object, turning the output to a dict, and adding it to my dataframe using the df.append() method, and I'm doing that for each participant (10).

Is there a faster way to do this?

Improve this question
\$\endgroup\$
3
  • 1
    \$\begingroup\$ Probably, but you need to show much more context for a meaningful review. Ideally: sample input and output data, and code from the entire program. \$\endgroup\$
    – Reinderien
    Jun 19 at 18:36
  • \$\begingroup\$ The sample input was a bit too large, hoping for something more theoretical. \$\endgroup\$
    – Jack
    Jun 19 at 18:49
  • 1
    \$\begingroup\$ Python provides a number of tools to measure code execution time. Some good read: Python Timer Functions: Three Ways to Monitor Your Code. Each of the three statements inside your loop should be clocked separately. But the data matters too. Your question should be edited to provide more context and some sample data that can be used by reviewers to reproduce the issue. Maybe there is a better way to achieve the intended result, but we are not seeing the whole picture here. \$\endgroup\$
    – Kate
    Jun 19 at 19:13

2 Answers 2

Reset to default
0
\$\begingroup\$

Without knowing what info['participants'] looks like, I've made this assumption.

As others have mentioned it would help the reviewers if you could provide a small sample of the data or a function to generate something that represents it.

info = {
    "participants": [{f"foo_{i}": f"bar"} for i in range(200)]
    + [{f"spam_{i}": f"eggs"} for i in range(200)]
}

It appears you are using the FlatDict delimiter='_' to split keys in your object.

With that I've ditched FlatDict for a python generator function

import pandas as pd


info = ...
info_participants = info["participants"]

def participants_to_dataframe(guest_list: list[dict[str, any]]) -> pd.DataFrame:
    def generate() -> tuple[str, ...]:
        for guest in guest_list:
            for key, value in guest.items():
                yield *key.split("_"), value

    return pd.DataFrame(generate(), columns=["name", "id", "value"]).set_index(
        ["name", "id"]
    )


def start():
    df = participants_to_dataframe(info_participants)
    print(df)


if __name__ == "__main__":
    start()
Improve this answer
\$\endgroup\$
0
\$\begingroup\$ 
info_participants = info['participants']
list_of_dicts = []
for participant in info_participants:
    info_gamestatistics_table = flatdict.FlatDict(participant, delimiter='_').as_dict()
    list_of_dicts.append(info_gamestatistics_table)

info_gamestatistics_df = pd.concat([info_gamestatistics_df, pd.DataFrame(list_of_dicts)], ignore_index=True)

These changes increased the performance by 10x.

For anyone that might be interested, answer as above, don't waste time converting it to a dict manually, you can just add the .as_dict() "method" (Not sure thats right?) to save time as already implemented in the flatdict docs.

info_gamestatistics_table = flatdict.FlatDict(participant, delimiter='_').as_dict()

Secondly, you should save the rows you intend to insert to your DataFrame in a seperate list (list_of_dicts) and then concatenate that list with the original dataframe. (pd.concat([info_gamestatistics_df, pd.DataFrame(list_of_dicts)], ignore_index=True))

Improve this answer
\$\endgroup\$

Not the answer you're looking for? Browse other questions tagged or ask your own question.