4
\$\begingroup\$

This is a problem I find a lot!! Can I achieve this goal without consuming so much time?

My code below achieves what I want it to achieve. However, I believe it could be a lot more efficient and Pythonic.

PROBLEM: I want to extract summary data from a larger dataset and I only know how to do so utilizing nested For loops. For example, I have a large dataset containing golf data, and I would like to extract summary statistics for the individual golf holes.

This code creates a scoring distribution and mean score for each Season-Hole-Round-Score vs. Par combination (48 rows in total).

import numpy as np
import pandas as pd
import itertools

seasons = [2001,2001,2001,2001,2002,2002,2002,2002]
holes = [1,1,2,2,1,1,2,2]
rounds = [3,4,3,4,3,4,3,4]
scores = [1,-1,0,0,0,1,-1,1] # actual scores vs. par

df = pd.DataFrame({'season' : seasons, 'hole': holes, 'round':rounds, 'score': scores})


all_seasons = set(seasons); all_holes = set(holes);  all_scores = [-1,0,1]
all_rounds = ["R3","R4","Weekend"] #some averages combine rounds
round_iter = np.arange(0,4) #position of rounds list
round_ids = [[3],[4],[3,4]] # weekend incldues rounds 3 and 4

hold_list = [] #blank list

for season,round,hole in itertools.product(all_seasons,round_iter,all_holes):

    hold_data = df[((df['season'] == season) & (df['hole'] == hole))
    & (df['round'].isin(round_ids[round-1]))]

    mean_score = hold_data['score'].mean()
    vspar_distro = hold_data['score'].value_counts().to_dict()
    for score in all_scores:
        count_score = 0
        if score in vspar_distro:
            count_score = vspar_distro[score]
        hold_list.append([season,all_rounds[round-1]
                        ,hole,mean_score,score,count_score])


historical_df = pd.DataFrame(hold_list,columns 
            = ['season','round','hole','mean_score','vspar_score','count'])

This produces the df that I desire (here are the first 5 rows), but applying this to a file with 100k+ records takes a long time and I believe there is a more efficient way. Thanks!

   season    round  hole  mean_score  vspar_score  count
0    2001  Weekend     1         0.0           -1      1
1    2001  Weekend     1         0.0            0      0
2    2001  Weekend     1         0.0            1      1
3    2001  Weekend     2         0.0           -1      0
4    2001  Weekend     2         0.0            0      2
\$\endgroup\$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.