2
\$\begingroup\$

Coming from a heavy MATLAB background so excuse my un-pythonic ways. I'd like to get an idea of how a python coder would approach this simple set-up of a dataframe.

I'm importing a .mat file from experimental data, it is made such that each row pertains to a single trial for a set of conditions (subjectID,group etc...).

My goal is to create a pandas dataframe such that a single row contains an entire session (all trials for a given set of conditions)- this will hopefully be clearer in my code. This code works well, but I'd like to implement this elegantly given the capabilities of python/pandas/numpy.

import scipy.io
import pandas as pd
import numpy as np

raw_dat = scipy.io.loadmat('big_data_file.mat')
big_data = raw_dat['big_data']

#Define Constants
SUB = 0
GROUP = 1
SESS = 5
MEDS = 6
GAME = 8
EOB = 9
DIR = 12
COL = 13
REW = 14
RT = 15
c_set = [SUB, GROUP, SESS, MEDS, EOB]

#Filter for strategic version of task
st_data = big_data[big_data[:,GAME] == 2,:]

#Get unique conditional sets (subject,group,meds,eob,session)
unique_sets = np.unique(st_data[:,c_set],axis=0).astype(int)
valid_ind= (st_data[:,DIR] == 0) | (st_data[:,DIR] == 1)

#Set up string identities for groups/EoB
str_EOB = ('eye','button') 
str_groups = ('young','elderly','parkinsons','BPD','adult_male','adult_female')

#Set up dictionary for creating DataFrame    

dict_data = []    
for line in unique_sets:

 #Select of data matching conditions 
    set_ind = np.all(st_data[:,c_set].astype(int) == line,axis=1)
    select_data = st_data[np.logical_and(set_ind,valid_ind),:]

    #From this, create a dictionary of values to append into dataframe 
    dict_data.append(
        {'subject_id' : line[0],'group' : str_groups[line[1]],'session' : line[2],'meds' : line[3],
         'eob' : str_EOB[line[4]],'dir_choice' : select_data[:,DIR],'col_choice' : select_data[:,COL],
         'outcomes' : select_data[:,REW],'RT' : select_data[:,RT]}
    )

df = pd.DataFrame.from_dict(dict_data)
df = df[['subject_id','group','meds','session','eob','dir_choice','col_choice','outcomes','RT']]

Where dir_choice, col_choice, outcomes, RT contain column vectors as its datatype.

Not sure if this will affect the methodology but the .mat file is approximately 200,000 rows

\$\endgroup\$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.