Coming from a heavy MATLAB background so excuse my un-pythonic ways. I'd like to get an idea of how a python coder would approach this simple set-up of a dataframe.

I'm importing a .mat file from experimental data, it is made such that each row pertains to a single trial for a set of conditions (subjectID,group etc...).

My goal is to create a pandas dataframe such that a single row contains an entire session (all trials for a given set of conditions)- this will hopefully be clearer in my code. This code works well, but I'd like to implement this elegantly given the capabilities of python/pandas/numpy.

import scipy.io
import pandas as pd
import numpy as np

raw_dat = scipy.io.loadmat('big_data_file.mat')
big_data = raw_dat['big_data']

#Define Constants
SUB = 0
SESS = 5
MEDS = 6
GAME = 8
EOB = 9
DIR = 12
COL = 13
REW = 14
RT = 15

#Filter for strategic version of task
st_data = big_data[big_data[:,GAME] == 2,:]

#Get unique conditional sets (subject,group,meds,eob,session)
unique_sets = np.unique(st_data[:,c_set],axis=0).astype(int)
valid_ind= (st_data[:,DIR] == 0) | (st_data[:,DIR] == 1)

#Set up string identities for groups/EoB
str_EOB = ('eye','button') 
str_groups = ('young','elderly','parkinsons','BPD','adult_male','adult_female')

#Set up dictionary for creating DataFrame    

dict_data = []    
for line in unique_sets:

 #Select of data matching conditions 
    set_ind = np.all(st_data[:,c_set].astype(int) == line,axis=1)
    select_data = st_data[np.logical_and(set_ind,valid_ind),:]

    #From this, create a dictionary of values to append into dataframe 
        {'subject_id' : line[0],'group' : str_groups[line[1]],'session' : line[2],'meds' : line[3],
         'eob' : str_EOB[line[4]],'dir_choice' : select_data[:,DIR],'col_choice' : select_data[:,COL],
         'outcomes' : select_data[:,REW],'RT' : select_data[:,RT]}

df = pd.DataFrame.from_dict(dict_data)
df = df[['subject_id','group','meds','session','eob','dir_choice','col_choice','outcomes','RT']]

Where dir_choice, col_choice, outcomes, RT contain column vectors as its datatype.

Not sure if this will affect the methodology but the .mat file is approximately 200,000 rows


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.