# Set up of pandas dataframe with mixed datatypes including arrays from .mat file in a pythonic/numpy way

Coming from a heavy MATLAB background so excuse my un-pythonic ways. I'd like to get an idea of how a python coder would approach this simple set-up of a dataframe.

I'm importing a .mat file from experimental data, it is made such that each row pertains to a single trial for a set of conditions (subjectID,group etc...).

My goal is to create a pandas dataframe such that a single row contains an entire session (all trials for a given set of conditions)- this will hopefully be clearer in my code. This code works well, but I'd like to implement this elegantly given the capabilities of python/pandas/numpy.

import scipy.io
import pandas as pd
import numpy as np

big_data = raw_dat['big_data']

#Define Constants
SUB = 0
GROUP = 1
SESS = 5
MEDS = 6
GAME = 8
EOB = 9
DIR = 12
COL = 13
REW = 14
RT = 15
c_set = [SUB, GROUP, SESS, MEDS, EOB]

#Filter for strategic version of task
st_data = big_data[big_data[:,GAME] == 2,:]

#Get unique conditional sets (subject,group,meds,eob,session)
unique_sets = np.unique(st_data[:,c_set],axis=0).astype(int)
valid_ind= (st_data[:,DIR] == 0) | (st_data[:,DIR] == 1)

#Set up string identities for groups/EoB
str_EOB = ('eye','button')


#Set up dictionary for creating DataFrame

dict_data = []
for line in unique_sets:

#Select of data matching conditions
set_ind = np.all(st_data[:,c_set].astype(int) == line,axis=1)
select_data = st_data[np.logical_and(set_ind,valid_ind),:]

#From this, create a dictionary of values to append into dataframe
dict_data.append(
{'subject_id' : line[0],'group' : str_groups[line[1]],'session' : line[2],'meds' : line[3],
'eob' : str_EOB[line[4]],'dir_choice' : select_data[:,DIR],'col_choice' : select_data[:,COL],
'outcomes' : select_data[:,REW],'RT' : select_data[:,RT]}
)

df = pd.DataFrame.from_dict(dict_data)
df = df[['subject_id','group','meds','session','eob','dir_choice','col_choice','outcomes','RT']]


Where dir_choice, col_choice, outcomes, RT contain column vectors as its datatype.

Not sure if this will affect the methodology but the .mat file is approximately 200,000 rows