# Calculating frequencies of each obs in the data

I am currently attempting to make some code more maintainable for a research project I am working on. I am definitely looking to create some more functions, and potentially create a general class to conduct such data calculations.

The code works, however, I am new to Python, and was wondering if someone could review my code for efficiency, formatting, readability, maintainability, scaling, etc. The data I am taking in are .txt files with 59 columns and some number of rows depending on how many observations are within the file (an example of a normal amount of rows is around 4400).

One of my main concerns is how to limit the amount of opening and closing of files. I am writing these files to store them in case I need to reference them to justify my calculations. Is there any way I can do this in parallel maybe, or even utilize threading?

Also, can I write it out in binary? Would that enhance the script's speed as well?

Here is an example of optimizations I would like to make:

I have created a small function. This way, a user can input and automagically have the script create column headers based on the size of the data frame taken in:

def createDFNames(numCols, colName):
'''Generates column names for the dataframe based on how many columns there are'''
colNames = []
col = 1
for col in range(numCols):
colNames.append('{}{}'.format(colName , col + 1))
return colNames


I am utilizing anaconda in a Jupyter notebook to conduct these calcs:

# Combining Script 1, and Script 2 to make one main script

# Begin reading files into a dataframe one by one, perform the necessary calculations, and save the results to a
# output file

count = 1

# Read the file into the dataframe

for file in stateFiles:

dfStateMat = pd.read_table(file, sep='\t', lineterminator = '\n')

numObs = len(dfStateMat.index)
numCols = len(dfStateMat.index)

print("Current File Being Processed is: " + file)

# Rename the columns of the dataframe - Hardcoded for now, but can create a func that
# Reads how many cols there are, then runs through a loop up to this number renaming each col name
# Can call func rename columns ;)

dfStateMat.columns = ['R1', 'R2', 'R3', 'R4', 'R5', 'R6', 'R7', 'R8', 'R9', 'R10', 'R11', 'R12','R13','R14',
'R15', 'R16', 'R17', 'R18', 'R19', 'R20', 'R21', 'R22', 'R23', 'R24', 'R25', 'R26',
'R27', 'R28', 'R29', 'R30', 'R31', 'R32', 'R33', 'R34', 'R35', 'R36', 'R37', 'R38', 'R39', 'R40',
'R41', 'R42', 'R43', 'R44', 'R45', 'R46', 'R47', 'R48', 'R49', 'R50', 'R51', 'R52', 'R53',
'R54', 'R55', 'R56', 'R57', 'R58', 'R59']

# Record the frequencies of every observation for each state matrix aka find the mode of each observation and

with open('5-14-2014streamW{}PPstateFreqs.txt'.format(count), "w") as outFile, open('ModesandFreqs{}.txt'.format(count), 'w') as outFile1:
# Loops through the rows in the df we created, up to the last row
print('stateFreqs and ModesandFreq files created {}'.format(count))
for i in dfStateMat.index:

# Creates a counter object via the counter in the collections package for each row, ie provides a list of
# numbers and their frequencies,then turns this object into a dictionary and stores it into var sensorFreqs

sensorFreqs = dict(Counter(dfStateMat.loc[i]))

sortedSensorFreqs = sorted(sensorFreqs.items(), key=operator.itemgetter(1), reverse=True)

outFile.write(str(sortedSensorFreqs) + '\n')

# Creates an output File called ModesandFreqs for each state matrix - Should propbably separate this out from the for loop?

outFile1.write('{}'.format(str(sortedSensorFreqs[0]).strip('()') + ', ' + str(sortedSensorFreqs[1]).strip('()') + ',' + '\n'))

print('stateFreqs and ModesandFreq files written too and closed {}'.format(count))

with open('5-14-2014streamW{}PPTimeSeconds.txt'.format(count), "r") as inFile3, open('ModesandFreqs{}.txt'.format(count), "r") as inFile4:

print('ModesandFreqsWithTime files created {}'.format(count))

timeSeconds = pd.read_table(inFile3, sep = ' ', lineterminator = '\n')
timeSeconds.drop(timeSeconds.columns[[0]], axis=1, inplace=True)
timeSeconds.drop(timeSeconds.columns[[1]], axis=1, inplace=True)

ModesandFreq = pd.read_table(inFile4, sep = ',')
ModesandFreq.drop(ModesandFreq.columns[[4]], axis=1, inplace=True)
ModesandFreq

result = pd.concat([ModesandFreq, timeSeconds], axis=1, join='inner')

result.columns = ['Popular_Sensor1', 'Strength1', 'Popular_Sensor2', 'Strength2', 'Time_in_Secs']

# This commented line below would create a subset of the dataframe that we could then easily export
# dfFinalMat2 = dfFinalMat[['Popular_Sensor1', 'Strength1', 'Popular_Sensor2']]

dfFinalMat = pd.concat([dfStateMat, result], axis=1, join='inner')

# Examine the relationship between the Queens + breed Male and the general colony

dfFinalMat['QueenAtPopSens1'] = np.where(dfFinalMat['R1'] == dfFinalMat['Popular_Sensor1'], 1, 0)

dfFinalMat['Queen2AtPopSens1'] = np.where(dfFinalMat['R2'] == dfFinalMat['Popular_Sensor1'], 1, 0)

dfFinalMat['BreedMaleAtPopSens1'] = np.where(dfFinalMat['R3'] == dfFinalMat['Popular_Sensor1'], 1, 0)

# Find the total number of occurences the specific animals were with one another

dfFinalMat['Queen1isWithQueen2'] = np.where(dfFinalMat['R1'] == dfFinalMat['R2'], 1, 0)

dfFinalMat['Queen1isWithBreedMale'] = np.where(dfFinalMat['R1'] == dfFinalMat['R3'], 1, 0)

dfFinalMat['Queen2isWithBreedMale'] = np.where(dfFinalMat['R2'] == dfFinalMat['R3'], 1, 0)

# Find the total number of occurences that each animal is at that specific mean

print('Final Mat calcs completed {}'.format(count))

with open("QueensAndBreedSumsAtModeSens.txt", 'a') as outFile3:
outFile3.write("{}, {}, {}, {} \n".format(dfFinalMat['QueenAtPopSens1'].sum(), dfFinalMat['Queen2AtPopSens1'].sum(), dfFinalMat['BreedMaleAtPopSens1'].sum(), len(dfFinalMat.index)))

with open("QueensAndBreedSumsWithOneAnother.txt", 'a') as outFile4:
outFile4.write("{}, {}, {}, {} \n".format(dfFinalMat['Queen1isWithQueen2'].sum(), dfFinalMat['Queen1isWithBreedMale'].sum(), dfFinalMat['Queen2isWithBreedMale'].sum(), len(dfFinalMat.index)))

print('Both the QueensAndBreedSums files were appended {}'.format(count))

count += 1


Specs on the script reads in 800 and creates 1602 files in approximately 12 minutes and only utilized 30% of CPU on a core i5 processor ThinkPad T420 with 9.7 GB of RAM.

If you need a screenshot or an example of the data, please let me know.