I am currently attempting to make some code more maintainable for a research project I am working on. I am definitely looking to create some more functions, and potentially create a general class to conduct such data calculations.

The code works, however, I am new to Python, and was wondering if someone could review my code for efficiency, formatting, readability, maintainability, scaling, etc. The data I am taking in are .txt files with 59 columns and some number of rows depending on how many observations are within the file (an example of a normal amount of rows is around 4400).

One of my main concerns is how to limit the amount of opening and closing of files. I am writing these files to store them in case I need to reference them to justify my calculations. Is there any way I can do this in parallel maybe, or even utilize threading?

Also, can I write it out in binary? Would that enhance the script's speed as well?

Here is an example of optimizations I would like to make:

I have created a small function. This way, a user can input and automagically have the script create column headers based on the size of the data frame taken in:

def createDFNames(numCols, colName):
    '''Generates column names for the dataframe based on how many columns there are'''
    colNames = []
    col = 1
    for col in range(numCols):
        colNames.append('{}{}'.format(colName , col + 1))
    return colNames

I am utilizing anaconda in a Jupyter notebook to conduct these calcs:

# Combining Script 1, and Script 2 to make one main script 

# Begin reading files into a dataframe one by one, perform the necessary calculations, and save the results to a 
# output file

count = 1

# Read the file into the dataframe 

for file in stateFiles:

dfStateMat = pd.read_table(file, sep='\t', lineterminator = '\n')

numObs = len(dfStateMat.index)
numCols = len(dfStateMat.index)

print("Current File Being Processed is: " + file)

# Rename the columns of the dataframe - Hardcoded for now, but can create a func that 
# Reads how many cols there are, then runs through a loop up to this number renaming each col name 
# Can call func rename columns ;)

dfStateMat.columns = ['R1', 'R2', 'R3', 'R4', 'R5', 'R6', 'R7', 'R8', 'R9', 'R10', 'R11', 'R12','R13','R14',
         'R15', 'R16', 'R17', 'R18', 'R19', 'R20', 'R21', 'R22', 'R23', 'R24', 'R25', 'R26',
         'R27', 'R28', 'R29', 'R30', 'R31', 'R32', 'R33', 'R34', 'R35', 'R36', 'R37', 'R38', 'R39', 'R40',
         'R41', 'R42', 'R43', 'R44', 'R45', 'R46', 'R47', 'R48', 'R49', 'R50', 'R51', 'R52', 'R53',
         'R54', 'R55', 'R56', 'R57', 'R58', 'R59']

# Record the frequencies of every observation for each state matrix aka find the mode of each observation and 

with open('5-14-2014streamW{}PPstateFreqs.txt'.format(count), "w") as outFile, open('ModesandFreqs{}.txt'.format(count), 'w') as outFile1:
    # Loops through the rows in the df we created, up to the last row
    print('stateFreqs and ModesandFreq files created {}'.format(count))
    for i in dfStateMat.index:  

        # Creates a counter object via the counter in the collections package for each row, ie provides a list of 
        # numbers and their frequencies,then turns this object into a dictionary and stores it into var sensorFreqs 

        sensorFreqs = dict(Counter(dfStateMat.loc[i]))                  

        sortedSensorFreqs = sorted(sensorFreqs.items(), key=operator.itemgetter(1), reverse=True)

        outFile.write(str(sortedSensorFreqs) + '\n')

        # Creates an output File called ModesandFreqs for each state matrix - Should propbably separate this out from the for loop? 

        outFile1.write('{}'.format(str(sortedSensorFreqs[0]).strip('()') + ', ' + str(sortedSensorFreqs[1]).strip('()') + ',' + '\n'))

print('stateFreqs and ModesandFreq files written too and closed {}'.format(count))

with open('5-14-2014streamW{}PPTimeSeconds.txt'.format(count), "r") as inFile3, open('ModesandFreqs{}.txt'.format(count), "r") as inFile4:

    print('ModesandFreqsWithTime files created {}'.format(count))

    timeSeconds = pd.read_table(inFile3, sep = ' ', lineterminator = '\n')
    timeSeconds.drop(timeSeconds.columns[[0]], axis=1, inplace=True)
    timeSeconds.drop(timeSeconds.columns[[1]], axis=1, inplace=True)

    ModesandFreq = pd.read_table(inFile4, sep = ',')
    ModesandFreq.drop(ModesandFreq.columns[[4]], axis=1, inplace=True)

    result = pd.concat([ModesandFreq, timeSeconds], axis=1, join='inner')

    result.columns = ['Popular_Sensor1', 'Strength1', 'Popular_Sensor2', 'Strength2', 'Time_in_Secs']

    # This commented line below would create a subset of the dataframe that we could then easily export
    # dfFinalMat2 = dfFinalMat[['Popular_Sensor1', 'Strength1', 'Popular_Sensor2']]

    result.to_csv('/home/zackymo/Desktop/ModesAndFreqs/ModesandFreqsWithTime{}'.format(count), header=False, index=False)

    dfFinalMat = pd.concat([dfStateMat, result], axis=1, join='inner')

# Examine the relationship between the Queens + breed Male and the general colony 

dfFinalMat['QueenAtPopSens1'] = np.where(dfFinalMat['R1'] == dfFinalMat['Popular_Sensor1'], 1, 0)

dfFinalMat['Queen2AtPopSens1'] = np.where(dfFinalMat['R2'] == dfFinalMat['Popular_Sensor1'], 1, 0)

dfFinalMat['BreedMaleAtPopSens1'] = np.where(dfFinalMat['R3'] == dfFinalMat['Popular_Sensor1'], 1, 0)

# Find the total number of occurences the specific animals were with one another 

dfFinalMat['Queen1isWithQueen2'] = np.where(dfFinalMat['R1'] == dfFinalMat['R2'], 1, 0)

dfFinalMat['Queen1isWithBreedMale'] = np.where(dfFinalMat['R1'] == dfFinalMat['R3'], 1, 0)

dfFinalMat['Queen2isWithBreedMale'] = np.where(dfFinalMat['R2'] == dfFinalMat['R3'], 1, 0)

# Find the total number of occurences that each animal is at that specific mean

print('Final Mat calcs completed {}'.format(count))

with open("QueensAndBreedSumsAtModeSens.txt", 'a') as outFile3:
    outFile3.write("{}, {}, {}, {} \n".format(dfFinalMat['QueenAtPopSens1'].sum(), dfFinalMat['Queen2AtPopSens1'].sum(), dfFinalMat['BreedMaleAtPopSens1'].sum(), len(dfFinalMat.index)))

with open("QueensAndBreedSumsWithOneAnother.txt", 'a') as outFile4:
    outFile4.write("{}, {}, {}, {} \n".format(dfFinalMat['Queen1isWithQueen2'].sum(), dfFinalMat['Queen1isWithBreedMale'].sum(), dfFinalMat['Queen2isWithBreedMale'].sum(), len(dfFinalMat.index)))    

print('Both the QueensAndBreedSums files were appended {}'.format(count))

count += 1

Specs on the script reads in 800 and creates 1602 files in approximately 12 minutes and only utilized 30% of CPU on a core i5 processor ThinkPad T420 with 9.7 GB of RAM.

If you need a screenshot or an example of the data, please let me know.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.