6
\$\begingroup\$

The below code runs a t-statistic within a large dataframe (rnadf) based on masked values from another dataframe (cnvdf_maked). Before, I was putting my results into a dictionary and turning it into a pandas dataframe. These dataframes became too large for my memory, so I had to switch to writing the results to a file line by line, as below on the last line.

The above code takes a little less than a week to produce one complete file. I have many dataframes to run this on, as in many pairs of the DFs rnadf and cnvdf. I have 138 pairs of these DFs pickled in a directory, so I was making my script "parallel" by running each DF pair's ttest caluclation at once in its own screen session -- for a total of 138 screen sessions. This was a clunky way to do it. When result files get to be around 2.6Gb, they fail with an OS Error 30.

I've since moved to a new server with about 1Tb of memory, so I'm no longer practically constrained by memory. I'm open to doing two things to improve optimization: 1) make each calculated ttest faster, and 2) parallelize this process to run on many different dataframes at once without using separate screen sessions.

from scipy import stats
import pandas as pd
import numpy as np
import itertools

# sample dataframes
rnadf = pd.DataFrame(np.random.randint(0,100,size=(100, 26)), columns=list('ABCDEFGHIJKLMNOPQRSTUVWXYZ'))
cnvdf = pd.DataFrame(np.random.randint(0,1000,size=(100, 25)), columns=list('BCDEFGHIJKLMNOPQRSTUVWXY5'))
cnvdf_mask = cnvdf <= 500

inter = list(set(cnvdf_mask.columns).intersection(rnadf.columns))
cnvdf_mask, rnadf = cnvdf_mask[inter], rnadf[inter]

with open('out_tab.txt', 'w') as f:
    for pr in itertools.product(rnadf.index, cnvdf_mask.index):
        rnaPos = np.array(rnadf.loc[pr[0]][cnvdf_mask.loc[pr[1]]].dropna())
        rnaNeg = np.array(rnadf.loc[pr[0]][~cnvdf_mask.loc[pr[1]]].dropna())
        t, p = stats.ttest_ind(rnaPos, rnaNeg)
        f.write('{}\t{}\t{}\n'.format(t, p, pr))  # changed pr table to tuple of str indices joined by '&'
\$\endgroup\$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.