This script is to calculate the Spearman correlation of genes that are stored in rows of a pandas dataframe. The original df has around 2000 row names. With my current code, this requires 2000*2000 iterations to compute the correlation value for each possible comparison row-wise. I would like to get some hints in order to reduce the run time of this code.

import sys
import pandas as pd
from scipy import stats

def spearman_correlation():
    script = sys.argv[0]
    input_f = sys.argv[1]

    # create a new file name... it's that easy!
    positive = "positive_significant_pairs.csv"
    negative = "negative_significant_pairs.csv"

    positive_f = open(positive, 'a')
    negative_f = open(negative, 'a')

    df =pd.read_csv(input_f,index_col=0) #index_col=0 makes first row as row index
    for i in df.T.columns: #loop on the columns (transformed df)

    for i in lst1:
        for j in lst2:
            if i!=j: #avoid correlation of gene1,gene1
                r, p = stats.spearmanr(df.T[i], df.T[j])
                if r > 0 and p < 0.01:
                    positive_f.write(''.join(str(i) + ',' + str(j) + ',' + str(r) + ',' + str(p)+'\n'))
                if r < 0 and p < 0.01:
                    negative_f.write(''.join(str(i) + ',' + str(j) + ',' + str(r) + ',' + str(p)+'\n'))

if __name__ == '__main__':
  • \$\begingroup\$ Could you please include some sample data and current output? \$\endgroup\$
    – kubatucka
    Oct 22, 2021 at 20:45
  • 1
    \$\begingroup\$ Do not loop. Pass the entire df.T at once. The first argument to scipy.stats.spearmanr coould be a 2D array, just for the cases like yours. \$\endgroup\$
    – vnp
    Oct 22, 2021 at 21:44
  • \$\begingroup\$ Exactly I can pass the entire df.T and it gives me r and p values in a list. But then How I can retrieve rows that have specific r and p values from the 2D array? \$\endgroup\$
    – Apex
    Oct 24, 2021 at 19:06
  • \$\begingroup\$ The output that I want to have is in this format of row1,row2,r-value,p-value \$\endgroup\$
    – Apex
    Oct 24, 2021 at 19:07

1 Answer 1


Sorry if my comment looked too cryptic. Let's take a closer look at the Spearman statistics.

The Spearman coefficient between two sets of raw data is \$\dfrac{\mathrm{Cov}(R(T_i), R(T_j))}{\sigma_{R(T_i)}\sigma_{R(T_j)}}\$. It means that each time you call stats.spearmanr, it converts each row into the rank row, and computes its standard deviation.

In other words, each row is converted to the rank row 2000 times, and its standard deviation is also computed 2000 times, obviously with the same result.

Passing the entire dataset allows stats.spearmanr to extract these repeated computations, and perform them just once per a row. Unfortunately, computing covariances is still quadratic.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.