# Calculate correlation of genes stored in dataframe rows

This script is to calculate the Spearman correlation of genes that are stored in rows of a pandas dataframe. The original df has around 2000 row names. With my current code, this requires 2000*2000 iterations to compute the correlation value for each possible comparison row-wise. I would like to get some hints in order to reduce the run time of this code.

import sys
import pandas as pd
from scipy import stats

def spearman_correlation():
script = sys.argv[0]
input_f = sys.argv[1]

# create a new file name... it's that easy!
positive = "positive_significant_pairs.csv"
negative = "negative_significant_pairs.csv"

positive_f = open(positive, 'a')
negative_f = open(negative, 'a')

df =pd.read_csv(input_f,index_col=0) #index_col=0 makes first row as row index

lst1=[]
lst2=[]
for i in df.T.columns: #loop on the columns (transformed df)
lst1.append(i)
lst2.append(i)

for i in lst1:
for j in lst2:
if i!=j: #avoid correlation of gene1,gene1
r, p = stats.spearmanr(df.T[i], df.T[j])
if r > 0 and p < 0.01:
positive_f.write(''.join(str(i) + ',' + str(j) + ',' + str(r) + ',' + str(p)+'\n'))
if r < 0 and p < 0.01:
negative_f.write(''.join(str(i) + ',' + str(j) + ',' + str(r) + ',' + str(p)+'\n'))

if __name__ == '__main__':
spearman_correlation()
$$$$

• Could you please include some sample data and current output? Oct 22, 2021 at 20:45
• Do not loop. Pass the entire df.T at once. The first argument to scipy.stats.spearmanr coould be a 2D array, just for the cases like yours.
– vnp
Oct 22, 2021 at 21:44
• Exactly I can pass the entire df.T and it gives me r and p values in a list. But then How I can retrieve rows that have specific r and p values from the 2D array?
– Apex
Oct 24, 2021 at 19:06
• The output that I want to have is in this format of row1,row2,r-value,p-value
– Apex
Oct 24, 2021 at 19:07

The Spearman coefficient between two sets of raw data is $$\\dfrac{\mathrm{Cov}(R(T_i), R(T_j))}{\sigma_{R(T_i)}\sigma_{R(T_j)}}\$$. It means that each time you call stats.spearmanr, it converts each row into the rank row, and computes its standard deviation.
Passing the entire dataset allows stats.spearmanr` to extract these repeated computations, and perform them just once per a row. Unfortunately, computing covariances is still quadratic.