Here is the slow part of my code:
def Prepare_WCfst_file():
content_WCfst = "iteration\tlocus\tFromSite\tToSite\tWCFst\n"
nbchunks = nbsites/nbsitesPerChunk
if nbchunks%1 != 0:
print("\n\nThe number of sites per chunk for WCFst is not a divisor of the number of sites per locus.\nProcess is aborted!\n\n")
sys.exit(1)
nbchunks = int(nbchunks)
nbsitesPerChunk_minus1 = nbsitesPerChunk - 1
for iteration in xrange(nbiterations):
for locus in xrange(nbloci):
FromSite = 0
ToSite = nbsitesPerChunk_minus1
for chunk in xrange(nbchunks):
content_WCfst += str(iteration) + "\t" + str(locus) + "\t" + str(FromSite) + "\t" + str(ToSite) + "\n"
FromSite = ToSite + 1
ToSite = FromSite + nbsitesPerChunk_minus1
return content_WCfst
Typically, nbiterations
and nbloci
take values between 1 and 100. nbchunks
however can take values of the order of 10^6 to 10^9. This function will be called about 100 to 10,000 times in total.
Suggested data to benchmark
nbloci = 10^6
nbiterations = 20
nbloci = 2
nbsitesPerChunk = 10
What the function does
Basically, what this piece of code does is to create a long string content_WCfst
(which will afterward be written on a file). The file needs to contain 4 columns: iteration
, locus
, FromSite
and ToSite
. The difference between the columns FromSite
and ToSite
is always nbsitesPerChunk_minus1
. By the end the number of lines is nbiterations * nbloci * nbchunks
.
My thoughts about what to improve
I think the part that can be improved is the three loops or eventually only the most internal loop (for chunk in xrange(nbchunks):
)
Note that I've tried to replace
str(iteration) + "\t" + str(locus) + "\t" + str(FromSite) + "\t" + str(ToSite) + "\n"
by
"\t".join(str(iteration), str(locus), str(FromSite), str(ToSite), "\n")
but it was even slower.
Note also that the function doesn't receive any arguments and therefore, all parameters are defined globally outside this function. I don't expect it to be an issue as they are not called over and over again.
I am hoping that a solution can exist so that the looping occurs in C, using NumPy eventually.