I am writing a long piece of code, which is taking way too long to execute. I used cProfile on the code, I found that the following function is called 150 times and takes 1.3 seconds per call, leading to around 200 seconds to this function alone. The function is -
def makeGsList(sentences,org): gs_list1= gs_list2= for s in sentences: if s.startswith(tuple(StartWords)): s = s.lower() if org=='m': gs_list1 = [k for k in m_phrases if k in s] if org=='h': gs_list1 = [k for k in h_phrases if k in s] for gs_element in gs_list1: gs_list2.append(gs_element) gs_list3 = list(set(gs_list2)) return gs_list3
The code is supposed to take a list of sentences and a flag
org. Then it goes through each line, checks if it starts with any of the words present in the list
StartWords, and then lower-cases it. Then, depending on the value of
org, it makes a list of all phrases in the current sentence which are also present in either
h_phrases. It keeps appending these phrases to another list
gs_list2. Finally it makes a set of
gs_list2 and returns it.
Can someone give me any suggestion as to how I can optimize this function to reduce the time taken to execute?
Some examples -
StartWords = ['!Series_title','!Series_summary','!Series_overall_design','!Sample_title','!Sample_source_name_ch1','!Sample_characteristics_ch1'] sentences = [u'!Series_title\t"Transcript profiles of DCs of PLOSL patients show abnormalities in pathways of actin bundling and immune response"\n', u'!Series_summary\t"This study was aimed to identify pathways associated with loss-of-function of the DAP12/TREM2 receptor complex and thus gain insight into pathogenesis of PLOSL (polycystic lipomembranous osteodysplasia with sclerosing leukoencephalopathy). Transcript profiles of PLOSL patients\' DCs showed differential expression of genes involved in actin bundling and immune response, but also for the stability of myelin and bone remodeling."\n', u'!Series_summary\t"Keywords: PLOSL patient samples vs. control samples"\n', u'!Series_overall_design\t"Transcript profiles of in vitro differentiated DCs of three controls and five PLOSL patients were analyzed."\n', u'!Series_type\t"Expression profiling by array"\n', u'!Sample_title\t"potilas_DC_A"\t"potilas_DC_B"\t"potilas_DC_C"\t"kontrolli_DC_A"\t"kontrolli_DC_C"\t"kontrolli_DC_D"\t"potilas_DC_E"\t"potilas_DC_D"\n', u'!Sample_characteristics_ch1\t"in vitro differentiated DCs"\t"in vitro differentiated DCs"\t"in vitro differentiated DCs"\t"in vitro differentiated DCs"\t"in vitro differentiated DCs"\t"in vitro differentiated DCs"\t"in vitro differentiated DCs"\t"in vitro differentiated DCs"\n', u'!Sample_description\t"DAP12mut"\t"DAP12mut"\t"DAP12mut"\t"control"\t"control"\t"control"\t"TREM2mut"\t"TREM2mut"\n'] h_phrases = ['pp1665', 'glycerophosphodiester phosphodiesterase domain containing 5', 'gde2', 'PLOSL patients', 'actin bundling', 'glycerophosphodiester phosphodiesterase 2', 'glycerophosphodiester phosphodiesterase domain-containing protein 5']
m_phrases are similar. Assume in this case, org=
Regarding sizes -
The length of both lists
m_phrases is around 250,000. And each element in the lists is on an average 2 words long. The list of sentences is around 10-20 sentences long and I have provided an example list to give you the idea of how big each sentence can be.