Separation of concerns. You're mingling code to parse your data with code to process it once it is parsed. Instead, parse everything into the appropriate data structures first, and then do the processing.
Use appropriate libraries and data structures. When you're doing linear algebra, that means numpy arrays. Numpy is a dedicated, LA library that has heavily optimised routines for things like dotvarious matrix products, so you don't need to roll your own. Numpy arrays will also use quite a bit less memory than Python lists.
Separation of concerns. You're mingling code to parse your data However, what you're working with code to process it onceis labelled data (and some of it is parsed. Insteadlabelled in two dimensions), parse everything intowhich means the appropriatebest data structuresstructure is probably a firstpandas dataframe - it's basically like your dict of strings to lists in
pat_mRNA, but much more optimised, and then do the processingfully compatible with numpy's matrix manipulations.
This is a list of lists of strings. For most of the rows, you don't seem to care about column 0 at all. Column 1 contains strings identifying the row labels, and corresponding to keys in your dictionary pat_mRNA. The rest of it is numeric data, currently stored as strings. A better data structure for this would be something more like your pat_mRNA: a dictionary mapping the string identifier to a vector of floats (except I would use a numpy array instead of a list to represent the vector). Since the strings are apparently names of genes, I assume they have a standard format and a maximum (or even fixed) length? If that is the case, you could use a numpy record array - essentially a matrix-like structure where the columns can have different types - which may make things a little easier depending if you need to do anything else with this data.
It also seems to containcontains 'header' rows - you only keep one of those around, so presumably you only expect one to be in there. SoI'm guessing that is column labels, which a DataFrame lets you include directly in line with pointthe same data structure.
The only trick parsing this into a DataFrame is that it is a little easier (2and, apparently, more efficient) aboveto swap the columns and rows from how you have them currently. That will affect how it prints, let'sand which methods you call to relabel things, but not much else. This is how I would parse this fileit into a dictionary mapping strings to np arrays.DataFrame:
exp_matdata = {}pd.DataFrame()
headers = []
for line in ccle:
row = line.rstrip().split('\t')
if row[0] == 'Name':
ccle_exp_headheaders = row
elif row[1] in pat_mRNA:
exp_mat[row[1]]data[row[1]] = np.array([float(f) for f in row[2:]])
if headers:
data.index = headers
I've maintained a temporary 'headers' list, because your code is currently robust against those appearing anywhere in the file. A better way to maintain that robustness would be if you know - or, preferably, can parse from your file before now - how many variables you have data for. Then you could do away with that headers list and do this:
data = pd.DataFrame(index=range(nvars))
for line in ccle:
row = line.rstrip().split('\t')
if row[0] == 'Name':
data.index = row
elif row[1] in pat_mRNA:
data[row[1]] = [float(f) for f in row[2:]]
In either case, I am assuming that you have one header for each variable. If that's a bad assumption, you may need to adjust this code.
As an aside, as a matter of commonly-accepted Python style (and in accordance with PEP 8), I have placed all of the code for each branch of the if in indented blocks, even if it is only one line. At most, you should only put it immediately after the : if there is only one branch to the if (no else or elifs), and even then, consider an indented block anyway.
I would change pat_mRNA to also be a matching data structure to thisDataFrame. FortunatelyYou could change how you currently build it, this is a lot easier - add an appropriateor you could just pass the whole dictionary you have at the moment into nppd.arrayDataFrame() call to the code that current builds it, so that each list becomes an array just before it goes in the dict.
Once you've done that, you can use numpy features to do your productLA work. As written, your entire strange product becomes this:
dot_prod_total = sum(np.outer(exp_mat[key]data[key], pat_mRNA[key]).flat for key in exp_matdata)