My goal is to reduce processing time significantly (if possible) by making this working code more efficient. Currently 50k row by 105 column data taking about overall 2 hours to process. Share of this piece is 95%.
This piece is a key part of my Python 3.6.3 script that compares two set of list of lists element by element regardless of datatype. I spent long hours but seems I reached my limits here. This is running on Windows 10.
Sorry about lots of variables. Here is description:
Bu - list of lists. Each sublist
- may contain any datatype (usually String, Number, Null, Date).
- 1st element of list within list is always unique string.
- has equal number of elements as other lists
- each list in
Aphas corresponding list in
Bu(corresponding in here means 1st element and ID element (not necessarily other elements) of a sublist of
Apmatches that of
Bu, that's considered there is corresponding match)
- each sublist will contain unique ID in the same position. Meaning index of ID is same in every sublist (be it Ap or Bu).
prx - is index of a list within
urx - corresponding/matching index of a list within
Bu, as evidenced by
cx - is index of an element in a single list of Au
ux - is a corresponding element index of an element in a matching list of
as evidenced by
ux = l_ux.index(cx)
rng_lenAp - is
rng_ls - is
range(individual list within Ap)
To visualize (just example):
Ap = [['egg', 12/12/2000, 10, ID1, NULL], ['goog', 23, 100, ID2,12/12/2000]] Bu = [['goog', '3434', 100, ID2, 12/12/2000], ['egg', 12/12/2000, 45, ID1, NULL]]
for prx in rng_lenAp: urx = l_urx.index (prx) if Ap[prx] == Bu[urx]: for cx in rng_ls: ux = l_ux.index(cx) #If not header, non-matching cells get recorded with their current value if cx!=0 and Ap[prx][cx] != Bu[urx][ux]: output[prx].append (str(Ap[prx][cx] + '^' + str(Bu[urx][ux])) #Unless it is row header or ID in column, matching cells gets 'ok' elif cx!=0 and prx!=0 and urx !=0 and Ap[prx][cx] == Bu[urx][ux]: output[prx].append ('ok' +'^' + 'ok') # Anything else gets recorded with their current value else: output[prx].append (str(Ap[prx][cx] + '^' + str(Bu[urx][ux]))
There must a way to reduce processing time drastically. Currently the cell by cell comparison of 50k row by 100 column data to 50k row by 100 column data is takes about 2 hours. Expected under 30 min. 3.1 GHz, 4 CPUs (8196MB RAM).