I have a tab delimited file which contains about 2207 rows with 4 columns of
Id pos value1 value2 01 1123 0.76483 0.9001 02 1124 0.20 0.9800
Another file is to be compared with which contains about 27041 rows with also 4 columns:
name Id pos1 pos2 s13e 01 234 800 zz12 01 800 866
Now, my goal is to find all possible consecutive rows of 3 or higher in which:
- Their contents of the first column of the first file (Id) are equal to the second column of the second file (Id).
- Their contents of the second column of the first file (pos) is in between the range of column 3 and column 4 of the second file.
My approach is to do the following:
- Loop through a sliding window of length 3 to the entire length number of rows of the first file (that's my outer loop)
- Second loop is to iterate and take each element of the sliding window of some specific length like 3 for instance and then
- Check if that element satisfy the conditions above if it is satisfy then break the loop to take the next element of the first file
- Otherwise, continue searching for the second file to find the target
For a length 300 rows of the first file the whole sliding window thing with the checking to the second file takes about 5 minutes. However, I am looking for a faster and more efficient way to do that.
An example of a satisfied result is the following:
let's say we have performed a window sliding of a length 3, Note that I have to iterate over the entire length of the file (number of rows) and the length of 3 for the sliding window is just an example!
From file 1:
idW = (01,01,01) posW = (127, 192,199) val1W = (.9,.01,.23) val2W = (.2,.03,.43)
From file 2:
name Id pos1 pos2 s13e 01 120 200
For a successful result, the following will have to be met:
- Consecutive rows of 3 rows and (true)
- Id of file1 are equal file2 and (true)
posWfrom the first file are in between the values of
The result is satisfy because all conditions above are true and met.
This code is working fine but I want it to be the fastest possible and more efficient:
# borrowed from one answer from stack overflow def sliding_window (s, n): it = iter(seq) result = tuple(islice(it, n)) if len(result) == n: yield result for elem in it: result = result[1:] + (elem,) yield result Ids, pos, val1, val2 = zip(*Data) # len_Data is the entire length of the file for windwCounts in range(3,len_Data): file2count = 0 swindow_id = sliding_window(Ids, windwCounts) swindow_pos = sliding_window(pos, windwCounts) swindow_val1 = sliding_window(val1, windwCounts) swindow_val2 = sliding_window(val2, windwCounts) slidingWindws = zip(swindow_id, swindow_pos, swindow_val1, swindow_val2) for idW, posW, val1W, val2W in slidingWindws: while (file2count <= len_file2): name,id2,pos1,pos2 = file2[file2count] # No point of moving down if the pos of file1 is less than pos2 of file2 if any(int(ps) < pos2 for ps in posW): break # All conditions must be met:- ids_cond = all(a_id == id2 for a_id in idW) within_data = all(pos1 <= int(ps) <= pos2 for ps in posW) if ids_cond and within_data: print "satisfy" break else: print "not satisfy" file2count += 1