I have a tab delimited file which contains about 2207 rows with 4 columns of floats or int types:

Id    pos    value1    value2
01    1123   0.76483   0.9001
02    1124   0.20      0.9800

Another file is to be compared with which contains about 27041 rows with also 4 columns:

name    Id    pos1    pos2
s13e    01    234     800
zz12    01    800     866

Now, my goal is to find all possible consecutive rows of 3 or higher in which:

  1. Their contents of the first column of the first file (Id) are equal to the second column of the second file (Id).
  2. Their contents of the second column of the first file (pos) is in between the range of column 3 and column 4 of the second file.

My approach is to do the following:

  1. Loop through a sliding window of length 3 to the entire length number of rows of the first file (that's my outer loop)
  2. Second loop is to iterate and take each element of the sliding window of some specific length like 3 for instance and then
  3. Check if that element satisfy the conditions above if it is satisfy then break the loop to take the next element of the first file
  4. Otherwise, continue searching for the second file to find the target

For a length 300 rows of the first file the whole sliding window thing with the checking to the second file takes about 5 minutes. However, I am looking for a faster and more efficient way to do that.

An example of a satisfied result is the following:

let's say we have performed a window sliding of a length 3, Note that I have to iterate over the entire length of the file (number of rows) and the length of 3 for the sliding window is just an example!

From file 1:

idW = (01,01,01)
posW = (127, 192,199)
val1W = (.9,.01,.23)
val2W = (.2,.03,.43)

From file 2:

name    Id    pos1    pos2
s13e    01    120     200

For a successful result, the following will have to be met:

  • Consecutive rows of 3 rows and (true)
  • Id of file1 are equal file2 and (true)
  • posW from the first file are in between the values of pos1 and pos2 (true)

The result is satisfy because all conditions above are true and met.

This code is working fine but I want it to be the fastest possible and more efficient:

# borrowed from one answer from stack overflow 
def sliding_window (s, n):
 it = iter(seq)
 result = tuple(islice(it, n))
 if len(result) == n:
    yield result
 for elem in it:
    result = result[1:] + (elem,)
    yield result

Ids, pos, val1, val2 = zip(*Data)
# len_Data is the entire length of the file 
for windwCounts in range(3,len_Data):
    file2count = 0
    swindow_id = sliding_window(Ids, windwCounts)
    swindow_pos = sliding_window(pos, windwCounts)
    swindow_val1 = sliding_window(val1, windwCounts)
    swindow_val2 = sliding_window(val2, windwCounts)
    slidingWindws = zip(swindow_id, swindow_pos,
                        swindow_val1, swindow_val2)
    for idW, posW, val1W, val2W in slidingWindws:
     while (file2count <= len_file2):
            name,id2,pos1,pos2 = file2[file2count]
            # No point of moving down if the pos of file1 is less than pos2 of file2
            if any(int(ps) < pos2 for ps in posW):
            # All conditions must be met:-
            ids_cond = all(a_id == id2 for a_id in idW)
            within_data = all(pos1 <= int(ps) <= pos2
                                       for ps in posW)
            if ids_cond and within_data:

                  print "satisfy"

                  print "not satisfy"
            file2count += 1
  • \$\begingroup\$ I encountered your post when looking for old, unanswered question. It appears your question is incomplete. The imports are missing and you're not opening any files here. I'm perfectly willing to analyse your code to dust, but I'll need something complete to run to verify whether my improvements are actually faster than your approach. If you're willing to add that information, that would be great. You may be interested in reading our FAQ on getting the most out of Code Review. \$\endgroup\$ – Mast Dec 19 '17 at 23:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.