I have some code here that basically takes in a list of IDs (sequencing read names) using STDIN, and then filters
fin (a gzipped FASTQ file) to just those reads with a read name which matches one of the IDs, and then writes the name and sequence to
fout (a gzipped FASTA file). Fairly standard data processing problem in bioinformatics.
The code below can run through about 100k/sec lines of
fin is about 500M lines, so it still takes over an hour.
Can this be sped up significantly? I'm hoping for a 10x or better improvement in performance. I may be taking a performance hit running
line.decode('utf-8'), however it's necessary for matching since gzip files are read as bytestring. I'm willing to drop down and rewrite this in C if this will substantially increase speed.
import sys import gzip import re id_array =  for line in sys.stdin: id_array.append(line.rstrip()) ids = set(id_array) with gzip.open(sys.argv, "r") as fin: with gzip.open(sys.argv, "w") as fout: take_next_line = False for line in fin.readlines(): seq_search = re.search('[@+]([^\s]+)', line.decode('utf-8')) if take_next_line: fout.write(line) take_next_line = False elif seq_search and seq_search.group(1) in ids: take_next_line = True fout.write('>' + line[1:])