I have written a function to selectively extract data from a file. I want to be able to extract only from a certain line and only given rows.
Would convert this function into a generator improve the overhead when I need to process large files?
import itertools
import csv
def data_extraction(filename,start_line,lenght,span_start,span_end):
with open(filename, "r") as myfile:
file_= csv.reader(myfile, delimiter=' ') #extracts data from .txt as lines
return (x for x in [filter(lambda a: a != '', row[span_start:span_end]) \
for row in itertools.islice(file_, start_line, lenght)])