# Using generator for buffered read of large file in Python

I have a large file that I need to parse - and since it will be regenerated from external queries every time script runs so there is no way to parse it once and cache the results.

I want to save memory footprint and read and parse only logical "chunks" of that file everything between open 'product' and closing curly bracket. Not sure what is the canonical way in Python (I am new to the language).

Here's what I tried so far:

def read_chunk(file_name, pattern_open_line, pattern_close_line):
with open(file_name,"r") as in_file:
chunk = []
in_chunk = False
open_line = re.compile(pattern_open_line);
close_line = re.compile(pattern_close_line)
try:
for line in in_file:
line = line.strip()
if in_chunk:
chunk.append(line)
if close_line.match(line):
yield chunk
if open_line.match(line):
chunk = []
chunk.append(line)
in_chunk = True
continue
except StopIteration:
pass

def get_products_buffered(infile):
chunks = read_chunk(infile, r'^product\s*$', r'^\s*\}\s*') products = [] for lines in chunks: for line in lines: if line.startswith('productNumber:'): productNumber = line[len('productNumber:'):].strip().rstrip(';').strip('"') products.append(productNumber) continue return products def get_products_unbuffered(infile): with open(infile) as f: lines = f.readlines() f.close() products = [] for line in lines: if line.startswith('productNumber:'): productNumber = line[len('productNumber:'):].strip().rstrip(';').strip('"') products.append(productNumber) continue return products  I profiled both runs and while unbuffered reading is faster: Buffered reading Found 9370 products: Execution time: 3.0031037185720177 Unbuffered reading Found 9370 products: Execution time: 1.2247122452647523  it also incurs a much bigger memory hit when file is essentially read into memory: Line # Mem usage Increment Line Contents ================================================ 29 28.2 MiB 0.0 MiB @profile 30 def get_products_buffered(infile): 31 28.2 MiB 0.0 MiB chunks = read_chunk(infile, '^product\s*$', '^\s*\}\s*')
32     28.2 MiB      0.0 MiB       products = []
33     30.1 MiB      1.9 MiB       for lines in chunks:


versus:

Line #    Mem usage    Increment   Line Contents
================================================
42     29.2 MiB      0.0 MiB   @profile
43                             def get_products_unbuffered(infile):
44     29.2 MiB      0.0 MiB       with open(infile) as f:
45    214.5 MiB    185.2 MiB           lines = f.readlines()


I would be grateful for any pointers/suggestions.

You called it unbuffered, but these lines:

with open(infile) as f:
f.close()


slurp the entire file into memory, while your 'buffered' version only pulls in a line at a time, returning chunks.

I note that you're not doing anything with the entire chunk, just the lines starting with 'productNumber:', so I think a rework of your 'unbuffered' code will actually be fastest, as well as clearest:

def get_products_unbuffered(infile):
products = []
with open(infile) as f:
for line in f:
if line.startswith('productNumber:'):
productNumber = line[len('productNumber:'):].strip().rstrip(';').strip('"')
products.append(productNumber)
return products


as this will read the file a line at a time and only keep desired info (productNumbers) in memory..

• Thank you! I do agree with the critique - that nothing meaningful is done after reading in a "chunk" - I gutted the example - real life code did a lot of pattern matching. "productNumber" was just the first one to provide key for all data pulled out of chunk. And yes - "unbuffered" was bad name - should call it "in_memory" or sth like that. My main question was that there must be a canonical way to deal with such problem. – Tom N Apr 18 '18 at 19:42