I have a matrix of fewer than 1021 rows. I'd like to take its transpose, but it is too big for GNU
awk solutions to fit into memory.
To keep memory overhead low, my thought is that I keep a list of read-only file pointers to the start of each row. I then read out bytes from each pointer until I hit a delimiter (tab or newline).
Because I have fewer than 1021 rows, I won't hit the usual 1024 OS-based file pointer limit.
Once I have read out a field from all file pointers, I write a newline character and start over, reading a field from all file pointers, and again, until no more bytes are available:
#!/usr/bin/env python import sys import os try: in_fn = sys.argv except ValueError as ve: sys.exit(-1) def get_size(fn): st = os.stat(fn) return st.st_size # # 1) Read in file offsets to start of new line # 2) Open up a file pointer to that offset # 3) Process each file pointer to get a value until a delimiter is hit, then write it as a row of output # size = get_size(in_fn) fps =  new_fp = open(in_fn, 'r') new_fp.seek(0, 0) fps.append(new_fp) with open(in_fn, 'r') as f: byte = f.read(1) while byte: if byte == '\n': new_offset = f.tell() if new_offset < size: new_fp = open(in_fn, 'r') new_fp.seek(new_offset, 0) fps.append(new_fp) else: break byte = f.read(1) while size > 0: for fi, f in enumerate(fps): byte = f.read(1) size -= 1 while byte: if byte == '\t' or byte == '\n': if fi != len(fps) - 1: sys.stdout.write('\t') break sys.stdout.write('%s' % (byte)) byte = f.read(1) size -= 1 sys.stdout.write('\n') for f in fps: f.close()
Is there anything I can do to improve the performance of this? Reading and processing a set of file pointers one byte at a time seems quite expensive. However, I need to find newline characters to build offsets and file pointers. Is there a cleverer/faster way (in Python) to find the byte offsets of newlines?