I have a matrix of fewer than 1021 rows. I'd like to take its transpose, but it is too big for GNU datamash and awk solutions.

My thought is that I keep a list of read-only file pointers to the start of each row. I then read out bytes from each pointer until I hit a delimiter (tab or newline).

Because I have fewer than 1021 rows, I won't hit the usual 1024 OS-based file pointer limit.

Once I have read out a field from all file pointers, I write a newline character and start over, reading a field from all file pointers, and again, until no more bytes are available:

#!/usr/bin/env python                                                                                                                                                                                                                               

import sys
import os

try:
  in_fn = sys.argv[1]
except ValueError as ve:
  sys.exit(-1)

def get_size(fn):
  st = os.stat(fn)
  return st.st_size

#                                                                                                                                                                                                                                                   
# 1) Read in file offsets to start of new line                                                                                                                                                                                                      
# 2) Open up a file pointer to that offset                                                                                                                                                                                                          
# 3) Process each file pointer to get a value until a delimiter is hit, then write it as a row of output                                                                                                                                            
#                                                                                                                                                                                                                                                   

size = get_size(in_fn)

fps = []
new_fp = open(in_fn, 'r')
new_fp.seek(0, 0)
fps.append(new_fp)

with open(in_fn, 'r') as f:
  byte = f.read(1)
  while byte:
    if byte == '\n':
      new_offset = f.tell()
      if new_offset < size:
        new_fp = open(in_fn, 'r')
        new_fp.seek(new_offset, 0)
        fps.append(new_fp)
      else:
        break
    byte = f.read(1)

while size > 0:
  for fi, f in enumerate(fps):
    byte = f.read(1)
    size -= 1
    while byte:
      if byte == '\t' or byte == '\n':
        if fi != len(fps) - 1: sys.stdout.write('\t')
        break
      sys.stdout.write('%s' % (byte))
      byte = f.read(1)
      size -= 1
  sys.stdout.write('\n')

for f in fps:
  f.close()

Is there anything I can do to improve the performance of this? Reading and processing a set of file pointers one byte at a time seems quite expensive. However, I need to find newline characters to build offsets and file pointers. Is there a cleverer/faster way (in Python) to find the byte offsets of newlines?