Your solution looks quite ok to me. unfortunately you do not provide all necessary parameters to give an exact review. I do miss the number of lines in the file, the number or percentage of unique values. also there is no hint about the frequency of calls to that function. the points to discuss when knowing the numbers are
file write
If the whole problem fits into memory and there is a performance problem because of frequent calls you could write each file in a single flush. If there is no performance problem stick to the KISS variant. If you are not sure if it will always fit into memory stick to the kiss variant.
file read
You do open the imput file twice. This look a little inefficient. If you do this by intention to get better separation between software modules this is a reasonable idea. If you have latencies on file fetch it may be a very bad idea. I this peticular case I would do header analysis and data processing in a single file open.
regex split
Your regex split can most probably be done by the onboard split()
. Depending on your data line.split("\t")
or [x for x in line.split("\t") if x]
may fit. If this is a tab separated file the regex looks suspicous. the regex is missing documentation and/or examples.
algorithm over all
your solution is O(n), your memory footprint is low, you have a minimum of open files. all good. unless you have real performance problems this is fine.
other topics:
Counting keys
When you want to count keys in a dict()
you always end up with special handling for the first appearence of the key. There is Counter
existing in collections
which may be incremented even if the key is not exsting yet.
counting lines or other iterables you are iterating over
initializing a variable like rows
outside a loop and increment inside is error prone. Use enumerate()
.
variable names
your names are mostly fine and explaining the code. the exceptions are index
(for what?), cell_data
(are we talking about spreadsheets? and what is the data in this column?) and unique_values
(unique is fine but what values?)
cleaning up code
Whenever you think a program is done clean up. remove unnecessary code, clean/add comments and docstrings. you have unused counters for rows and unique values. you have comments raising questions, not answering them.
EDIT:
you may drop your unique_values
completely if you test the opened file for emptyness
with open(target, 'a') as outfile:
if f.tell() == 0:
outfile.write(first_line)
outfile.write(line)
Also, after you added some numbers, we know the problem does not fit into memory. So your algorithm is fine. If and only if you have performanceproblems you coud work on chunks (say 10.000 lines) of the inputfile buffering file writes and flush to disk at the end of every chunk only. you could change unique_values
to a defaultdict(list)
and append lines. at the end of the chunk you iterate over the unique_values, do path calculation from the key and write all lines. again you test for file emptyness to write the header.