Your solution looks quite ok to me. unfortunately you do not provide all necessary parameters to give an exact review. I do miss the number of lines in the file, the number or percentage of unique values. also there is no hint about the frequency of calls to that function. the points to discuss when knowing the numbers are
If the whole problem fits into memory and there is a performance problem because of frequent calls you could write each file in a single flush. If there is no performance problem stick to the KISS variant. If you are not sure if it will always fit into memory stick to the kiss variant.
You do open the imput file twice. This look a little inefficient. If you do this by intention to get better separation between software modules this is a reasonable idea. If you have latencies on file fetch it may be a very bad idea. I this peticular case I would do header analysis and data processing in a single file open.
Your regex split can most probably be done by the onboard
split(). Depending on your data
[x for x in line.split("\t") if x] may fit. If this is a tab separated file the regex looks suspicous. the regex is missing documentation and/or examples.
algorithm over all
your solution is O(n), your memory footprint is low, you have a minimum of open files. all good. unless you have real performance problems this is fine.
When you want to count keys in a
dict() you always end up with special handling for the first appearence of the key. There is
collections which may be incremented even if the key is not exsting yet.
counting lines or other iterables you are iterating over
initializing a variable like
rows outside a loop and increment inside is error prone. Use
your names are mostly fine and explaining the code. the exceptions are
index (for what?),
cell_data (are we talking about spreadsheets? and what is the data in this column?) and
unique_values (unique is fine but what values?)
cleaning up code
Whenever you think a program is done clean up. remove unnecessary code, clean/add comments and docstrings. you have unused counters for rows and unique values. you have comments raising questions, not answering them.
you may drop your
unique_values completely if you test the opened file for emptyness
with open(target, 'a') as outfile:
if f.tell() == 0:
Also, after you added some numbers, we know the problem does not fit into memory. So your algorithm is fine. If and only if you have performanceproblems you coud work on chunks (say 10.000 lines) of the inputfile buffering file writes and flush to disk at the end of every chunk only. you could change
unique_values to a
defaultdict(list) and append lines. at the end of the chunk you iterate over the unique_values, do path calculation from the key and write all lines. again you test for file emptyness to write the header.