I am splitting a csv file where the first 3 columns will be common for all the output files.

input file:

h1 h2 h3 o1 o2 ....
a  b  c  d  e  ....
a1 b1 c1 d1 e1 ....

output files:


h1 h2 h3 o1
a  b  c  d  
a1 b1 c1 d1  


h1 h2 h3 o2
a  b  c  e
a1 b1 c1 e1 

So if there are n columns in the input file , the code creates n-3 output files. However my code is inefficient and is quite slow. It takes 20 seconds for 50000 rows.


# reading and writing headers    
headers_line=$(head -n 1 "$csv_file")
IFS=$DELIMITER read -r -a headers <<< $headers_line

for header in "${headers[@]:3}"
   # writing headers to every file
   echo $common_headers$DELIMITER$header > "$header$START_TIME".csv

# reading csv file line by line
while IFS=$DELIMITER read -r -a row_data
    test $i -eq 1 && ((i++)) && continue      # ignoring headers


    for val in "${row_data[@]:3}"
        #  appending row to every new csv file
        echo $common_data$DELIMITER$val >> "${headers[(($j+3))]}$START_TIME".csv

done < $csv_file

Any suggestions are appreciated.


1 Answer 1


Bash is not efficient for processing large files line by line. For small data it's fine, but when a script starts to feel heavy, it's good to look for other alternatives. Also note that the line by line processing and breaking into columns is not easy to get right, I bet you spent quite some time on this. You wrote it well, but the result is not particularly easy to read, and I'm afraid this is as good as it gets with Bash.

So what's the alternative? Try with cut in a loop. Yes that will imply reading the file n-3 times, but I bet it will be faster than the pure Bash solution. And it will be nicely readable too, which is an extremely important benefit.

A few notes about technique:

  • Use $(...) instead of `...`
  • You took care to save IFS and then restore at the end, but it was unnecessary: when you do var=... somecmd, the value of var is only set in the environment of somecmd, it is unchanged for the current script. That being said, what you did is safe, so it's fine.
  • The incrementing i variable in the loop is a bit misleading, because i is a common name in counting loops, and at first I thought the count itself has some purpose. But it doesn't, this variable is used only to distinguish the first line from the others. I would write differently, to make the intention perfectly obvious.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.