Changing file encoding, removing mid-line LF, and converting DOS CR-LF to Unix LF

Question

We have several thousand large (10M<lines) text files produced by a windows machine.

We need to change the file encoding of these files from cp1252 to utf-8, replace any bare Unix LF sequences (i.e. \n) with spaces, then replace the DOS line end sequences ("CR-LF", i.e \r\n) with Unix line end sequences (i.e. \n).

The dos2unix utility is not available for this task.

We've written a bash function to package these operations together using iconv and sed, with iconv doing the encoding and sed dealing with the LF/CRLF sequences.

It first changes the encoding, then replaces \r\n sequences with a placeholder. It then replaces remaining \n with , and finally replaces the placeholders with \n:

post_extract() {

    iconv --from-code=CP1252 --to-code=UTF-8 $1 | \
    sed 's/\\r\\n/@PLACEHOLDER@/g' | \
    sed 's/\\n/ /g' | \
    sed 's/@PLACEHOLDER@/\\n/g' > $2

}

I'm not used to writing shell functions.

There may be many edge cases and other considerations not handled here but this first pass works. To be specific with how this function is used in practice, it is sent to the --to-command= argument of a call to tar, such that the above process is performed for every file extracted from the tar archive. Use case is preprocessing tabular data for upload to a database.

I'm not sure if we can do the operation in place or not.

Open to using tools other than sed, e.g. tr, awk or perl (though I don't think the latter is necessary).

(Highly simplified) example input:

apple|orange|\n|lemon\r\nrasperry|strawberry|mango|\n\r\n

Example output:

apple|orange| |lemon\nrasperry|strawberry|mango| \n

Answer 1

There's nothing Bash-specific here, so consider using plain sh for lower overheads.

The unquoted expansions of $1 and $2 are subject to word-splitting, and you probably don't want that - use "$1" and "$2" respectively.

It looks like you're over-escaping \r\n, unless you actually meant to handle those four characters (backslash, r, backslash, n). If we're handling ASCII CR and NL, then we need single backslashes for sed (the single quotes prevent shell expanding them).

There's a small risk that the input might contain the magic @PLACEHOLDER@ sequence, and no matter how unlikely you make it, there's always a finite probability that it will turn up. It's worth cultivating a habit that such things aren't needed - that's the sort of "unlikely" input that gets exploited for security breaches.

We're never going to match a newline in our sed commands, because sed is line-oriented. We would need an entirely different sed program, that removes \r$ (i.e. CR at end of line), and for lines that don't end in CR, read in the following line using the N command and replace the resulting newline with space).

In any case, sed always writes complete lines of output - if input finishes with an incomplete line (no newline) then there's no way to prevent sed from corrupting that.

An alternative would be to use Perl (I know you said that might be overkill). It can simply replace all newlines not preceded by CR by using a negative lookbehind assertion: s/(?<!\r)\n/ /g. And can slurp the entire file rather than operating on individual lines:

perl -g -pe 's/(?<!\r)\n/ /g; s/\r\n/\n/g;'

We might be able to get Perl to do the character-code conversion too. perl -C6 ensures that standard output and error are UTF-8, so if we can put its input stream into windows-1252, we'll have a short single program rather than needing a pipeline with iconv. I'm not sure we can do that with -p - we might need to provide the read-execute-write loop ourselves in that case.