Regex to match each line in a file, with windows and/or linux line break included, even for missing line break at EOF

Question

My requirement is to match each line of a text file, including the line terminator of each, at most excluding the terminator of the last line, to take into account the crippled, non POSIX-compiant files generated on Windows; each line terminator can be either \n or \r\n.

As a consequence, no character in the file should be left unmatched.

The best regex I could come up with is this:

\n|\r\n|[^\r\n]++(\r\n|\n)?

Is this the best I can write, performance-wise?

Please, if you use the ^/$ anchors or similar, comment about that, because their behavior is dependent on whether the engine considers them as multiline by default.

Answer 1

I think this would work:

.*(\r?\n|$)

.* matches anything except line breaks (at least '\n', but some regex engines also treat other characters as line breaks).

(\r?\n|$) matches the line break or the end of the string (in case the last line is missing a line break.

Answer 2

According to the official document of open,

newline controls how universal newlines mode works (it only applies to text mode). It can be None, '', '\n', '\r', and '\r\n'. It works as follows:

• When reading input from the stream, if newline is None, universal newlines mode is enabled. Lines in the input can end in '\n', '\r', or '\r\n', and these are translated into '\n' before being returned to the caller. If it is '', universal newlines mode is enabled, but line endings are returned to the caller untranslated. If it has any of the other legal values, input lines are only terminated by the given string, and the line ending is returned to the caller untranslated.

IIUC, adding newline='' to open is what you need. To iterate over the file line by line, you could simply do for line in f. To read all lines at once, the readlines function could be used.

with open("text.txt", "w") as f:
    f.write("a\r\nb\nc")
with open("text.txt", "r", newline="") as f:
    print([line for line in f])
with open("text.txt", "r", newline="") as f:
    print(f.readlines())

Output:

['a\r\n', 'b\n', 'c']
['a\r\n', 'b\n', 'c']