1
\$\begingroup\$

My requirement is to match each line of a text file, including the line terminator of each, at most excluding the terminator of the last line, to take into account the crippled, non POSIX-compiant files generated on Windows; each line terminator can be either \n or \r\n.

As a consequence, no character in the file should be left unmatched.

The best regex I could come up with is this:

\n|\r\n|[^\r\n]++(\r\n|\n)?

Is this the best I can write, performance-wise?

Please, if you use the ^/$ anchors or similar, comment about that, because their behavior is dependent on whether the engine considers them as multiline by default.

\$\endgroup\$
1
  • \$\begingroup\$ What is the regex engine you are using for this task? The best expression will depend on the regex flavor. \$\endgroup\$ Sep 18, 2020 at 18:01

2 Answers 2

1
\$\begingroup\$

I think this would work:

.*(\r?\n|$)

.* matches anything except line breaks (at least '\n', but some regex engines also treat other characters as line breaks).

(\r?\n|$) matches the line break or the end of the string (in case the last line is missing a line break.

\$\endgroup\$
8
  • \$\begingroup\$ This seems interesting. Any comment on the performances? Since you assume . is not matching line breaks, maybe .*+ would improve the performance (as long as . doesn't match \r either). \$\endgroup\$
    – Enlico
    Sep 17, 2020 at 18:14
  • 1
    \$\begingroup\$ I don't think using a possessive qualifier would speed things up in this case, because there shouldn't be any backtracking. Maybe use [^\r\n]* instead of .*. \$\endgroup\$
    – RootTwo
    Sep 17, 2020 at 23:04
  • \$\begingroup\$ no, no backtracking indeed, but the non possessive quantifier means that reach status is saved in case backtracking read needed. No? \$\endgroup\$
    – Enlico
    Sep 17, 2020 at 23:06
  • 1
    \$\begingroup\$ I imagine that would depend on the implementation of the regex engine. Best bet would be to test them and see which is faster. \$\endgroup\$
    – RootTwo
    Sep 17, 2020 at 23:11
  • 1
    \$\begingroup\$ "there shouldn't be any backtracking": it depends on the regex flavor and even additional options. If it is Java, and UNIX_LINES flag is used, . also matches CR and there will be backtracking. Same thing concerns the .NET regex flavor, where . matches CR symbol. This can also happen in PCRE. \$\endgroup\$ Sep 18, 2020 at 18:03
1
\$\begingroup\$

According to the official document of open,

newline controls how universal newlines mode works (it only applies to text mode). It can be None, '', '\n', '\r', and '\r\n'. It works as follows:

  • When reading input from the stream, if newline is None, universal newlines mode is enabled. Lines in the input can end in '\n', '\r', or '\r\n', and these are translated into '\n' before being returned to the caller. If it is '', universal newlines mode is enabled, but line endings are returned to the caller untranslated. If it has any of the other legal values, input lines are only terminated by the given string, and the line ending is returned to the caller untranslated.

IIUC, adding newline='' to open is what you need. To iterate over the file line by line, you could simply do for line in f. To read all lines at once, the readlines function could be used.

with open("text.txt", "w") as f:
    f.write("a\r\nb\nc")
with open("text.txt", "r", newline="") as f:
    print([line for line in f])
with open("text.txt", "r", newline="") as f:
    print(f.readlines())

Output:

['a\r\n', 'b\n', 'c']
['a\r\n', 'b\n', 'c']
\$\endgroup\$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.