3
\$\begingroup\$

I have this code that reads a file and after processing a few lines writes the output to a second file:

num_reads = 7
with open('data.txt') as read_file:
    with open('new_data.txt', 'w') as write_file:

        while (True):
            lines = []
            try:       # expect errors if the number of lines in the file are not a multiplication of num_reads
                for i in range(num_reads):     
                    lines.append(next(read_file))  # when the file finishes an exception occurs here

                #do sutff with the lines (exactly num_reads number of lines)
                processed = " ".join(list(map(lambda x: x.replace("\n", ''), lines)))
                write_file.write(processed + '\n')

            except StopIteration:     # here we process the (possibly) insufficent last lines
                #do stuff with the lines (less that  num_reads)
                processed = " ".join(list(map(lambda x: x.replace("\n", ''), lines)))
                write_file.write(processed + '\n')
                break

Here is the input file (data.txt):

line1
line2
line3
line4
line5
line7
line8
line9

And this is the output file that has the desired state:

line1 line2 line3 line4 line5 line7
line8 line9

This works correctly but as I wish to do the same processing and writing procedure in both cases (when the number of elements is 7 and when the file finishes and the exception is raised) I think the above code violates DRY principle even if I define a new function and call it once in try block and once in except before break. Any other ordering that I could come up with was either causing an infinite loop or losing the final lines. I appreciate any comments on handling this issue, as it is not limited to this case and I had faced it in other cases as well.

\$\endgroup\$
  • \$\begingroup\$ @200_success done! :) \$\endgroup\$ – Farzad Vertigo Jan 18 at 5:23
  • 1
    \$\begingroup\$ (Welcom to Code Review!) \$\endgroup\$ – greybeard Jan 18 at 7:49
3
\$\begingroup\$

Disclaimer: This question belongs to Stack Overflow, and I voted to migrate it. Therefore, the answer is not a review.

Keep in mind that principles are there to guide you. They should be treated like guard rails, rather than roadblocks.

I would argue that

    while (....) {
        foo(7);
    }
    foo(3);

does not violate DRY. Your situation is pretty much the same.

That said, your idea of defining function is valid. You just factoring out the wrong code. Factor out reading. Consider

    def read_n_lines(infile, n):
        lines = []
        try:
            for _ in range(n):
                lines.append(next(infile))
        except StopIteration:
            pass
        return lines

and use it as

    while True:
        lines = read_n_lines(infile, 7)
        if len(lines) == 0:
            break
        process_lines(lines)
\$\endgroup\$
  • \$\begingroup\$ Thank you very much. Beautiful idea. I appreciate it. \$\endgroup\$ – Farzad Vertigo Jan 18 at 6:05
6
\$\begingroup\$

You should avoid writing code with exception-handling altogether. Usually, when you want to write a fancy loop in Python, the itertools module is your friend. In this case, I would take advantage of itertools.groupby() to form groups of lines, assisted by itertools.count() to provide the line numbers.

import itertools

def chunks(iterable, n):
    i = itertools.count()
    for _, group in itertools.groupby(iterable, lambda _: next(i) // n):
        yield group

with open('data.txt') as read_f, open('new_data.txt', 'w') as write_f:
    for group in chunks(read_f, 7):
        print(' '.join(line.rstrip() for line in group), file=write_f)

A few other minor changes:

  • You only need one with block to open both files.
  • line.rstrip() is more convenient than lambda x: x.replace("\n", '')
  • print(…, file=write_file) is slightly more elegant than write_file.write(… + '\n').
\$\endgroup\$
  • 2
    \$\begingroup\$ Isn't the grouper recipe more appropriate to make fixed-length chunks? Or did you purposefully avoid it to avoid dealing with the fill values at the end of the iteration? \$\endgroup\$ – Mathias Ettinger Jan 18 at 8:10
  • 1
    \$\begingroup\$ @MathiasEttinger The grouper() recipe works best for complete groups; you would have to specify a fillvalue, then strip out that padding. \$\endgroup\$ – 200_success Jan 18 at 8:19
  • \$\begingroup\$ @Graipher I don't see any reason to copy a recipe that doesn't do what we want, then work around the unwanted behavior by stripping off junk. \$\endgroup\$ – 200_success Jan 18 at 15:06
  • \$\begingroup\$ @200_success: I agree now that it is too cumbersome. We should probably clean up the comments. \$\endgroup\$ – Graipher Jan 18 at 15:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.