5
\$\begingroup\$

I'm very new to Python, and far away from writing my own scripts. For my work with lilypond I needed a script that parses a text file, searches for a string and inserts another textfile before the match. I have been searching quite a lot for this kinda script and I did not find any. I ended up combining the snippets I found on here and other sites and came up with this script, which is working:

#!/usr/bin/env python

# usage:
# $ python thisfile.py text.txt searchstring insert.txt

import sys

f2 = open(sys.argv[3])
data = f2.read()
f2.close()

with open(sys.argv[1], "r+") as f1:
    a = [x.rstrip() for x in f1]
    index = 0
    for item in a:
        if item.startswith(sys.argv[2]):
            a.insert(index, data)
            break
        index += 1
    f1.seek(0)
    f1.truncate()
    for line in a:
        f1.write(line + "\n")

I also got a very detailed answer on Stack Overflow, telling what is actually going on in the code, before I was far away from understanding any detail.

What I got out of it so far is the following problem:

If anything would go wrong with reading in the data from f1 or f2, f1.truncate() would delete the original content of f2, then not being able to (re)write the appropriate content the content would get lost. A much more secure way would be using some kind of temporary file, or at least moving the original content of f1 there before calling truncate().

I would be glad for any comments on this "problem", and any others if there are.

\$\endgroup\$
5
\$\begingroup\$

The problem is a general one in data processing, one that programmers have to think about all the time! When changing a file \$F\$ from \$A\$ to \$B\$ it's tempting to implement it like this:

  1. Read \$A\$ from \$F\$.
  2. Compute \$B\$ from \$A\$.
  3. Delete \$F\$.
  4. Write \$B\$ to \$F\$.

But we need to consider the possibility that something will go wrong. Maybe the user will type control-C on the keyboard and interrupt the program? Maybe there will be a power cut? Maybe the disk will not have enough room for \$B\$? If any of these things happened after step 3 and before step 4, then you would be left in a situation where \$F\$ contains neither \$A\$ nor \$B\$. So you have lost your data and can't get it back.

This is why we try to design systems so that operations are atomic — either they succeed completely or they fail completely. In this case we would use the following procedure:

  1. Read \$A\$ from \$F\$.
  2. Compute \$B\$ from \$A\$.
  3. Write \$B\$ to a temporary location \$G\$.
  4. Replace \$F\$ with \$G\$.

This works because operating systems (usually!) give us an atomic implementation of step 4. In Python we can use os.rename, where you can see that the documentation says:

If successful, the renaming will be an atomic operation

In this design, if something goes wrong before step 4, the file \$F\$ still contains \$A\$, and so we haven't lost our data, and so we have a chance to fix the problem and try again.

So in this case, I'd write something like this (but this is not tested, so don't use it blindly!):

import os
import shutil
import sys
import tempfile

# usage:
# $ python thisfile.py text.txt searchstring insert.txt
text_file, searchstring, insert_file = sys.argv[1:]

with tempfile.NamedTemporaryFile('w', delete=False) as temp:
    with open(text_file) as f1:
        inserted = False # Have we inserted insert_file yet?
        for f1_line in f1:
            if not inserted and f1_line.startswith(searchstring):
                with open(insert_file) as f2:
                    for f2_line in f2:
                        temp.write(f2_line)
                inserted = True
            temp.write(f1_line)

os.rename(temp.name, text_file)

Here I've used the library function tempfile.NamedTemporaryFile to choose somewhere to put the temporary file.

 Update

So here's another reason why it's a good idea to make operations atomic — you might have made a programming error! The code I wrote above works correctly on my operating system (macOS) but as it says in the os.rename documentation:

The operation may fail on some Unix flavors if src and dst are on different filesystems

So I'm guessing that you're on some kind of Linux system. On these systems you've got to ensure that the temporary file \$G\$ is on the same filesystem as \$F\$, and the only reliable way to do that is to put it in the same directory as \$F\$:

import os
import shutil
import sys
import tempfile

# usage:
# $ python thisfile.py text.txt searchstring insert.txt
text_file, searchstring, insert_file = sys.argv[1:]

# Directory and name of text_file.
dirname, basename = os.path.split(text_file)

# Create temporary file in same directory as text_file.
with tempfile.NamedTemporaryFile('w', dir=dirname, prefix=basename,
                                 delete=False) as temp:
    with open(text_file) as f1:
        inserted = False # Have we inserted insert_file yet?
        for f1_line in f1:
            if not inserted and f1_line.startswith(searchstring):
                with open(insert_file) as f2:
                    for f2_line in f2:
                        temp.write(f2_line)
                inserted = True
            temp.write(f1_line)

os.rename(temp.name, text_file)

Writing reliable code that works on different platforms is not easy!

\$\endgroup\$
  • \$\begingroup\$ First of all: thanks a lot for this detailed answer! since I registered on here my learning curve started to go up quite steep. \$\endgroup\$ – nath Jul 15 '17 at 14:05
  • \$\begingroup\$ I just run the script you suggested on some test-files. And yes, the files remain untouched when I cancel the script. Though it seems to have a problem renaming the temp.name back to text_file. I get the following error: os.rename(temp.name, text_file) [Errno 18] Invalid cross-device link \$\endgroup\$ – nath Jul 15 '17 at 14:12
  • \$\begingroup\$ @nath: See updated answer. This stuff is hard! \$\endgroup\$ – Gareth Rees Jul 15 '17 at 15:12
  • \$\begingroup\$ WOW yea you are guessing absolutely right. I'm on a LINUX-System where the system itself including python is run on a different partition then the files and the script. If I understand the problem, it comes from the fact that os.rename is creating the tempfile on the partition where python is running and then having trouble finding the path to F. Much thanks again for this lesson, I'm pretty amazed, seriously. THANKS \$\endgroup\$ – nath Jul 16 '17 at 12:16
  • \$\begingroup\$ ... not to forget, the script is running perfectly. Well I told in the beginning I'm not at the state of writing scripts yet, but I'm learning to read them: For my general snipet repository especially this elagant way with text_file, searchstring, insert_file = sys.argv[1:] is going to be used quite frequently from now on, since it allows to make the script itself easier to understand then having it stuffed up with sys.argv[1] sys.argv[2] sys.argv[3] etc... \$\endgroup\$ – nath Jul 16 '17 at 12:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.