OutputStream class for use with subprocess.Popen

Question

I've recently been writing an application which does a lot of work with the output of commands. To call the commands I've been using the subprocess module, and originally, was using subprocess.PIPEs for the stdout and stderr, and then using the communicate method once the process was complete to retrieve the output to work with it, log it, etc. Unfortunately, I had a problem with deadlocks when a particular command generated a long output stream.

For this reason, I've written an OutputStream class, an instance of which can be provided as the stdout and stderr params of subprocess.Popen. In the background, it writes to a cStringIO buffer which I can retrieve the value from later, and it seems to work well. However, this is the first time I've worked with anything like this and I'm concerned that I might be doing something potentially unsafe. I'd appreciate any feedback anybody has.

from cStringIO import StringIO
import os
import threading

class OutputStream(threading.Thread):
    def __init__(self):
        super(OutputStream, self).__init__()
        self.done = False
        self.buffer = StringIO()
        self.read, self.write = os.pipe()
        self.reader = os.fdopen(self.read)
        self.start()

    def fileno(self):
        return self.write

    def run(self):
        while not self.done:
            self.buffer.write(self.reader.readline())

        self.reader.close()

    def close(self):
        self.done = True
        os.close(self.write)

I would then use it something like this:

import subprocess
import time

stream = OutputStream()
proc = subprocess.Popen(['ls', '-l'], stdout=stream, stderr=subprocess.STDOUT)

while proc.poll() is None:
    time.sleep(0.05)

stream.close()
output = stream.buffer.getvalue()
print output

Edit: When running the test code above, Python doesn't exit once the command output has been printed. Such behaviour isn't exhibited by my application in which this class is being used.

Answer 1

I don't know enough about threading to help with that part, but it looks like your class could benefit from implementing the __enter__ and __exit__ functions that would allow it to be used with the with keyword.

There's a short section in the docs about it as well as an entry in the PEP from when it was added to Python. with allows you to instantiate an object that involves some connection or file that needs to be closed and will always ensure that even if errors arise, the object will be closed accordingly regardless of those errors. It's like wrapping your document with try: finally: stream.close(), but without needing to add that every time you call the stream.

Here's how it might look in your object:

def __enter__(self):
    # Theoretically could be used to set up things not in __init__
    return self

def __exit__(self, exc_type, exc_val, exc_tb):
    self.close()

The parameters that __exit__ takes will be passed to it are just exception information, you could use them if you wanted to but they have to be included in the definition as they'll be passed as part of the with syntax.

Now you could use the object like this:

with OutputStream() as stream:
    proc = subprocess.Popen(['ls', '-l'], stdout=stream, stderr=subprocess.STDOUT)

    while proc.poll() is None:
        time.sleep(0.05)