0
\$\begingroup\$

I'm new to threading in Python. I want to use it to speed up taking images on my lab computer. I have the following dummy code:

#%%
import threading
import logging
import time
import numpy as np


camera_lock = threading.Lock()
file_lock = threading.Lock()

class Camera:
    def __init__(self, shuttertime=3):
        self.shuttertime = shuttertime
    
    def take_picture(self):
        time.sleep(self.shuttertime)
        return np.random.randint(0, 10, size=30)

def take_and_save_image(thread_id, cam):
    with camera_lock:
        logging.info("Thread %s: taking image...", thread_id)
        time.sleep(.1)
        
    data = cam.take_picture()
    
    with camera_lock:
        logging.info("Thread %s: finished taking image", thread_id)
        time.sleep(.1)
        
    logging.info("Thread %s: waiting for lock...", thread_id)
    with file_lock:
        logging.info("Thread %s: saving to disk...", thread_id)
        np.save('temp/thread{}.npy'.format(thread_id), data)
        time.sleep(3)
        logging.info("Thread %s: finished saving to disk", thread_id)

# if __name__ == "__main__":
format = "%(asctime)s: %(message)s"
logging.basicConfig(format=format, level=logging.INFO,
                    datefmt="%H:%M:%S")

threads = list()
cameras = list()
for index in range(4):
    cam = Camera()
    x = threading.Thread(target=take_and_save_image, args=(index, cam))
    threads.append(x)
    x.start()

The code is modified from https://realpython.com/intro-to-python-threading/#basic-synchronization-using-lock

What happens:

  1. I want to take and save 4 images
  2. First a signal is sent to the camera to take the image (0.1 s waiting time)
  3. The camera waits for exposure time (3s)
  4. The camera sends back a response (.1s)
  5. The file is saved (3s)

These waiting times are arbitrary.

I have the following questions.

  1. What pitfalls can I expect in such a piece of code? I know there can be deadlock and racing conditions. Are these impossible in this piece of code?
  2. Are the camera_lock and file_lock necessary? The cameras are connected by USB if that helps, but I'd like to keep this question quite general to be of use to other people.
  3. Is a __name__=='__main__' if statement necessary? I know it is for multiprocessing and the link I provided also uses it, but it doesn't seem to be necessary here.
\$\endgroup\$
3
  • 1
    \$\begingroup\$ What do you mean by "dummy code"? It doesn't appear to be the kind of example code that's specifically off-topic here, but you could improve your question with an edit to summarise its purpose, especially the title. We're more interested in the why than the how when reviewing code (and the how is inferrable in a way that the why isn't). \$\endgroup\$ Commented Jun 18 at 12:53
  • \$\begingroup\$ @TobySpeight If used all the code directly from my lab computer, this code could not be run by other people because it connects to a physical camera. I used timeouts to simulate this. The purpose is explained under "what happens:". Could you elaborate what more I should add? \$\endgroup\$ Commented Jun 18 at 12:59
  • 1
    \$\begingroup\$ I changed the title so that it describes what the code does per site goals: "State what your code does in your title, not your main concerns about it.". Please check that I haven't misrepresented your code, and correct it if I have. BTW, if __name__=='__main__' is unrelated to multiprocessing - it's to allow your code to be imported from other Python code, making it reusable. \$\endgroup\$ Commented Jun 19 at 6:54

1 Answer 1

2
\$\begingroup\$

regrettable locking

Please don't sleep() while holding a mutex -- that's not polite.

        np.save(f'temp/thread{thread_id}.npy', data)

This file is explicitly a thread-local resource. Please don't perform that operation while holding a mutex. There's no need to block out other threads while that's happening. (Oh, and prefer an f-string.)

GIL

It's important to understand that current cPython interpreters hold the Global Interpreter Lock when executing bytecode. The practical effect of this is that execution of python threads will mostly be non-overlapping. This is quite different from a situation where e.g. several C++ threads may be simultaneously executing on several cores. Indeed, a library such as numpy might warm up several cores during a big matrix multiply, only to single-thread under the GIL once it goes back to interpreted bytecode.

Python threads are currently a better fit for something like a webserver which spends lots of time waiting on client responses, than for compute-bound workloads. Without knowing the total framerate, it's hard to say whether python threads will be much help for your use case. You may be better off with multiprocessing, where each child process writes pixels directly to the filesystem, without sending them through a pipe to the parent process.

conventional init

threads = list()

nit: Prefer to conventionally assign threads = [] instead.

meaningful names

    x = threading.Thread( ... )
    threads.append(x)
    x.start()

In this context, x is not a real name, it is just the Author being lazy. Please call it something sensible, such as thread.

main guard

Please protect the top-level verbs with the usual if __name__ == "__main__": idiom. Why do we do that? So other code, perhaps an automated test suite, can safely import this module to test a helper function without risking inconvenient side effects.

multiprocessing

You didn't reveal how many locally attached cameras are of interest, nor how the code interacts with them, nor how long it takes to obtain the pixel data. The details matter.

Possibly your issue is merely one of scheduling capture events so they happen approximately isochronously. Possibly what you're really trying to do is overlap I/O events to max out a USB bus or a memory bus in order to achieve high frame rate. The OP code should answer such questions, but it fails to do so.

concrete code

    def take_picture(self):
        time.sleep(self.shuttertime)
        return np.random.randint(0, 10, size=30)

That clearly doesn't interact with any hardware to take a picture.

Missing Review Context: Code Review requires concrete code from a project, with enough code and / or context for reviewers to understand how that code is used. Pseudocode, stub code, hypothetical code ... are outside the scope of this site.

\$\endgroup\$
3
  • \$\begingroup\$ Some useful suggestions! It is particularly nice to know that I can open multiple files in different threads without having to worry about that causing conflicts. I think multiprocessing is overkill here, since it places some awkward constraints on the code (having to call from main, running entire main file in thread), and I'm mostly interested in optimizing the waiting times for a singe shot capture. I used .format() because I inherited a lot code that is still in Python 2.7. I will migrate to Python 3 soon but right now I have to use 2.7 \$\endgroup\$ Commented Jun 19 at 8:04
  • \$\begingroup\$ You are right about providing more details: I will add more realistic waiting times today if I have time. \$\endgroup\$ Commented Jun 19 at 8:05
  • \$\begingroup\$ 2.7 support lasted through 2019, and 3.7 support is similarly at EOL. There's little point in using an ancient python interpreter to create new code. A conda virtual environment will install a modern interpreter with low effort. \$\endgroup\$
    – J_H
    Commented Jun 19 at 19:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.