-1
\$\begingroup\$

The code shows here is a demo of my application. The func1 always hang somewhere with reason I haven't figured out by now. Now I just want to write a watchdog for this process. If it didn't generate any data for some time (10 minutes here), watchdog will kill it and start a new one.

All the code is running for few hours with nothing error happened. But I want to know is there any better method for this watchdog. What's the worst case I should to take care of?

from multiprocessing import Value, Queue, Process
import time
import redis

redis_queue = redis.Redis(host="localhost", port=6379, password="password", 
    db=0)


THREAD_TIMEOUT = 60 * 10

def func1():
    data2 = do-something1()
    redis_queue.sadd("queue1", data2)

def func2():
    data2 = redis_queue.spop("queue1")
    data3 = do-something2(data2)
    redis_queue.sadd("queue2", data3)

def func3():
    data3 = redis_queue.spop("queue2")
    do-something3()


def watch_dog():
    # watch dog for process p1, timeout is 10 minutes
    time_now = time.time()
    while True:
        queue1_length = len(queue1.smembers('queue1'))
        if queue1_length == 0:
            if time.time() - time_now > THREAD_TIMEOUT:
                print("queue1 has no message for {} minutes".format(
                    THREAD_TIMEOUT / 60))
                watch_dog_q.put("KILL WORKER")
        else:
            time_now = time.time()


if __name__ == "__main__":
    watch_dog_q = Queue()
    p1 = Process(target=func1)
    p2 = Process(target=func2)
    p3 = Process(target=func3)
    wdog = Process(target=watch_dog)
    wdog.daemon = True
    wdog.start()
    p1.start()
    p2.start()
    p3.start()
    p2.join()
    p3.join()
    wdog.join()
    while True:
        msg = watch_dog_q.get()
        if msg == "KILL WORKER":
            print("Terminating timeout process")
            p1.terminate()
            time.sleep(0.1)
            if not p1.is_alive():
                print("p1 process is a goner")
                p1 = join(timeout=1.0)
                p1 = Process(target=p1)
                p1 = start()
\$\endgroup\$
2
  • 1
    \$\begingroup\$ This code is broken, def func3(); won't allow the program to run. It's also unclear what do-something3() wants to do, is that meant to be a function? Or are you doing some weird ass mutations with it? \$\endgroup\$
    – Peilonrayz
    Commented Jun 24, 2017 at 17:04
  • 1
    \$\begingroup\$ @Peilonrayz Thanks, fixed. It's Just some functions will run for a long time. \$\endgroup\$
    – Tony Wang
    Commented Jun 24, 2017 at 19:34

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.