The code shows here is a demo of my application. The func1 always hang somewhere with reason I haven't figured out by now. Now I just want to write a watchdog for this process. If it didn't generate any data for some time (10 minutes here), watchdog will kill it and start a new one.
All the code is running for few hours with nothing error happened. But I want to know is there any better method for this watchdog. What's the worst case I should to take care of?
from multiprocessing import Value, Queue, Process
import time
import redis
redis_queue = redis.Redis(host="localhost", port=6379, password="password",
db=0)
THREAD_TIMEOUT = 60 * 10
def func1():
data2 = do-something1()
redis_queue.sadd("queue1", data2)
def func2():
data2 = redis_queue.spop("queue1")
data3 = do-something2(data2)
redis_queue.sadd("queue2", data3)
def func3():
data3 = redis_queue.spop("queue2")
do-something3()
def watch_dog():
# watch dog for process p1, timeout is 10 minutes
time_now = time.time()
while True:
queue1_length = len(queue1.smembers('queue1'))
if queue1_length == 0:
if time.time() - time_now > THREAD_TIMEOUT:
print("queue1 has no message for {} minutes".format(
THREAD_TIMEOUT / 60))
watch_dog_q.put("KILL WORKER")
else:
time_now = time.time()
if __name__ == "__main__":
watch_dog_q = Queue()
p1 = Process(target=func1)
p2 = Process(target=func2)
p3 = Process(target=func3)
wdog = Process(target=watch_dog)
wdog.daemon = True
wdog.start()
p1.start()
p2.start()
p3.start()
p2.join()
p3.join()
wdog.join()
while True:
msg = watch_dog_q.get()
if msg == "KILL WORKER":
print("Terminating timeout process")
p1.terminate()
time.sleep(0.1)
if not p1.is_alive():
print("p1 process is a goner")
p1 = join(timeout=1.0)
p1 = Process(target=p1)
p1 = start()
def func3();
won't allow the program to run. It's also unclear whatdo-something3()
wants to do, is that meant to be a function? Or are you doing some weird ass mutations with it? \$\endgroup\$