I'm trying to design a multithreaded daemon for an industrial automation related project.

The daemon will be using a number of 3rd party libs like MQTT, mysql, etc.. My idea is to have worker threads (or callbacks registered with these 3rd party libs) relay events to the main thread using a global mutex and conditional variable based synchronisation mechanism.

The main thread then performs some global state management and some lite processing that I'm willing to restrict to the main thread (like MySQL stuff). After that the main thread defers the processing to a pool of worker threads.

I haven't posted the logging and much of the error checking code here. I'm planning on using the start-stop-daemon facility for daemonizing.

I wanted to know if I'm headed in the right direction. Constructive criticism is welcome.

#include <stdio.h>
#include <errno.h>

#define RESPONSE_MAX_LENGTH 256
/**
* Internal data structures and enums
*/

typedef enum myEventType_ {
MY_EVENT_TYPE_NONE,
MY_EVENT_TYPE_STOP,
} myEventTypeT;

typedef struct myEvent_ {
int is_triggered;
myEventTypeT type;
void *data;        /* may include a callback */
} myEventT;

const myEventT myEventInitializer = {
.is_triggered = 0,
.type = MY_EVENT_TYPE_NONE,
.data = NULL
};

/**
* Global variables
*/
static myEventT myEvent;
/* Mutex and cond_var */

/*
* Triggers an internal myEvent
* Blocks calling thread till event acknowledgement
*/
int trigger_event(myEventTypeT type, void *data) {
myEvent.type = type;
myEvent.data = data;
// Trigger event!
myEvent.is_triggered = 1;
// Block till current trigger has been acknowledged
while(myEvent.is_triggered);
return 0;
}

void *worker_loop() {
while(1) {
sleep(5);       /* Do something worthwhile */
trigger_event(MY_EVENT_TYPE_NONE, NULL);
}
}

/*
* Called from main thread, mainly for state management & db related processing,
* i.e. stuff we want to restrict to the main thread.
*/
int handle_event(myEventT *event, char *response) {
// handle the event, fill response if required
return 0;
}

int main(int argc, char* argv[]) {
// Initialize globals
myEvent = myEventInitializer;

return -1;
}

// Main loop: wait for event
while(1) {

// Wait for signal from worker threads/ callbacks
if (res_wait == EINVAL) {   // TODO: Consolidated error handling
return -1;
}

/**
* Event handling construct
*
*/
if (myEvent.is_triggered) {

char response_string[RESPONSE_MAX_LENGTH];  //TODO macro for size!
if(handle_event(&myEvent, response_string) >= 0) {
// Initiate deferred processing on another thread (maybe callback)
}

// Clear event triggered flag and reinit event data
myEvent = myEventInitializer;

}
}

}


# Bugs

I can see three problems with your program:

1. Your main thread releases the mutex and then uses myEvent without holding the mutex. If another thread grabs the mutex and modifies myEvent, for example by calling trigger_event(), the main thread will be reading/writing the same structure and unpredictable things will happen.
2. The way you use your condition variable, there is a possibility that you will get stuck forever because another thread signalled the condition when you weren't waiting on it.
3. If two threads call trigger_event(), the first event will get overwritten by the second event.

I will elaborate more on each problem.

# Using shared variables properly

One thing you should always make sure is that a shared variable is never read/written without holding the mutex. In your case, the shared variable is myEvent. Keeping that in mind, I can see two possible fixes for your main loop. The first fix is simple: just keep holding the mutex while you use myEvent. However, perhaps handle_event() is lengthy and you don't want to hold the mutex for that long. If so, then the second fix is to make a copy of myEvent while you still hold the mutex. Here is an example of how that would look. Notice that I also moved the part where you clear myEvent to where the mutex is still being held:

    if (myEvent.is_triggered) {
myEventT eventCopy = myEvent;
myEvent = myEventInitializer;

char response_string[RESPONSE_MAX_LENGTH];  //TODO macro for size!
if(handle_event(&myEventCopy, response_string) >= 0) {
// Initiate deferred processing on another thread (maybe callback)
}

}


When using a mutex with a condition variable, your code should always look like this:

pthread_mutex_lock(&mutex);
while (someVariable) {
}


This kind of loop guarantees that you will never miss another thread signalling the condition variable. Notice in particular how someVariable is checked while the mutex is being held.

Now let's look at your code and see where it could fail:

    if (myEvent.is_triggered) {

char response_string[RESPONSE_MAX_LENGTH];  //TODO macro for size!
if(handle_event(&myEvent, response_string) >= 0) {
// Initiate deferred processing on another thread (maybe callback)
}

// Clear event triggered flag and reinit event data
myEvent = myEventInitializer;

//
//

}


I've marked above where the danger lies. If another thread calls trigger_event() when the main thread is at the comment, that other thread will set myEvent.is_triggered and signal the condition, but the main thread won't notice.

    pthread_attr_destroy(&attr);

// Main loop: wait for event
while(1) {
while (!myEvent.is_triggered) {
// Wait for signal from worker threads/ callbacks
&my_event_mutex);
if (res_wait == EINVAL) {
return -1;
}
}

// Mutex is held at this point, so it is safe to modify myEvent.
myEventT eventCopy = myEvent;
myEvent = myEventInitializer;

/**
* Event handling construct
*/
char response_string[RESPONSE_MAX_LENGTH];
if(handle_event(&eventCopy, response_string) >= 0) {
// Initiate deferred processing on another thread.
}
}


# Handling multiple events

Right now, you only have one variable myEvent, which means that if two or more threads trigger events at the same time, myEvent will get overwritten with the latest event and you will lose the earlier event.

The fix for this is simple: use a queue of events instead of a single event. In trigger_event(), while holding the mutex you push an event onto the queue. In the main loop, while holding the mutex, you pop an event off the queue. You can do this where my code above currently copies myEvent into eventCopy.

• This is what I was hoping for! Thanks for taking the time for reviewing my code and writing this. Truly appreciate it. As for the queue, could you suggest if I should use a simple linked list based queue or use something readymade like a POSIX message queue? – user3490458 Nov 15 '15 at 5:27
• @user3490458 It could be a simple linked list or array. Just as long as you guard access to it using the mutex. – JS1 Nov 15 '15 at 5:48

# MySQL and the main thread

and some lite processing that I'm willing to restrict to the main thread (like MySQL stuff)

Don't consider MySQL stuff a lite processing. Don't consider anything that goes out to the network or external processes a lite processing. Consider using some asynchronous interface to DB or a separate thread to connect to DB, or connecting to DB directly in the threads (if you use separate connections and are a bit careful you'll be OK. That's what most larger web applications do.

• I actually wanted to say that the mysql processing that I will be doing will be lite. But either ways, I think you're right. I have no RT constraints and would prefer to keep away from using multiple connections and all the synchronisation stuff that will come with it. Using a separate thread for DB is a good idea. I will try this. Thanks. – user3490458 Nov 15 '15 at 12:55