My server receives requests in one thread and responds in another. I am using thread pools of pthread-s for both receiving and sending. The performance is not that bad, however...

When I am doing the same algorithm in one thread only(MULTITHREAD not defined) so that both recvs and sends are in the same worker thread, my app runs about 10% better.

Strangely, when I am using separate threads(worker and operation) but include printf in any operation, the performance raises (yes, raises - not declines) some 20%. My printfs goes over SSH.(*)

(Packet analyses shows some 100 micros of delay of ACKing responses (not separate ACKs). This delay does not happen with the printfs, neither it happens in a single-thread mode. It seems like SSH triggers something in the TCP stack. )

I tried to use the TCP_QUICKACK option after every recv() but the performance goes down!(**) I am using Centos7. The application is very concurrent - it is SMB server.

The algorithm is as follow:

One thread that get the message with epoll_wait() and then execute the first worker thread, this thread parse the message and launch the operation thread(create/read/write etc.).

static void mainThreadBodySocketTransport(CSTransport * transport)
{
struct epoll_event e;
int res;

while (true)
{
void * context = NULL;              /* to pass to thread */
res = epoll_wait(transport->epoll, &e, 1, (int)*60 * 12 * 1000));
if (res == 1)
{
*context = e.data.ptr;
}
/* wrapper to execute the worker thread */
}
}


static void threadBodySocketTransport(void * context)
{
msgBuffer * message;

message = parseMSG(context);
switch (message->cmd)
{
case WRITE:
doWriteOp(message);
break;
break;
case CREATE:
doCreateOp(message);
break;
}
}

void doWriteOp(msgBuffer * message)
{
/* wrapper to execute the worker thread */
#else
/* write to file */
#endif
}


static void threadBodyWrite(void * context)
{
msgBuffer * message = (msgBuffer *)context;
int fd;

.
.(get the fd)
.
/* printf("w\n");*/
write(fd, message->write.data , message->write.len);
sendResponse(message->cmd, true);
}


void threadStart(SYThread *taskIdPtr, void (*startpoint)(void), NQ_BOOL background)
{

}


Maybe I need to use different thread attribute??

Packet sequence:
(*)ack time - request time = ~50 micros
request
response(builtin ack)
request
request
ack
respons

(**)ack time - request time = ~50 micros
request
ack
response
request
request
ack
respons

• Have you considered "merging" the main and worker thread and using an actor pattern, i.e., after receiving a message the worker starts a new which receives and perform the next message. The whole idea is to reduce system calls and thread context switches as much as possible. – Mikael Patel Nov 22 '16 at 12:22
• @MikaelPatel First of all my platform is Multi-core computer so my idea is to push the use all my resources . Secondly the i have only 1 main thread that do epoll_wait for every socket that register to his Epoll, and after one of the socket get an interrupt, the main thread start the worker thread(10 threads, can be modified, so several workers can "work" simultaneously), and number of op thread can be modified too. Thirdly i can change the thread priority for each "type", now all the thread has SCHED_FIFO , the priority is as follow: main - 1 , worker - 5 , op - 15. – Eliad Cohen Nov 23 '16 at 8:39
• What I am suggesting is very much the same solution with a pool of worker threads. The major difference is that there is no main thread. Instead the worker threads one at a time do the epoll_wait and then service the message. The "one at a time" can be achieved with for instance a semaphore. Therefore several works can be processing messages at the same time. – Mikael Patel Nov 23 '16 at 15:11
• The current architecture just do what you suggest but instead of worker thread that do the epoll_wait the main thread can launch several workers one at a time. [epoll_wait -> execute(worker thread 1), epoll_wait->execute(worker thread 2) in this scenario worker 1 and worker 2 process messages in the same time.] – Eliad Cohen Nov 24 '16 at 6:22