My server receives requests in one thread and responds in another. I am using thread pools of pthread-s for both receiving and sending. The performance is not that bad, however...When I am doing the same algorithm in one thread only(MULTITHREAD not defined) so that both recvs and sends are in the same worker thread, my app runs about 10% better. Strangely, when I am using separate threads(worker and operation) but include printf in any operation, the performance raises (yes, raises - not declines) some 20%. My printfs goes over SSH.(*)(Packet analyses shows some 100 micros of delay of ACKing responses (not separate ACKs). This delay does not happen with the printfs, neither it happens in a single-thread mode. It seems like SSH triggers something in the TCP stack. )I tried to use the TCP_QUICKACK option after every recv() but the performance goes down!(**)I am using Centos7. The application is very concurrent - it is SMB server. The algorithm is as follow:One thread that get the message with epoll_wait() and then execute the first worker thread, this thread parse the message and launch the operation thread(create/read/write etc.).main thread:static void mainThreadBodySocketTransport(CSTransport * transport){    struct epoll_event e;    int res;    while (true)    {        void * context = NULL;              /* to pass to thread */        res = epoll_wait(transport->epoll, &e, 1, (int)*60 * 12 * 1000));        if (res == 1)        {            *context = e.data.ptr;        }    /* wrapper to execute the worker thread */        cmThreadPoolExecute(transport->threadPool, context);    }}worker threadstatic void threadBodySocketTransport(void * context){   msgBuffer * message;   message = parseMSG(context);   switch (message->cmd)   {   case WRITE:      doWriteOp(message);   break;   case READ:      doReadOp(message);   break;   case CREATE:      doCreateOp(message);   break;   }}void doWriteOp(msgBuffer * message){#ifdef MULTITHREAD    /* wrapper to execute the worker thread */    cmThreadPoolExecute(threadPoolWrite, (void *)message);#else    /* write to file */    threadBodyWrite(message);#endif}operation threadstatic void threadBodyWrite(void * context){   msgBuffer * message = (msgBuffer *)context;   int fd;   .   .(get the fd)   .   /* printf(w\n);*/   write(fd, message->write.data , message->write.len);   sendResponse(message->cmd, true);}pthread createvoid threadStart(SYThread *taskIdPtr, void (*startpoint)(void), NQ_BOOL background){    pthread_attr_t attr;    pthread_attr_init(&attr);    pthread_attr_setdetachstate(&attr, PTHREAD_CREATE_DETACHED);    pthread_create(taskIdPtr, &attr, (void * (*)(void *))startpoint, NULL)    pthread_attr_destroy(&attr);}Maybe I need to use different thread attribute??Packet sequence: (*)ack time - request time = ~50 microsrequestresponse(builtin ack)requestrequestackrespons(**)ack time - request time = ~50 microsrequestackresponserequestrequestackrespons
Linux performance using multi-threads (vs one thread) and ssh message
c;multithreading;linux;tcp;network file transfer