Q:

Thread pool extreme perfomance lag

There have been quite a few discussions on the issue, but they can't seem to explain my particular problem. I'm having serious performence issues when threading useing ThreadPool instead of Thread class.
The details: 
I have built a tcp server, when the tcp server accepts a new client it spawns a new thread to handle that client. All Fairly simple, but it takes my server way too long to handle many concurrent clients. 30 seconds for about 35 simple clients that just send a 2048 byte buffer, recieve it and close.
After many stopwathes I found that ThreadPool.QueueUserWorkItem takes up to 26 seconds. I use it to spawn new threads for handling new clients.
After replacing ThreadPool.QueueUserWorkItem with new Thread() my performance improved to less then a second.
I would love some explanation to why this is happening.
Clarification:
The Delay has nothing to do with the client code, from the moment ThreadPool.QueueUserWorkItem is called until clientMsgHandler.HandleIncomingMsgs is started 20 seconds can pass.
The lag starts at very first threads, and actually improves slightly as the test continues. I'm less interested in solutions and more interested in an explanation to why it's happening. 
The client does block, but for a very short while.
The server code:
private void AddTcpClientMsgHandler(TcpClient tcpClient)
    {
        //lock so no addition of client and closure can occur concurrently
        Stopwatch watch = new Stopwatch();
        watch.Start();
        Monitor.Enter(this);
        int pWatchIdx =  watchIDX++;
        if (!isOpen)
            throw new ObjectDisposedException(ResourceAlreadyClosed);

        TcpClientMsgHandler clientMsgHandler = CreateClientHandler(tcpClient);                                         
        clientMsgHandlerManager.AddTcpClientMsgHandler(clientMsgHandler);
        //ThreadPool.QueueUserWorkItem(clientMsgHandler.HandleIncomingMsgs); takes 20 seconds to run
        Thread thread = new Thread(clientMsgHandler.HandleIncomingMsgs);
        thread.Start();
        watch.Stop();
        Monitor.Exit(this);
        Console.WriteLine(string.Format("Iteration {0} took {1} Client {2}", pWatchIdx.ToString(),watch.Elapsed, tcpClient.Client.RemoteEndPoint));

    }

A:

Blocking code is the enemy of the ThreadPool. From the example you posted, it's not possible to tell where the blocking is happening, but I suggest that you review the code path to find out where the code is blocking. Run your server in the debugger until it starts showing high latency then break execution and take a look in the threads panel of VS. This will show you where the threads have blocked. Most likely this is on synchronous IO. Consider replacing with asynchronous code.

