Q:

Performance - Windows Server 2008 R2 vs Windows Server 2012

we have two different server that runs the same server application. They have to handle lot of connections (2000+) in a custom protocol, our users have our program to connect to our servers.
Machine 1 now is handling 3300 connection and it have an Intel i7 3770 with 16gb ram with Windows 2008 R2
Machine 2 now is handling 2400 connection and it have an Intel i7 4770 with 32gb ram with Windows 2012
Both the two machines uses about 6-9% of the CPU, less than 1% of the network, 0% HD, similar ping (66ms and 73ms) from our offices and the same network card.
Machine 2 is more powerful and have to handle less connections, but is much slower than machine 1!
I can't measure in numbers how much is slower, but our service needs a quick response to work well and Machine 2 is significantly slower.
Probably the problem is inside our .exe (even if its the same for both the machines) but we're considering the problem can be Windows 2012 too.
Does someone have any idea about what can produce this difference?
edit: thank you for your answers and interest in this! I'm adding the info you're requesting:

our program create a one-directional compressed stream of data between two or more users, so they must be both connected on the same machine.
we checked the performance in "Performance Monitor" and can't find any bottle neck to that, all is always below 10% usage (cpu, memory, hd, network)
we have no network bottleneck, our servers have a 1gpbs access to the internet and we normally use only 2 or 3 mbps. I know that 1gpbs is the network card speed, but also is our server's internet access speed.
quantifying the performance differences is not easy, but on Machine 1 our users can login in less than 100ms, on Machine 2 the same login uses about 2 seconds. Login is not the only thing slowed, the entire connection session is slower

Thank you @theunpaidBill, we'll try to do a better packet capture to understand what's happening!
Another edit: I know this is already closed but i'm adding this last piece of information!
We downgraded the Windows 2012 server to Windows 2008 R2 and we get a performance boost with our server. I still don't know why but thats what happen.

A:

The first step would be to isolate where your bottleneck is; network, disk, cpu.
My first suspect would be the network.  Verify that your server is connecting to the network at the proper speed.  
If it is, I would put a protocol analyzer on the server and run a packet capture.  In my experience custom protocols are usually a recipe for heartache.  
The packet capture will tell you if the network traffic is causing delays due to retransmits, extraneous handshaking, etc.
If it's not the network I would get with the developers about running it in debug mode.
Isolating a performance issue like this is very challenging, good luck.

