Q:

Bottleneck in 3000+ thread .NET Application (HttpWebReqsuest.BeginGetResponse)

So I'm creating an application which uses an HTTP API to get the status of each of about ~50,000 accounts. The code loops through each account, sends an HTTP request for it. When I receive responses for the http requests, based on some criteria, i create a new thread to manage that account.
Now normally, i could easily be sending out about 1K requests per second, but once i hit ~3,100 active threads, the http request loop begins slowing down and freezing to only about 1 request per second. Then it suddenly jumps back up to 3k per second, then freezes for a few seconds...etc. The important thing is that it doesn't seem like a gradual degradation. It happens very suddenly.
Clearly there is a bottleneck somewhere, but I'm not sure where. I've made sure that the TCP paramaters (maximum number of usable ports) is set to high limits. I have the servicepoint.defaultconnection limit set to int.maxvalue.
My CPU is a 4-core (8 thread with HT) with a dedicated 1Gbps. I'm considering moving over to a much larger (32 core 2x CPU) machine. But I'm not sure if it will have any benefit. I'm wondering if someone is aware of some other bottlenecks i may be encountering.
I'm not even close to using up all my bandwidth or memory, I know it's not an issue there.
This is roughly what my code looks like
Sub Main()
    While 1
        For each account As Account in GetAccountsFromDatabase()'~50K Accounts
            dim request = HttpWebRequest.Create("http://api.com?id=" & account.name)
            request.BeginGetResponse(New AsyncCallback(AddressOf HandleResponse), request)
            RequestsSent += 1
            Console.Writeline("Request")

            'After ~3000 active threads in the process, this loop begins freezing/slowing down.

            if RequestsSent > 5000 then 'Limit
                Thread.Sleep(5000)
                RequestsSent = 0
            end if

        Next
    End While
End Sub

Sub HandleResponse(ByVal asynchronousResult As IAsyncResult)
    Dim webRequest As HttpWebRequest = DirectCast(asynchronousResult.AsyncState, HttpWebRequest)
    Dim webResponse As HttpWebResponse = webRequest.EndGetResponse(asynchronousResult)
    Dim stream As New StreamReader(webResponse.GetResponseStream())
    Dim response = stream.ReadToEnd

    if response.contains("somestuff") then    
        dim t As New Thread(AddressOf ProcessAccount)
        t.Start(account) 
    end if

End Sub

Sub ProcessAccount(acc As Account)
    'Process the account. Involves some other loops, http requests...etc
End Sub

A:

Well, you are asynchronously starting the request but once BeginGetResponse completes you synchronously process the request. This is likely to consume a lot of threads from the pool eventually. Maybe the pool limit in your configuration is around 3000 threads.
You need to throw this away. Once you get into the hundreds of threads, and definitely in the 1000s, you need to switch to async non-blocking IO. With await this has become quite easy.
Make the stream reading part async (ReadToEnd). Probably, you should delete all that code and replace it with var str = await new HttpClient().GetAsync(url); (C#).
Next, it is unwise to run 50,000 (!) threads. That alone will set you back 50 GB (!) of memory for stacks. The OS starts to have trouble scheduling all those threads as well which I found out during testing. The mouse stops moving for seconds for example although the mouse driver has very high priority. Clearly, the kernel team doesn't care much about this scenario.
Probably, you should process accounts at a given degree of parallelism. For example, process only 100 concurrently.
If you require processing them at the same time for some reason then ProcessAccount must be async as well. No other way.

