Q:

Error when trying to parse HTML

I trying to parse site"https://www.crunchbase.com". But this site has an "Antibot protection". And i don't know how to get any html element from the page. 
First i made a "ssl" security channel.
ServicePointManager.Expect100Continue = true;

ServicePointManager.SecurityProtocol = SecurityProtocolType.Ssl3 | SecurityProtocolType.Tls | SecurityProtocolType.Tls11 | SecurityProtocolType.Tls12;

Then i made a HttpRequest with my browser's user agent string.
var request = (HttpWebRequest)WebRequest.Create("https://www.crunchbase.com");

request.UserAgent = "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:52.0) Gecko/20100101 Firefox/52.0";

request.Timeout = 10000;

HttpWebResponse response = (HttpWebResponse)request.GetResponse();

Console.WriteLine("Server status code: " + response.StatusCode);

And used a StreamWriter to load the page:
using (StreamReader sr = new StreamReader(response.GetResponseStream()))
{
    string result = sr.ReadToEnd();
    Console.WriteLine(result);
}

But result is:
enter image description here
And finally i tried to get all Urls from the page:
HtmlWeb web = new HtmlWeb();

HtmlDocument document = web.Load(response.ResponseUri.AbsoluteUri);

string respUri = response.ResponseUri.ToString();

HtmlNode[] nodes = document.DocumentNode.SelectNodes("//a").ToArray();

foreach (var item in nodes)
{
    Console.WriteLine(item.InnerHtml);
}

But apllication throws Unhadled Exception.

A:

I assume the upper part in your console window is the output from Console.WriteLine(result), and this shows pretty much the antibot protection. Whatever you see when browsing this site, it's not in this HTML which has an almost empty body (when this is rendered, it gives... nothing). The actual content of the web page is probably loaded dynamically by one of the Javascript code pieces referred by the HTML content. On the other hand, the HtmlWeb parser (from HTML Agility Pack, I presume) does not execute this Javascript code, and thus does not reach the actual content that includes the  elements you are looking for. In other words, the protection works...

