<story><title>Detecting Chrome headless, the game goes on</title><url>https://antoinevastel.com/bot%20detection/2019/07/19/detecting-chrome-headless-v3.html</url></story><parent_chain></parent_chain><comment><author>jadell</author><text>It seems he&amp;#x27;s doing something with header detection. I used Puppeteer to play around with the site and various configurations I use when scraping.&lt;p&gt;In headless Chrome, the &amp;quot;Accept-Language&amp;quot; header is not sent. In Puppeteer, one can force the header to be sent by doing:&lt;p&gt;&lt;pre&gt;&lt;code&gt;  page.setExtraHTTPHeaders({ &amp;#x27;Accept-Language&amp;#x27;: &amp;#x27;en-US,en;q=0.9&amp;#x27; })

&lt;/code&gt;&lt;/pre&gt;
However, Puppeteer sends that header as lowercase:&lt;p&gt;&lt;pre&gt;&lt;code&gt;  accept-language: en-US,en;q=0.9

&lt;/code&gt;&lt;/pre&gt;
So it seems the detection is as simply: if there is no &amp;#x27;Accept-Language&amp;#x27; header (case-sensitive), then &amp;quot;Headless Chrome&amp;quot;; else, &amp;quot;Not Headless Chrome&amp;quot;.&lt;p&gt;This is a completely server-side check, which is why he can say the fpcollect client-side javascript library isn&amp;#x27;t involved.&lt;p&gt;Here are some curl commands that demonstrate:&lt;p&gt;Detected: not headless&lt;p&gt;&lt;pre&gt;&lt;code&gt;  curl &amp;#x27;https:&amp;#x2F;&amp;#x2F;arh.antoinevastel.com&amp;#x2F;bots&amp;#x2F;areyouheadless&amp;#x27; \
  -H &amp;#x27;User-Agent: Mozilla&amp;#x2F;5.0 (X11; Linux x86_64) AppleWebKit&amp;#x2F;537.36 (KHTML, like Gecko) Chrome&amp;#x2F;76.0.3803.0 Safari&amp;#x2F;537.36&amp;#x27; \
  -H &amp;#x27;Accept-Language: en-US,en;q=0.9&amp;#x27;
&lt;/code&gt;&lt;/pre&gt;
Detected: headless&lt;p&gt;&lt;pre&gt;&lt;code&gt;  curl &amp;#x27;https:&amp;#x2F;&amp;#x2F;arh.antoinevastel.com&amp;#x2F;bots&amp;#x2F;areyouheadless&amp;#x27; \
  -H &amp;#x27;User-Agent: Mozilla&amp;#x2F;5.0 (X11; Linux x86_64) AppleWebKit&amp;#x2F;537.36 (KHTML, like Gecko) Chrome&amp;#x2F;76.0.3803.0 Safari&amp;#x2F;537.36&amp;#x27;
&lt;/code&gt;&lt;/pre&gt;
Detected: headless&lt;p&gt;&lt;pre&gt;&lt;code&gt;  curl &amp;#x27;https:&amp;#x2F;&amp;#x2F;arh.antoinevastel.com&amp;#x2F;bots&amp;#x2F;areyouheadless&amp;#x27; \
  -H &amp;#x27;User-Agent: Mozilla&amp;#x2F;5.0 (X11; Linux x86_64) AppleWebKit&amp;#x2F;537.36 (KHTML, like Gecko) Chrome&amp;#x2F;76.0.3803.0 Safari&amp;#x2F;537.36&amp;#x27; \
  -H &amp;#x27;accept-language: en-US,en;q=0.9&amp;#x27;&lt;/code&gt;&lt;/pre&gt;</text></comment>
<story><title>Detecting Chrome headless, the game goes on</title><url>https://antoinevastel.com/bot%20detection/2019/07/19/detecting-chrome-headless-v3.html</url></story><parent_chain></parent_chain><comment><author>Pinbenterjamin</author><text>I run the division at my company that builds crawlers for websites with public records. We scrape this information on-demand when a case is requested, and we handle an enormous volume of different sites (or sources as we call them). We recently passed 700 total custom scrapers.&lt;p&gt;Recently, we have seen a spike in sites that detect, and block our crawlers with some sort of Javascript we cannot identify. We use headless Chrome and selenium to build out most of our integrations, I&amp;#x27;m starting to wonder if the science of blocking scraping is getting more popular...&lt;p&gt;I don&amp;#x27;t think what I&amp;#x27;m doing is subversive at all, we&amp;#x27;re running background checks on people, and we can reduce business costs by eliminating error-prone researchers with smart scrapers that run all day.&lt;p&gt;I don&amp;#x27;t want to seem like the bad guy here, but what if I wanted to do the opposite of this research? Where do I start? Study the chromium source? Can anyone recommend a few papers?</text></comment>