<story><title>Detecting Chrome headless, the game goes on</title><url>https://antoinevastel.com/bot%20detection/2019/07/19/detecting-chrome-headless-v3.html</url></story><parent_chain><item><author>Pinbenterjamin</author><text>I run the division at my company that builds crawlers for websites with public records. We scrape this information on-demand when a case is requested, and we handle an enormous volume of different sites (or sources as we call them). We recently passed 700 total custom scrapers.&lt;p&gt;Recently, we have seen a spike in sites that detect, and block our crawlers with some sort of Javascript we cannot identify. We use headless Chrome and selenium to build out most of our integrations, I&amp;#x27;m starting to wonder if the science of blocking scraping is getting more popular...&lt;p&gt;I don&amp;#x27;t think what I&amp;#x27;m doing is subversive at all, we&amp;#x27;re running background checks on people, and we can reduce business costs by eliminating error-prone researchers with smart scrapers that run all day.&lt;p&gt;I don&amp;#x27;t want to seem like the bad guy here, but what if I wanted to do the opposite of this research? Where do I start? Study the chromium source? Can anyone recommend a few papers?</text></item></parent_chain><comment><author>curun1r</author><text>&amp;gt; Where do I start? Study the chromium source?&lt;p&gt;I&amp;#x27;m curious why you&amp;#x27;d jump straight to browser detection as the most likely culprit. When I was doing scraping, the far more common case was bot detection by origin and access patterns. It&amp;#x27;s just very difficult to make an automated scraper look like a residential or business user.&lt;p&gt;Where do you run your scraping operation? Is it in AWS or some other hosting provider, because that will get you blocked quickly by a lot of sites? Do you rate limit, including adding random jitter to mimic the way a human might use a browser?&lt;p&gt;There&amp;#x27;s scraping services available that essentially use a network of browsers on residential connections with their extension installed to get around scraping detection. It&amp;#x27;s much slower, but it&amp;#x27;s much more reliable. We also had some success by signing up with a bunch of the VPN providers (PIA, NordVPN, ExpressVPN, etc) and cycling through their servers frequently. Anything to avoid creating patterns that look automated or being tied to an IP that can be blacklisted. I&amp;#x27;d start there before I&amp;#x27;d worry about hacky javascript detection like in this story being what&amp;#x27;s tripping you up.</text></comment>
<story><title>Detecting Chrome headless, the game goes on</title><url>https://antoinevastel.com/bot%20detection/2019/07/19/detecting-chrome-headless-v3.html</url></story><parent_chain><item><author>Pinbenterjamin</author><text>I run the division at my company that builds crawlers for websites with public records. We scrape this information on-demand when a case is requested, and we handle an enormous volume of different sites (or sources as we call them). We recently passed 700 total custom scrapers.&lt;p&gt;Recently, we have seen a spike in sites that detect, and block our crawlers with some sort of Javascript we cannot identify. We use headless Chrome and selenium to build out most of our integrations, I&amp;#x27;m starting to wonder if the science of blocking scraping is getting more popular...&lt;p&gt;I don&amp;#x27;t think what I&amp;#x27;m doing is subversive at all, we&amp;#x27;re running background checks on people, and we can reduce business costs by eliminating error-prone researchers with smart scrapers that run all day.&lt;p&gt;I don&amp;#x27;t want to seem like the bad guy here, but what if I wanted to do the opposite of this research? Where do I start? Study the chromium source? Can anyone recommend a few papers?</text></item></parent_chain><comment><author>floatingatoll</author><text>Reducing your business costs by scraping a public access website is often considered an &lt;i&gt;alternative&lt;/i&gt; to paying the business costs of the website operator.&lt;p&gt;Are you saving money at the expense of the site operator by scraping their site for public records, or are you saving money as well as the site operator?&lt;p&gt;If you&amp;#x27;re costing them money to reduce your own bottom line without their express written consent, that makes you &amp;quot;the bad guy&amp;quot;. Offsetting costs onto an unwitting, non-consenting third party is an unethical approach to doing business.&lt;p&gt;I interpret your request as a similar problem to &amp;quot;help me with my homework problem&amp;quot;. I could dig up papers and studies, but at the end of the day, you need to go do your homework. Reach out to each municipality and figure out a business arrangement with them that satisfies your needs. It&amp;#x27;s possible they do not wish you to perform this activity, in which case you will either need to violate their intent for your own profit using scraping or accede to their wishes and stop scraping their municipality. That&amp;#x27;s your homework as a for-profit business.</text></comment>