<story><title>So you want to scrape like the big boys (2021)</title><url>https://incolumitas.com/2021/11/03/so-you-want-to-scrape-like-the-big-boys/</url></story><parent_chain></parent_chain><comment><author>KieranMac</author><text>I&amp;#x27;m a lawyer that works in the web-scraping space, and I always chuckle when I read threads like this. Almost every company that we now consider a monopolist (or their affiliates) in the tech space used scraping a part of their process to build their business, and almost every one of those same monopolists now prohibits startups and competitors from scraping their data (which, invariably, is not actually &amp;quot;their&amp;quot; data in any sort of legally cognizable sense). And so perhaps the ethics of web scraping are not so straightforward. And neither are the legal issues associated with it.&lt;p&gt;I wrote an article about that last fall that got some attention here.&lt;p&gt;&lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;news.ycombinator.com&amp;#x2F;item?id=37264676&quot;&gt;https:&amp;#x2F;&amp;#x2F;news.ycombinator.com&amp;#x2F;item?id=37264676&lt;/a&gt;</text></comment>
<story><title>So you want to scrape like the big boys (2021)</title><url>https://incolumitas.com/2021/11/03/so-you-want-to-scrape-like-the-big-boys/</url></story><parent_chain></parent_chain><comment><author>anyfactor</author><text>I was a professional web scraper. I still keep up to date with the industry.&lt;p&gt;These days, you do not make money by doing web scraping; you make money selling services to web scrapers. There are tons of web scraping SAAS and services out there, as well as dozens of residential proxy providers.&lt;p&gt;Most anti-bot mechanisms evolve so quickly that you can make a decent income just by working in a traditional software engineering role dedicated entirely to engineering anti-anti-bot solutions. As these mechanisms evolve rapidly, working for a web scraping company is more stable than pursuing web scraping as a profession.&lt;p&gt;Web scrapers get paid by projects, making it an unstable job in the long run. High-level web scraping requires operational investments in residential proxies and renting out servers. Additionally, low-end jobs pay very little. Brightdata hosting a conference on web scraping, which should indicate the profitability of selling services in large-scale web scraping.</text></comment>