<story><title>Show HN: Till – Unblock and scale your web scrapers, with minimal code changes</title><url>https://github.com/DataHenHQ/till</url></story><parent_chain><item><author>cube00</author><text>&amp;gt; Till helps you circumvent detected as a web scraper by identifying your scraper as a real web browser. It does this by generating random user-agent headers and randomizing proxy IPs (that you supply) on every HTTP request.&lt;p&gt;How did we start with consent using robots.txt and end up here?&lt;p&gt;A good neighbor doesn&amp;#x27;t use circumvention.</text></item></parent_chain><comment><author>realusername</author><text>We ended up here since some people wrongly decided that only Google has a right to scrape.</text></comment>
<story><title>Show HN: Till – Unblock and scale your web scrapers, with minimal code changes</title><url>https://github.com/DataHenHQ/till</url></story><parent_chain><item><author>cube00</author><text>&amp;gt; Till helps you circumvent detected as a web scraper by identifying your scraper as a real web browser. It does this by generating random user-agent headers and randomizing proxy IPs (that you supply) on every HTTP request.&lt;p&gt;How did we start with consent using robots.txt and end up here?&lt;p&gt;A good neighbor doesn&amp;#x27;t use circumvention.</text></item></parent_chain><comment><author>ok2938</author><text>We ended up here, because a) the promise of easy to use structured data - that is inherent in data processing systems - has not been fulfilled: you have to scrape HTML to reconstruct some relational table, and b) information, despite being almost free to transmit in large amounts today, is treated as if it were a physical good, and each copy costs money. There is just lots of money in there, because our economic system looks more backwards than forwards.</text></comment>