<story><title>Avoiding Webscraping Throttling Using Python and Tor as a Proxy</title><url>https://boredhacking.com/tor-webscraping-proxy/</url></story><parent_chain><item><author>CWuestefeld</author><text>While this is intellectually interesting, I&amp;#x27;m troubled by the fact that the author seems not to have given the slightest thought that he&amp;#x27;s breaking the site&amp;#x27;s T&amp;amp;C, or of how much this abuse costs the service.&lt;p&gt;I&amp;#x27;m particularly sensitive to this because I&amp;#x27;m constantly dealing with scraper bots from competitors that are trying to monitor our pricing. Without our ongoing policing, and a fair amount of developer time going into it, the traffic coming from these bots - and hence the amount it costs us to operate the site - is significantly larger than that of actual customers. Let me say that again: scraper bots account for more traffic on our sites than do legitimate customers.</text></item></parent_chain><comment><author>meritt</author><text>&amp;gt; breaking the site&amp;#x27;s T&amp;amp;C&lt;p&gt;T&amp;amp;C&amp;#x27;s have absolutely no bearing on publicly accessible information [1][2][3]. They only apply to registered users accessing login-required portions of your website or application. Browsewrap does not represent a legally binding contract, no matter how much you pay your lawyers to write up your irrelevant T&amp;amp;C&amp;#x27;s.&lt;p&gt;[1] &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;www.eff.org&amp;#x2F;deeplinks&amp;#x2F;2018&amp;#x2F;04&amp;#x2F;scraping-just-automated-access-and-everyone-does-it&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;www.eff.org&amp;#x2F;deeplinks&amp;#x2F;2018&amp;#x2F;04&amp;#x2F;scraping-just-automate...&lt;/a&gt;&lt;p&gt;[2] &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;www.eff.org&amp;#x2F;deeplinks&amp;#x2F;2018&amp;#x2F;04&amp;#x2F;dc-court-accessing-public-information-not-computer-crime&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;www.eff.org&amp;#x2F;deeplinks&amp;#x2F;2018&amp;#x2F;04&amp;#x2F;dc-court-accessing-pub...&lt;/a&gt;&lt;p&gt;[3] &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;www.eff.org&amp;#x2F;deeplinks&amp;#x2F;2018&amp;#x2F;01&amp;#x2F;ninth-circuit-doubles-down-violating-websites-terms-service-not-crime&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;www.eff.org&amp;#x2F;deeplinks&amp;#x2F;2018&amp;#x2F;01&amp;#x2F;ninth-circuit-doubles-...&lt;/a&gt;</text></comment>
<story><title>Avoiding Webscraping Throttling Using Python and Tor as a Proxy</title><url>https://boredhacking.com/tor-webscraping-proxy/</url></story><parent_chain><item><author>CWuestefeld</author><text>While this is intellectually interesting, I&amp;#x27;m troubled by the fact that the author seems not to have given the slightest thought that he&amp;#x27;s breaking the site&amp;#x27;s T&amp;amp;C, or of how much this abuse costs the service.&lt;p&gt;I&amp;#x27;m particularly sensitive to this because I&amp;#x27;m constantly dealing with scraper bots from competitors that are trying to monitor our pricing. Without our ongoing policing, and a fair amount of developer time going into it, the traffic coming from these bots - and hence the amount it costs us to operate the site - is significantly larger than that of actual customers. Let me say that again: scraper bots account for more traffic on our sites than do legitimate customers.</text></item></parent_chain><comment><author>vbezhenar</author><text>Another problem is that blocking tor nodes is very easy. And with abusing tor nodes as free proxies, it makes a huge disservice to the Tor network, because more websites will ban it and ordinary users won&amp;#x27;t be able to use it.</text></comment>