<story><title>Web Scraping to Create Open Data</title><url>https://blog.scrapinghub.com/2016/03/30/web-scraping-to-create-open-data/</url></story><parent_chain></parent_chain><comment><author>jnotarstefano</author><text>I had to use a similar approach when creating a cluster analysis of the amendments in the Italian Senate [0].&lt;p&gt;The Italian Senate offers a SPARQL endpoint [1], which unfortunately doesn&amp;#x27;t offer access to the texts of the amendments. So I had to roll my own and create a small spider for them using Scrapy [2].&lt;p&gt;[0]: &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;github.com&amp;#x2F;jacquerie&amp;#x2F;senato.py&amp;#x2F;blob&amp;#x2F;master&amp;#x2F;analysis.ipynb&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;github.com&amp;#x2F;jacquerie&amp;#x2F;senato.py&amp;#x2F;blob&amp;#x2F;master&amp;#x2F;analysis....&lt;/a&gt;&lt;p&gt;[1]: &lt;a href=&quot;http:&amp;#x2F;&amp;#x2F;dati.senato.it&amp;#x2F;23&quot; rel=&quot;nofollow&quot;&gt;http:&amp;#x2F;&amp;#x2F;dati.senato.it&amp;#x2F;23&lt;/a&gt;&lt;p&gt;[2]: &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;github.com&amp;#x2F;jacquerie&amp;#x2F;senato.py&amp;#x2F;blob&amp;#x2F;master&amp;#x2F;senato&amp;#x2F;spiders&amp;#x2F;senato_spider.py&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;github.com&amp;#x2F;jacquerie&amp;#x2F;senato.py&amp;#x2F;blob&amp;#x2F;master&amp;#x2F;senato&amp;#x2F;sp...&lt;/a&gt;</text></comment>
<story><title>Web Scraping to Create Open Data</title><url>https://blog.scrapinghub.com/2016/03/30/web-scraping-to-create-open-data/</url></story><parent_chain></parent_chain><comment><author>minimaxir</author><text>I&amp;#x27;m not fond of the implication at the end that scraping is justifiable because old websites are dinosaurs without APIs, and those websites are &lt;i&gt;jerks&lt;/i&gt; for not doing so, and therefore scraping is the &lt;i&gt;moral&lt;/i&gt; thing to do.&lt;p&gt;I&amp;#x27;ve scraped my share of BuzzFeed data and Foursquare data to make data visualizations (with the latter explicitly saying &amp;quot;don&amp;#x27;t scrape&amp;quot; in their Terms). But if either one told me to stop and take down my results, I would not contest, since data is what drives the Internet ecosystem.&lt;p&gt;(For the record, neither service did; in fact, both tried to recruit me as a result of the visualizations. The difference is that I am not using the data to create a direct competitor that could cause them to lose business.)</text></comment>