<story><title>Link rot and content drift are endemic to the web</title><url>https://www.theatlantic.com/technology/archive/2021/06/the-internet-is-a-collective-hallucination/619320/</url></story><parent_chain><item><author>uniqueuid</author><text>By the way, the technical side of this is very interesting. If you look at the tools mentioned (the wayback machine, but also perma.cc and other archival solutions), almost all of them rely on a single semi-modern tech stack that produces WARCs (web archives - ISO - ISO 28500:2017 &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;iipc.github.io&amp;#x2F;warc-specifications&amp;#x2F;specifications&amp;#x2F;warc-format&amp;#x2F;warc-1.1&amp;#x2F;&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;iipc.github.io&amp;#x2F;warc-specifications&amp;#x2F;specifications&amp;#x2F;wa...&lt;/a&gt;).&lt;p&gt;The main crawler still seems to be heritrix3 (&lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;github.com&amp;#x2F;internetarchive&amp;#x2F;heritrix3&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;github.com&amp;#x2F;internetarchive&amp;#x2F;heritrix3&lt;/a&gt;), but there&amp;#x27;s a great little ecosystem with tools such as webrecorder and warcprox.&lt;p&gt;Still, I&amp;#x27;ve read through the code of these tools and am feeling that they are failing in the face of the modern web with single page apps, mobile phone apps and walled gardens. Even newer iterations with browser automation are getting increasingly throttled and blocked and excluded from walled gardens.&lt;p&gt;Perhaps the time has come for a coordinated, decentralized but omnipresent approach to archival.</text></item></parent_chain><comment><author>PaulHoule</author><text>WARC can record and replay single-page apps,  but it struggles with knowing where a &amp;quot;page&amp;quot; begins and ends.&lt;p&gt;There was a time when I was furious with the web going to hell and I investigated the possibility of &amp;quot;web without browsers&amp;quot; that started with making a WARC capture of page and putting pages through extensive filtering and classification before the user sees anything.&lt;p&gt;With interactive capturing you can push a button to indicate that a page is done &amp;quot;loading&amp;quot; but with automated capturing you can&amp;#x27;t really know that the page is done or that you got a good capture.  That ended the project right there.</text></comment>
<story><title>Link rot and content drift are endemic to the web</title><url>https://www.theatlantic.com/technology/archive/2021/06/the-internet-is-a-collective-hallucination/619320/</url></story><parent_chain><item><author>uniqueuid</author><text>By the way, the technical side of this is very interesting. If you look at the tools mentioned (the wayback machine, but also perma.cc and other archival solutions), almost all of them rely on a single semi-modern tech stack that produces WARCs (web archives - ISO - ISO 28500:2017 &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;iipc.github.io&amp;#x2F;warc-specifications&amp;#x2F;specifications&amp;#x2F;warc-format&amp;#x2F;warc-1.1&amp;#x2F;&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;iipc.github.io&amp;#x2F;warc-specifications&amp;#x2F;specifications&amp;#x2F;wa...&lt;/a&gt;).&lt;p&gt;The main crawler still seems to be heritrix3 (&lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;github.com&amp;#x2F;internetarchive&amp;#x2F;heritrix3&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;github.com&amp;#x2F;internetarchive&amp;#x2F;heritrix3&lt;/a&gt;), but there&amp;#x27;s a great little ecosystem with tools such as webrecorder and warcprox.&lt;p&gt;Still, I&amp;#x27;ve read through the code of these tools and am feeling that they are failing in the face of the modern web with single page apps, mobile phone apps and walled gardens. Even newer iterations with browser automation are getting increasingly throttled and blocked and excluded from walled gardens.&lt;p&gt;Perhaps the time has come for a coordinated, decentralized but omnipresent approach to archival.</text></item></parent_chain><comment><author>shadowgovt</author><text>Honestly, it would be a better use of surplus resources than crypto mining.&lt;p&gt;If only there are a way to algorithmically tie a proof of work for a new cryptocurrency to archival of the internet in a way that wouldn&amp;#x27;t be easily gamed (by people archiving easy to access content or highly redundant archival of trivia).</text></comment>