<story><title>Internet Archive Infrastructure</title><url>https://archive.org/details/jonah-edwards-presentation</url></story><parent_chain><item><author>tkgally</author><text>This is a recent video presentation by Jonah Edwards, who runs the Core Infrastructure Team at the Internet Archive. He explains the IA’s server, storage, and networking infrastructure, and then takes questions from other people at the Archive.&lt;p&gt;I found it all interesting. But the main takeaway for me is his response to Brewster Kahle’s question, beginning at 13:06, about why the IA does everything in-house rather than having its storage and processing hosted by, for example, AWS. His answer: lower cost, greater control, and greater confidence that their users are not being tracked.</text></item></parent_chain><comment><author>7800</author><text>I have ADD and typically eschew watching video if I can get the same, quality content faster in text.&lt;p&gt;I loved this and watched it to the end.&lt;p&gt;To any that feel the need to hide or disparage this because it seems to promote doing things on your own vs in the cloud, this isn’t some tech ops Total Money Makeover where you read a book and you’re suddenly in some sort of anti-credit cult. This is hard shit, and it’s the basics of the hard shit that I grew up with as being the only shit.&lt;p&gt;Yes, you can serve your own data. No one should fault you for doing that if you want. It takes the humble intelligence of the core team and everyone at IA to pull that off at this scale. If you don’t want to do the hard things, you could use the cloud. There are financial reasons also for one or the other, just as there are reasons people live with their family, rent, lease, and buy homes and office space- an imperfect analogy of course.&lt;p&gt;I hope that some of the others that could go on to work at the big guys or have been working there and want a challenge consider applying to IA when there’s an opening. They’ve done an incredible job, and I look forward to the cool things they accomplish in the future.</text></comment>
<story><title>Internet Archive Infrastructure</title><url>https://archive.org/details/jonah-edwards-presentation</url></story><parent_chain><item><author>tkgally</author><text>This is a recent video presentation by Jonah Edwards, who runs the Core Infrastructure Team at the Internet Archive. He explains the IA’s server, storage, and networking infrastructure, and then takes questions from other people at the Archive.&lt;p&gt;I found it all interesting. But the main takeaway for me is his response to Brewster Kahle’s question, beginning at 13:06, about why the IA does everything in-house rather than having its storage and processing hosted by, for example, AWS. His answer: lower cost, greater control, and greater confidence that their users are not being tracked.</text></item></parent_chain><comment><author>ignoramous</author><text>Speaking of Infrastructure, it is amazing that the initial set of the Apache BigData projects started at &lt;i&gt;Internet Archive&lt;/i&gt; [0] whilst &lt;i&gt;Alexa Internet&lt;/i&gt;, a startup Brewster Kahle sold to &lt;i&gt;Amazon&lt;/i&gt; in 1999, formed the basis of &lt;i&gt;Alexa Web Information Service&lt;/i&gt;, one of the first &amp;#x27;&amp;#x27;AWS&amp;#x27;&amp;#x27; products [1] which is still up? &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;aws.amazon.com&amp;#x2F;awis&amp;#x2F;&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;aws.amazon.com&amp;#x2F;awis&amp;#x2F;&lt;/a&gt;&lt;p&gt;[0] &lt;a href=&quot;http:&amp;#x2F;&amp;#x2F;radar.oreilly.com&amp;#x2F;2015&amp;#x2F;04&amp;#x2F;coming-full-circle-with-bigtable-and-hbase.html&quot; rel=&quot;nofollow&quot;&gt;http:&amp;#x2F;&amp;#x2F;radar.oreilly.com&amp;#x2F;2015&amp;#x2F;04&amp;#x2F;coming-full-circle-with-big...&lt;/a&gt;&lt;p&gt;[1] &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;web.stanford.edu&amp;#x2F;class&amp;#x2F;ee204&amp;#x2F;Publications&amp;#x2F;Amazon-EE353-2008-1.pdf&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;web.stanford.edu&amp;#x2F;class&amp;#x2F;ee204&amp;#x2F;Publications&amp;#x2F;Amazon-EE3...&lt;/a&gt;</text></comment>