<story><title>Why Tarsnap doesn&apos;t use Glacier</title><url>http://www.daemonology.net/blog/2012-09-04-why-tarsnap-doesnt-use-glacier.html</url><text></text></story><parent_chain><item><author>ChuckMcM</author><text>While I think this is a completely reasonable thing (not storing tarsnap data in Glacier), I&apos;m not sure if the reasoning holds beyond, &quot;For the way we designed tarsnap it doesn&apos;t make sense.&quot;&lt;p&gt;When you look at how GFS is implemented or Bigtable, or Blekko&apos;s NoSQL data store, there are layers, with the meta data as a pretty separable layer from the data. In all of the cases for these large stores that separation was put in to facilitate putting the meta data into lower latency storage than the &apos;bulk&apos; data which facilitates fast access.&lt;p&gt;As cperciva relates, tarsnap spends a lot of time de-duplicating data (or writing only one copy of a block with identical contents) which is great for reducing your overall storage footprint which lowers your costs. But if the cost of storage is much much smaller than the retrieval cost, then your design methodology would be different.&lt;p&gt;So I would not be surprised if there was a way to build a function equivalent product to tarsnap that had lower storage costs if the bulk data was in glacier and the index data was in S3, or if the index data was designed such that a document recovery was exactly two retrievals (a catalog, and then the data). And of course such a system would not de-duplicate as that would result in potentially more retrievals.&lt;p&gt;It seems that if the price delta ratio between Glacier and S3 was higher than the de-dupe ratio, then Glacier would &apos;win&apos; with no de-dupe. Else de-dupe would still win. Thoughts?</text></item></parent_chain><comment><author>cperciva</author><text>&lt;i&gt;I&apos;m not sure if the reasoning holds beyond, &quot;For the way we designed tarsnap it doesn&apos;t make sense.&quot;&lt;/i&gt;&lt;p&gt;I agree -- that&apos;s exactly what I was saying.  The title of the blog post was &quot;Why Tarsnap doesn&apos;t use Glacier&quot;, not &quot;Why you shouldn&apos;t use Glacier&quot;.&lt;p&gt;I think Glacier is a great service, just not a particularly good fit to Tarsnap.</text></comment>
<story><title>Why Tarsnap doesn&apos;t use Glacier</title><url>http://www.daemonology.net/blog/2012-09-04-why-tarsnap-doesnt-use-glacier.html</url><text></text></story><parent_chain><item><author>ChuckMcM</author><text>While I think this is a completely reasonable thing (not storing tarsnap data in Glacier), I&apos;m not sure if the reasoning holds beyond, &quot;For the way we designed tarsnap it doesn&apos;t make sense.&quot;&lt;p&gt;When you look at how GFS is implemented or Bigtable, or Blekko&apos;s NoSQL data store, there are layers, with the meta data as a pretty separable layer from the data. In all of the cases for these large stores that separation was put in to facilitate putting the meta data into lower latency storage than the &apos;bulk&apos; data which facilitates fast access.&lt;p&gt;As cperciva relates, tarsnap spends a lot of time de-duplicating data (or writing only one copy of a block with identical contents) which is great for reducing your overall storage footprint which lowers your costs. But if the cost of storage is much much smaller than the retrieval cost, then your design methodology would be different.&lt;p&gt;So I would not be surprised if there was a way to build a function equivalent product to tarsnap that had lower storage costs if the bulk data was in glacier and the index data was in S3, or if the index data was designed such that a document recovery was exactly two retrievals (a catalog, and then the data). And of course such a system would not de-duplicate as that would result in potentially more retrievals.&lt;p&gt;It seems that if the price delta ratio between Glacier and S3 was higher than the de-dupe ratio, then Glacier would &apos;win&apos; with no de-dupe. Else de-dupe would still win. Thoughts?</text></item></parent_chain><comment><author>jerf</author><text>&quot;But if the cost of storage is much much smaller than the retrieval cost, then your design methodology would be different.&quot;&lt;p&gt;If you read between the lines a bit on how the second-to-last paragraph would manifest in code, I think that&apos;s what he&apos;s already getting at. A Glaiciated machine would probably not get global de-duping, but merely a machine-level de-duping, if even that. (The retrieval cost structure is quite bizarre.)</text></comment>