<story><title>How much memory does a graph take?</title><url>https://jazco.dev/2024/04/15/in-memory-graphs/</url></story><parent_chain></parent_chain><comment><author>lmeyerov</author><text>Extra fun: We find most enterprise&amp;#x2F;gov graph analytics work only requires 1-2 attributes to go along with the graph index, and those attributes often are already numeric (time, $, ...) or can be dictionary-encoded as discussed here (categorical, ID, ...). The result is that even &amp;#x27;tough&amp;#x27; billion scale graphs are fine on 1 gpu.&lt;p&gt;Ex: Most of our users are happy to do 100K-100M edges quickly on a single node, imagine a quick parquet read, so goes surprisingly far. For $0.5&amp;#x2F;hr, you can get 16GB of GPU RAM on aws, stuff in 100M+ edges comfortably, and have room to compute all sorts of things. With a more typical one, 1B is easy.&lt;p&gt;Early, but that&amp;#x27;s been the basic thinking into our new GFQL system: slice into the columns you want, and then do all the in-GPU traversals you want. In our V1, we keep things dataframe-native, including the in-GPU data representation, and we are now working on adding the first extensions to support on-the-fly switching to more graph-native indexing as needed.&lt;p&gt;Ex: &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;github.com&amp;#x2F;graphistry&amp;#x2F;pygraphistry&amp;#x2F;blob&amp;#x2F;master&amp;#x2F;demos&amp;#x2F;gfql&amp;#x2F;benchmark_hops_cpu_gpu.ipynb&quot;&gt;https:&amp;#x2F;&amp;#x2F;github.com&amp;#x2F;graphistry&amp;#x2F;pygraphistry&amp;#x2F;blob&amp;#x2F;master&amp;#x2F;demos...&lt;/a&gt;</text></comment>
<story><title>How much memory does a graph take?</title><url>https://jazco.dev/2024/04/15/in-memory-graphs/</url></story><parent_chain></parent_chain><comment><author>fl0ki</author><text>I&amp;#x27;ve been doing this for years in projects internal to my employers. It&amp;#x27;s nice to see it catching on, because it&amp;#x27;ll be easier to explain to people if I can point to a blog post saying the same thing.&lt;p&gt;Other tips worth trying:&lt;p&gt;* You&amp;#x27;ll do even better by optimizing your ID type. Even if it has to be a string, it can at least be a string with SSO, or a long-lived string reference, or an interned string.&lt;p&gt;* Try the ahash crate instead of the default cryptogically secure hash function in Rust.&lt;p&gt;* Consider HashSet&amp;lt;ByRef&amp;lt;Arc&amp;lt;T&amp;gt;&amp;gt;&amp;gt; instead of having to take keys to a different map to resolve the actual object.</text></comment>