<story><title>Welcome to the New AWS AI Blog</title><url>https://aws.amazon.com/blogs/ai/welcome-to-the-new-aws-ai-blog/</url></story><parent_chain><item><author>niklasrde</author><text>We have recently set up a TensorFlow assessment function in AWS lambda, and got &lt;i&gt;very&lt;/i&gt; close to the maximum allowed size of a lambda function (250MB) with the trained model currently being 85MB, and the TensorFlow libraries and binaries taking up another 140 or so megabytes by default (&amp;lt;- which we cracked down a bit, but that&amp;#x27;s quite a hack).&lt;p&gt;I feel like Amazon could do some work in this area to support users to use their own engines and not be bound to AWS AI Platforms and Services.&lt;p&gt;This could be as simple as publicly documenting the time lambda&amp;#x27;s stay &amp;#x27;warm&amp;#x27; for and retain data in &amp;#x2F;tmp persisting through multiple invocations or some other examples on how an AI workflow could be implemented with popular custom engines such as TensorFlow.&lt;p&gt;Does anybody else have any experience in this regard?</text></item></parent_chain><comment><author>tcas</author><text>I&amp;#x27;ve hacked around lambda quite a bit (I think the compressed size of one function is a tad under the max allowed). My hacks I remember are:&lt;p&gt;- Run strip all .so libraries -- many aren&amp;#x27;t stripped fully&lt;p&gt;- In Python I manually deleted sub packages of numpy&amp;#x2F;scipy I didn&amp;#x27;t need&lt;p&gt;- If you&amp;#x27;re loading large models at initialize, numpy load routines are _much_ faster than cPickle. Have it load at module initialization, not during each invocation.&lt;p&gt;I should really write a blog post about my experience with it.&lt;p&gt;At a certain point I decided I was doing something that lambda really wasn&amp;#x27;t designed for -- I&amp;#x27;m looking at migrating off, but the current implementation makes capacity planning super easy. Provisioning 1000 machines with 1GB of RAM for 15 minutes every day to read off a queue isn&amp;#x27;t a trivial problem.&lt;p&gt;(Also, if anyone from AWS is reading, being able to limit the max concurrency of a single function vs account level limits would be super useful).</text></comment>
<story><title>Welcome to the New AWS AI Blog</title><url>https://aws.amazon.com/blogs/ai/welcome-to-the-new-aws-ai-blog/</url></story><parent_chain><item><author>niklasrde</author><text>We have recently set up a TensorFlow assessment function in AWS lambda, and got &lt;i&gt;very&lt;/i&gt; close to the maximum allowed size of a lambda function (250MB) with the trained model currently being 85MB, and the TensorFlow libraries and binaries taking up another 140 or so megabytes by default (&amp;lt;- which we cracked down a bit, but that&amp;#x27;s quite a hack).&lt;p&gt;I feel like Amazon could do some work in this area to support users to use their own engines and not be bound to AWS AI Platforms and Services.&lt;p&gt;This could be as simple as publicly documenting the time lambda&amp;#x27;s stay &amp;#x27;warm&amp;#x27; for and retain data in &amp;#x2F;tmp persisting through multiple invocations or some other examples on how an AI workflow could be implemented with popular custom engines such as TensorFlow.&lt;p&gt;Does anybody else have any experience in this regard?</text></item></parent_chain><comment><author>ranman</author><text>The keep warm time isn&amp;#x27;t static AFAIK but some frameworks like Zappa have chosen 5 minutes (I work for AWS). If you raise an issue with support there may be other ways around this. There are also other reasons for not relying on functions being warm (spike in concurrent invocations, AZ outages, latency, etc.). If you open a support ticket and email me the # I&amp;#x27;ll see what I can find out: randhunt at amazon dot com&lt;p&gt;I&amp;#x27;m curious what sort of things you did to shrink your model?&lt;p&gt;Have you considered pulling the model data from S3 outside of the main lambda handler and seeing if that negatively impacts performance -- with a cloud watch event running every 5 minutes or so to keep the function warm?</text></comment>