<story><title>Fixing memory leaks in popular Python libraries</title><url>https://www.paulsprogrammingnotes.com/2021/12/python-memory-leaks.html</url></story><parent_chain></parent_chain><comment><author>js2</author><text>FWIW, Python will close a socket automatically when the reference to the socket is GC&amp;#x27;d. (Of course, you _should_ always close a socket explicitly and not rely on GC to handle it for you.)&lt;p&gt;So the larger issue here was that the original code was written such that when shutdown() raised an exception, the reference to the socket object wasn&amp;#x27;t released—i.e., the line of code that set `self.socket = None` was not reached.&lt;p&gt;I had to follow the link to the PR to see what was going on, because this wasn&amp;#x27;t clear to me from the minimal example in the blog post. The fix to redis-py wasn&amp;#x27;t 100% necessary. Here&amp;#x27;s the redis-py code before the PR:&lt;p&gt;&lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;github.com&amp;#x2F;pawl&amp;#x2F;redis-py&amp;#x2F;blob&amp;#x2F;82bad1686177c4c543818a8bfac35c6fdfc9ddf1&amp;#x2F;redis&amp;#x2F;connection.py#L715&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;github.com&amp;#x2F;pawl&amp;#x2F;redis-py&amp;#x2F;blob&amp;#x2F;82bad1686177c4c543818a...&lt;/a&gt;&lt;p&gt;&lt;pre&gt;&lt;code&gt;   def disconnect(self):
        &amp;quot;Disconnects from the Redis server&amp;quot;
        self._parser.on_disconnect()
        if self._sock is None:
            return
        try:
            if os.getpid() == self.pid:
                self._sock.shutdown(socket.SHUT_RDWR)
            self._sock.close()
        except OSError:
            pass
        self._sock = None
&lt;/code&gt;&lt;/pre&gt;
Notice that self._sock is set to None even if shutdown() raises an exception. This will eventually cause the socket object to be GC&amp;#x27;d, and thus closed. No memory leak here. But it&amp;#x27;s still good to call close explicitly, so I&amp;#x27;m not disagree&amp;#x27;ing with the PR, just that there wasn&amp;#x27;t a memory leak:&lt;p&gt;&lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;github.com&amp;#x2F;pawl&amp;#x2F;redis-py&amp;#x2F;commit&amp;#x2F;d7f4ac33217a95541ddf2a0e22367afb33dfac0a#diff-45b6ba995a827b484d3ef97cb806d336bf058ff9678b04cf371200ea53e584d0&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;github.com&amp;#x2F;pawl&amp;#x2F;redis-py&amp;#x2F;commit&amp;#x2F;d7f4ac33217a95541ddf...&lt;/a&gt;&lt;p&gt;Conversely, py-amqp looked like this:&lt;p&gt;&lt;pre&gt;&lt;code&gt;    def close(self):
        if self.sock is not None:
            self._shutdown_transport()
            # Call shutdown first to make sure that pending messages
            # reach the AMQP broker if the program exits after
            # calling this method.
            self.sock.shutdown(socket.SHUT_RDWR)
            self.sock.close()
            self.sock = None
        self.connected = False
&lt;/code&gt;&lt;/pre&gt;
Any exception in this method prevents `self.sock = None` from running. That, _combined with the fact that_ Celery never released the py-amqp object is where the leak was.</text></comment>
<story><title>Fixing memory leaks in popular Python libraries</title><url>https://www.paulsprogrammingnotes.com/2021/12/python-memory-leaks.html</url></story><parent_chain></parent_chain><comment><author>wyldfire</author><text>Seems like a context manager is what you really want, right?  some other exception could be thrown from some deeper frame while sock is in scope and this wouldn&amp;#x27;t ensure it gets closed.</text></comment>