Q:

JVM Buffer pool only grows

I'm ElasticSearch in production and using prometheus to scrape the metrics. Looking at the graphs I could see the jvm_buffer_pool metric just grow until finally crashed
As I understood the buffer pool is outside of GC, but how to clean it up?

A:

The JVM has Direct ByteBuffers which are on heap objects which proxy off-heap memory. The ByteBuffer is tens of bytes even if the off-heap memory is 1 GB.  When the GC cleans up this proxy object because it is no longer referenced, the off-heap memory is also released.
If the off-heap memory isn't being released, it because;

it's on heap proxies are being retained. i.e. the memory is needed.
ElasticSearch is allocating off-heap memory directly and the library has a leak (unlikely)

I would try allowing more direct memory to see if this helps. -XX:MaxDirectMemorySize=64g or whatever you can spare.

