GCP calls OOMKiller for java app however it doesn t consume max allowed memory

Question

Input: GCP, Kubernetes, java 11 spring boot 2 application

Container is started with memory limit 1.6GB. Java application is limiting memory as well -XX:MaxRAMPercentage=80.0. Under a "heavy" (not really) load - about 1 http request per 100 ms during about 4 hours application is killed by OOMKiller. Internal diagnostic tools is showing that memory is far from limit: POD contains only java app (+jaeger agent). The odd thing that after restart GCP shows almost maximum memory usage instead of slowly growing if it was memory leak.

Answer 1

As per the log file there are more than 10,000 started threads. This is a lot considering that there are 2CPU/core for each container. Each thread and its stack is allocated in memory separate from the heap. It is quite possible that the large number of started threads is the cause for the OOM problem.

The JVM is started with Native Memory Tracking related options

(-XX:NativeMemoryTracking=detail, -XX:+UnlockDiagnosticVMOptions, -XX:+PrintNMTStatistics)

that could help to see the memory usage include what's consumed by those threads. I would highly recommend to not have that many threads.