Kubernetes can t start due to too many open files in system

Question

I am trying create a bunch of pods, services and deployment using Kubernetes, but keep getting these errors when I run the kubectl describe command.

for "POD" with RunContainerError: "runContainer: API error (500): Cannot start container bbdb58770a848733bf7130b1b230d809fcec3062b2b16748c5e4a8b12cc0533a: [8] System error: too many open files in system\n"

I am not an Linux expert, so I am just wondering how do I find all the open files and close them?

Answer 1

You can confirm which process is hogging file descriptors by running:

lsof | awk '{print $2}' | sort | uniq -c | sort -n

That will give you a sorted list of open FD counts with the pid of the process. Then you can look up each process

ps -p <pid>

Answer 2

You could probably try more powerful machines with higher CPU and larger memory. Let me know if it works. Thanks

Answer 3

Seems like a kernel config issue. If there's nothing important running on the cluster delete the old one and bring up a new cluster.