Cadvisor container root partition monitoring when using overlay2 with size limits

0 votes

I am trying to monitoring root partitions inside containers using gcr.io/cadvisor/cadvisor:v0.37.0 and Prometheus.

Docker is set up with:

"storage-driver": "overlay2",
"storage-opts": ["overlay2.size=5G"],

For example, for the following container, I would like to get the 5% (actually 4.06% when computed exactly)

[root@infra-05 docker-compose]# docker exec -ti infra-05_logging-elasticsearch df
Filesystem 1K-blocks Used Available Use% Mounted on
overlay 5242880 213172 5029708 5% /
tmpfs 65536 0 65536 0% /dev
tmpfs 8023452 0 8023452 0% /sys/fs/cgroup
/dev/mapper/vg0-varlibdockerlv 41932800 30960284 10972516 74% /etc/hosts
shm 65536 0 65536 0% /dev/shm
/dev/mapper/vg0-vardockerelasticsearchlv 146785280 24772532 122012748 17% /usr/share/elasticsearch/data
/dev/mapper/vg0-lv_root 3997376 3023344 747936 81% /usr/share/elasticsearch/config/log4j2.properties
tmpfs 8023452 0 8023452 0% /proc/acpi
tmpfs 8023452 0 8023452 0% /proc/scsi
tmpfs 8023452 0 8023452 0% /sys/firmware

Following query in Prometheus:

container_fs_usage_bytes{instance="infra-05.node.consul:9103",name=".elasticsearch."}/container_fs_limit_bytes{instance="infra-05.node.consul:9103",name=".elasticsearch."}*100

brings back a single line:

{container_label_co_elastic_logs_module="elasticsearch",container_label_com_docker_compose_config_hash="ad29cbce151d36e15c4b13d72b1818bc0808e1936890e51e46e9c6e0816bb6a7",container_label_com_docker_compose_container_number="1",container_label_com_docker_compose_oneoff="False",container_label_com_docker_compose_project="docker-compose",container_label_com_docker_compose_service="logging-elasticsearch",container_label_com_docker_compose_version="1.21.2",container_label_org_label_schema_build_date="2019-11-26T01:06:52.520070Z",container_label_org_label_schema_license="Elastic-License",container_label_org_label_schema_name="Elasticsearch",container_label_org_label_schema_schema_version="1.0",container_label_org_label_schema_url="https://www.elastic.co/products/elasticsearch",container_label_org_label_schema_usage="https://www.elastic.co/guide/en/elasticsearch/reference/index.html",container_label_org_label_schema_vcs_ref="e9ccaed468e2fac2275a3761849cbee64b39519f",container_label_org_label_schema_vcs_url="https://github.com/elastic/elasticsearch",container_label_org_label_schema_vendor="Elastic",container_label_org_label_schema_version="7.5.0",container_label_org_opencontainers_image_created="2019-11-26T01:06:52.520070Z",container_label_org_opencontainers_image_documentation="https://www.elastic.co/guide/en/elasticsearch/reference/index.html",container_label_org_opencontainers_image_licenses="Elastic-License",container_label_org_opencontainers_image_revision="e9ccaed468e2fac2275a3761849cbee64b39519f",container_label_org_opencontainers_image_source="https://github.com/elastic/elasticsearch",container_label_org_opencontainers_image_title="Elasticsearch",container_label_org_opencontainers_image_url="https://www.elastic.co/products/elasticsearch",container_label_org_opencontainers_image_vendor="Elastic",container_label_org_opencontainers_image_version="7.5.0",customer="Xenit",device="/dev/mapper/vg0-varlibdockerlv",environment="Xenit_PROD",id="/docker/677f57566219f459f30d6c4abac0235b3ab43c0f8031747bf676dfa9b33d1047",image="elasticsearch:7.5.0",instance="infra-05.node.consul:9103",job="cadvisor",name="infra-05_logging-elasticsearch",service="cadvisor"} | 0.5084897741147741

I noticed that the root partition in that container is available if I query for device=~overlay.*:

container_fs_usage_bytes{instance="infra-05.node.consul:9103",device="overlay.*"}/container_fs_limit_bytes{instance="infra-05.node.consul:9103",device="overlay.*"}*100

{customer="Xenit",device="overlay_0-109",environment="Xenit_PROD",id="/",instance="infra-05.node.consul:9103",job="cadvisor",service="cadvisor"} | 4.0659332275390625
......................

But how to link the device "overlay_0-109" with the container infra-05_logging-elasticsearch?

Sep 9 in Docker by anghelutar
• 120 points
59 views

Hi@anghelutar,

You are trying to run elasticsearch with the Prometheus server in a docker container. After that can you explain your query a little bit more?

No answer to this question. Be the first to respond.

Your answer

Your name to display (optional):
Privacy: Your email address will only be used for sending these notifications.

Related Questions In Docker

+2 votes
2 answers
0 votes
1 answer

Using SSH keys inside docker container

Turns out when using Ubuntu, the ssh_config ...READ MORE

answered Jul 18, 2018 in Docker by Nilesh
• 6,980 points
7,147 views
0 votes
1 answer

I lose my data when I get out of the container

No its not installation issue. You need to commit the ...READ MORE

answered Jul 26, 2018 in Docker by Kalgi
• 51,970 points
272 views
+1 vote
1 answer

Docker Container Networking with Docker-in-Docker

Given your example docker-in-docker setup above, you ...READ MORE

answered Jul 30, 2018 in Docker by Kalgi
• 51,970 points
1,246 views
0 votes
1 answer

Using GPU from a docker container?

Preparation Install nvidia driver and cuda on your ...READ MORE

answered Jul 31, 2018 in Docker by Kalgi
• 51,970 points
555 views
+2 votes
1 answer
+2 votes
1 answer

Deploy Docker Containers from Docker Cloud

To solve this problem, I followed advice ...READ MORE

answered Sep 3, 2018 in AWS by Priyaj
• 57,680 points
838 views