Glusterfs fuse client consuming high memory - memory leak

Question

I have set up my glusterfs cluster as Striped-Replicated on GCP Servers, but I am facing memory leak issue with Glusterfs fuse client. Its memory consumption is increasing every day.

Both glusterfs server and glusterfs fuse client is using the latest version(Client-4.1.5, Server- 4.1), but the below process is consuming high memory on the client servers.

glusterfs --fopen-keep-cache=off --volfile-server=gluster1 --volfile-id=/+ 

Every day I can see that memory consumption of the above process is increasing, a temporary fix is to unmount and kill the process.

In my glusterfs client server, I am getting 'Stale File Handle error'. Not sure this error caused the memory leak.

[2018-10-21 04:08:46.921985] W [fuse-bridge.c:1201:fuse_setattr_cbk] 0-glusterfs-fuse: 3705309: SETATTR() /Production/example.com/wp-content/cache/wpsol-cache/4bd4f0bf132901ecb17261f388864fd3 => -1 (Stale file handle)

Also, my glusterfs server is using default settings. Please help if there is any patches or fix.

Comments

Thank you, let me downgrade it.

---

I think 3.12.6. is not available to download, can I use 3.12.15 ? I am using ubuntu client machines.

---

Try 3.12.8 if .6 is not possible.

---

@kalgi, it is also not available
https://launchpad.net/~gluster/+archive/ubuntu/glusterfs-3.12

---

@kalgi, It is also not available

https://launchpad.net/~gluster/+archive/ubuntu/glusterfs-3.12

Available versions are .15, .12, .5

Answer 1

'Stale File Handle error' occurs when the directory is deleted. So basically the pointer your shell has to that particular inode does not exist. It's just a way system treats the invalid inode pointers. In your case, I think this error occurs because you're unmounting and killing the process.

Comments

Okay, that is inode issue. We used unmounting to temporarily fix the load issue

Answer 2

@Kalgi, you mean the backend error, yes am continuously getting this error in backend

Answer 3

Heyy @Jithin,

Try taking statedumbs from different intervals and post your readings.