Glusterfs fuse client consuming high memory - memory leak

+4 votes

I have set up my glusterfs cluster as Striped-Replicated on GCP Servers, but I am facing memory leak issue with Glusterfs fuse client. Its memory consumption is increasing every day.

Both glusterfs server and glusterfs fuse client is using the latest version(Client-4.1.5, Server- 4.1), but the below process is consuming high memory on the client servers.

glusterfs --fopen-keep-cache=off --volfile-server=gluster1 --volfile-id=/+ 

Every day I can see that memory consumption of the above process is increasing, a temporary fix is to unmount and kill the process.

In my glusterfs client server, I am getting 'Stale File Handle error'. Not sure this error caused the memory leak.

[2018-10-21 04:08:46.921985] W [fuse-bridge.c:1201:fuse_setattr_cbk] 0-glusterfs-fuse: 3705309: SETATTR() /Production/example.com/wp-content/cache/wpsol-cache/4bd4f0bf132901ecb17261f388864fd3 => -1 (Stale file handle)

Also, my glusterfs server is using default settings. Please help if there is any patches or fix.

Oct 22, 2018 in Linux Administration by Edureka
• 200 points
2,669 views
what is the gluster version you are using?
For server 4.1.4, client(Fuse) is using 4.1.5. Both are having latest version.

try downgrading it to 

3.12.6
@priyaj, Why? is there any dependency problem with the latest versions?
Did you get this error before or after unmounting and killing the process?
This problem was initially found in 3.12 version itself but this was fixed. You can try using the 3.12 version.

try putting glusterfs inside cgroup with memory limit according to your resource availability using systemd and let me know if it works..

@priyaj,   I think the problem already exists with the old version as well. Is there any patches available ?
Are you sure that downgrading, also are you referring to glusterfs fuse version or server version?
@Jithin, yess try downgrading to either 3.12.8 or 3.12.6 as the issue was sorted in these versions, seems more promising.
Are u referring to server version or client version ?
@vardhan yes there are some unresolved bugs with the version 3.12.x and above
I'm referring the client version here @Jithin.
actually the problem was for high consumption of RAM but it was fixed in version 3.12.2 but again was taken care in 3.12.6
Thank you, let me downgrade it.
I think 3.12.6. is not available to download, can I use 3.12.15 ? I am using ubuntu client machines.
Try 3.12.8 if .6 is not possible.
@kalgi, It is also not available

https://launchpad.net/~gluster/+archive/ubuntu/glusterfs-3.12

Available versions are .15, .12, .5

3 answers to this question.

+1 vote
'Stale File Handle error' occurs when the directory is deleted. So basically the pointer your shell has to that particular inode does not exist. It's just a way system treats the invalid inode pointers. In your case, I think this error occurs because you're unmounting and killing the process.
answered Oct 22, 2018 by Kalgi
• 52,350 points
Okay, that is inode issue. We used unmounting to temporarily fix the load issue
0 votes

@Kalgi, you mean the backend error, yes am continuously getting this error in backend

answered Oct 22, 2018 by Edureka
• 200 points
0 votes
Heyy @Jithin,

Try taking statedumbs from different intervals and post your readings.
answered Oct 22, 2018 by Hannah
• 18,570 points

edited Oct 22, 2018 by Hannah

Related Questions In Linux Administration

0 votes
1 answer

How to increase Swap Memory in CentOS 7?

Follow the below steps and procedures: Prerequisites Must have ...READ MORE

answered Oct 5, 2018 in Linux Administration by Frankie
• 9,830 points
9,412 views
0 votes
1 answer

What is the linux command to find Memory and CPU usage in percent for last 30 days?

Try the following: Cat proc/meminfo top top -i less /proc/memin ...READ MORE

answered Oct 14, 2020 in Linux Administration by Kim
555 views
0 votes
0 answers

Linux "free -m": Total, used and free memory values don't add up [closed]

On a linux system, while using "free", ...READ MORE

Apr 26 in Linux Administration by Edureka
• 13,640 points
22 views
0 votes
1 answer

Is it possible to create a memory leak with Java

Here's a good way to create a memory ...READ MORE

answered May 31, 2018 in Java by Parth
• 4,630 points
1,658 views
0 votes
1 answer

Getting memory leak error on AWS EMR - Saving Dataframe into hive external table on S3 -

You can start by creating  a temp ...READ MORE

answered Aug 9, 2018 in AWS by Priyaj
• 58,080 points
820 views
0 votes
1 answer

How can I solve this: To prevent a memory leak, the JDBC Driver has been forcibly unregistered

What can you do? Ignore these warnings. Tomcat ...READ MORE

answered Jan 8, 2019 in Java by Daisy
• 8,120 points
24,467 views
0 votes
1 answer
0 votes
1 answer

How to create a memory leak in Java?

Hello @kartik, A simple thing to do is ...READ MORE

answered Jul 28, 2020 in Java by Niroj
• 82,720 points
202 views
0 votes
1 answer

How does the HDFS Client knows the block size while writing?

HDFS is designed in a way where ...READ MORE

answered Mar 27, 2018 in Big Data Hadoop by kurt_cobain
• 9,390 points
351 views
webinar REGISTER FOR FREE WEBINAR X
Send OTP
REGISTER NOW
webinar_success Thank you for registering Join Edureka Meetup community for 100+ Free Webinars each month JOIN MEETUP GROUP