Hadoop java io IOException File could only be replicated to 0 nodes instead of minReplication 1

Question

While running a job i am getting the following given below error ,

java.io.IOException: File /user/ashsshar/olhcache/loaderMap9b663bd9 could only be replicated to 0 nodes instead of minReplication (=1). There are 3 datanode(s) running and 3 node(s) are excluded in this operation. at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget(BlockManager.java:1325)

I tried restarting the DataNodes but I still get the same error. 

PS: There is a lot of space available

Answer 1

Try this, first stop all the daemons, remove all the file from /var/lib/hadoop-hdfs/cache/hdfs/dfs/name, format the Namenode and start the daemons:

for x in `cd /etc/init.d ; ls hadoop*` ; do sudo service $x stop ; done
sudo rm -r /var/lib/hadoop-hdfs/cache/
sudo -u hdfs hdfs namenode -format
for x in `cd /etc/init.d ; ls hadoop*` ; do sudo service $x start ; done