Format HDFS Namenode Error Could not find or load main class -Djava library path home hadoop hadoop-3 2 1 lib native

Question

I am building a single node HDFS on Ubuntu 18.04 and am getting the following error when I try to format the HDFS name node using the command:

hdfs namenode -format

Error: Could not find or load main class ”-Djava.library.path=.home.hadoop.hadoop-3.2.1.lib.native”

I have the following configurations files:

.bashrc

export HADOOP_HOME=/home/hadoop/hadoop-3.2.1
export HADOOP_INSTALL=$HADOOP_HOME
export HADOOP_MAPRED_HOME=$HADOOP_HOME
export HADOOP_COMMON_HOME=$HADOOP_HOME
export HADOOP_HDFS_HOME=$HADOOP_HOME
export YARN_HOME=$HADOOP_HOME
export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native
export PATH=$PATH:$HADOOP_HOME/sbin:$HADOOP_HOME/bin
export HADOOP_OPTS=”-Djava.library.path=$HADOOP_HOME/lib/native”

hadoop-env.sh

export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64

core-site.xml

<configuration>
<property>
  <name>hadoop.tmp.dir</name>
  <value>/home/hadoop/tmpdata</value>
</property>
<property>
  <name>fs.default.name</name>
  <value>hdfs://127.0.0.1:9000</value>
</property>
</configuration>

hdfs-site.xml

<configuration>
<property>
  <name>dfs.data.dir</name>
  <value>/home/hadoop/dfsdata/namenode</value>
</property>
<property>
  <name>dfs.data.dir</name>
  <value>/home/hadoop/dfsdata/datanode</value>
</property>
<property>
  <name>dfs.replication</name>
  <value>1</value>
</property>
</configuration>

mapred-site.xml

<configuration> 
<property> 
  <name>mapreduce.framework.name</name> 
  <value>yarn</value> 
</property> 
</configuration>

yarn-site.xml

<configuration>
<property>
  <name>yarn.nodemanager.aux-services</name>
  <value>mapreduce_shuffle</value>
</property>
<property>
  <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
  <value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>
<property>
  <name>yarn.resourcemanager.hostname</name>
  <value>127.0.0.1</value>
</property>
<property>
  <name>yarn.acl.enable</name>
  <value>0</value>
</property>
<property>
  <name>yarn.nodemanager.env-whitelist</name>   
  <value>JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,CLASSPATH_PERPEND_DISTCACHE,HADOOP_YARN_HOME,HADOOP_MAPRED_HOME</value>
</property>
</configuration>

I have created the namenode and datanode folders referenced in hdfs-site.xml.  I have been chasing this for several hours without a solutions.  Thanks for suggested fixes.

Comments

when i type in this command it works fine

$ java -version
openjdk version "1.8.0_265"
OpenJDK Runtime Environment (build 1.8.0_265-8u265-b01-0ubuntu2~18.04-b01)
OpenJDK 64-Bit Server VM (build 25.265-b01, mixed mode)

but when i type hadoop version this error shows up again.

$hadoop version
Error: Could not find or load main class ”-Djava.library.path=.home.hdoop.hadoop-3.2.1.lib.”

---

#Hadoop Related Options
export HADOOP_HOME=/home/hdoop/hadoop-3.2.1
export HADOOP_INSTALL=$HADOOP_HOME
export HADOOP_MAPRED_HOME=$HADOOP_HOME
export HADOOP_COMMON_HOME=$HADOOP_HOME
export HADOOP_HDFS_HOME=$HADOOP_HOME
export YARN_HOME=$HADOOP_HOME
export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native
export PATH=$PATH:$HADOOP_HOME/sbin:$HADOOP_HOME/bin
export HADOOP_OPTS=”-Djava.library.path=$HADOOP_HOME/lib”
export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64

done ...still same error

~$ hdfs namenode -format
Error: Could not find or load main class ”-Djava.library.path=.home.hdoop.hadoop-3.2.1.lib”

---

@Bezz,

Are you using the command as hadoop version? 

Try as:

hadoop -version

It should work

---

Hi@bezz,

You didn't set the path for Java according to your .bashrc file. Check this env. variable. If not present then try to add first and then try.

---

Hi @akhtar,@GItIka and @MD thank you all for your help..It has finally worked. It started working when i changed the double quotation into single quotation in .bashrc file.,

export HADOOP_OPTS="-Djava.library.path=$HADOOP_HOME/lib"

changed to this

export HADOOP_OPTS='-Djava.library.path=$HADOOP_HOME/lib'

and it worked and am done with the installation opened it in browser.

Answer 1

Hi@fwood,

According to your configuration, you didn't set JAVA_HOME & PATH. Try to set these variables in .bashrc file. You can use the below-given command to find your JAVA_HOME & PATH.

$ which java

Also, change the value of HADOOP_OPTS as given below.

export HADOOP_OPTS="-Djava.library.path=$HADOOP_HOME/lib/"

In your hadoop-env.sh file export the value of HADOOP_CONF_DIR. I suggest you don't change the variable name. You change the variable name of HADOOP_CONF_DIR to HADOOP_INSTALL. Hadoop internally searches these variables and if it finds different values, it may give an error.

Comments

Thank You MD:  That did the trick, I have Hadoop up and running with yarn.  For those looking at this thread later Here are my updated files:

.bashrc

export HADOOP_HOME=/home/hadoop/hadoop-3.2.1
export HADOOP_INSTALL=$HADOOP_HOME
export HADOOP_MAPRED_HOME=$HADOOP_HOME
export HADOOP_COMMON_HOME=$HADOOP_HOME
export HADOOP_HDFS_HOME=$HADOOP_HOME
export YARN_HOME=$HADOOP_HOME
export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native
export PATH=$PATH:$HADOOP_HOME/sbin:$HADOOP_HOME/bin
export HADOOP_OPTS="-Djava.library.path=$HADOOP_HOME/lib/"
export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64

hadoop-env.sh

export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64

export HADOOP_CONF_DIR=${HADOOP_HOME}/etc/hadoop

Now that I have hadoop up and running I plan to install ZooKeeper and Accumulo.  The overall project goal is to build a basic docker container app and get it to talk to data in the Hadoop store through the Accumulo security structure.  Thanks Again for the help.

---

thank you so much! command worked like a charm,have being trying to install hadoop to windows,but switched later to ubuntu. Encountered the same kind of error and the above command took care of it. Saved me 3 hours of sleep =)