How to run a jar file in hadoop

Question

I have created a jar file using the java file
javac -classpath /usr/local/hadoop/hadoop-core-1.0.3.jar -d /home/user1/dir Dictionary.java
/usr/lib/jvm/jdk1.7.0_07/bin/jar cf Dictionary.jar /home/hduser/dir
Now i have tried running this jar with various commands
user1@ubuntu:~$ /usr/local/hadoop/bin/hadoop jar Dictionary.jar
Output:
Warning: $HADOOP_HOME is deprecated.

RunJar jarFile [mainClass] args...  
user1@ubuntu:~$ /usr/local/hadoop/bin/hadoop jar Dictionary.jar Dictionary
Output:
Warning: $HADOOP_HOME is deprecated.

Exception in thread "main" java.lang.ClassNotFoundException: Dictionary
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:423)
at java.lang.ClassLoader.loadClass(ClassLoader.java:356)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:264)
at org.apache.hadoop.util.RunJar.main(RunJar.java:149)
How can i run the jar in hadoop?

Answer 1

Basically, the directory that you are packaging into the jar is confusing the jar file in locating the main class file.
Instead if trying do the following :
/usr/lib/jvm/jdk1.7.0_07/bin/jar cf Dictionary.jar /home/user1/dir/Dictionary.class
i.e. package the class file specifically into the jar and then run:
/usr/local/hadoop/bin/hadoop jar Dictionary.jar Dictionary
It just works fine provided that you have a main function in your class called Dictionary.
The problem is when you package a full directory inside a jar then the jar also needs to be aware of the directory structure to locate the class file. For this, we need to have a well defined package hierarchy to define the class location. So, when you are packaging /home/hduser/dir/ into the jar, the jar is not aware of the location of the class file which is located deep inside this directory structure. For this you need to add a package name to your .java file according to the directory structure , for example home.hduser.dir and while running the hadoop jar command specify the class name with the package structure, for example home.user1.dir.Dictionary.

Answer 2

You can try running jar from terminal like this:

hadoop jar <jarFileName> <mainClassname> <AnyCommandLineArguements>

Answer 3

While creating the jar file, mention the output as .class file instead of .java file.

Answer 4

As Vilas mentioned, jar file should be created as class. Here is what you can do:

javac -classpath /usr/local/hadoop/hadoop-core-1.0.3.jar -d /home/user1/dir Dictionary.class

Answer 5

You can use the  RunJar() function to run your jar file:

RunJar(Dictionary.jar)

Answer 6

Here's a syntax from the docs:

$ hadoop jar $HADOOP_HOME/hadoop-streaming.jar \
    -input myInputDirs \
    -output myOutputDir \
    -mapper org.apache.hadoop.mapred.lib.IdentityMapper \
    -reducer /bin/wc

Source: https://mapr.com/docs/52/ReferenceGuide/hadoop-jar.html

Answer 7

I used this command to run my wordcount jar. This could be helpful to you:

hadoop jar /opt/mapr/hadoop/hadoop-0.20.2/hadoop-0.20.2-dev-examples.jar wordcount /myvolume/in /myvolume/out