How to run a jar file in hadoop

Question

I have created a jar file using the java file
javac -classpath /usr/local/hadoop/hadoop-core-1.0.3.jar -d /home/user1/dir Dictionary.java
/usr/lib/jvm/jdk1.7.0_07/bin/jar cf Dictionary.jar /home/hduser/dir
Now i have tried running this jar with various commands
user1@ubuntu:~$ /usr/local/hadoop/bin/hadoop jar Dictionary.jar
Output:
Warning: $HADOOP_HOME is deprecated.

RunJar jarFile [mainClass] args...  
user1@ubuntu:~$ /usr/local/hadoop/bin/hadoop jar Dictionary.jar Dictionary
Output:
Warning: $HADOOP_HOME is deprecated.

Exception in thread "main" java.lang.ClassNotFoundException: Dictionary
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:423)
at java.lang.ClassLoader.loadClass(ClassLoader.java:356)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:264)
at org.apache.hadoop.util.RunJar.main(RunJar.java:149)
How can i run the jar in hadoop?

Answer 1

Basically, the directory that you are packaging into the jar is confusing the jar file in locating the main class file.
Instead if trying do the following :
/usr/lib/jvm/jdk1.7.0_07/bin/jar cf Dictionary.jar /home/user1/dir/Dictionary.class
i.e. package the class file specifically into the jar and then run:
/usr/local/hadoop/bin/hadoop jar Dictionary.jar Dictionary
It just works fine provided that you have a main function in your class called Dictionary.
The problem is when you package a full directory inside a jar then the jar also needs to be aware of the directory structure to locate the class file. For this, we need to have a well defined package hierarchy to define the class location. So, when you are packaging /home/hduser/dir/ into the jar, the jar is not aware of the location of the class file which is located deep inside this directory structure. For this you need to add a package name to your .java file according to the directory structure , for example home.hduser.dir and while running the hadoop jar command specify the class name with the package structure, for example home.user1.dir.Dictionary. Learn more about Big Data Architect and its concepts from the Big data architect course.

Answer 2

You can try running jar from terminal like this:

hadoop jar <jarFileName> <mainClassname> <AnyCommandLineArguements>

Answer 3

While creating the jar file, mention the output as .class file instead of .java file.

Answer 4

As Vilas mentioned, jar file should be created as class. Here is what you can do:

javac -classpath /usr/local/hadoop/hadoop-core-1.0.3.jar -d /home/user1/dir Dictionary.class

Answer 5

You can use the  RunJar() function to run your jar file:

RunJar(Dictionary.jar)

Answer 6

Here's a syntax from the docs:

$ hadoop jar $HADOOP_HOME/hadoop-streaming.jar \
    -input myInputDirs \
    -output myOutputDir \
    -mapper org.apache.hadoop.mapred.lib.IdentityMapper \
    -reducer /bin/wc

Answer 7

I used this command to run my wordcount jar. This could be helpful to you:

hadoop jar /opt/mapr/hadoop/hadoop-0.20.2/hadoop-0.20.2-dev-examples.jar wordcount /myvolume/in /myvolume/out