How to call MapReduce program using a simple Java Program?

0 votes

I have been trying to call a mapreduce job from a simple java program in the same package.. I tried to refer the mapreduce jar file in my java program and call it using the runJar(String args[]) method by also passing the input and output paths for the mapreduce job.. But the program dint work..


How do I run such a program where I just use pass input, output and jar path to its main method?? Is it possible to run a mapreduce job (jar) through it?? I want to do this because I want to run several mapreduce jobs one after another where my java program vl call each such job by referring its jar file.. If this gets possible, I might as well just use a simple servlet to do such calling and refer its output files for the graph purpose..


/*
 * To change this template, choose Tools | Templates
 * and open the template in the editor.
 */

/**
 *
 * @author root
 */
import org.apache.hadoop.util.RunJar;
import java.util.*;

public class callOther {

    public static void main(String args[])throws Throwable
    {

        ArrayList arg=new ArrayList();

        String output="/root/Desktp/output";

        arg.add("/root/NetBeansProjects/wordTool/dist/wordTool.jar");

        arg.add("/root/Desktop/input");
        arg.add(output);

        RunJar.main((String[])arg.toArray(new String[0]));

    }
}
Sep 4 in Big Data Hadoop by nitinrawat895
• 10,670 points
13 views

1 answer to this question.

0 votes

Because map and reduce run on different machines, all your referenced classes and jars must move from machine to machine.

If you have package jar, and run on your desktop, @ThomasJungblut's answer is OK. But if you run in Eclipse, right-click your class and run, it doesn't work.

Instead of:

job.setJarByClass(Mapper.class);

Use:

job.setJar("build/libs/hdfs-javac-1.0.jar");

At the same time, your jar's manifest must include Main-Class property, which is your main class.

For Gradle users, can put these lines in build.gradle:

jar {
manifest {
    attributes("Main-Class": mainClassName)
}}
answered Sep 4 by ravikiran
• 4,560 points

Related Questions In Big Data Hadoop

0 votes
1 answer

How to write a file in HDFS using Java Programming language?

Define the HADOOP_CONF_DIR environment variable to your Hadoop configuration ...READ MORE

answered Sep 28, 2018 in Big Data Hadoop by Frankie
• 9,810 points
167 views
0 votes
1 answer

How to delete a file from Hadoop hdfs using java?

Try this: FileSystem fs = FileSystem.get(getConf()); fs.delete(new ...READ MORE

answered Nov 28, 2018 in Big Data Hadoop by Omkar
• 67,520 points
466 views
0 votes
1 answer

How to run Map Reduce program using Ubuntu terminal?

 I used the following steps to execute it ...READ MORE

answered Aug 7, 2018 in Big Data Hadoop by Neha
• 6,280 points
98 views
0 votes
1 answer

How to write a file in hdfs with Java?

You could pass the URI when getting ...READ MORE

answered Sep 26, 2018 in Big Data Hadoop by digger
• 26,550 points
123 views
0 votes
1 answer

Moving files in Hadoop using the Java API?

I would recommend you to use FileSystem.rename(). ...READ MORE

answered Apr 15, 2018 in Big Data Hadoop by Shubham
• 13,290 points
804 views
0 votes
1 answer

Hadoop giving java.io.IOException, in mkdir Java code.

I am not sure about the issue. ...READ MORE

answered May 3, 2018 in Big Data Hadoop by Shubham
• 13,290 points
404 views
0 votes
1 answer

Hadoop Mapreduce word count Program

Firstly you need to understand the concept ...READ MORE

answered Mar 16, 2018 in Data Analytics by nitinrawat895
• 10,670 points
2,754 views
0 votes
1 answer

hadoop.mapred vs hadoop.mapreduce?

org.apache.hadoop.mapred is the Old API  org.apache.hadoop.mapreduce is the ...READ MORE

answered Mar 16, 2018 in Data Analytics by nitinrawat895
• 10,670 points
289 views
0 votes
1 answer

How to Sqoop in a Java Program?

You can use the following sample code for ...READ MORE

answered Jul 22 in Big Data Hadoop by ravikiran
• 4,560 points
70 views
0 votes
2 answers

How to use scoop in a Java Program

There is a trick which worked out ...READ MORE

answered Sep 4 in Big Data Hadoop by ravikiran
• 4,560 points
37 views