What is the usage of Configured class in Hadoop programs

Question

Most of Hadoop MapReduce programs are like this:

public class MyApp extends Configured Implements Tool {
    @Override
    public int run(String[] args) throws Exception {
        Job job = new Job(getConf());
        /* process command line options */
        return job.waitForCompletion(true) ? 0 : 1;
    }
    public static void main(String[] args) throws Exception {
        int exitCode = ToolRunner.run(new MyApp(), args);
        System.exit(exitCode);
    }
}

What is the usage of Configured? As Tool and Configured both have getConf() and setConf() in common.

Answer 1

Configured is a default implementation of the Configurable interface - basically its setConfmethod retains a private instance variable to the passed Configuration object and getConf()returns that reference

Tool is an extension of the Configurable interface, providing an addition run(..) method and is used with ToolRunner to parse out command line options (using the GenericOptionsParser) and build a Configuration object which is then passed to the setConf(..) method.

Your main class will typically extend Configured such that the Configurable interface methods required in Tool will be implemented for you.

In general you should be using the ToolRunner utility class to launch your MapReduce jobs as it handles the common task of parsing out command line arguments and building the Configuration object. I'd look at the API Docs for ToolRunner for more info.