Find Hadoop Configuration in Java Util

Question

I am planning to write a Java Util that should be accessing HDFS. I have a requirement for a configuration Object

Configuration conf = new Configuration()

I had used the above code to create one. This method was unsuccessful and threw out the following info.

fs.getHomeDirectory() 

I have even tried to make some changes in 

• Core-site.xml
• mapred-site.xml
• yarn-site.xml
• hfds-site.xml

None of the modifications helped out.

please help me resovle this issue.

Answer 1

I understand your issue of HDFS, I feel your Core-Site.xml and HDFS-site.xml are not correctly assigned.

This following code must help you.

Configuration conf = new Configuration();
conf.addResource(new Path("file:///etc/hadoop/conf/core-site.xml")); // Replace with actual path
conf.addResource(new Path("file:///etc/hadoop/conf/hdfs-site.xml")); // Replace with actual path

Path pt = new Path("."); // HDFS Path
FileSystem fs = pt.getFileSystem(conf);

System.out.println("Home directory :"+fs.getHomeDirectory());

This should help, Even then if you face the same trouble, Then try adding the configuration files using addResource method. Use set method instead of add method in the value fs.defaultFS in your core-site.xml

conf.set("fs.defaultFS","hdfs://<Namenode-Host>:<Port>");  // Refer you core-site.xml file and replace <Namenode-Host> and <Port> with your cluster namenode and Port (default port number should be `8020`).