How to read HDFS and local files with the same code in Java

+1 vote

I want to use my Java code to read the hdfs and local files. Now I use separate functions for it because hdfs comes with hdfs:// prefix. Is there a way I can use the same code to read both files?​

Nov 22, 2018 in Big Data Hadoop by slayer
• 29,300 points
2,790 views

1 answer to this question.

+2 votes

You can try something like this:

    Configuration conf = new Configuration();
    conf.addResource(new Path("/hadoop/projects/hadoop-1.0.4/conf/core-site.xml"));
    conf.addResource(new Path("/hadoop/projects/hadoop-1.0.4/conf/hdfs-site.xml"));
    file_path = //assign path to your file
    Path path = new Path(file_path);
    FileSystem fs = path.getFileSystem(conf);
    FSDataInputStream inputStream = fs.open(path);
    System.out.println(inputStream.available());
    fs.close();
answered Nov 22, 2018 by Omkar
• 69,110 points
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.apache.hadoop.security.authentication.util.KerberosUtil (file:/home/wisal/hadoop-2.7.3/share/hadoop/common/lib/hadoop-auth-2.7.3.jar) to method sun.security.krb5.Config.getInstance()
WARNING: Please consider reporting this to the maintainers of org.apache.hadoop.security.authentication.util.KerberosUtil
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
2019-12-05 12:47:08,795 [AWT-EventQueue-0] WARN  org.apache.hadoop.util.NativeCodeLoader - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
7

Related Questions In Big Data Hadoop

0 votes
1 answer

How to display the Access Control Lists (ACLs) of files and directories in HDFS?

Hi@akhtar, You can see the Access Control Lists (ACLs) ...READ MORE

answered Oct 19, 2020 in Big Data Hadoop by MD
• 95,140 points
107 views
0 votes
1 answer

How to write a file in hdfs with Java?

You could pass the URI when getting ...READ MORE

answered Sep 26, 2018 in Big Data Hadoop by digger
• 26,700 points
2,367 views
0 votes
1 answer

How to create new directory in hdfs using java code?

Try this: { Configuration config ...READ MORE

answered Dec 6, 2018 in Big Data Hadoop by Omkar
• 69,110 points
1,632 views
0 votes
1 answer

How to copy files in hdfs with wildcards?

You can use the hdfs command: hdfs fs ...READ MORE

answered Dec 11, 2018 in Big Data Hadoop by Omkar
• 69,110 points
1,019 views
0 votes
1 answer

Moving files in Hadoop using the Java API?

I would recommend you to use FileSystem.rename(). ...READ MORE

answered Apr 15, 2018 in Big Data Hadoop by Shubham
• 13,480 points
1,683 views
0 votes
1 answer

Hadoop giving java.io.IOException, in mkdir Java code.

I am not sure about the issue. ...READ MORE

answered May 3, 2018 in Big Data Hadoop by Shubham
• 13,480 points
1,234 views
+1 vote
1 answer

Hadoop Mapreduce word count Program

Firstly you need to understand the concept ...READ MORE

answered Mar 16, 2018 in Data Analytics by nitinrawat895
• 11,380 points
7,087 views
0 votes
1 answer

hadoop.mapred vs hadoop.mapreduce?

org.apache.hadoop.mapred is the Old API  org.apache.hadoop.mapreduce is the ...READ MORE

answered Mar 16, 2018 in Data Analytics by nitinrawat895
• 11,380 points
1,139 views
0 votes
1 answer

Hadoop: How to copy directory from local system to hdfs using Java code?

Just use the FileSystem's copyFromLocalFile method. If the source Path ...READ MORE

answered Nov 14, 2018 in Big Data Hadoop by Omkar
• 69,110 points
2,885 views
0 votes
1 answer

How to concatenate hdfs files and store in output file?

You can use a combination of cat and put command. Something ...READ MORE

answered Dec 5, 2018 in Big Data Hadoop by Omkar
• 69,110 points
2,024 views