How to write a file in HDFS using Java Programming language

Question

I want to create a file in HDFS and write data in that. I used this code:

Configuration config = new Configuration();     
FileSystem fs = FileSystem.get(config); 
Path filenamePath = new Path("input.txt");  
try {
    if (fs.exists(filenamePath)) {
        fs.delete(filenamePath, true);
    }

    FSDataOutputStream fin = fs.create(filenamePath);
    fin.writeUTF("hello");
    fin.close();
}

It creates the file, but it doesn't write anything in it. I searched a lot but didn't find anything. What is my problem? Do I need any permission to write in HDFS?

Answer 1

Define the HADOOP_CONF_DIR environment variable to your Hadoop configuration folder or add the following 2 lines in your code :

config.addResource(new Path("/HADOOP_HOME/conf/core-site.xml"));
config.addResource(new Path("/HADOOP_HOME/conf/hdfs-site.xml"));

If you don't add this, your client will try to write to the local FS, hence resulting into the permission denied exception.

You can also try this approach:

FileSystem fs = path.getFileSystem(conf);
SequenceFile.Writer inputWriter = new SequenceFile.Writer(fs, conf, path, LongWritable.class, MyWritable.class);
inputWriter.append(new LongWritable(uniqueId++), new MyWritable(data));
inputWriter.close();

Comments

WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.apache.hadoop.security.authentication.util.KerberosUtil (file:/home/wisal/hadoop-2.7.3/share/hadoop/common/lib/hadoop-auth-2.7.3.jar) to method sun.security.krb5.Config.getInstance()
WARNING: Please consider reporting this to the maintainers of org.apache.hadoop.security.authentication.util.KerberosUtil
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
2019-12-05 03:36:17,623 [AWT-EventQueue-0] WARN  org.apache.hadoop.util.NativeCodeLoader - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable