How can I append data to an existing file in HDFS

0 votes

I am facing issues in appending data to an existing file in HDFS. I want to append data to an existing file, but instead, it is creating a new file.

I have embedded my code in this query, kindly go through that.

if (!file.exists(path)){
   file.createNewFile(path);
}
FSDataOutputStream fileOutputStream = file.append(path); 
BufferedWriter br = new BufferedWriter(new OutputStreamWriter(fileOutputStream));
br.append("Content: " + content + "\n");
br.close();

The code I used to run it is as follows.

RunTimeCalculationHdfsWrite.hdfsWriteFile("RunTimeParserLoaderMapperTest2", "Error message test 2.2", context, null);

1st parameter is the file name, and the second parameter is the message and you can ignore the others.

Jul 25, 2019 in Big Data Hadoop by nitinrawat895
• 11,380 points
7,998 views

1 answer to this question.

0 votes

You have to do some configurations as shown below :

Set dfs.support.append as true in hdfs-site.xml :

<property>
   <name>dfs.support.append</name>
   <value>true</value>
</property>

Stop all your daemon services using stop-all.sh and restart it again using start-all.sh

If you have a single-node cluster, you have to set the replication factor to 1.

You can use the following command line.

./hdfs dfs -setrep -R 1 filepath/directory

Or you can do the same at run time through java code:

fShell.setrepr((short) 1, filePath);  

Follow this code to append data into your File

public void createAppendHDFS() throws IOException {
    Configuration hadoopConfig = new Configuration();
    hadoopConfig.set("fs.defaultFS", hdfsuri);
    FileSystem fileSystem = FileSystem.get(hadoopConfig);
    String filePath = "/test/doc.txt";
    Path hdfsPath = new Path(filePath);
    fShell.setrepr((short) 1, filePath); 
    FSDataOutputStream fileOutputStream = null;
    try {
        if (fileSystem.exists(hdfsPath)) {
            fileOutputStream = fileSystem.append(hdfsPath);
            fileOutputStream.writeBytes("appending into file. \n");
        } else {
            fileOutputStream = fileSystem.create(hdfsPath);
            fileOutputStream.writeBytes("creating and writing into file\n");
        }
    } finally {
        if (fileSystem != null) {
            fileSystem.close();
        }
        if (fileOutputStream != null) {
            fileOutputStream.close();
        }
    }
}

I hope this helps.

answered Jul 25, 2019 by ravikiran
• 4,620 points

Related Questions In Big Data Hadoop

0 votes
1 answer

How can I put file to HDFS directly without copying it local disk?

Can use pipe from wget to hdfs. You ...READ MORE

answered Apr 15, 2018 in Big Data Hadoop by kurt_cobain
• 9,390 points
4,217 views
0 votes
1 answer

How can I use my host machine’s web browser to check my HDFS services running in the VM?

The sole purpose of the virtual machine ...READ MORE

answered Apr 18, 2018 in Big Data Hadoop by Shubham
• 13,490 points
1,068 views
0 votes
1 answer

How can I change data in HDFS?

You can not modified data once stored ...READ MORE

answered Feb 19, 2019 in Big Data Hadoop by Rishi
4,023 views
0 votes
1 answer

How can I get the respective Bitcoin value for an input in USD when using c#

Simply make call to server and parse ...READ MORE

answered Mar 25, 2018 in Big Data Hadoop by charlie_brown
• 7,720 points
752 views
+1 vote
1 answer

Hadoop Mapreduce word count Program

Firstly you need to understand the concept ...READ MORE

answered Mar 16, 2018 in Data Analytics by nitinrawat895
• 11,380 points
10,555 views
0 votes
1 answer

hadoop.mapred vs hadoop.mapreduce?

org.apache.hadoop.mapred is the Old API  org.apache.hadoop.mapreduce is the ...READ MORE

answered Mar 16, 2018 in Data Analytics by nitinrawat895
• 11,380 points
2,184 views
+2 votes
11 answers

hadoop fs -put command?

Hi, You can create one directory in HDFS ...READ MORE

answered Mar 16, 2018 in Big Data Hadoop by nitinrawat895
• 11,380 points
104,201 views
–1 vote
1 answer

Hadoop dfs -ls command?

In your case there is no difference ...READ MORE

answered Mar 16, 2018 in Big Data Hadoop by kurt_cobain
• 9,390 points
4,260 views
0 votes
1 answer

I need to copy data from one HDFS to another HDFS. Can you help me do so?

I understood your issue. Let me help you ...READ MORE

answered May 16, 2019 in Big Data Hadoop by ravikiran
• 4,620 points
2,146 views
0 votes
1 answer

How does Hadoop process data which is split across multiple boundaries in an HDFS?

I found some comments: from the Hadoop ...READ MORE

answered Jul 1, 2019 in Big Data Hadoop by ravikiran
• 4,620 points
723 views
webinar REGISTER FOR FREE WEBINAR X
REGISTER NOW
webinar_success Thank you for registering Join Edureka Meetup community for 100+ Free Webinars each month JOIN MEETUP GROUP