error org apache flume sink hdfs HDFSEventSink process HDFSEventSink java 456 HDFS IO error java io IOException

0 votes

I'm trying to run flume with an hdfs sink. But when I run flume and send events to it I got the following error.

[WARN - org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:456)] HDFS IO error
java.io.IOException: Callable timed out after 5000 ms
    at org.apache.flume.sink.hdfs.HDFSEventSink.callWithTimeout(HDFSEventSink.java:352)
    at org.apache.flume.sink.hdfs.HDFSEventSink.append(HDFSEventSink.java:727)
    at org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:430)
    at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
    at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
    at java.lang.Thread.run(Thread.java:679)
Caused by: java.util.concurrent.TimeoutException
    at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:258)

How can I solve this error?
 

Apr 23, 2020 in Big Data Hadoop by akhtar
• 38,230 points
1,468 views

1 answer to this question.

0 votes

Hi@akhtar,

There may be problem in firewall, so make sure iptables is off.

$ iptables -F

If it will not work, then check the following properties in flume-conf.properties file.

agent.sinks.hdfs-sink.hdfs.path = hdfs://ip:9000/user/hdfs/flume
agent.sinks.hdfs-sink.hdfs.callTimeout = 180000

Set hdfs path and time accordingly your set up.

answered Apr 23, 2020 by MD
• 95,440 points

Related Questions In Big Data Hadoop

0 votes
1 answer

Hadoop Java: org.apache.hadoop.conf.Configuration does not exist error

To solve this error, add hadoop-common to ...READ MORE

answered Nov 21, 2018 in Big Data Hadoop by Omkar
• 69,210 points
6,927 views
0 votes
1 answer

Hadoop: namenode -format error "java.io.IOException"

Run the command as sudo or add the ...READ MORE

answered Nov 26, 2018 in Big Data Hadoop by Omkar
• 69,210 points
3,039 views
+1 vote
1 answer

Hadoop Mapreduce word count Program

Firstly you need to understand the concept ...READ MORE

answered Mar 16, 2018 in Data Analytics by nitinrawat895
• 11,380 points
10,523 views
0 votes
1 answer

hadoop.mapred vs hadoop.mapreduce?

org.apache.hadoop.mapred is the Old API  org.apache.hadoop.mapreduce is the ...READ MORE

answered Mar 16, 2018 in Data Analytics by nitinrawat895
• 11,380 points
2,166 views
+2 votes
11 answers

hadoop fs -put command?

Hi, You can create one directory in HDFS ...READ MORE

answered Mar 16, 2018 in Big Data Hadoop by nitinrawat895
• 11,380 points
103,821 views
–1 vote
1 answer

Hadoop dfs -ls command?

In your case there is no difference ...READ MORE

answered Mar 16, 2018 in Big Data Hadoop by kurt_cobain
• 9,390 points
4,234 views
0 votes
1 answer
webinar REGISTER FOR FREE WEBINAR X
REGISTER NOW
webinar_success Thank you for registering Join Edureka Meetup community for 100+ Free Webinars each month JOIN MEETUP GROUP