Which is the Real Time Monitoring tool API for Hadoop

0 votes
I have been recently using Hadoop. I wanted to know if there is any API or Monitoring Tool that can be used to collect real-time statistics regarding tasks and HDFS (like data movement among slaves/master).

I can only find this information from log files but I want this information in real-time (not do post analysis).
Sep 4, 2018 in Big Data Hadoop by Neha
• 6,300 points
1,200 views

1 answer to this question.

0 votes

If you're using Yarn, there's a rest API that I'd use before screen scraping the job tracker, Hadoop YARN - Introduction to the web services REST API's. If you're using 1.3, I don't know of anything. There is a bug opened on Apache's Jira asking for a said feature, but it's marked as resolved in MRv2, so I wouldn't expect any progress towards it.

Regarding Ganglia/Nagios, the pair doesn't track job flow, it tracks the health of the system. If it has the capability to do job tracking buried among its innards, I haven't found it.

You can either scrape the information from the Jobtracker web UI (for tasks) or write a small Java program using the API's to access the JobTracker and poll it to grab the information. In terms of HDFS events, you'll need to tail & parse the log file, or possibly scrape some of the information from the Namenode web UI. Possibly use JMX to get metrics from each of the datanodes, depending on what you are after

answered Sep 4, 2018 by Frankie
• 9,830 points

Related Questions In Big Data Hadoop

0 votes
1 answer

How to create a project for the first time in Hadoop.?

If you want to learn Hadoop framework ...READ MORE

answered Jul 27, 2018 in Big Data Hadoop by Neha
• 6,300 points
1,140 views
0 votes
1 answer

Which is the most preferable language for Hadooop Map-Reduce programs?

MapReduce is a programming model to perform ...READ MORE

answered Aug 4, 2018 in Big Data Hadoop by Neha
• 6,300 points
3,684 views
+3 votes
5 answers

Hadoop DistributedCache is deprecated - what is the preferred API?

I had the same problem. And not ...READ MORE

answered Oct 12, 2018 in Big Data Hadoop by Rohan
2,288 views
+1 vote
1 answer

Hadoop Mapreduce word count Program

Firstly you need to understand the concept ...READ MORE

answered Mar 16, 2018 in Data Analytics by nitinrawat895
• 11,380 points
10,979 views
0 votes
1 answer

hadoop.mapred vs hadoop.mapreduce?

org.apache.hadoop.mapred is the Old API  org.apache.hadoop.mapreduce is the ...READ MORE

answered Mar 16, 2018 in Data Analytics by nitinrawat895
• 11,380 points
2,497 views
+2 votes
11 answers

hadoop fs -put command?

Hi, You can create one directory in HDFS ...READ MORE

answered Mar 16, 2018 in Big Data Hadoop by nitinrawat895
• 11,380 points
108,548 views
–1 vote
1 answer

Hadoop dfs -ls command?

In your case there is no difference ...READ MORE

answered Mar 16, 2018 in Big Data Hadoop by kurt_cobain
• 9,390 points
4,586 views
0 votes
1 answer

Which is the easiest way for text analytics with hadoop?

Apache pig provides CSVExcelStorage class for loading ...READ MORE

answered Nov 22, 2018 in Big Data Hadoop by Frankie
• 9,830 points
871 views
0 votes
1 answer

Which Windows client is used for Cloudera Hadoop Cluster?

You can very well use VM linux ...READ MORE

answered Sep 4, 2018 in Big Data Hadoop by Frankie
• 9,830 points
894 views
webinar REGISTER FOR FREE WEBINAR X
REGISTER NOW
webinar_success Thank you for registering Join Edureka Meetup community for 100+ Free Webinars each month JOIN MEETUP GROUP