which one to choose for log analysis?

0 votes
I saw that there are various tools available in big data ecosystem such as flume, kafka, map reduce (or) if there is anything else to analyse logs. I have my logs present in the production serve.

I am confused between the tools available. Which tool should I used for analysing logs. Could anyone suggest me which one would be the best option please ?
Apr 6, 2018 in Big Data Hadoop by kurt_cobain
• 9,240 points
30 views

1 answer to this question.

0 votes

It is not about that you can use any tool for log processing. Every tool has its own functionalities so it depends on your use case.

Flume and Kafka are actually two quite different products. Kafka is a general purpose publish-subscribe model messaging system, which offers strong durability, scalability and fault-tolerance support.

While, Flume is a distributed, reliable, and available system for efficiently collecting, aggregating, and moving large amounts of data from many different sources to a centralized data store, such as HDFS or HBase. It is more tightly integrated with Hadoop ecosystem.

When to use Flume and Kafka:

Flume: When working with non-relational data sources such as log files which are to be streamed into Hadoop.
Kafka: When in need of highly reliable and scalable enterprise messaging system to connect many multiple systems (Including Hadoop)

Go through this link to learn more: https://www.linkedin.com/pulse/flume-kafka-real-time-event-processing-lan-jiang/

answered Apr 6, 2018 by nitinrawat895
• 10,490 points

Related Questions In Big Data Hadoop

0 votes
1 answer
+10 votes
11 answers

Hadoop “Unable to load native-hadoop library for your platform” warning

modify the glibc version.CentOS provides safe softwares ...READ MORE

answered Sep 10, 2018 in Big Data Hadoop by bug_seeker
• 15,320 points
9,557 views
0 votes
1 answer

Hadoop dfs -ls command?

In your case there is no difference ...READ MORE

answered Mar 16, 2018 in Big Data Hadoop by kurt_cobain
• 9,240 points
884 views
0 votes
1 answer

Hadoop Mapreduce word count Program

Firstly you need to understand the concept ...READ MORE

answered Mar 16, 2018 in Data Analytics by nitinrawat895
• 10,490 points
2,328 views
0 votes
1 answer

How to get started with Hadoop?

Well, hadoop is actually a framework that ...READ MORE

answered Mar 21, 2018 in Big Data Hadoop by coldcode
• 2,020 points
75 views
0 votes
10 answers

hadoop fs -put command?

put syntax: put <localSrc> <dest> copy syntax: copyFr ...READ MORE

answered Dec 7, 2018 in Big Data Hadoop by Aditya
11,979 views
0 votes
2 answers
0 votes
1 answer

How to create a FileSystem object that can be used for reading from and writing to HDFS?

Read operation on HDFS In order to read ...READ MORE

answered Mar 21, 2018 in Big Data Hadoop by nitinrawat895
• 10,490 points

edited Mar 21, 2018 by nitinrawat895 245 views