which one to choose for log analysis?

0 votes
I saw that there are various tools available in big data ecosystem such as flume, kafka, map reduce (or) if there is anything else to analyse logs. I have my logs present in the production serve.

I am confused between the tools available. Which tool should I used for analysing logs. Could anyone suggest me which one would be the best option please ?
Apr 6, 2018 in Big Data Hadoop by kurt_cobain
• 9,260 points
25 views

1 answer to this question.

Your answer

Your name to display (optional):
Privacy: Your email address will only be used for sending these notifications.
0 votes

It is not about that you can use any tool for log processing. Every tool has its own functionalities so it depends on your use case.

Flume and Kafka are actually two quite different products. Kafka is a general purpose publish-subscribe model messaging system, which offers strong durability, scalability and fault-tolerance support.

While, Flume is a distributed, reliable, and available system for efficiently collecting, aggregating, and moving large amounts of data from many different sources to a centralized data store, such as HDFS or HBase. It is more tightly integrated with Hadoop ecosystem.

When to use Flume and Kafka:

Flume: When working with non-relational data sources such as log files which are to be streamed into Hadoop.
Kafka: When in need of highly reliable and scalable enterprise messaging system to connect many multiple systems (Including Hadoop)

Go through this link to learn more: https://www.linkedin.com/pulse/flume-kafka-real-time-event-processing-lan-jiang/

answered Apr 6, 2018 by nitinrawat895
• 9,030 points

Related Questions In Big Data Hadoop

0 votes
1 answer
+10 votes
10 answers
0 votes
1 answer

How to set a custom install directory for a deb package with fpm

Here's something that you can try... the last ...READ MORE

answered Mar 26, 2018 in Big Data Hadoop by Amrinder
• 140 points
43 views
0 votes
1 answer

Hadoop dfs -ls command?

In your case there is no difference ...READ MORE

answered Mar 16, 2018 in Big Data Hadoop by kurt_cobain
• 9,260 points
547 views
0 votes
1 answer

Hadoop Mapreduce word count Program

Firstly you need to understand the concept ...READ MORE

answered Mar 16, 2018 in Data Analytics by nitinrawat895
• 9,030 points
1,633 views
0 votes
1 answer

How to get started with Hadoop?

Well, hadoop is actually a framework that ...READ MORE

answered Mar 21, 2018 in Big Data Hadoop by coldcode
• 1,980 points
39 views
0 votes
10 answers

hadoop fs -put command?

copy command can be used to copy files ...READ MORE

answered Dec 7, 2018 in Big Data Hadoop by Sujay
7,922 views
0 votes
2 answers
0 votes
1 answer

How to create a FileSystem object that can be used for reading from and writing to HDFS?

Read operation on HDFS In order to read ...READ MORE

answered Mar 21, 2018 in Big Data Hadoop by nitinrawat895
• 9,030 points

edited Mar 21, 2018 by nitinrawat895 143 views

© 2018 Brain4ce Education Solutions Pvt. Ltd. All rights Reserved.
"PMP®","PMI®", "PMI-ACP®" and "PMBOK®" are registered marks of the Project Management Institute, Inc. MongoDB®, Mongo and the leaf logo are the registered trademarks of MongoDB, Inc.