which one to choose for log analysis?

0 votes
I saw that there are various tools available in big data ecosystem such as flume, kafka, map reduce (or) if there is anything else to analyse logs. I have my logs present in the production serve.

I am confused between the tools available. Which tool should I used for analysing logs. Could anyone suggest me which one would be the best option please ?
Apr 6, 2018 in Big Data Hadoop by kurt_cobain
• 9,260 points
38 views

1 answer to this question.

0 votes

It is not about that you can use any tool for log processing. Every tool has its own functionalities so it depends on your use case.

Flume and Kafka are actually two quite different products. Kafka is a general purpose publish-subscribe model messaging system, which offers strong durability, scalability and fault-tolerance support.

While, Flume is a distributed, reliable, and available system for efficiently collecting, aggregating, and moving large amounts of data from many different sources to a centralized data store, such as HDFS or HBase. It is more tightly integrated with Hadoop ecosystem.

When to use Flume and Kafka:

Flume: When working with non-relational data sources such as log files which are to be streamed into Hadoop.
Kafka: When in need of highly reliable and scalable enterprise messaging system to connect many multiple systems (Including Hadoop)

Go through this link to learn more: https://www.linkedin.com/pulse/flume-kafka-real-time-event-processing-lan-jiang/

answered Apr 6, 2018 by nitinrawat895
• 10,710 points

Related Questions In Big Data Hadoop

0 votes
1 answer

which one is better bigdata or devops for career

I think Big Data will be good ...READ MORE

answered Aug 26 in Big Data Hadoop by Karan
85 views
0 votes
1 answer
0 votes
1 answer

Hadoop dfs -ls command?

In your case there is no difference ...READ MORE

answered Mar 16, 2018 in Big Data Hadoop by kurt_cobain
• 9,260 points
1,194 views
0 votes
1 answer

Hadoop Mapreduce word count Program

Firstly you need to understand the concept ...READ MORE

answered Mar 16, 2018 in Data Analytics by nitinrawat895
• 10,710 points
3,321 views
0 votes
1 answer

How to get started with Hadoop?

Well, hadoop is actually a framework that ...READ MORE

answered Mar 21, 2018 in Big Data Hadoop by coldcode
• 2,020 points
103 views
0 votes
10 answers

hadoop fs -put command?

put syntax: put <localSrc> <dest> copy syntax: copyFr ...READ MORE

answered Dec 7, 2018 in Big Data Hadoop by Aditya
16,396 views
0 votes
2 answers
0 votes
1 answer

How to create a FileSystem object that can be used for reading from and writing to HDFS?

Read operation on HDFS In order to read ...READ MORE

answered Mar 21, 2018 in Big Data Hadoop by nitinrawat895
• 10,710 points

edited Mar 21, 2018 by nitinrawat895 355 views