which one to choose for log analysis

0 votes
I saw that there are various tools available in big data ecosystem such as flume, kafka, map reduce (or) if there is anything else to analyse logs. I have my logs present in the production serve.

I am confused between the tools available. Which tool should I used for analysing logs. Could anyone suggest me which one would be the best option please ?
Apr 6, 2018 in Big Data Hadoop by kurt_cobain
• 9,390 points

1 answer to this question.

0 votes

It is not about that you can use any tool for log processing. Every tool has its own functionalities so it depends on your use case.

Flume and Kafka are actually two quite different products. Kafka is a general purpose publish-subscribe model messaging system, which offers strong durability, scalability and fault-tolerance support.

While, Flume is a distributed, reliable, and available system for efficiently collecting, aggregating, and moving large amounts of data from many different sources to a centralized data store, such as HDFS or HBase. It is more tightly integrated with Hadoop ecosystem.

When to use Flume and Kafka:

Flume: When working with non-relational data sources such as log files which are to be streamed into Hadoop.
Kafka: When in need of highly reliable and scalable enterprise messaging system to connect many multiple systems (Including Hadoop)

Go through this link to learn more: https://www.linkedin.com/pulse/flume-kafka-real-time-event-processing-lan-jiang/

answered Apr 6, 2018 by nitinrawat895
• 11,380 points

Related Questions In Big Data Hadoop

0 votes
1 answer

which one is better bigdata or devops for career

I think Big Data will be good ...READ MORE

answered Aug 26, 2019 in Big Data Hadoop by Karan
0 votes
1 answer
–1 vote
1 answer

Hadoop dfs -ls command?

In your case there is no difference ...READ MORE

answered Mar 16, 2018 in Big Data Hadoop by kurt_cobain
• 9,390 points
+1 vote
1 answer

Hadoop Mapreduce word count Program

Firstly you need to understand the concept ...READ MORE

answered Mar 16, 2018 in Data Analytics by nitinrawat895
• 11,380 points
0 votes
1 answer

How to get started with Hadoop?

Well, hadoop is actually a framework that ...READ MORE

answered Mar 21, 2018 in Big Data Hadoop by coldcode
• 2,080 points
+2 votes
11 answers

hadoop fs -put command?

Hi, You can create one directory in HDFS ...READ MORE

answered Mar 16, 2018 in Big Data Hadoop by nitinrawat895
• 11,380 points
0 votes
2 answers
0 votes
1 answer

How to create a FileSystem object that can be used for reading from and writing to HDFS?

Read operation on HDFS In order to read ...READ MORE

answered Mar 21, 2018 in Big Data Hadoop by nitinrawat895
• 11,380 points

edited Mar 22, 2018 by nitinrawat895 2,797 views
webinar_success Thank you for registering Join Edureka Meetup community for 100+ Free Webinars each month JOIN MEETUP GROUP