which one to choose for log analysis

Question

I saw that there are various tools available in big data ecosystem such as flume, kafka, map reduce (or) if there is anything else to analyse logs. I have my logs present in the production serve.

I am confused between the tools available. Which tool should I used for analysing logs. Could anyone suggest me which one would be the best option please ?

Answer 1

It is not about that you can use any tool for log processing. Every tool has its own functionalities so it depends on your use case.

Flume and Kafka are actually two quite different products. Kafka is a general purpose publish-subscribe model messaging system, which offers strong durability, scalability and fault-tolerance support.

While, Flume is a distributed, reliable, and available system for efficiently collecting, aggregating, and moving large amounts of data from many different sources to a centralized data store, such as HDFS or HBase. It is more tightly integrated with Hadoop ecosystem.

When to use Flume and Kafka:

Flume: When working with non-relational data sources such as log files which are to be streamed into Hadoop.
Kafka: When in need of highly reliable and scalable enterprise messaging system to connect many multiple systems (Including Hadoop)

Go through this link to learn more: https://www.linkedin.com/pulse/flume-kafka-real-time-event-processing-lan-jiang/