How can we send data from MongoDB to Hadoop

Question

In my scenario, I want to get the daily inserted data from MongoDB (roughly around 10MB) and put that all into Hadoop. Also MongoDB node and Hadoop node runs on the same server.

I'm not getting how to do this? Please help me out.

Answer 1

The MongoDB Connector for Hadoop reads data directly from MongoDB. You can configure multiple input splits to read data from the same collection in parallel. The Mapper and Reducer jobs are run by Hadoop's Map/Reduce engine, not MongoDB's Map/Reduce.

If your data estimate is correct (only 10MB per day?) that is a small amount to ingest and the job may be faster if you don't have any input splits calculated.

You should be wary of Hadoop and MongoDB competing for resources on the same server, as contention for memory or disk can affect the efficiency of your data transfer.