How do I run Hadoop with Docker for both DEV and PROD environments

Question

Can you help me out with the easiest way that works for both DEV and Real-Time environment to run Hadoop with Docker?

That is for local development environment and real environment difference should be only destination machine.

Answer 1

Your requirements show that you are using Hadoop in fully distributed mode. I'll redirect you to a site which must help you with the image you need.

Here containers can be run as both master and worker.

#docker network create hadoop

#docker run -d -p 50070:50070 -p 8088:8088 \
#    --network hadoop --name hadoop-dotnet-master \
#    sixeyed/hadoop-dotnet master

#docker run -d -p 50075:50075 -p 8142:8042 -p 19888:19888 \
#    --network hadoop 
#    sixeyed/hadoop-dotnet worker

Or if you wish to run a fully distributed cluster with a docker composer file, then I this must be the solution