Can I run Hadoop with Docker for both DEV and PROD environments

Question

Is there an easiest way that works for both dev' and real environments so that to run Hadoop with Docker?

This query is for the local development environment and real environment difference should be reflected only on the destination machine.

Answer 1

Hadoop Docker Hub provides Hadoop Images, You can try them out or if you have something suitable for different environments, then you might require to run on a fully distributed environment.i.e. with a container for the HDFS and YARN master node and multiple containers for the worker nodes. 

I can help you with an Image that I have. You can see from the Dockerfile that it starts from the Java base image, installs Hadoop and uses a startup script so containers can be run as a master or a worker.

That means you can run a distributed cluster with Docker, using the latest networking stack:

docker network create hadoop

docker run -d -p 50070:50070 -p 8088:8088 \
    --network hadoop --name hadoop-dotnet-master \
    sixeyed/hadoop-dotnet master

docker run -d -p 50075:50075 -p 8142:8042 -p 19888:19888 \
    --network hadoop 
    sixeyed/hadoop-dotnet worker

You can run a fully-distributed cluster with a Docker Compose file.

I hope this should help. 

Happy Learning...;-)