Can I run Hadoop with Docker for both DEV and PROD environments?

0 votes
Is there an easiest way that works for both dev' and real environments so that to run Hadoop with Docker?

This query is for the local development environment and real environment difference should be reflected only on the destination machine.
Jun 26 in Big Data Hadoop by nitinrawat895
• 10,670 points
22 views

1 answer to this question.

0 votes

Hadoop Docker Hub provides Hadoop Images, You can try them out or if you have something suitable for different environments, then you might require to run on a fully distributed environment.i.e. with a container for the HDFS and YARN master node and multiple containers for the worker nodes. 

I can help you with an Image that I have. You can see from the Dockerfile that it starts from the Java base image, installs Hadoop and uses a startup script so containers can be run as a master or a worker.

That means you can run a distributed cluster with Docker, using the latest networking stack:

docker network create hadoop

docker run -d -p 50070:50070 -p 8088:8088 \
    --network hadoop --name hadoop-dotnet-master \
    sixeyed/hadoop-dotnet master

docker run -d -p 50075:50075 -p 8142:8042 -p 19888:19888 \
    --network hadoop 
    sixeyed/hadoop-dotnet worker

You can run a fully-distributed cluster with a Docker Compose file.

I hope this should help. 

Happy Learning...;-)

answered Jun 26 by ravikiran
• 4,560 points

Related Questions In Big Data Hadoop

0 votes
0 answers

Can I run Hadoop with Docker for both DEV and PROD environments?

Is there an easiest way that works ...READ MORE

Jun 26 in Big Data Hadoop by nitinrawat895
• 10,670 points
15 views
0 votes
1 answer

How can I download only hdfs and not hadoop?

No, you cannot download HDFS alone because ...READ MORE

answered Mar 15, 2018 in Big Data Hadoop by nitinrawat895
• 10,670 points
83 views
0 votes
1 answer

Hadoop Mapreduce word count Program

Firstly you need to understand the concept ...READ MORE

answered Mar 16, 2018 in Data Analytics by nitinrawat895
• 10,670 points
2,733 views
0 votes
1 answer

hadoop.mapred vs hadoop.mapreduce?

org.apache.hadoop.mapred is the Old API  org.apache.hadoop.mapreduce is the ...READ MORE

answered Mar 16, 2018 in Data Analytics by nitinrawat895
• 10,670 points
287 views
0 votes
10 answers

hadoop fs -put command?

put syntax: put <localSrc> <dest> copy syntax: copyFr ...READ MORE

answered Dec 7, 2018 in Big Data Hadoop by Aditya
13,534 views
0 votes
1 answer

Hadoop dfs -ls command?

In your case there is no difference ...READ MORE

answered Mar 16, 2018 in Big Data Hadoop by kurt_cobain
• 9,240 points
989 views
0 votes
1 answer

How do I run Hadoop with Docker for both DEV and PROD environments?

Your requirements show that you are using ...READ MORE

answered May 30 in Big Data Hadoop by ravikiran
• 4,560 points
31 views