Can I run Hadoop with Docker for both DEV and PROD environments

0 votes
Is there an easiest way that works for both dev' and real environments so that to run Hadoop with Docker?

This query is for the local development environment and real environment difference should be reflected only on the destination machine.
Jun 26, 2019 in Big Data Hadoop by nitinrawat895
• 11,380 points
85 views

1 answer to this question.

0 votes

Hadoop Docker Hub provides Hadoop Images, You can try them out or if you have something suitable for different environments, then you might require to run on a fully distributed environment.i.e. with a container for the HDFS and YARN master node and multiple containers for the worker nodes. 

I can help you with an Image that I have. You can see from the Dockerfile that it starts from the Java base image, installs Hadoop and uses a startup script so containers can be run as a master or a worker.

That means you can run a distributed cluster with Docker, using the latest networking stack:

docker network create hadoop

docker run -d -p 50070:50070 -p 8088:8088 \
    --network hadoop --name hadoop-dotnet-master \
    sixeyed/hadoop-dotnet master

docker run -d -p 50075:50075 -p 8142:8042 -p 19888:19888 \
    --network hadoop 
    sixeyed/hadoop-dotnet worker

You can run a fully-distributed cluster with a Docker Compose file.

I hope this should help. 

Happy Learning...;-)

answered Jun 26, 2019 by ravikiran
• 4,620 points

Related Questions In Big Data Hadoop

0 votes
1 answer

Can I run Hadoop with Docker for both DEV and PROD environments?

Hi, Yes, you can run Hadoop with Docker ...READ MORE

answered Jan 24, 2020 in Big Data Hadoop by MD
• 95,060 points
87 views
0 votes
1 answer

How can I download only hdfs and not hadoop?

No, you cannot download HDFS alone because ...READ MORE

answered Mar 15, 2018 in Big Data Hadoop by nitinrawat895
• 11,380 points
292 views
+1 vote
1 answer

Hadoop Mapreduce word count Program

Firstly you need to understand the concept ...READ MORE

answered Mar 16, 2018 in Data Analytics by nitinrawat895
• 11,380 points
6,823 views
0 votes
1 answer

hadoop.mapred vs hadoop.mapreduce?

org.apache.hadoop.mapred is the Old API  org.apache.hadoop.mapreduce is the ...READ MORE

answered Mar 16, 2018 in Data Analytics by nitinrawat895
• 11,380 points
1,093 views
+2 votes
11 answers

hadoop fs -put command?

Hi, You can create one directory in HDFS ...READ MORE

answered Mar 16, 2018 in Big Data Hadoop by nitinrawat895
• 11,380 points
48,006 views
–1 vote
1 answer

Hadoop dfs -ls command?

In your case there is no difference ...READ MORE

answered Mar 16, 2018 in Big Data Hadoop by kurt_cobain
• 9,390 points
2,589 views
0 votes
1 answer

How do I run Hadoop with Docker for both DEV and PROD environments?

Your requirements show that you are using ...READ MORE

answered May 30, 2019 in Big Data Hadoop by ravikiran
• 4,620 points
292 views
0 votes
1 answer