HDFS on docker containers - is it possible?

0 votes
we are using Hadoop cluster based on HDP version - `2.6.5` with ambari platform

we want to know if we can use the following HDFS components on docker containers as:

    1. journal nodes
    2. ZK fail controller
    3. NameNode ( HA with active namenode and standby namenode )

after some searching , we not found so much documentation about HDFS on  docker containers

so I am assume that `HDFS` on docker containers isnt good implementation

I will happy to get more ideas about this?

if its can be done on containers then please advice if we can use it on production systems?
Feb 11 in Big Data Hadoop by Judy
• 120 points
353 views

1 answer to this question.

0 votes

Hi @judy,

It is possible to Simplifying Hadoop deployments with the DCOS framework. Here Portworx makes it easy to deploy specific data services, in this case, Hadoop, by understanding how these all work together. We will see how to set up a 3 node Hadoop cluster running on DCOS:

  • First, you need to add the repository for the services for that you need to login to a node that has the dcos cli installed and is authenticated to the DCOS cluster.
  • Run the command to add the repository to your DCOS cluster:
 $ dcos package repo add –index=0 hadoop-px
  • Once run the command you will see the Hadoop-PX service available in your universe.
  • If you click on the Hadoop-PX service you should be able to look at the status of the nodes being created, and one service for the scheduler.
  • Once the Scheduler service and all the Hadoop containers nodes are in Running (green) status, you should be ready to start using the Hadoop cluster.
  • Once you check your cluster, you should see multiple volumes that were automatically created.

I hope this explanation will help you and to know much more about it you can visit here and check https://docs.portworx.com/install-with-other/docker/stateful-applications/hadoopandhdfs/

answered Feb 12 by Gitika
• 29,690 points

Related Questions In Big Data Hadoop

0 votes
1 answer

Is it possible to only install Hadoop HDFS?

First of all think of Hadoop as ...READ MORE

answered Mar 27, 2018 in Big Data Hadoop by kurt_cobain
• 9,310 points
316 views
0 votes
11 answers

What is the default Namenode port of HDFS. Is it 8020 or 9000 or 50070?

8020/9000 hadoop namenode metadata port number. or ...READ MORE

answered Dec 31, 2018 in Big Data Hadoop by SM
23,196 views
0 votes
1 answer

What metadata is stored on a DataNode when a block is written to it?

Let me explain you step by step.  Each ...READ MORE

answered Jul 23, 2018 in Big Data Hadoop by nitinrawat895
• 10,920 points
554 views
0 votes
1 answer

Is it possible to start a zookeeper server instance in process?

To start ZooKeeper you have to execute ZooKeeperServerMain class. You can use ...READ MORE

answered Oct 9, 2018 in Big Data Hadoop by Omkar
• 69,060 points
77 views
0 votes
1 answer

Is it possible to integrate Kdb+ and hadoop?

kdb is a column oriented database plus ...READ MORE

answered Oct 9, 2018 in Big Data Hadoop by Frankie
• 9,810 points
191 views
0 votes
1 answer

How to find the port on which hdfs is running?

If you are using hadoop 2.7 or ...READ MORE

answered Jan 24, 2019 in Big Data Hadoop by Omkar
• 69,060 points
132 views
0 votes
1 answer

Is it possible to run Apache Spark without Hadoop?

Though Spark and Hadoop were the frameworks designed ...READ MORE

answered May 2, 2019 in Big Data Hadoop by ravikiran
• 4,600 points
200 views
0 votes
1 answer

Is it possible to run Apache Spark without Apache Hadoop?

First of all, Let us get a ...READ MORE

answered Jun 17, 2019 in Big Data Hadoop by ravikiran
• 4,600 points
140 views
0 votes
1 answer

How to check your file is created on HDFS?

Hey,  There is a process or steps you ...READ MORE

answered May 7, 2019 in Big Data Hadoop by Gitika
• 29,690 points
56 views
0 votes
1 answer

Is it possible to change the default location of a managed table?

Hi, Yes, you can do it by using ...READ MORE

answered Jun 18, 2019 in Big Data Hadoop by Gitika
• 29,690 points
256 views