HDFS on docker containers - is it possible

Question

we are using Hadoop cluster based on HDP version - `2.6.5` with ambari platform

we want to know if we can use the following HDFS components on docker containers as:

    1. journal nodes
    2. ZK fail controller
    3. NameNode ( HA with active namenode and standby namenode )

after some searching , we not found so much documentation about HDFS on  docker containers

so I am assume that `HDFS` on docker containers isnt good implementation

I will happy to get more ideas about this?

if its can be done on containers then please advice if we can use it on production systems?

Answer 1

Hi @judy,

It is possible to Simplifying Hadoop deployments with the DCOS framework. Here Portworx makes it easy to deploy specific data services, in this case, Hadoop, by understanding how these all work together. We will see how to set up a 3 node Hadoop cluster running on DCOS:

• First, you need to add the repository for the services for that you need to login to a node that has the dcos cli installed and is authenticated to the DCOS cluster.
• Run the command to add the repository to your DCOS cluster:

 $ dcos package repo add –index=0 hadoop-px

• Once run the command you will see the Hadoop-PX service available in your universe.
• If you click on the Hadoop-PX service you should be able to look at the status of the nodes being created, and one service for the scheduler.
• Once the Scheduler service and all the Hadoop containers nodes are in Running (green) status, you should be ready to start using the Hadoop cluster.
• Once you check your cluster, you should see multiple volumes that were automatically created.

I hope this explanation will help you and to know much more about it you can visit here and check https://docs.portworx.com/install-with-other/docker/stateful-applications/hadoopandhdfs/