Operation to simulate a Hadoop Production Cluster

0 votes
I want to run Hadoop jobs on my development workstation for testing before I submit them to my production cluster. Which mode of operation in Hadoop allows to most closely simulate a production cluster while using a single machine?

Can anyone help?
Aug 8, 2018 in Big Data Hadoop by Meci Matt
• 9,420 points
40 views

1 answer to this question.

0 votes
In this case, what you can do is, you can run all the nodes in your production cluster as virtual machines on your development workstation.

In large-scale cloud infrastructures, there is another deployment pattern: local VMs on desktop systems or other development machines. This is a good tactic if your physical machines run windows and you need to bring up a Linux system running Hadoop, and/or you want to simulate the complexity of a small Hadoop cluster.
 
Have enough RAM for the VM to not swap. Don't try and run more than one VM per physical host, it will only make things slower. use file: URLs to access persistent input and output data. consider making the default filesystem a file: URL so that all storage is really on the physical host. It's often faster and preserves data better.

Hope it will answer your query to some extent.
answered Aug 8, 2018 by nitinrawat895
• 10,690 points

Related Questions In Big Data Hadoop

0 votes
1 answer

How to install and configure a multi-node Hadoop cluster?

I would recommend you to install Cent ...READ MORE

answered Mar 21, 2018 in Big Data Hadoop by Shubham
• 13,300 points
675 views
0 votes
1 answer

Sending hdfs file to a non hadoop cluster

There are different ways to do this. ...READ MORE

answered Jul 9 in Big Data Hadoop by Tarun
27 views
0 votes
1 answer

Hadoop dfs -ls command?

In your case there is no difference ...READ MORE

answered Mar 16, 2018 in Big Data Hadoop by kurt_cobain
• 9,260 points
1,128 views
0 votes
1 answer

Hadoop Mapreduce word count Program

Firstly you need to understand the concept ...READ MORE

answered Mar 16, 2018 in Data Analytics by nitinrawat895
• 10,690 points
3,092 views
0 votes
1 answer

How to get started with Hadoop?

Well, hadoop is actually a framework that ...READ MORE

answered Mar 21, 2018 in Big Data Hadoop by coldcode
• 2,020 points
96 views
0 votes
10 answers

hadoop fs -put command?

put syntax: put <localSrc> <dest> copy syntax: copyFr ...READ MORE

answered Dec 7, 2018 in Big Data Hadoop by Aditya
15,175 views
0 votes
1 answer
0 votes
1 answer

How to access different directories in a Hadoop cluster?

You need to configure the client to ...READ MORE

answered Sep 18, 2018 in Big Data Hadoop by nitinrawat895
• 10,690 points
73 views