Operation to simulate a Hadoop Production Cluster

0 votes
I want to run Hadoop jobs on my development workstation for testing before I submit them to my production cluster. Which mode of operation in Hadoop allows to most closely simulate a production cluster while using a single machine?

Can anyone help?
Aug 8, 2018 in Big Data Hadoop by Meci Matt
• 9,400 points
32 views

1 answer to this question.

0 votes
In this case, what you can do is, you can run all the nodes in your production cluster as virtual machines on your development workstation.

In large-scale cloud infrastructures, there is another deployment pattern: local VMs on desktop systems or other development machines. This is a good tactic if your physical machines run windows and you need to bring up a Linux system running Hadoop, and/or you want to simulate the complexity of a small Hadoop cluster.
 
Have enough RAM for the VM to not swap. Don't try and run more than one VM per physical host, it will only make things slower. use file: URLs to access persistent input and output data. consider making the default filesystem a file: URL so that all storage is really on the physical host. It's often faster and preserves data better.

Hope it will answer your query to some extent.
answered Aug 8, 2018 by nitinrawat895
• 10,110 points

Related Questions In Big Data Hadoop

0 votes
1 answer

How to install and configure a multi-node Hadoop cluster?

I would recommend you to install Cent ...READ MORE

answered Mar 21, 2018 in Big Data Hadoop by Shubham
• 13,190 points
546 views
0 votes
1 answer

Sending hdfs file to a non hadoop cluster

There are different ways to do this. ...READ MORE

answered Jul 9 in Big Data Hadoop by Tarun
15 views
0 votes
1 answer

Hadoop dfs -ls command?

In your case there is no difference ...READ MORE

answered Mar 16, 2018 in Big Data Hadoop by kurt_cobain
• 9,240 points
768 views
0 votes
1 answer

Hadoop Mapreduce word count Program

Firstly you need to understand the concept ...READ MORE

answered Mar 16, 2018 in Data Analytics by nitinrawat895
• 10,110 points
2,055 views
0 votes
1 answer

How to get started with Hadoop?

Well, hadoop is actually a framework that ...READ MORE

answered Mar 21, 2018 in Big Data Hadoop by coldcode
• 2,010 points
63 views
0 votes
10 answers

hadoop fs -put command?

copy command can be used to copy files ...READ MORE

answered Dec 7, 2018 in Big Data Hadoop by Sujay
10,515 views
0 votes
1 answer
0 votes
1 answer

How to access different directories in a Hadoop cluster?

You need to configure the client to ...READ MORE

answered Sep 18, 2018 in Big Data Hadoop by nitinrawat895
• 10,110 points
52 views