Big Data and Hadoop (168 Blogs) Become a Certified Professional
AWS Global Infrastructure

Big Data

Topics Covered
  • Big Data and Hadoop (144 Blogs)
  • Hadoop Administration (7 Blogs)
  • Apache Storm (4 Blogs)
  • Apache Spark and Scala (29 Blogs)
SEE MORE

How to Plan the Capacity of a Hadoop Cluster?

Published on Feb 20,2020 1.1K Views
Ravi Kiran
Tech Enthusiast working as a Research Analyst at Edureka. Curious about learning... Tech Enthusiast working as a Research Analyst at Edureka. Curious about learning more about Data Science and Big-Data Hadoop.

Hadoop Cluster is the most vital asset with strategic and high-caliber performance when you have to deal with storing and analyzing huge loads of Big Data in distributed Environment. In this article, we will about Hadoop Cluster Capacity Planning with maximum efficiency considering all the requirements.

 

What is a Hadoop Cluster?

A cluster is basically a collection. A computer cluster is a collection of computers interconnected to each other over a network. Similarly, a Hadoop Cluster is a collection of extraordinary computational systems designed and deployed to store, optimise, and analyse petabytes of Big Data with astonishing agility

 

Factors deciding the Hadoop Cluster Capacity

Now that we know what exactly a Hadoop Cluster is, let us now learn why exactly we need to plan a Hadoop Cluster and what are various factors we need to look into, in order to plan an efficient Hadoop Cluster with optimum performance

  • Volume of Data

If you ever wonder how Hadoop even came into existence, it is because of the huge volume of data that the traditional data processing systems could not handle. Since the introduction of Hadoop, the volume of data also increased exponentially.

So, it is important for a Hadoop Admin to know about the volume of Data he needs to deal with and accordingly plan, organize, and set up the Hadoop Cluster with the appropriate number of nodes for an Efficient Data Management

  • Data Retention

Data Retention is all about storing only the important and valid data. There are many situations where the data arrived will be incomplete or invalid that may affect the process of Data Analysis. So, there is no point in storing such data.

Data Retention is a process where the user gets to remove outdated, invalid, and unnecessary data from the Hadoop Storage to save space and improve cluster computation speeds.

  • Data Storage

Data Storage is one of the crucial factors that come into picture when you are into planning a Hadoop Cluster. Data is never stored directly as it is obtained. It undergoes through a process called Data Compression.

Here, the obtained data is encrypted and compressed using various Data Encryption and Data Compression algorithms so that the data security is achieved and the space consumed to save the data is as minimal as possible.

  • Type of Work Load

This factor is purely performance-oriented. All this factor deals with is the performance of the cluster. the Work Load on the processor can be classified into three types. Intensive, normal, and low.

Some jobs like Data Storage cause low workload on the processor. Jobs like Data Querying will have intense workloads on both the processor and the storage units of the Hadoop Cluster.

 

Hardware Requirements for Hadoop Cluster

We have discussed Hadoop Cluster and the factors involved in planning an effective Hadoop Cluster. Now, we will discuss the standard hardware requirements needed by the Hadoop Components. Hadoop’s Architecture basically has the following components.

  • NameNode
  • Job Tracker
  • DataNode
  • Task Tracker

 

NameNode/Secondary NameNode/Job Tracker. 

NameNode and Secondary NameNode are the crucial parts of any Hadoop Cluster. They are expected to be highly available. The NameNode and Secondary NameNode servers are dedicated to storing the namespace storage and edit-log journaling.

ComponentRequirement
Operating System1 Terabyte Harddisk Space
FS-Image2 Terabyte Harddisk Space
Other Softwares(Zookeeper)1 Terabyte Harddisk Space
ProcessorOcta-Core Processor 2.5 GHz
RAM128 GB
Intenet10 GBPS

 

DataNode/Task Tracker

Followed by the NameNode and Job Tracker, the next crucial components in a Hadoop Cluster where the actual data is stored and the Hadoop jobs get executed are data nodes and Task Tacker respectively. Let us now discuss the Hardware requirements for DataNode and Task Tracker.

ComponentRequirement
Number of Nodes24 nodes(4 Terabytes each)
ProcessorOcta-Core Processor 2.5 GHz
RAM128 GB
Internet10 GBPS

 

Operating System Requirement

When it comes to software, then the Operating System becomes most important. You can set up your Hadoop cluster using the operating system of your choice. Few of the most recommended operating Systems to set up a Hadoop Cluster are,

  • Solaris
  • Ubuntu
  • Fedora
  • RedHat
  • CentOS

Now, let us understand a sample use case

 

Sample Hadoop Cluster Plan

Now that we have understood The Hardware and the Software requirements for Hadoop Cluster Capacity Planning, we will now plan a sample Hadoop Cluster for a better understanding. The following problem is based on the same.

Let us assume that we have to deal with the minimum data of 10 TB and assume that there is a gradual growth of data, say 25% per every 3 months. In future, assuming that the data grows per every year and data in year 1 is 10,000 TB.

By then end of 5 years, let us assume that it may grow to 25,000 TB. If we assume 25% of year-by-year growth and 10,000 TB data per year, then after 5 years, the resultant data is nearly 100,000 TB.

So, how exactly can we even estimate the number of data nodes that we might require to tackle all this data? The answer is simple. Using the formula as mentioned below.

Hadoop Storage (HS) = CRS / (1-i)

Where

  • C= Compression Ratio
  • R= Replication Factor
  • S= Size of the data to be moved into Hadoop
  • i= Intermediate Factor

Calculating the number of nodes required.

Assuming that we will not be using any sort of Data Compression, hence, C is 1.

The standard replication factor for Hadoop is 3.

The Intermediate factor is 0.25, then the calculation for Hadoop, in this case, will result as follows

HS = (1*3*S) / (1-(1/4)

HS = 4S

The expected Hadoop Storage instance, in this case, is 4 times the initial storage. The following formula can be used to estimate the number of data nodes.

N = HS/D = (CRS/(1-i)) / D

Where D is Diskspace available per Node.

Let us assume that 25 TB is the available Diskspace per single node. Each Node Comprising of 27 Disks of 1 TB each. (2 TB is dedicated to Operating System). Also assuming the initial Data Size to be 5000 TB.

N = 5000/25 = 200

Hence, We need 200 Nodes in this scenario.

 

Hadoop Admin Responsibilities

Hadoop Developer Roles and Responsibilities Who

  • Responsible for implementation and administration of Hadoop Administration.
  • Testing MapReduce, Hive, Pig and Acess for Hadoop Applications.
  • Cluster maintenance tasks like backup, Recovery, Upgrading, Patching.
  • Performance Tuning and Capacity planning for clusters.
  • Monitor Hadoop Cluster and deploy Security.

With this, we come to an end of this article. I hope I have thrown some light on to your knowledge on the Hadoop Cluster Capacity Planning along with Hardware and Software required.

Now that you have understood Big data and its Technologies, check out the Hadoop training by Edureka, a trusted online learning company with a network of more than 250,000 satisfied learners spread across the globe. The Edureka Big Data Hadoop Certification Training course helps learners become expert in HDFS, Yarn, MapReduce, Pig, Hive, HBase, Oozie, Flume and Sqoop using real-time use cases on Retail, Social Media, Aviation, Tourism, Finance domain.

If you have any query related to this “Hadoop Cluster Capacity Planning” article, then please write to us in the comment section below and we will respond to you as early as possible.

Comments
0 Comments

Join the discussion

Browse Categories

webinar REGISTER FOR FREE WEBINAR
REGISTER NOW
webinar_success Thank you for registering Join Edureka Meetup community for 100+ Free Webinars each month JOIN MEETUP GROUP

Subscribe to our Newsletter, and get personalized recommendations.