how to make an existing hadoop cluster self adaptive

Question

i have a hadoop cluster of 10 slaves nodes and i want it to be self adaptive. That is, it will either shrink or expand depending on the size of dataset file in master node. how can i proceed with it?

all the ip of slaves nodes are stored in ect/host file.

Comments

Hi@Sanjana,

In Hadoop we don't have the concept of block as a service. So you can't able to do partitions in your cluster.

Can you kindly explain more about your query on shrink or expand?

Thank You

Answer 1

Hey, @Sanjana,

You can't shrink or expand nodes when your are using bare-metal servers, You can do that when your cluster is hosted on the cloud but that will not happen automatically, you need to do it manually. Increase/decrease the disk size but this doesn't happen in a productive environment.