What is Hadoop Performance Tuning

Question

I increased the input split size from 128MB to 256MB. The execution time of the job has been decreased by a minute.

But I could not understand the behavior. Why it is happening? In what scenarios, we can tune the input split size?

Answer 1

Is it consistent or one-off reading? Is this on your local hadoop installation or on a cluster?

I would suggest to record number of mappers when input split size is 128MB and 256MB for number of runs. That may have a possible hint as to why the execution time is decreased by a minute.

The number of input splits corresponds to the number of mappers needed to process the input. If this number is higher than the map slots available on your cluster, job has to wait until one set of mappers are run before it can process remaining ones. However, if a number of input splits are less ( e.g 256MB in your case) then accordingly number of map tasks to be run are lesser than earlier case. If this number is lesser than or equal to the number of map slots on your cluster then there are chances that all of your map tasks running simultaneously which may better your job execution time.