How to achieve maximum memory to disk I O ratio

Question

I am working on a Hadoop project and while running a map reduce job, I observe that the number of spilled records from map tasks for exceeds the number of map output records. In this case the child heap size is 1 GB and my io.sort.mb value is set to 100MB. Can someone tell me that how would I tune io.sort.mb value to achieve maximum memory to disk I/O ratio?

Answer 1

You need to tune io.sort.mb value until you observe that the number of spilled records equals (or is as close to equals) the number of map output records.

Hope this will help.