Average function is not commutative and associative

Question

Hi,

I am not getting what is wrong with the code while finding the average, the average function is not showing commutative and associative. Can someone help how can I change the code to make it work properly?

Here is the code given:

def sum(x, y):

return x+y;

total = myrdd.reduce(sum);

avg = total / myrdd.count();

 

Answer 1

Hey,

I guess the only problem with the code is that the total might become very big thus overflow. So, I would rather divide each number by count and then sum in the following way.

You can use this code to see the result in a better manner:

cnt = myrdd.count();

def devideByCnd(x):

return x/cnt;

myrdd1 = myrdd.map(devideByCnd);

avg = myrdd.reduce(sum);