Array of RDD

Question

Hi. I have to pass different RDDs from one part of the application to another. I am using a loop for this right now. Is there a way that I can send all of them at once? Like a special Data-structure or something?

Answer 1

You can create an array of RDDs and pass the array. And for this, you can use Glom. Glom on spark rdd which allows you to treat a partition as an array rather as single row at time. This allows you speed up some operations with some increased memory usage. 

val maxValue = dataRDD.glom().map((value:Array[Double]) => value.max)
 .reduce(_ max _)