Spark Error StackOverflowError Exception in thread main java lang StackOverflowError at org apache spark rdd UnionRDD anonfun 1 apply

Question

I have a Scala program which uses a REST API to get data batch by batch. I guess when the number of batches are high then the program is throwing this error.

This is the sample code:

def main(args: Array[String]) = {
     val conf = new SparkConf().setAppName("Union test").setMaster("local[1]")
     val sc = new SparkContext(conf)
     val limit = 1000;
     var rdd = sc.emptyRDD[Int]
     for (x <- 1 to limit) {
       val currentRdd = sc.parallelize(x to x + 3)
       rdd = rdd.union(currentRdd)
     }
     println(rdd.sum())
   }

Can anyone give the solution to this.

Answer 1

Hey,

It already has SparkContent.union and it does know how to compute a union of multiple RDD's. You can use this below lines:

val rdds = List.tabulate(limit + 1)(x => sc.parallelize(x to x + 3))
val rdd = sc.union(rdds)

Otherwise, you can see one more way to do this:

val rdds = List.tabulate(limit + 1)(x => sc.parallelize(x to x + 3))
val rdd = balancedReduce(rdds)(_ union _)