Question

1)Given sfpd RDD, to create a pair RDD consisting of tuples of the form (Category. 1) in scala ,which of the following is used?

a) val pairs =sfpd.parallelize()

b) val pairs =sfpd.map(x=>(x(Category),1))

c) val pairs =sfpd.map(x=>x.parallize))

d) val pairs =sfpd.map(x=>x.unparallelize))

Answer 1

Hi, @Ritu,

When creating a pair RDD from an in-memory collection in Scala and Python, we only need to call SparkContext.parallelize() on a collection of pairs. To create a pair RDD in Java from an in-memory collection, we instead use SparkContext.parallelizePairs().

Option c for your query.

Hope this helps!

To know more about Spark Scala, It's recommended to join Spark training online today.

Thanks!!