How can I convert Spark Dataframe to Spark RDD

Question

Initially I was unaware that Spark RDD functions cannot be applied on Spark Dataframe. So, I was getting error when I was trying to execute Spark RDD methods on Spark Dataframe.

Can anyone tell how can I convert Spark DataFrame to Spark RDD?

Answer 1

To convert Spark Dataframe to Spark RDD use .rdd method.

val rows: RDD[row] = df.rdd

Comments

how to do this one in python(dataframe to rdd)

---

You can use PySpark like this:

rdd = df.rdd.map(tuple)

or

rdd = df.rdd.map(list)

Answer 2

Assuming your RDD[row] is called rdd, you can use:

val sqlContext = new SQLContext(sc) 
import sqlContext.implicits._
rdd.toDF()