How can I remove headers from dataframe

Question

Hi ,
I have been trying to remove the headers from dataframe below is my code:

val file_source_read1=spark.read.option("header",false).option("delimiter",source_del).csv(source_path)
val file_source_read2=file_source_read1.first()
val file_source_read3 = file_source_read1.except(file_source_read2)

but it is throwing an error. Would you please help on this? If possible please tell me how to do it with PySpark

Answer 1

You can use filter to do this. Something like this:​​

val header = data.first 
val rows = data.filter(line => line != header)