Spark Saving file csv

Question

It would be great if you can suggest to me what I am doing wrong in the below code. I just want to save output in Ans3AppleStore.csv. I think it is last the last part of the code where I need some change.

def main(args: Array[String])
{
val conf = new SparkConf().setAppName('mod5sol')
val sc:SparkContext = new SparkContext(conf)
val sqlContext: SQLContext = new SQLContext(sc)
val df: DataFrame = sqlContext.read.format('csv').option('header', 'true').load('AppleStore.csv')
df.registerTempTable('Apple5')
var dfsize=sqlContext.sql('select size_bytes Size, (size_bytes/1024) In_MB, ((size_bytes/1024)/1024) In_GB from Apple5').write.format('csv').save('Ans3Apple
Store.csv')}

Answer 1

 If you need a single output file (still in a folder) you can repartition (preferred if upstream data is large, but requires a shuffle):

df
      .repartition(1)
      .write.format("com.databricks.spark.csv")
      .option("header", "true")
      .save("mydata.csv")
      

or coalesce:

df
      .coalesce(1)
      .write.format("com.databricks.spark.csv")
      .option("header", "true")
      .save("mydata.csv")

All data will be written to mydata.csv/part-00000.