Json and snappy compression

Question

When trying to write json file using snappy compression the below method is not working.

sqlContext.setConf("spark.sql.json.compression.codec","snappy")
filterStatus.write.json("/user/hduser_212418/heorder_json")

what changes to be done for the above code for it to save in snappy compression format. Only the below one works.

filterStatus.toJSON.rdd.saveAsTextFile("/user/hduser_212418/heorder_json",classOf[org.apache.hadoop.io.compress.SnappyCodec])

Input to the abov is :

val filterStatus = rdFile.filter("order_status like "%Y%"")
filterStatus: org.apache.spark.sql.Dataset[org.apache.spark.sql.Row] = [order_id: int, order_date: bigint ... 2 more fields]

Answer 1

The issue you're facing arises because the Spark SQL configuration sqlContext.setConf("spark.sql.json.compression.codec", "snappy") is not recognized by the default DataFrame .write.json() operation for controlling the compression codec. The correct way to specify compression for DataFrame writes involves using option() settings on the write operation. Here's how you can modify your code:

filterStatus.write
  .option("compression", "snappy")
  .json("/user/hduser_212418/heorder_json")

Explanation:

• Instead of setting the codec at the sqlContext level, the option("compression", "snappy") method ensures that the JSON write operation uses Snappy compression.
• This method is simpler and fits directly into the .write.json() logic.

Why the Original toJSON.rdd.saveAsTextFile() Works:

• The filterStatus.toJSON.rdd.saveAsTextFile() approach works because it explicitly transforms the DataFrame to an RDD of JSON strings, and then the saveAsTextFile method uses Hadoop's SnappyCodec for compression.

Key Differences:

• The .write.json() with .option("compression", "snappy") approach is more idiomatic for Spark's DataFrame API.
• toJSON.rdd.saveAsTextFile() is lower-level and converts the data to an RDD of strings before saving. It offers more control but can be less optimized compared to the built-in DataFrame write methods.