Json and snappy compression

–1 vote

When trying to write json file using snappy compression the below method is not working.

sqlContext.setConf("spark.sql.json.compression.codec","snappy")
filterStatus.write.json("/user/hduser_212418/heorder_json")

what changes to be done for the above code for it to save in snappy compression format. Only the below one works.

filterStatus.toJSON.rdd.saveAsTextFile("/user/hduser_212418/heorder_json",classOf[org.apache.hadoop.io.compress.SnappyCodec])

Input to the abov is :

val filterStatus = rdFile.filter("order_status like "%Y%"")
filterStatus: org.apache.spark.sql.Dataset[org.apache.spark.sql.Row] = [order_id: int, order_date: bigint ... 2 more fields]
Jan 10, 2019 in Big Data Hadoop by digger
• 26,720 points
1,889 views

1 answer to this question.

0 votes

Refer the below screenshot for your above requirement

image

image

image

The above approach works

answered Jan 10, 2019 by Omkar
• 69,150 points

Related Questions In Big Data Hadoop

0 votes
10 answers

What is the difference between Mongodb and Hadoop?

MongoDB is a NoSQL database, whereas Hadoop is ...READ MORE

answered Jun 20, 2018 in Big Data Hadoop by jenny_code
7,998 views
0 votes
1 answer

How can I download only hdfs and not hadoop?

No, you cannot download HDFS alone because ...READ MORE

answered Mar 15, 2018 in Big Data Hadoop by nitinrawat895
• 11,380 points
385 views
+2 votes
10 answers

Is there any difference between “hdfs dfs” and “hadoop fs” shell commands?

Yes, there's a difference between hadoop fs and ...READ MORE

answered Dec 7, 2018 in Big Data Hadoop by Kunal
21,764 views
0 votes
1 answer

How to install and configure a multi-node Hadoop cluster?

I would recommend you to install Cent ...READ MORE

answered Mar 21, 2018 in Big Data Hadoop by Shubham
• 13,480 points
1,424 views
+1 vote
2 answers
+1 vote
1 answer

Hadoop Mapreduce word count Program

Firstly you need to understand the concept ...READ MORE

answered Mar 16, 2018 in Data Analytics by nitinrawat895
• 11,380 points
7,620 views
0 votes
1 answer

hadoop.mapred vs hadoop.mapreduce?

org.apache.hadoop.mapred is the Old API  org.apache.hadoop.mapreduce is the ...READ MORE

answered Mar 16, 2018 in Data Analytics by nitinrawat895
• 11,380 points
1,258 views
+2 votes
11 answers

hadoop fs -put command?

Hi, You can create one directory in HDFS ...READ MORE

answered Mar 16, 2018 in Big Data Hadoop by nitinrawat895
• 11,380 points
58,086 views
0 votes
5 answers

Hadoop hdfs: list all files in a directory and its subdirectories

Hi, You can try this command: hadoop fs -ls ...READ MORE

answered Aug 1, 2019 in Big Data Hadoop by Dinish
12,017 views
0 votes
1 answer

Hadoop: What is the difference between `hadoop dfs` and `hadoop fs`?

You can find the definition here: $HADOOP_HOME/bin/hadoop ... elif [ ...READ MORE

answered Nov 6, 2018 in Big Data Hadoop by Omkar
• 69,150 points
844 views