Json and snappy compression

–1 vote

When trying to write json file using snappy compression the below method is not working.

sqlContext.setConf("spark.sql.json.compression.codec","snappy")
filterStatus.write.json("/user/hduser_212418/heorder_json")

what changes to be done for the above code for it to save in snappy compression format. Only the below one works.

filterStatus.toJSON.rdd.saveAsTextFile("/user/hduser_212418/heorder_json",classOf[org.apache.hadoop.io.compress.SnappyCodec])

Input to the abov is :

val filterStatus = rdFile.filter("order_status like "%Y%"")
filterStatus: org.apache.spark.sql.Dataset[org.apache.spark.sql.Row] = [order_id: int, order_date: bigint ... 2 more fields]
Jan 11, 2019 in Big Data Hadoop by digger
• 26,720 points
2,301 views

1 answer to this question.

0 votes

Refer the below screenshot for your above requirement

image

image

image

The above approach works

answered Jan 11, 2019 by Omkar
• 69,190 points

Related Questions In Big Data Hadoop

0 votes
10 answers

What is the difference between Mongodb and Hadoop?

MongoDB is a NoSQL database, whereas Hadoop is ...READ MORE

answered Jun 20, 2018 in Big Data Hadoop by jenny_code
9,267 views
0 votes
1 answer

How can I download only hdfs and not hadoop?

No, you cannot download HDFS alone because ...READ MORE

answered Mar 16, 2018 in Big Data Hadoop by nitinrawat895
• 11,380 points
528 views
+2 votes
10 answers

Is there any difference between “hdfs dfs” and “hadoop fs” shell commands?

Yes, there's a difference between hadoop fs and ...READ MORE

answered Dec 7, 2018 in Big Data Hadoop by Kunal
25,056 views
0 votes
1 answer

How to install and configure a multi-node Hadoop cluster?

I would recommend you to install Cent ...READ MORE

answered Mar 22, 2018 in Big Data Hadoop by Shubham
• 13,490 points
1,602 views
+1 vote
2 answers
+1 vote
1 answer

Hadoop Mapreduce word count Program

Firstly you need to understand the concept ...READ MORE

answered Mar 16, 2018 in Data Analytics by nitinrawat895
• 11,380 points
8,626 views
0 votes
1 answer

hadoop.mapred vs hadoop.mapreduce?

org.apache.hadoop.mapred is the Old API  org.apache.hadoop.mapreduce is the ...READ MORE

answered Mar 16, 2018 in Data Analytics by nitinrawat895
• 11,380 points
1,524 views
+2 votes
11 answers

hadoop fs -put command?

Hi, You can create one directory in HDFS ...READ MORE

answered Mar 16, 2018 in Big Data Hadoop by nitinrawat895
• 11,380 points
76,190 views
0 votes
5 answers

Hadoop hdfs: list all files in a directory and its subdirectories

Hi, You can try this command: hadoop fs -ls ...READ MORE

answered Aug 1, 2019 in Big Data Hadoop by Dinish
13,548 views
0 votes
1 answer

Hadoop: What is the difference between `hadoop dfs` and `hadoop fs`?

You can find the definition here: $HADOOP_HOME/bin/hadoop ... elif [ ...READ MORE

answered Nov 6, 2018 in Big Data Hadoop by Omkar
• 69,190 points
1,033 views
webinar REGISTER FOR FREE WEBINAR X
Send OTP
REGISTER NOW
webinar_success Thank you for registering Join Edureka Meetup community for 100+ Free Webinars each month JOIN MEETUP GROUP