How to convert a json file structure with values in single quotes to quoteless

+1 vote
Something as this json object:

"Territory": "'CA'",

I have to get rid of ' ' beside CA.

And in here i am moving the so processed file into a s3 bucket.

I have already established connection. i need help with the code of pyspark to run on the custer that can help change this.
Oct 4, 2019 in Apache Spark by anonymous
• 160 points
3,959 views

1 answer to this question.

+1 vote

You can do this by turning off the default escape of double-quote character with a backslash (\). Try the following:

dataframe
.write
.option("header", true)
.option("quote", "\u0000") 
.csv("Path to csv file")

What this does is, changes the "quote" character to the null character(Unicode \u0000). 

answered Oct 4, 2019 by Jisha

Related Questions In Apache Spark

0 votes
1 answer
+1 vote
1 answer

How to convert JSON file to AVRO file and vise versa

Try including the package while starting the ...READ MORE

answered Aug 26, 2019 in Apache Spark by Karan
3,123 views
+15 votes
2 answers

Git management technique when there are multiple customers and need multiple customization?

Consider this - In 'extended' Git-Flow, (Git-Multi-Flow, ...READ MORE

answered Mar 27, 2018 in DevOps & Agile by DragonLord999
• 8,450 points
3,439 views
+2 votes
1 answer
0 votes
1 answer

How to remove the elements with a key present in any other RDD?

Hey, You can use the subtractByKey () function to ...READ MORE

answered Jul 22, 2019 in Apache Spark by Gitika
• 65,910 points
3,746 views
+1 vote
1 answer

How to read a data from text file in Spark?

Hey, You can try this: from pyspark import SparkContext SparkContext.stop(sc) sc ...READ MORE

answered Aug 6, 2019 in Apache Spark by Gitika
• 65,910 points
4,673 views
webinar REGISTER FOR FREE WEBINAR X
REGISTER NOW
webinar_success Thank you for registering Join Edureka Meetup community for 100+ Free Webinars each month JOIN MEETUP GROUP