How to convert a json file structure with values in single quotes to quoteless

+1 vote
Something as this json object:

"Territory": "'CA'",

I have to get rid of ' ' beside CA.

And in here i am moving the so processed file into a s3 bucket.

I have already established connection. i need help with the code of pyspark to run on the custer that can help change this.
Oct 4, 2019 in Apache Spark by anonymous
• 160 points

1 answer to this question.

+1 vote

You can do this by turning off the default escape of double-quote character with a backslash (\). Try the following:

.option("header", true)
.option("quote", "\u0000") 
.csv("Path to csv file")

What this does is, changes the "quote" character to the null character(Unicode \u0000). 

answered Oct 4, 2019 by Jisha

Related Questions In Apache Spark

0 votes
1 answer
+1 vote
1 answer

How to convert JSON file to AVRO file and vise versa

Try including the package while starting the ...READ MORE

answered Aug 26, 2019 in Apache Spark by Karan
+15 votes
2 answers

Git management technique when there are multiple customers and need multiple customization?

Consider this - In 'extended' Git-Flow, (Git-Multi-Flow, ...READ MORE

answered Mar 27, 2018 in DevOps & Agile by DragonLord999
• 8,450 points
+2 votes
1 answer
0 votes
1 answer

How to remove the elements with a key present in any other RDD?

Hey, You can use the subtractByKey () function to ...READ MORE

answered Jul 22, 2019 in Apache Spark by Gitika
• 65,970 points
+1 vote
1 answer

How to read a data from text file in Spark?

Hey, You can try this: from pyspark import SparkContext SparkContext.stop(sc) sc ...READ MORE

answered Aug 6, 2019 in Apache Spark by Gitika
• 65,970 points