How to convert a json file structure with values in single quotes to quoteless

Question

Something as this json object:

"Territory": "'CA'",

I have to get rid of ' ' beside CA.

And in here i am moving the so processed file into a s3 bucket.

I have already established connection. i need help with the code of pyspark to run on the custer that can help change this.

Answer 1

You can do this by turning off the default escape of double-quote character with a backslash (\). Try the following:

dataframe
.write
.option("header", true)
.option("quote", "\u0000") 
.csv("Path to csv file")

What this does is, changes the "quote" character to the null character(Unicode \u0000).