How to parse a textFile to csv in pyspark

0 votes
import pyspark

from pyspark import SparkContext

sc=SparkContext()

data=sc.textFile("/user/edureka_968359/appleStore/appleStore.csv")

I have read a csv file as a textfile and now want to parse it to csv
Apr 11, 2020 in Apache Spark by anonymous
• 120 points
2,130 views

1 answer to this question.

0 votes

Hi,

Use this below given code, it will help you.

import pyspark
from pyspark import SparkContext
from pyspark import SparkConf
sc = SparkContext.getOrCreate(SparkConf())
df = sc.textFile('file:/root/read.csv')
df.saveAsTextFile('/root/name.csv')

After this check your output, it will be in csv format.

Hope this helps!

To know more about Pyspark, it's recommended that you join Pyspark course online.

Thanks.

answered Apr 13, 2020 by MD
• 95,340 points

Related Questions In Apache Spark

+1 vote
1 answer
0 votes
5 answers

How to change the spark Session configuration in Pyspark?

You aren't actually overwriting anything with this ...READ MORE

answered Dec 14, 2020 in Apache Spark by Gitika
• 65,850 points
80,090 views
0 votes
1 answer
0 votes
1 answer

How to add third party java jars for use in PySpark?

You can add external jars as arguments ...READ MORE

answered Jul 4, 2018 in Apache Spark by nitinrawat895
• 11,380 points

edited Nov 19, 2021 by Sarfaraz 7,007 views
0 votes
1 answer

How to get ID of a map task in Spark?

you can access task information using TaskContext: import org.apache.spark.TaskContext sc.parallelize(Seq[Int](), ...READ MORE

answered Nov 20, 2018 in Apache Spark by Frankie
• 9,830 points
2,278 views
0 votes
1 answer

How to restrict a group to only view in Spark?

You can do it dynamically be setting ...READ MORE

answered Mar 15, 2019 in Apache Spark by Raj
206 views
0 votes
2 answers

How to execute a function in apache-scala?

Function Definition : def test():Unit{ var a=10 var b=20 var c=a+b } calling ...READ MORE

answered Aug 5, 2020 in Apache Spark by Ramkumar Ramasamy
286 views
0 votes
1 answer

How to load data of .csv file in MySQL Database Table?

You can do it using a code ...READ MORE

answered Jul 22, 2019 in Apache Spark by Vishwa
1,122 views
0 votes
1 answer
0 votes
1 answer

How to create a not null column in case class in spark

Hi@Deepak, In your test class you passed empid ...READ MORE

answered May 14, 2020 in Apache Spark by MD
• 95,340 points
2,420 views
webinar REGISTER FOR FREE WEBINAR X
Send OTP
REGISTER NOW
webinar_success Thank you for registering Join Edureka Meetup community for 100+ Free Webinars each month JOIN MEETUP GROUP