How to parse a textFile to csv in pyspark

0 votes
import pyspark

from pyspark import SparkContext

sc=SparkContext()

data=sc.textFile("/user/edureka_968359/appleStore/appleStore.csv")

I have read a csv file as a textfile and now want to parse it to csv
Apr 11, 2020 in Apache Spark by anonymous
• 120 points
4,058 views

1 answer to this question.

0 votes

Hi,

Use this below given code, it will help you.

import pyspark
from pyspark import SparkContext
from pyspark import SparkConf
sc = SparkContext.getOrCreate(SparkConf())
df = sc.textFile('file:/root/read.csv')
df.saveAsTextFile('/root/name.csv')

After this check your output, it will be in csv format.

Hope this helps!

To know more about Pyspark, it's recommended that you join PySpark course online.

Thanks.

answered Apr 13, 2020 by MD
• 95,460 points

Related Questions In Apache Spark

+1 vote
1 answer
0 votes
5 answers

How to change the spark Session configuration in Pyspark?

You aren't actually overwriting anything with this ...READ MORE

answered Dec 14, 2020 in Apache Spark by Gitika
• 65,890 points
125,186 views
0 votes
1 answer
0 votes
1 answer

How to add third party java jars for use in PySpark?

You can add external jars as arguments ...READ MORE

answered Jul 4, 2018 in Apache Spark by nitinrawat895
• 11,380 points

edited Nov 19, 2021 by Sarfaraz 8,642 views
0 votes
1 answer

How to get ID of a map task in Spark?

you can access task information using TaskContext: import org.apache.spark.TaskContext sc.parallelize(Seq[Int](), ...READ MORE

answered Nov 20, 2018 in Apache Spark by Frankie
• 9,830 points
3,390 views
0 votes
1 answer

How to restrict a group to only view in Spark?

You can do it dynamically be setting ...READ MORE

answered Mar 15, 2019 in Apache Spark by Raj
601 views
0 votes
2 answers

How to execute a function in apache-scala?

Function Definition : def test():Unit{ var a=10 var b=20 var c=a+b } calling ...READ MORE

answered Aug 5, 2020 in Apache Spark by Ramkumar Ramasamy
947 views
0 votes
1 answer

How to load data of .csv file in MySQL Database Table?

You can do it using a code ...READ MORE

answered Jul 22, 2019 in Apache Spark by Vishwa
1,860 views
0 votes
1 answer
0 votes
1 answer

How to create a not null column in case class in spark

Hi@Deepak, In your test class you passed empid ...READ MORE

answered May 14, 2020 in Apache Spark by MD
• 95,460 points
4,925 views
webinar REGISTER FOR FREE WEBINAR X
REGISTER NOW
webinar_success Thank you for registering Join Edureka Meetup community for 100+ Free Webinars each month JOIN MEETUP GROUP