How to parse a textFile to csv in pyspark

0 votes
import pyspark

from pyspark import SparkContext

sc=SparkContext()

data=sc.textFile("/user/edureka_968359/appleStore/appleStore.csv")

I have read a csv file as a textfile and now want to parse it to csv
Apr 11, 2020 in Apache Spark by anonymous
• 120 points
1,177 views

1 answer to this question.

0 votes

Hi,

Use this below given code, it will help you.

import pyspark
from pyspark import SparkContext
from pyspark import SparkConf
sc = SparkContext.getOrCreate(SparkConf())
df = sc.textFile('file:/root/read.csv')
df.saveAsTextFile('/root/name.csv')

After this check your output, it will be in csv format.

answered Apr 13, 2020 by MD
• 95,160 points

Related Questions In Apache Spark

+1 vote
1 answer
0 votes
5 answers

How to change the spark Session configuration in Pyspark?

You aren't actually overwriting anything with this ...READ MORE

answered Dec 13, 2020 in Apache Spark by Gitika
• 65,870 points
55,106 views
0 votes
1 answer
0 votes
1 answer

How to add third party java jars for use in PySpark?

You can add external jars as arguments ...READ MORE

answered Jul 4, 2018 in Apache Spark by nitinrawat895
• 11,380 points
5,683 views
0 votes
1 answer

How to get ID of a map task in Spark?

you can access task information using TaskContext: import org.apache.spark.TaskContext sc.parallelize(Seq[Int](), ...READ MORE

answered Nov 20, 2018 in Apache Spark by Frankie
• 9,810 points
1,735 views
0 votes
1 answer

How to restrict a group to only view in Spark?

You can do it dynamically be setting ...READ MORE

answered Mar 15, 2019 in Apache Spark by Raj
135 views
0 votes
2 answers

How to execute a function in apache-scala?

Function Definition : def test():Unit{ var a=10 var b=20 var c=a+b } calling ...READ MORE

answered Aug 5, 2020 in Apache Spark by Ramkumar Ramasamy
194 views
0 votes
1 answer

How to load data of .csv file in MySQL Database Table?

You can do it using a code ...READ MORE

answered Jul 22, 2019 in Apache Spark by Vishwa
821 views
0 votes
1 answer
0 votes
1 answer

How to create a not null column in case class in spark

Hi@Deepak, In your test class you passed empid ...READ MORE

answered May 14, 2020 in Apache Spark by MD
• 95,160 points
1,267 views