How to parse a textFile to csv in pyspark?

0 votes
import pyspark

from pyspark import SparkContext

sc=SparkContext()

data=sc.textFile("/user/edureka_968359/appleStore/appleStore.csv")

I have read a csv file as a textfile and now want to parse it to csv
Apr 11 in Apache Spark by anonymous
• 120 points
93 views

1 answer to this question.

0 votes

Hi,

Use this below given code, it will help you.

import pyspark
from pyspark import SparkContext
from pyspark import SparkConf
sc = SparkContext.getOrCreate(SparkConf())
df = sc.textFile('file:/root/read.csv')
df.saveAsTextFile('/root/name.csv')

After this check your output, it will be in csv format.

answered Apr 13 by MD
• 23,050 points

Related Questions In Apache Spark

+1 vote
1 answer
0 votes
4 answers

How to change the spark Session configuration in Pyspark?

You can dynamically load properties. First create ...READ MORE

answered Dec 10, 2018 in Apache Spark by Vini
30,513 views
0 votes
1 answer
0 votes
1 answer

How to add third party java jars for use in PySpark?

You can add external jars as arguments ...READ MORE

answered Jul 4, 2018 in Apache Spark by nitinrawat895
• 10,920 points
2,960 views
0 votes
1 answer

How to get ID of a map task in Spark?

you can access task information using TaskContext: import org.apache.spark.TaskContext sc.parallelize(Seq[Int](), ...READ MORE

answered Nov 20, 2018 in Apache Spark by Frankie
• 9,810 points
975 views
0 votes
1 answer

How to restrict a group to only view in Spark?

You can do it dynamically be setting ...READ MORE

answered Mar 15, 2019 in Apache Spark by Raj
76 views
0 votes
1 answer

How to execute a function in apache-scala?

Hi, Here is a simple example of how ...READ MORE

answered Jul 1, 2019 in Apache Spark by Gitika
• 29,290 points
78 views
0 votes
1 answer

How to load data of .csv file in MySQL Database Table?

You can do it using a code ...READ MORE

answered Jul 22, 2019 in Apache Spark by Vishwa
234 views
0 votes
1 answer

How to create a not null column in case class in spark

Hi@Deepak, In your test class you passed empid ...READ MORE

answered May 14 in Apache Spark by MD
• 23,050 points
71 views
0 votes
1 answer

How to create multiple producers in apache kafka?

Hi@akhtar, To create multiple producer you have to ...READ MORE

answered Feb 6 in Apache Spark by MD
• 23,050 points
169 views