Scala: save filtered data row by row using saveAsTextFile

0 votes

Hello Team,

I want to save filtered data row by row using saveAsTextFile. Kindly help.

I tried with flatmap but it's flattening every column.

val rdd1=sc.textFile("/user/edureka_40114/AppleStore.csv")

val rdd3=apple.map(x=>x.split(",")).filter(x=>x(12).equals("\"Games\"")).map(x=>(x(0),x(1)))

rdd3.collect()

I want the output to be saved in hdfs file as below but using flatmap it's giving everything in a separate line.

"1","281656475"

"6","283619399"

"10","284736660"
Aug 2 in Apache Spark by Hari
16 views

1 answer to this question.

0 votes

Try this code, it worked for me:

val sqlContext = new org.apache.spark.sql.SQLContext(sc)

import sqlContext.implicits._​

val data = sqlContext.read.format("csv").option("header", "false").load("/user/edureka_425640/AppleStore.csv")​

data.write.format("csv").save("mobil_out")​
answered Aug 2 by Karan

Related Questions In Apache Spark

0 votes
1 answer

Scala: CSV file to Save data into HBase

Check the reference code mentioned below: def main(args: ...READ MORE

answered Jul 25 in Apache Spark by Hari
22 views
0 votes
1 answer

Scala pass input data as arguments

Please refer to the below code as ...READ MORE

answered Jun 19 in Apache Spark by Lisa
29 views
0 votes
1 answer

How can we iterate any function using "foreach" function in scala?

Hi, Yes, "foreach" function you use because it will ...READ MORE

answered Jul 5 in Apache Spark by Gitika
• 25,300 points
32 views
0 votes
0 answers

Need to load 40 GB data to elasticsearch using spark

I am working in psedo distributed spark ...READ MORE

Jul 16 in Apache Spark by Amit
• 120 points
37 views
0 votes
1 answer
0 votes
1 answer

Hadoop Mapreduce word count Program

Firstly you need to understand the concept ...READ MORE

answered Mar 16, 2018 in Data Analytics by nitinrawat895
• 10,510 points
2,408 views
0 votes
1 answer

hadoop.mapred vs hadoop.mapreduce?

org.apache.hadoop.mapred is the Old API  org.apache.hadoop.mapreduce is the ...READ MORE

answered Mar 16, 2018 in Data Analytics by nitinrawat895
• 10,510 points
246 views
0 votes
10 answers

hadoop fs -put command?

put syntax: put <localSrc> <dest> copy syntax: copyFr ...READ MORE

answered Dec 7, 2018 in Big Data Hadoop by Aditya
12,237 views
0 votes
1 answer

Spark comparing two big data files using scala

Try this and see if this does ...READ MORE

answered Apr 2 in Apache Spark by Omkar
• 67,290 points
193 views
0 votes
1 answer

Scala: Convert text file data into ORC format using data frame

Converting text file to Orc: Using Spark, the ...READ MORE

answered Aug 1 in Apache Spark by Esha
16 views