When not to use foreachPartition and mapPartition?

0 votes
Are there any scenarios where the use of map() and foreach() is preferred over mapPartition and foreachPartition?
Apr 30, 2018 in Apache Spark by shams
• 3,580 points
2,857 views

1 answer to this question.

0 votes
With mapPartion() or foreachPartition(), you can only modify/iterate the partition data. Nodes can't be invoked while executing the code as it will be executed on the executors.  This code should be executed only from the driver node. Thus only from the driver code you can access dataframes or spark session.
answered Apr 30, 2018 by Data_Nerd
• 2,370 points

Related Questions In Apache Spark

0 votes
1 answer

Scala: when to use x(2) and x._2?

In the above statement, x(2) is specifying an array ...READ MORE

answered Jul 22, 2019 in Apache Spark by Yogi
94 views
–1 vote
1 answer

Not able to use sc in spark shell

Seems like master and worker are not ...READ MORE

answered Jan 3, 2019 in Apache Spark by Omkar
• 68,860 points
232 views
0 votes
1 answer

where can i get spark-terasort.jar and not .scala file, to do spark terasort in windows.

Hi! I found 2 links on github where ...READ MORE

answered Feb 13, 2019 in Apache Spark by Omkar
• 68,860 points
180 views
0 votes
0 answers
0 votes
1 answer

How can we optimize and minimize the memory when work with scala use case?

Hi, There is a term in Scala that is ...READ MORE

answered Jul 5, 2019 in Apache Spark by Gitika
• 25,440 points
52 views
+5 votes
11 answers

Concatenate columns in apache spark dataframe

its late but this how you can ...READ MORE

answered Mar 21, 2019 in Apache Spark by anonymous
39,076 views
0 votes
1 answer

Changing Column position in spark dataframe

Yes, you can reorder the dataframe elements. You need ...READ MORE

answered Apr 19, 2018 in Apache Spark by Ashish
• 2,630 points
5,588 views
0 votes
1 answer

Which query to use for better performance, join in SQL or using Dataset API?

DataFrames and SparkSQL performed almost about the ...READ MORE

answered Apr 19, 2018 in Apache Spark by kurt_cobain
• 9,290 points
162 views
0 votes
1 answer

Can I read a CSV represented as a string into Apache Spark?

You can use the following command. This ...READ MORE

answered May 3, 2018 in Apache Spark by kurt_cobain
• 9,290 points
80 views
0 votes
1 answer

Is it possible to run Spark and Mesos along with Hadoop?

Yes, it is possible to run Spark ...READ MORE

answered May 29, 2018 in Apache Spark by Data_Nerd
• 2,370 points
90 views