When not to use foreachPartition and mapPartition?

0 votes
Are there any scenarios where the use of map() and foreach() is preferred over mapPartition and foreachPartition?
Apr 30, 2018 in Apache Spark by shams
• 3,580 points
1,992 views

1 answer to this question.

0 votes
With mapPartion() or foreachPartition(), you can only modify/iterate the partition data. Nodes can't be invoked while executing the code as it will be executed on the executors.  This code should be executed only from the driver node. Thus only from the driver code you can access dataframes or spark session.
answered Apr 30, 2018 by Data_Nerd
• 2,360 points

Related Questions In Apache Spark

0 votes
1 answer

Scala: when to use x(2) and x._2?

In the above statement, x(2) is specifying an array ...READ MORE

answered Jul 22 in Apache Spark by Yogi
26 views
0 votes
1 answer

Not able to use sc in spark shell

Seems like master and worker are not ...READ MORE

answered Jan 3 in Apache Spark by Omkar
• 67,290 points
117 views
0 votes
1 answer
0 votes
0 answers
0 votes
1 answer

How can we optimize and minimize the memory when work with scala use case?

Hi, There is a term in Scala that is ...READ MORE

answered Jul 5 in Apache Spark by Gitika
• 25,300 points
29 views
+5 votes
11 answers

Concatenate columns in apache spark dataframe

its late but this how you can ...READ MORE

answered Mar 21 in Apache Spark by anonymous
25,587 views
0 votes
1 answer

Changing Column position in spark dataframe

Yes, you can reorder the dataframe elements. You need ...READ MORE

answered Apr 19, 2018 in Apache Spark by Ashish
• 2,630 points
3,836 views
0 votes
1 answer

Which query to use for better performance, join in SQL or using Dataset API?

DataFrames and SparkSQL performed almost about the ...READ MORE

answered Apr 19, 2018 in Apache Spark by kurt_cobain
• 9,240 points
99 views
0 votes
1 answer

Can I read a CSV represented as a string into Apache Spark?

You can use the following command. This ...READ MORE

answered May 3, 2018 in Apache Spark by kurt_cobain
• 9,240 points
48 views
0 votes
1 answer

Is it possible to run Spark and Mesos along with Hadoop?

Yes, it is possible to run Spark ...READ MORE

answered May 29, 2018 in Apache Spark by Data_Nerd
• 2,360 points
39 views