When not to use foreachPartition and mapPartition?

0 votes
Are there any scenarios where the use of map() and foreach() is preferred over mapPartition and foreachPartition?
Apr 30, 2018 in Apache Spark by shams
• 3,630 points
4,513 views

1 answer to this question.

0 votes
With mapPartion() or foreachPartition(), you can only modify/iterate the partition data. Nodes can't be invoked while executing the code as it will be executed on the executors.  This code should be executed only from the driver node. Thus only from the driver code you can access dataframes or spark session.
answered Apr 30, 2018 by Data_Nerd
• 2,390 points

Related Questions In Apache Spark

0 votes
1 answer

Scala: when to use x(2) and x._2?

In the above statement, x(2) is specifying an array ...READ MORE

answered Jul 22, 2019 in Apache Spark by Yogi
241 views
–1 vote
1 answer

Not able to use sc in spark shell

Seems like master and worker are not ...READ MORE

answered Jan 3, 2019 in Apache Spark by Omkar
• 69,030 points
448 views
0 votes
1 answer

where can i get spark-terasort.jar and not .scala file, to do spark terasort in windows.

Hi! I found 2 links on github where ...READ MORE

answered Feb 13, 2019 in Apache Spark by Omkar
• 69,030 points
382 views
0 votes
0 answers

How can we optimize and minimize the memory when work with scala use case?

When we calculate some use case with ...READ MORE

Jul 5, 2019 in Apache Spark by nilam
101 views
0 votes
1 answer

How can we optimize and minimize the memory when work with scala use case?

Hi, There is a term in Scala that is ...READ MORE

answered Jul 5, 2019 in Apache Spark by Gitika
• 36,530 points
104 views
+5 votes
11 answers

Concatenate columns in apache spark dataframe

its late but this how you can ...READ MORE

answered Mar 21, 2019 in Apache Spark by anonymous
55,797 views
0 votes
1 answer

Changing Column position in spark dataframe

Yes, you can reorder the dataframe elements. You need ...READ MORE

answered Apr 19, 2018 in Apache Spark by Ashish
• 2,650 points
8,458 views
0 votes
1 answer

Which query to use for better performance, join in SQL or using Dataset API?

DataFrames and SparkSQL performed almost about the ...READ MORE

answered Apr 19, 2018 in Apache Spark by kurt_cobain
• 9,320 points
357 views
0 votes
1 answer

Can I read a CSV represented as a string into Apache Spark?

You can use the following command. This ...READ MORE

answered May 3, 2018 in Apache Spark by kurt_cobain
• 9,320 points
198 views
0 votes
1 answer

Is it possible to run Spark and Mesos along with Hadoop?

Yes, it is possible to run Spark ...READ MORE

answered May 29, 2018 in Apache Spark by Data_Nerd
• 2,390 points
172 views