Trending questions in Apache Spark

0 votes
1 answer

What does reduce action do in Spark?

Hey, Reduce action converts an RDD to a ...READ MORE

Jul 4, 2019 in Apache Spark by Gitika
• 65,910 points
456 views
0 votes
1 answer

How to print loop with condition in scala?

Hi, Yes, in scala there is a guard condition where ...READ MORE

Jul 6, 2019 in Apache Spark by Gitika
• 65,910 points
394 views
0 votes
1 answer

By which components spark ecosystem libraries are composed of?

Hi, Spark ecosystem libraries are composed of various ...READ MORE

Jul 1, 2019 in Apache Spark by Gitika
• 65,910 points
490 views
0 votes
1 answer

Difference between RDD as val and var

Variable declaration can be done in two ...READ MORE

May 23, 2019 in Apache Spark by Arun
2,152 views
0 votes
1 answer

what are the spark real time issues ?

Some of the issues I have faced ...READ MORE

Mar 18, 2019 in Apache Spark by Sharman
4,910 views
0 votes
1 answer

Spark CLI issue

For spark.read.textFile we need spark-2.x. Please try ...READ MORE

Jun 19, 2019 in Apache Spark by Maahi
502 views
0 votes
1 answer

Doubt in display(id, name, salary) before display function

The statement display(id, name, salary) is written before the display function ...READ MORE

Jun 19, 2019 in Apache Spark by Ritu
395 views
0 votes
1 answer

How to set extra JVM options for Spark application?

You cans set extra JVM options that ...READ MORE

Mar 28, 2019 in Apache Spark by Raj
3,697 views
0 votes
1 answer

How to store files in executor's working directory?

You have to specify a comma-separated list ...READ MORE

Mar 28, 2019 in Apache Spark by Raj
3,650 views
+1 vote
1 answer

map vs mapValues in Spark

There is a difference between the two: mapValues ...READ MORE

Jun 29, 2018 in Apache Spark by nitinrawat895
• 11,380 points
15,306 views
0 votes
1 answer

How to set cpu cores for spark task?

By default, each task is allocated with ...READ MORE

Mar 12, 2019 in Apache Spark by Veer
4,103 views
0 votes
1 answer

Starting Spark in Windows

Run below commands spark-class org.apache.spark.deploy.master.Master spark-class org.apache.spark.deploy.worker.Worker spark://192.168.254.1:7077 NOTE: The ...READ MORE

May 22, 2019 in Apache Spark by Reshma
797 views
0 votes
1 answer

Spark2-submit does not generate output file.

To generate the output file, you can ...READ MORE

Feb 24, 2019 in Apache Spark by Esha
4,469 views
0 votes
1 answer

How to change the location of Spark event logs?

You can change the location where you ...READ MORE

Mar 6, 2019 in Apache Spark by Rohit
3,997 views
0 votes
1 answer

Starting Spark Scala console

To get command prompt for Scala open ...READ MORE

May 24, 2019 in Apache Spark by Cassy
552 views
0 votes
1 answer

How to get Spark SQL configuration?

First create a Spark session like this: val ...READ MORE

Mar 18, 2019 in Apache Spark by John
3,112 views
0 votes
1 answer

What is Executor Memory in a Spark application?

Every spark application has same fixed heap ...READ MORE

Jan 5, 2019 in Apache Spark by Frankie
• 9,830 points
6,169 views
0 votes
1 answer

What is Spark Core?

It is not like a CPU to ...READ MORE

Mar 8, 2019 in Apache Spark by Raj
3,253 views
0 votes
1 answer

SparkContext.addFile() not able to update file.

Spark by default won't let you overwrite ...READ MORE

Mar 10, 2019 in Apache Spark by Siri
3,103 views
0 votes
1 answer

How to enable worker cleanup in Spark?

To enable cleanup, open the spark shell ...READ MORE

Mar 25, 2019 in Apache Spark by Hari
2,268 views
0 votes
1 answer

How to change commiter algorithm version in Spark?

To change to version 2, run the ...READ MORE

Mar 10, 2019 in Apache Spark by Siri
2,847 views
0 votes
1 answer

Disable Web UI for Spark Application

You can disable it like this: val sc ...READ MORE

Mar 6, 2019 in Apache Spark by Rohit
2,980 views
0 votes
3 answers

Can anyone explain fold() operation in Spark?

Fold in spark Fold is a very powerful ...READ MORE

Aug 23, 2018 in Apache Spark by samarth295
• 2,220 points
11,895 views
0 votes
1 answer

Increase Yarn wait time for Sparkcontext

The default time that the Yarn application waits ...READ MORE

Mar 27, 2019 in Apache Spark by Rohit
1,907 views
0 votes
1 answer

How can we use spark shell for scala without cluster?

You can run the Spark shell for ...READ MORE

Apr 28, 2019 in Apache Spark by Giri
490 views
0 votes
1 answer

Set archives to be extracted in executor directory

I don't think you can copy and ...READ MORE

Mar 28, 2019 in Apache Spark by Raj
1,669 views
0 votes
1 answer

How to enable Spark event logging?

To make Spark store the event logs, ...READ MORE

Mar 6, 2019 in Apache Spark by Rohit
2,600 views
0 votes
1 answer

Spark Yarn: Changing maximum number of time to submit application

By default, the maximum number of times ...READ MORE

Mar 28, 2019 in Apache Spark by Raj
1,655 views
0 votes
1 answer

How to enable SSL for Spark application?

You can do it dynamically like this: val ...READ MORE

Mar 15, 2019 in Apache Spark by Karan
2,162 views
0 votes
1 answer

Need help setting Spark yarn history server address

If you are running history server and ...READ MORE

Mar 27, 2019 in Apache Spark by Neha
1,612 views
0 votes
1 answer

what are the job optimization Technics in spark and scala ?

There are different methods to achieve optimization ...READ MORE

Mar 18, 2019 in Apache Spark by Veer
1,889 views
0 votes
1 answer

How to set keystore path?

You have to set the path to ...READ MORE

Mar 15, 2019 in Apache Spark by Karan
1,954 views
0 votes
1 answer

How to make Spark wait for more time for acknowledgement?

Use the following command to increase the ...READ MORE

Mar 11, 2019 in Apache Spark by Raj
2,099 views
0 votes
1 answer

How to change scheduling mode in Spark?

You can change the scheduling mode as ...READ MORE

Mar 12, 2019 in Apache Spark by Raj
2,087 views
0 votes
1 answer

How to set executors for static allocation in Spark Yarn?

Open Spark shell and run the following ...READ MORE

Mar 28, 2019 in Apache Spark by Raj
1,325 views
0 votes
1 answer

How to increase HDFS replication level in Spark?

Hi @Raunak. You can change the replication ...READ MORE

Mar 27, 2019 in Apache Spark by Yash
1,337 views
0 votes
1 answer

How to disable executor from fetching file from cache?

When a Spark application is running, the ...READ MORE

Mar 10, 2019 in Apache Spark by Siri
2,049 views
0 votes
1 answer

How to use Spark jars for Yarn distribution?

First, store upload this archive to hdfs and ...READ MORE

Mar 28, 2019 in Apache Spark by Raj
1,252 views
0 votes
1 answer

How to authenticate Spark internal connections using a secret key?

You need to set the secret key ...READ MORE

Mar 13, 2019 in Apache Spark by Venu
1,872 views
0 votes
0 answers

what are the memory issues in spark ?

Mar 18, 2019 in Apache Spark by satish kumar
• 180 points
1,697 views
0 votes
2 answers

In a Spark DataFrame how can I flatten the struct?

// Collect data from input avro file ...READ MORE

Jul 4, 2019 in Apache Spark by Dhara dhruve
5,676 views
0 votes
1 answer

Apache Spark, usage of yield.

Yield is used in sequence comprehensions. It is ...READ MORE

Feb 22, 2019 in Apache Spark by Saruj
2,675 views
0 votes
1 answer

How do spark extra listeners work?

Yes. You can use extra listeners by setting ...READ MORE

Feb 24, 2019 in Apache Spark by Rishi
2,576 views
0 votes
1 answer

How to access private key password with Spark?

Spark allows you to retrieve the key ...READ MORE

Mar 15, 2019 in Apache Spark by Karan
1,580 views
0 votes
1 answer

Increasing memory to use for Yarn application master?

You can increase the memory dynamically by ...READ MORE

Mar 26, 2019 in Apache Spark by Tina
1,094 views
0 votes
1 answer

Thread to use Yarn application master is limited

This is because the maximum number of ...READ MORE

Mar 28, 2019 in Apache Spark by Raj
997 views
0 votes
1 answer

How to connect to Zookeeper after setting Spark recovery mode?

You have set Zookeeper as the recovery ...READ MORE

Mar 25, 2019 in Apache Spark by Hari
1,099 views
0 votes
1 answer

How to set max executors for dynamic allocation?

You can set it by assigning the ...READ MORE

Mar 13, 2019 in Apache Spark by Venu
1,594 views
0 votes
1 answer

How to use ftp scheme using Yarn in Spark application?

In case Yarn does not support schemes ...READ MORE

Mar 28, 2019 in Apache Spark by Raj
908 views
0 votes
1 answer

Increase cores for yarn in Spark application

By default, only one core is used for ...READ MORE

Mar 26, 2019 in Apache Spark by Bhuvan
966 views