Trending questions in Apache Spark

0 votes
1 answer

How to use Spark jars for Yarn distribution?

First, store upload this archive to hdfs and ...READ MORE

Mar 28 in Apache Spark by Raj
182 views
0 votes
1 answer

How to set executors for static allocation in Spark Yarn?

Open Spark shell and run the following ...READ MORE

Mar 28 in Apache Spark by Raj
176 views
0 votes
1 answer

Spark Yarn: Changing maximum number of time to submit application

By default, the maximum number of times ...READ MORE

Mar 28 in Apache Spark by Raj
164 views
0 votes
1 answer

How to use ftp scheme using Yarn in Spark application?

In case Yarn does not support schemes ...READ MORE

Mar 28 in Apache Spark by Raj
131 views
0 votes
1 answer

Set archives to be extracted in executor directory

I don't think you can copy and ...READ MORE

Mar 28 in Apache Spark by Raj
95 views
0 votes
1 answer

what are the spark job and spark task and spark staging ?

In a Spark application, when you invoke ...READ MORE

Mar 18 in Apache Spark by Pavan
520 views
0 votes
1 answer

How to enable worker cleanup in Spark?

To enable cleanup, open the spark shell ...READ MORE

Mar 25 in Apache Spark by Hari
197 views
0 votes
1 answer

Need help setting Spark yarn history server address

If you are running history server and ...READ MORE

Mar 26 in Apache Spark by Neha
127 views
0 votes
1 answer

Thread to use Yarn application master is limited

This is because the maximum number of ...READ MORE

Mar 28 in Apache Spark by Raj
47 views
0 votes
1 answer

How to increase HDFS replication level in Spark?

Hi @Raunak. You can change the replication ...READ MORE

Mar 26 in Apache Spark by Yash
101 views
0 votes
1 answer

Set Library to launch Yarn master

You can make use of Special Library path to ...READ MORE

Mar 28 in Apache Spark by Raj
20 views
0 votes
1 answer

Increase Yarn wait time for Sparkcontext

The default time that the Yarn application waits ...READ MORE

Mar 26 in Apache Spark by Rohit
71 views
0 votes
1 answer

Increase cores for yarn in Spark application

By default, only one core is used for ...READ MORE

Mar 26 in Apache Spark by Bhuvan
60 views
0 votes
1 answer

How to retain Spark jar and app jar after staging?

By default, Spark jar, app jar, and ...READ MORE

Mar 26 in Apache Spark by Ginni
31 views
0 votes
1 answer

Increasing memory to use for Yarn application master?

You can increase the memory dynamically by ...READ MORE

Mar 26 in Apache Spark by Tina
29 views
0 votes
1 answer

How to cleanup application work directories faster?

By default, the cleanup time is set ...READ MORE

Mar 26 in Apache Spark by Jyoti
19 views
0 votes
1 answer

Setting directory for Spark recovery

You can set the directory to store ...READ MORE

Mar 25 in Apache Spark by Hari
59 views
0 votes
1 answer

Limit displaying completed applications

By default, the number of completed applications ...READ MORE

Mar 25 in Apache Spark by hari
46 views
0 votes
1 answer

How to change worker cleanup interval?

The default interval time is 1800 seconds ...READ MORE

Mar 25 in Apache Spark by Hari
36 views
0 votes
1 answer

How to set Spark recovery to Zookeeper?

There is a property of Spark which ...READ MORE

Mar 25 in Apache Spark by Hari
39 views
0 votes
1 answer

How to disable automatic remove of application of failures?

Yes, you have read it right. The ...READ MORE

Mar 25 in Apache Spark by Hari
31 views
0 votes
1 answer

How to connect to Zookeeper after setting Spark recovery mode?

You have set Zookeeper as the recovery ...READ MORE

Mar 25 in Apache Spark by Hari
32 views
0 votes
1 answer

Change number of completed drivers displayed

You can change the number of completed ...READ MORE

Mar 25 in Apache Spark by Hari
19 views
0 votes
1 answer

How to spread applications to less nodes?

You can limit the spread out by ...READ MORE

Mar 25 in Apache Spark by Hari
18 views
0 votes
1 answer

what are the job optimization Technics in spark and scala ?

There are different methods to achieve optimization ...READ MORE

Mar 18 in Apache Spark by Veer
305 views
0 votes
0 answers

what are the memory issues in spark ?

Mar 17 in Apache Spark by satish kumar
• 180 points
320 views
0 votes
1 answer

Set Spark executable for R scripts

To change the default executable, assign the ...READ MORE

Mar 19 in Apache Spark by Jai
98 views
0 votes
1 answer

Need to disable unpersist in Spark

You can dynamically change this function by ...READ MORE

Mar 19 in Apache Spark by Jai
52 views
0 votes
1 answer

How to set cpu cores for spark task?

By default, each task is allocated with ...READ MORE

Mar 12 in Apache Spark by Veer
355 views
0 votes
1 answer

How to get Spark SQL configuration?

First create a Spark session like this: val ...READ MORE

Mar 18 in Apache Spark by John
93 views
0 votes
1 answer

Spark to check if a particular string exists in a file

You can use this: lines = sc.textFile(“hdfs://path/to/file/filename.txt”); def isFound(line): if ...READ MORE

Mar 15 in Apache Spark by Raj
213 views
0 votes
1 answer

Prevent immediate shutdown of Streaming context after JVM shutdown

In technical terms, you want to gracefully shut down the ...READ MORE

Mar 19 in Apache Spark by Jai
38 views
0 votes
1 answer

How to enable write ahead logs for receivers?

To enable write-ahead logs, run the following ...READ MORE

Mar 19 in Apache Spark by Jai
30 views
0 votes
1 answer

Retaining the batch by status APIs before Garbage Collection

By default, 1000 batches are retained by ...READ MORE

Mar 19 in Apache Spark by Jai
27 views
0 votes
1 answer

Set maximum reading rate from Kafka partition

You can do it by using the ...READ MORE

Mar 19 in Apache Spark by Jai
27 views
0 votes
1 answer

How to get SQL configuration in Spark using Python?

You can get the configuration details through ...READ MORE

Mar 18 in Apache Spark by John
69 views
0 votes
1 answer

Change heart beat time to R process

The heartbeat interval is assigned to the ...READ MORE

Mar 19 in Apache Spark by Jai
24 views
0 votes
1 answer

Change number of threads use by R back end

Refer to the below commands to know ...READ MORE

Mar 19 in Apache Spark by Jai
23 views
0 votes
1 answer

Close file after writing write-ahead on driver

You can change the property to close ...READ MORE

Mar 19 in Apache Spark by Jai
21 views
0 votes
1 answer

How to change minimum rate per kafka partition?

Open spark shell using this command: $ spark-shell Then ...READ MORE

Mar 19 in Apache Spark by Jai
21 views
0 votes
1 answer

Get Spark SQL configuration in Java

You will need to use Spark session ...READ MORE

Mar 18 in Apache Spark by John
30 views
0 votes
1 answer

Using R to display configuration of Spark SQL

Try the below-mentioned code. sparkR.session() properties <- sql("SET -v") showDF(properties, ...READ MORE

Mar 18 in Apache Spark by John
20 views
0 votes
1 answer

How to set maximum receiving rate for backpressure mechanism?

You can set the maximum receiving rate ...READ MORE

Mar 18 in Apache Spark by John
20 views
0 votes
1 answer

Disable Web UI for Spark Application

You can disable it like this: val sc ...READ MORE

Mar 6 in Apache Spark by Rohit
485 views
0 votes
1 answer

How to change commiter algorithm version in Spark?

To change to version 2, run the ...READ MORE

Mar 10 in Apache Spark by Siri
336 views
0 votes
1 answer

How to access private key password with Spark?

Spark allows you to retrieve the key ...READ MORE

Mar 15 in Apache Spark by Karan
65 views
0 votes
1 answer

How to increase Spark listener bus event queue capacity?

The default capacity of listener bus is ...READ MORE

Mar 11 in Apache Spark by Raj
231 views
0 votes
1 answer

Cache() vs persist() in Spark

The cache() is used only the default storage level ...READ MORE

Mar 8 in Apache Spark by Raj
370 views
0 votes
1 answer

Can I set different protocol for SSL in Spark?

There is no protocol set by default. ...READ MORE

Mar 15 in Apache Spark by Karan
54 views
0 votes
1 answer

How to set client authentication for SSL?

By default, this feature is disabled. To ...READ MORE

Mar 15 in Apache Spark by Karan
49 views