Trending questions in Apache Spark

0 votes
1 answer

How to use Spark jars for Yarn distribution?

First, store upload this archive to hdfs and ...READ MORE

Mar 28 in Apache Spark by Raj
 • 182 views
0 votes
1 answer

How to set executors for static allocation in Spark Yarn?

Open Spark shell and run the following ...READ MORE

Mar 28 in Apache Spark by Raj
 • 176 views
0 votes
1 answer

Spark Yarn: Changing maximum number of time to submit application

By default, the maximum number of times ...READ MORE

Mar 28 in Apache Spark by Raj
 • 164 views
0 votes
1 answer

How to use ftp scheme using Yarn in Spark application?

In case Yarn does not support schemes ...READ MORE

Mar 28 in Apache Spark by Raj
 • 131 views
0 votes
1 answer

Set archives to be extracted in executor directory

I don't think you can copy and ...READ MORE

Mar 28 in Apache Spark by Raj
 • 95 views
0 votes
1 answer

what are the spark job and spark task and spark staging ?

In a Spark application, when you invoke ...READ MORE

Mar 18 in Apache Spark by Pavan
 • 520 views
0 votes
1 answer

How to enable worker cleanup in Spark?

To enable cleanup, open the spark shell ...READ MORE

Mar 25 in Apache Spark by Hari
 • 197 views
0 votes
1 answer

Need help setting Spark yarn history server address

If you are running history server and ...READ MORE

Mar 26 in Apache Spark by Neha
 • 127 views
0 votes
1 answer

Thread to use Yarn application master is limited

This is because the maximum number of ...READ MORE

Mar 28 in Apache Spark by Raj
 • 47 views
0 votes
1 answer

How to increase HDFS replication level in Spark?

Hi @Raunak. You can change the replication ...READ MORE

Mar 26 in Apache Spark by Yash
 • 101 views
0 votes
1 answer

Set Library to launch Yarn master

You can make use of Special Library path to ...READ MORE

Mar 28 in Apache Spark by Raj
 • 20 views
0 votes
1 answer

Increase Yarn wait time for Sparkcontext

The default time that the Yarn application waits ...READ MORE

Mar 26 in Apache Spark by Rohit
 • 71 views
0 votes
1 answer

Increase cores for yarn in Spark application

By default, only one core is used for ...READ MORE

Mar 26 in Apache Spark by Bhuvan
 • 60 views
0 votes
1 answer

How to retain Spark jar and app jar after staging?

By default, Spark jar, app jar, and ...READ MORE

Mar 26 in Apache Spark by Ginni
 • 31 views
0 votes
1 answer

Increasing memory to use for Yarn application master?

You can increase the memory dynamically by ...READ MORE

Mar 26 in Apache Spark by Tina
 • 29 views
0 votes
1 answer

How to cleanup application work directories faster?

By default, the cleanup time is set ...READ MORE

Mar 26 in Apache Spark by Jyoti
 • 19 views
0 votes
1 answer

Setting directory for Spark recovery

You can set the directory to store ...READ MORE

Mar 25 in Apache Spark by Hari
 • 59 views
0 votes
1 answer

Limit displaying completed applications

By default, the number of completed applications ...READ MORE

Mar 25 in Apache Spark by hari
 • 46 views
0 votes
1 answer

How to change worker cleanup interval?

The default interval time is 1800 seconds ...READ MORE

Mar 25 in Apache Spark by Hari
 • 36 views
0 votes
1 answer

How to set Spark recovery to Zookeeper?

There is a property of Spark which ...READ MORE

Mar 25 in Apache Spark by Hari
 • 39 views
0 votes
1 answer

How to disable automatic remove of application of failures?

Yes, you have read it right. The ...READ MORE

Mar 25 in Apache Spark by Hari
 • 31 views
0 votes
1 answer

How to connect to Zookeeper after setting Spark recovery mode?

You have set Zookeeper as the recovery ...READ MORE

Mar 25 in Apache Spark by Hari
 • 32 views
0 votes
1 answer

Change number of completed drivers displayed

You can change the number of completed ...READ MORE

Mar 25 in Apache Spark by Hari
 • 19 views
0 votes
1 answer

How to spread applications to less nodes?

You can limit the spread out by ...READ MORE

Mar 25 in Apache Spark by Hari
 • 18 views
0 votes
1 answer

what are the job optimization Technics in spark and scala ?

There are different methods to achieve optimization ...READ MORE

Mar 18 in Apache Spark by Veer
 • 305 views
0 votes
0 answers

what are the memory issues in spark ?

Mar 17 in Apache Spark by satish kumar
 • 180 points • 320 views
0 votes
1 answer

Set Spark executable for R scripts

To change the default executable, assign the ...READ MORE

Mar 19 in Apache Spark by Jai
 • 98 views
0 votes
1 answer

Need to disable unpersist in Spark

You can dynamically change this function by ...READ MORE

Mar 19 in Apache Spark by Jai
 • 52 views
0 votes
1 answer

How to set cpu cores for spark task?

By default, each task is allocated with ...READ MORE

Mar 12 in Apache Spark by Veer
 • 355 views
0 votes
1 answer

How to get Spark SQL configuration?

First create a Spark session like this: val ...READ MORE

Mar 18 in Apache Spark by John
 • 93 views
0 votes
1 answer

Spark to check if a particular string exists in a file

You can use this: lines = sc.textFile(“hdfs://path/to/file/filename.txt”); def isFound(line): if ...READ MORE

Mar 15 in Apache Spark by Raj
 • 213 views
0 votes
1 answer

Prevent immediate shutdown of Streaming context after JVM shutdown

In technical terms, you want to gracefully shut down the ...READ MORE

Mar 19 in Apache Spark by Jai
 • 38 views
0 votes
1 answer

How to enable write ahead logs for receivers?

To enable write-ahead logs, run the following ...READ MORE

Mar 19 in Apache Spark by Jai
 • 30 views
0 votes
1 answer

Retaining the batch by status APIs before Garbage Collection

By default, 1000 batches are retained by ...READ MORE

Mar 19 in Apache Spark by Jai
 • 27 views
0 votes
1 answer

Set maximum reading rate from Kafka partition

You can do it by using the ...READ MORE

Mar 19 in Apache Spark by Jai
 • 27 views
0 votes
1 answer

How to get SQL configuration in Spark using Python?

You can get the configuration details through ...READ MORE

Mar 18 in Apache Spark by John
 • 69 views
0 votes
1 answer

Change heart beat time to R process

The heartbeat interval is assigned to the ...READ MORE

Mar 19 in Apache Spark by Jai
 • 24 views
0 votes
1 answer

Change number of threads use by R back end

Refer to the below commands to know ...READ MORE

Mar 19 in Apache Spark by Jai
 • 23 views
0 votes
1 answer

Close file after writing write-ahead on driver

You can change the property to close ...READ MORE

Mar 19 in Apache Spark by Jai
 • 21 views
0 votes
1 answer

How to change minimum rate per kafka partition?

Open spark shell using this command: $ spark-shell Then ...READ MORE

Mar 19 in Apache Spark by Jai
 • 21 views
0 votes
1 answer

Get Spark SQL configuration in Java

You will need to use Spark session ...READ MORE

Mar 18 in Apache Spark by John
 • 30 views
0 votes
1 answer

Using R to display configuration of Spark SQL

Try the below-mentioned code. sparkR.session() properties <- sql("SET -v") showDF(properties, ...READ MORE

Mar 18 in Apache Spark by John
 • 20 views
0 votes
1 answer

How to set maximum receiving rate for backpressure mechanism?

You can set the maximum receiving rate ...READ MORE

Mar 18 in Apache Spark by John
 • 20 views
0 votes
1 answer

Disable Web UI for Spark Application

You can disable it like this: val sc ...READ MORE

Mar 6 in Apache Spark by Rohit
 • 485 views
0 votes
1 answer

How to change commiter algorithm version in Spark?

To change to version 2, run the ...READ MORE

Mar 10 in Apache Spark by Siri
 • 336 views
0 votes
1 answer

How to access private key password with Spark?

Spark allows you to retrieve the key ...READ MORE

Mar 15 in Apache Spark by Karan
 • 65 views
0 votes
1 answer

How to increase Spark listener bus event queue capacity?

The default capacity of listener bus is ...READ MORE

Mar 11 in Apache Spark by Raj
 • 231 views
0 votes
1 answer

Cache() vs persist() in Spark

The cache() is used only the default storage level ...READ MORE

Mar 8 in Apache Spark by Raj
 • 370 views
0 votes
1 answer

Can I set different protocol for SSL in Spark?

There is no protocol set by default. ...READ MORE

Mar 15 in Apache Spark by Karan
 • 54 views
0 votes
1 answer

How to set client authentication for SSL?

By default, this feature is disabled. To ...READ MORE

Mar 15 in Apache Spark by Karan
 • 49 views
Page:

Most popular tags

"PMP®","PMI®", "PMI-ACP®" and "PMBOK®" are registered marks of the Project Management Institute, Inc. MongoDB®, Mongo and the leaf logo are the registered trademarks of MongoDB, Inc.