Most answered questions in Apache Spark

0 votes
1 answer

How to enable worker cleanup in Spark?

To enable cleanup, open the spark shell ...READ MORE

Mar 25, 2019 in Apache Spark by Hari
2,268 views
0 votes
1 answer

How to increase worker timeout in Spark application?

By default, the timeout is set to ...READ MORE

Mar 25, 2019 in Apache Spark by Hari
8,365 views
0 votes
1 answer

How to disable automatic remove of application of failures?

Yes, you have read it right. The ...READ MORE

Mar 25, 2019 in Apache Spark by Hari
754 views
0 votes
1 answer

How to spread applications to less nodes?

You can limit the spread out by ...READ MORE

Mar 25, 2019 in Apache Spark by Hari
317 views
0 votes
1 answer

Change number of completed drivers displayed

You can change the number of completed ...READ MORE

Mar 25, 2019 in Apache Spark by Hari
360 views
0 votes
1 answer

Limit displaying completed applications

By default, the number of completed applications ...READ MORE

Mar 25, 2019 in Apache Spark by hari
437 views
0 votes
1 answer

Setting directory for Spark recovery

You can set the directory to store ...READ MORE

Mar 25, 2019 in Apache Spark by Hari
765 views
0 votes
1 answer

How to connect to Zookeeper after setting Spark recovery mode?

You have set Zookeeper as the recovery ...READ MORE

Mar 25, 2019 in Apache Spark by Hari
1,099 views
0 votes
1 answer

How to set Spark recovery to Zookeeper?

There is a property of Spark which ...READ MORE

Mar 25, 2019 in Apache Spark by Hari
529 views
0 votes
1 answer

Change heart beat time to R process

The heartbeat interval is assigned to the ...READ MORE

Mar 19, 2019 in Apache Spark by Jai
353 views
0 votes
1 answer

Set Spark executable for R scripts

To change the default executable, assign the ...READ MORE

Mar 19, 2019 in Apache Spark by Jai
774 views
0 votes
1 answer

Change number of threads use by R back end

Refer to the below commands to know ...READ MORE

Mar 19, 2019 in Apache Spark by Jai
398 views
0 votes
1 answer

Close file after writing write-ahead on driver

You can change the property to close ...READ MORE

Mar 19, 2019 in Apache Spark by Jai
789 views
0 votes
1 answer

Retaining the batch by status APIs before Garbage Collection

By default, 1000 batches are retained by ...READ MORE

Mar 19, 2019 in Apache Spark by Jai
494 views
0 votes
1 answer

How to change minimum rate per kafka partition?

Open spark shell using this command: $ spark-shell Then ...READ MORE

Mar 19, 2019 in Apache Spark by Jai
413 views
0 votes
1 answer

Set maximum reading rate from Kafka partition

You can do it by using the ...READ MORE

Mar 19, 2019 in Apache Spark by Jai
442 views
0 votes
1 answer

Prevent immediate shutdown of Streaming context after JVM shutdown

In technical terms, you want to gracefully shut down the ...READ MORE

Mar 19, 2019 in Apache Spark by Jai
926 views
0 votes
1 answer

Need to disable unpersist in Spark

You can dynamically change this function by ...READ MORE

Mar 19, 2019 in Apache Spark by Jai
768 views
0 votes
1 answer

How to enable write ahead logs for receivers?

To enable write-ahead logs, run the following ...READ MORE

Mar 19, 2019 in Apache Spark by Jai
620 views
0 votes
1 answer

Using R to display configuration of Spark SQL

Try the below-mentioned code. sparkR.session() properties <- sql("SET -v") showDF(properties, ...READ MORE

Mar 18, 2019 in Apache Spark by John
379 views
0 votes
1 answer

How to get SQL configuration in Spark using Python?

You can get the configuration details through ...READ MORE

Mar 18, 2019 in Apache Spark by John
958 views
0 votes
1 answer

Get Spark SQL configuration in Java

You will need to use Spark session ...READ MORE

Mar 18, 2019 in Apache Spark by John
766 views
0 votes
1 answer

How to set maximum receiving rate for backpressure mechanism?

You can set the maximum receiving rate ...READ MORE

Mar 18, 2019 in Apache Spark by John
393 views
0 votes
1 answer

How to get Spark SQL configuration?

First create a Spark session like this: val ...READ MORE

Mar 18, 2019 in Apache Spark by John
3,112 views
0 votes
1 answer

what are the job optimization Technics in spark and scala ?

There are different methods to achieve optimization ...READ MORE

Mar 18, 2019 in Apache Spark by Veer
1,889 views
0 votes
1 answer

what are the spark job and spark task and spark staging ?

In a Spark application, when you invoke ...READ MORE

Mar 18, 2019 in Apache Spark by Pavan
10,813 views
0 votes
1 answer

what are the spark real time issues ?

Some of the issues I have faced ...READ MORE

Mar 18, 2019 in Apache Spark by Sharman
4,910 views
0 votes
1 answer

How to disable credentials for services in Spark?

Disabling this feature will compromise the security ...READ MORE

Mar 15, 2019 in Apache Spark by Karan
879 views
0 votes
1 answer

How enable XSS protection for Spark application?

Spark has a built-in prevention system against XSS. ...READ MORE

Mar 15, 2019 in Apache Spark by Karan
746 views
0 votes
1 answer

Configuring location of credential provider

To configure the location of the credential ...READ MORE

Mar 15, 2019 in Apache Spark by Karan
397 views
0 votes
1 answer

How to access private key password with Spark?

Spark allows you to retrieve the key ...READ MORE

Mar 15, 2019 in Apache Spark by Karan
1,580 views
0 votes
1 answer

How to set client authentication for SSL?

By default, this feature is disabled. To ...READ MORE

Mar 15, 2019 in Apache Spark by Karan
763 views
0 votes
1 answer

Can I set different protocol for SSL in Spark?

There is no protocol set by default. ...READ MORE

Mar 15, 2019 in Apache Spark by Karan
872 views
0 votes
1 answer

How to set keystore path?

You have to set the path to ...READ MORE

Mar 15, 2019 in Apache Spark by Karan
1,955 views
0 votes
1 answer

Password for keystore in Spark

You can dynamically set a password to ...READ MORE

Mar 15, 2019 in Apache Spark by karan
579 views
0 votes
1 answer

Setting port for SSL for Spark application

You can set the port in the ...READ MORE

Mar 15, 2019 in Apache Spark by Karan
518 views
0 votes
1 answer

How to enable SSL for Spark application?

You can do it dynamically like this: val ...READ MORE

Mar 15, 2019 in Apache Spark by Karan
2,162 views
0 votes
1 answer

Spark to check if a particular string exists in a file

You can use this: lines = sc.textFile(“hdfs://path/to/file/filename.txt”); def isFound(line): if ...READ MORE

Mar 15, 2019 in Apache Spark by Raj
7,028 views
0 votes
1 answer

How to restrict a group to only view in Spark?

You can do it dynamically be setting ...READ MORE

Mar 15, 2019 in Apache Spark by Raj
427 views
0 votes
1 answer

How to give modify access to group?

You can do this using the following ...READ MORE

Mar 15, 2019 in Apache Spark by Vikas
369 views
0 votes
1 answer

How to give user only view access for Spark application?

You can give users only view permission ...READ MORE

Mar 14, 2019 in Apache Spark by Raj
1,246 views
0 votes
1 answer

How to add modify access for Web UI user?

For a user to have modification access ...READ MORE

Mar 14, 2019 in Apache Spark by Raj
402 views
0 votes
1 answer

How to check if user has permission in Web UI?

You can implement this as follows: First, add ...READ MORE

Mar 14, 2019 in Apache Spark by Raj
763 views
0 votes
1 answer

How to change encryption key value for local I/O?

There another property where you can set ...READ MORE

Mar 14, 2019 in Apache Spark by Raj
450 views
0 votes
1 answer

Enable encryption for local Input and Output

You can enable local I/O encryption like ...READ MORE

Mar 14, 2019 in Apache Spark by Raj
336 views
0 votes
1 answer

Client connection getting rejected due to SASL authentication

Seems like you have set the configuration ...READ MORE

Mar 14, 2019 in Apache Spark by Raj
1,138 views
0 votes
1 answer

How to enable SASL encryption?

You can use dynamic configuration setting to ...READ MORE

Mar 14, 2019 in Apache Spark by Raj
581 views
0 votes
1 answer

How to enable SASL authentication after Spark authentication?

You can do this by setting the ...READ MORE

Mar 13, 2019 in Apache Spark by Venu
832 views
0 votes
1 answer

Key Factor Algorithms used for encryption.

The default key factor algorithm used is PBKDF2WithHmacSHA1. You ...READ MORE

Mar 13, 2019 in Apache Spark by Venu
327 views
0 votes
1 answer

Change encryption key length fro Spark application

You can do this by running the ...READ MORE

Mar 13, 2019 in Apache Spark by Venu
462 views