Most voted questions in Apache Spark

0 votes
1 answer

Spark to check if a particular string exists in a file

You can use this: lines = sc.textFile(“hdfs://path/to/file/filename.txt”); def isFound(line): if ...READ MORE

Mar 15 in Apache Spark by Raj
330 views
0 votes
1 answer

How to restrict a group to only view in Spark?

You can do it dynamically be setting ...READ MORE

Mar 15 in Apache Spark by Raj
53 views
0 votes
1 answer

How to give modify access to group?

You can do this using the following ...READ MORE

Mar 15 in Apache Spark by Vikas
21 views
0 votes
1 answer

How to give user only view access for Spark application?

You can give users only view permission ...READ MORE

Mar 14 in Apache Spark by Raj
34 views
0 votes
1 answer

How to add modify access for Web UI user?

For a user to have modification access ...READ MORE

Mar 14 in Apache Spark by Raj
40 views
0 votes
1 answer

How to check if user has permission in Web UI?

You can implement this as follows: First, add ...READ MORE

Mar 14 in Apache Spark by Raj
26 views
0 votes
1 answer

How to change encryption key value for local I/O?

There another property where you can set ...READ MORE

Mar 14 in Apache Spark by Raj
37 views
0 votes
1 answer

Enable encryption for local Input and Output

You can enable local I/O encryption like ...READ MORE

Mar 14 in Apache Spark by Raj
22 views
0 votes
1 answer

Client connection getting rejected due to SASL authentication

Seems like you have set the configuration ...READ MORE

Mar 14 in Apache Spark by Raj
18 views
0 votes
1 answer

How to enable SASL encryption?

You can use dynamic configuration setting to ...READ MORE

Mar 14 in Apache Spark by Raj
34 views
0 votes
1 answer

How to enable SASL authentication after Spark authentication?

You can do this by setting the ...READ MORE

Mar 13 in Apache Spark by Venu
26 views
0 votes
1 answer

Key Factor Algorithms used for encryption.

The default key factor algorithm used is PBKDF2WithHmacSHA1. You ...READ MORE

Mar 13 in Apache Spark by Venu
35 views
0 votes
1 answer

Change encryption key length fro Spark application

You can do this by running the ...READ MORE

Mar 13 in Apache Spark by Venu
38 views
0 votes
1 answer

Network encryption for Spark

You can enable encryption for the Spark ...READ MORE

Mar 13 in Apache Spark by Venu
59 views
0 votes
1 answer

How to authenticate Spark internal connections using a secret key?

You need to set the secret key ...READ MORE

Mar 13 in Apache Spark by Venu
86 views
0 votes
1 answer

Authenticate internal connections in Spark

To make Spark authenticate internal connections, you ...READ MORE

Mar 13 in Apache Spark by Venu
20 views
0 votes
1 answer

Delay requesting new executor in dynamic allocation

You can set the duration like this: val ...READ MORE

Mar 13 in Apache Spark by Venu
25 views
0 votes
1 answer

Dynamic allocation: Set minimum number of executor

You cans et it dynamically like this:  val ...READ MORE

Mar 13 in Apache Spark by Venu
21 views
0 votes
1 answer

How to set max executors for dynamic allocation?

You can set it by assigning the ...READ MORE

Mar 13 in Apache Spark by Venu
142 views
0 votes
1 answer

How many executors run when dynamic allocation is enabled?

The number of executors running by default ...READ MORE

Mar 13 in Apache Spark by Venu
33 views
0 votes
1 answer

Remove idle executor automatically

Yes, it is possible and is already ...READ MORE

Mar 12 in Apache Spark by Veer
97 views
0 votes
1 answer

How to enable dynamic resource allocation in Spark?

To dynamically enable dynamic resource allocation, you ...READ MORE

Mar 12 in Apache Spark by veer
161 views
0 votes
1 answer

How to set stage abort in Spark?

You can do it as follows. Use ...READ MORE

Mar 12 in Apache Spark by veer
109 views
0 votes
1 answer

How to prevent executor from self-destructing?

I think there is a timeout set ...READ MORE

Mar 12 in Apache Spark by Veer
58 views
0 votes
1 answer

How to monitor interrupted Spark tasks?

To enable monitoring interrupted tasks, run the following ...READ MORE

Mar 12 in Apache Spark by Veer
25 views
0 votes
1 answer

How to set cpu cores for spark task?

By default, each task is allocated with ...READ MORE

Mar 12 in Apache Spark by Veer
488 views
0 votes
1 answer

When is speculation enabled in Spark?

Speculation is enabled when a fraction of ...READ MORE

Mar 12 in Apache Spark by veer
126 views
0 votes
1 answer

How to set time for task speculation?

By default, the check for task speculation ...READ MORE

Mar 12 in Apache Spark by Veer
28 views
0 votes
1 answer

How to relaunch tasks that are running slowly?

The technical term for what you want ...READ MORE

Mar 12 in Apache Spark by Veer
45 views
0 votes
1 answer

How to automatically kill executors on blacklisting?

You can set the property to directly ...READ MORE

Mar 12 in Apache Spark by Veer
54 views
0 votes
1 answer

Increasing retry before blacklisting a node

You can do it dynamically using the ...READ MORE

Mar 11 in Apache Spark by Raj
44 views
0 votes
1 answer

Increasing retry before blacklisting executor

You can do it like this: val sc ...READ MORE

Mar 11 in Apache Spark by Raj
56 views
0 votes
1 answer

Changing the blacklist time of executor

By default, the node or executor is ...READ MORE

Mar 11 in Apache Spark by Raj
104 views
0 votes
1 answer

How can I run a blacklisted task?

When a task results in too many ...READ MORE

Mar 11 in Apache Spark by Raj
25 views
0 votes
1 answer

How to increase Spark listener bus event queue capacity?

The default capacity of listener bus is ...READ MORE

Mar 11 in Apache Spark by Raj
329 views
0 votes
1 answer

How to change scheduling mode in Spark?

You can change the scheduling mode as ...READ MORE

Mar 11 in Apache Spark by Raj
242 views
0 votes
1 answer

How to make Spark skip node locality?

If you set the node wait time ...READ MORE

Mar 11 in Apache Spark by Raj
54 views
0 votes
1 answer

How to increase wait time to launch data-local task?

You can increase the locality wait time ...READ MORE

Mar 11 in Apache Spark by Raj
45 views
0 votes
1 answer

How to limit the cores being used by a cluster?

You can set the maximum number of ...READ MORE

Mar 11 in Apache Spark by Raj
48 views
0 votes
1 answer

How to make Spark wait for more time for acknowledgement?

Use the following command to increase the ...READ MORE

Mar 11 in Apache Spark by Raj
72 views
0 votes
1 answer

How to change block manager port for driver only?

Yes, it is possible to change the ...READ MORE

Mar 10 in Apache Spark by Siri
66 views
0 votes
1 answer

Changing port for Block Managers

By default, the port of which the ...READ MORE

Mar 10 in Apache Spark by Siri
21 views
0 votes
1 answer

How to change commiter algorithm version in Spark?

To change to version 2, run the ...READ MORE

Mar 10 in Apache Spark by Siri
421 views
0 votes
1 answer

How to disable existing directory check?

To disable this, run the below commands: val ...READ MORE

Mar 10 in Apache Spark by Siri
28 views
0 votes
1 answer

Not able to clone Hadoop configuration.

Run the following command in Spark shell ...READ MORE

Mar 10 in Apache Spark by Siri
40 views
0 votes
1 answer

SparkContext.addFile() not able to update file.

Spark by default won't let you overwrite ...READ MORE

Mar 10 in Apache Spark by Siri
310 views
0 votes
1 answer

How to disable executor from fetching file from cache?

When a Spark application is running, the ...READ MORE

Mar 10 in Apache Spark by Siri
74 views
0 votes
1 answer

How to make driver update metrics quickly to executor?

There's a heartbeat signal sent to the ...READ MORE

Mar 10 in Apache Spark by Siri
21 views
0 votes
1 answer

How to disable broadcast checksum?

Run the following in the Spark shell: val ...READ MORE

Mar 9 in Apache Spark by Siri
36 views
0 votes
1 answer

How do I get number of columns in each line from a delimited file??

Instead of spliting on '\n'. You should ...READ MORE

Aug 7 in Apache Spark by ashish
615 views