Trending questions in Apache Spark

0 votes
1 answer

Spark2-submit does not generate output file.

To generate the output file, you can ...READ MORE

Feb 23 in Apache Spark by Esha
645 views
0 votes
1 answer

Not able to clone Hadoop configuration.

Run the following command in Spark shell ...READ MORE

Mar 10 in Apache Spark by Siri
29 views
0 votes
1 answer

How to disable existing directory check?

To disable this, run the below commands: val ...READ MORE

Mar 10 in Apache Spark by Siri
23 views
0 votes
1 answer

Changing port for Block Managers

By default, the port of which the ...READ MORE

Mar 10 in Apache Spark by Siri
21 views
0 votes
1 answer

How to disable broadcast checksum?

Run the following in the Spark shell: val ...READ MORE

Mar 9 in Apache Spark by Siri
35 views
0 votes
1 answer

How to make driver update metrics quickly to executor?

There's a heartbeat signal sent to the ...READ MORE

Mar 10 in Apache Spark by Siri
17 views
0 votes
1 answer

How to compress serialized RDD partition?

Yes, you can do this by enabling ...READ MORE

Mar 7 in Apache Spark by Pavitra
108 views
0 votes
1 answer

What is Spark Core?

It is not like a CPU to ...READ MORE

Mar 8 in Apache Spark by Raj
64 views
0 votes
1 answer

Getting "buffer limit exceeded" exception inside Kryo.

Seems like the object being sent for ...READ MORE

Mar 7 in Apache Spark by Pavitra
73 views
0 votes
1 answer

Array of RDD

You can create an array of RDDs ...READ MORE

Mar 8 in Apache Spark by Raj
36 views
0 votes
1 answer

Components of Spark

Spark core: The base engine that offers ...READ MORE

Mar 8 in Apache Spark by Raj
34 views
0 votes
1 answer

What port the Spark dashboard run on?

Spark dashboard by default runs on port ...READ MORE

Mar 6 in Apache Spark by Rohit
77 views
0 votes
0 answers

Why doesn't my Spark Yarn client runs on all available worker machines?

I am running an application on Spark ...READ MORE

Feb 22 in Apache Spark by Uzair Ahmad

edited Feb 22 by Omkar 646 views
0 votes
1 answer

How to increase Spark memory for execution?

Probably the spill is because you have ...READ MORE

Mar 7 in Apache Spark by Pavitra

edited Mar 7 32 views
0 votes
1 answer

How to increase Garbage Collection speed?

The time interval between Garbage Collection is ...READ MORE

Mar 7 in Apache Spark by Pavitra
21 views
0 votes
1 answer

How to delay live entity updates on Spark ?

You can do this by increasing the ...READ MORE

Mar 6 in Apache Spark by Rohit
39 views
0 votes
1 answer

How to change default Spark dashboard port?

You can change it dynamically while using ...READ MORE

Mar 6 in Apache Spark by Rohit
30 views
0 votes
1 answer

Spark logs not overwriting

Spark does not allow you to overwrite ...READ MORE

Mar 6 in Apache Spark by Rohit
19 views
0 votes
1 answer

Spark event log location

Unless and until you have not changed ...READ MORE

Mar 6 in Apache Spark by Rohit
19 views
0 votes
1 answer

Log every block update in Spark

By default, Spark does not log all ...READ MORE

Mar 6 in Apache Spark by Rohit
19 views
0 votes
1 answer

Prevent jobs to be killed from Web UI

You need to be careful with this. ...READ MORE

Mar 6 in Apache Spark by Rohit
14 views
0 votes
1 answer

Spark workers are not accepting any job (Kubernetes-Docker-Spark)

When kubernetes picks 10.*.*.*/16 network as it's ...READ MORE

Mar 1 in Apache Spark by Hamza
• 180 points
192 views
0 votes
1 answer

How to increase the amount of data to be transferred to shuffle service at the same time?

The amount of data to be transferred ...READ MORE

Mar 1 in Apache Spark by Omkar
• 67,600 points
56 views
0 votes
1 answer

How do spark extra listeners work?

Yes. You can use extra listeners by setting ...READ MORE

Feb 23 in Apache Spark by Rishi
297 views
0 votes
1 answer

Spark shuffle service port number

The default port that shuffle service runs ...READ MORE

Mar 1 in Apache Spark by Omkar
• 67,600 points
28 views
0 votes
1 answer

Spark SQL in databricks

In sparkSql, we can use CASE when ...READ MORE

Feb 23 in Apache Spark by Rishi
211 views
0 votes
1 answer

Apache Spark, usage of yield.

Yield is used in sequence comprehensions. It is ...READ MORE

Feb 21 in Apache Spark by Saruj
208 views
0 votes
1 answer

Increase number of cores in Spark

Now that the job is already running, ...READ MORE

Feb 22 in Apache Spark by Reshma
146 views
0 votes
1 answer

Installing Spark on Ubuntu

Hey. Follow these steps to install Spark ...READ MORE

Feb 20 in Apache Spark by Omkar
• 67,600 points
239 views
0 votes
1 answer

Why is Spark map output compressed?

Spark thinks that it is a good ...READ MORE

Feb 23 in Apache Spark by Wasim
63 views
0 votes
1 answer

Companion objects in Scala

When a singleton object is named the ...READ MORE

Feb 23 in Apache Spark by Uma
48 views
0 votes
1 answer

Not able to preserve shuffle files in Spark

You lose the files because by default, ...READ MORE

Feb 23 in Apache Spark by Rana
44 views
+1 vote
1 answer

Facing out-of-memory errors in Spark driver

I am guessing that the configuration set ...READ MORE

Feb 22 in Apache Spark by Rishab
39 views
0 votes
1 answer

Passing condition dynamically to Spark application.

You can try this: d.filter(col("value").isin(desiredThings: _*)) and if you ...READ MORE

Feb 19 in Apache Spark by Omkar
• 67,600 points
168 views
0 votes
1 answer

Loading Spark properties dynamically

First, create an empty conf using this ...READ MORE

Feb 22 in Apache Spark by Mansoor
35 views
0 votes
1 answer

Parquet to ORC format in Spark

I appreciate that you want to try ...READ MORE

Feb 14 in Apache Spark by Anjali
282 views
0 votes
1 answer

How to select all columns with group by?

You can use the following to print ...READ MORE

Feb 18 in Apache Spark by Omkar
• 67,600 points
63 views
0 votes
1 answer

where can i get spark-terasort.jar and not .scala file, to do spark terasort in windows.

Hi! I found 2 links on github where ...READ MORE

Feb 13 in Apache Spark by Omkar
• 67,600 points
120 views
0 votes
1 answer

Multidimensional Array in Scala

Multidimensional array is an array which store ...READ MORE

Feb 11 in Apache Spark by Omkar
• 67,600 points
149 views
0 votes
1 answer

Spark context (sc) not found

Maybe the hadoop service didn't start properly. Try ...READ MORE

Feb 13 in Apache Spark by John
61 views
0 votes
1 answer

Error using double map.

You have forgotten to mention the case ...READ MORE

Feb 11 in Apache Spark by Omkar
• 67,600 points
30 views
0 votes
1 answer

Error reading avro dataset in spark

For avro, you need to download and ...READ MORE

Feb 4 in Apache Spark by Omkar
• 67,600 points
322 views
+1 vote
1 answer

Spark interview

Preparing for an interview? We have something ...READ MORE

Feb 7 in Apache Spark by Edureka
• 1,280 points
147 views
0 votes
1 answer

Query regarding a spark split logic

First, import the data in Spark and ...READ MORE

Feb 9 in Apache Spark by Omkar
• 67,600 points
31 views
0 votes
3 answers

I don't understand the reason behind Spark RDD being immutable.

There are few reasons for keeping RDD ...READ MORE

Apr 18 in Apache Spark by santlal561987@gmail.com
2,612 views
0 votes
1 answer

Error while using Spark SQL filter API

You have to use "===" instead of ...READ MORE

Feb 4 in Apache Spark by Omkar
• 67,600 points
32 views
0 votes
1 answer

Sliding function in spark

The sliding function is used when you ...READ MORE

Jan 29 in Apache Spark by Omkar
• 67,600 points
218 views
0 votes
3 answers

How to transpose Spark DataFrame?

Please check the below mentioned links for ...READ MORE

Dec 31, 2018 in Apache Spark by anonymous
6,140 views
0 votes
1 answer

Invalid syntax in spark

There's a problem with your syntax. There ...READ MORE

Jan 31 in Apache Spark by Omkar
• 67,600 points
49 views
–1 vote
1 answer

Deciding number of spark context objects

How many spark context objects you should ...READ MORE

Jan 16 in Apache Spark by Omkar
• 67,600 points
42 views