Latest questions in Apache Spark

0 votes
1 answer

Disable Web UI for Spark Application

You can disable it like this: val sc ...READ MORE

Mar 6, 2019 in Apache Spark by Rohit
 • 3,051 views
0 votes
1 answer

Spark logs not overwriting

Spark does not allow you to overwrite ...READ MORE

Mar 6, 2019 in Apache Spark by Rohit
 • 830 views
0 votes
1 answer

How to enable Spark event logging?

To make Spark store the event logs, ...READ MORE

Mar 6, 2019 in Apache Spark by Rohit
 • 2,632 views
0 votes
1 answer

How to change the location of Spark event logs?

You can change the location where you ...READ MORE

Mar 6, 2019 in Apache Spark by Rohit
 • 4,060 views
0 votes
1 answer

Spark event log location

Unless and until you have not changed ...READ MORE

Mar 6, 2019 in Apache Spark by Rohit
 • 464 views
0 votes
1 answer

Log every block update in Spark

By default, Spark does not log all ...READ MORE

Mar 6, 2019 in Apache Spark by Rohit
 • 762 views
0 votes
1 answer

How to increase the amount of data to be transferred to shuffle service at the same time?

The amount of data to be transferred ...READ MORE

Mar 1, 2019 in Apache Spark by Omkar
 • 69,230 points • 722 views
0 votes
1 answer

Spark shuffle service port number

The default port that shuffle service runs ...READ MORE

Mar 1, 2019 in Apache Spark by Omkar
 • 69,230 points • 646 views
0 votes
1 answer

Spark workers are not accepting any job (Kubernetes-Docker-Spark)

When kubernetes picks 10.*.*.*/16 network as it's ...READ MORE

Mar 1, 2019 in Apache Spark by Hamza
 • 200 points • 1,820 views
0 votes
1 answer

Spark2-submit does not generate output file.

To generate the output file, you can ...READ MORE

Feb 24, 2019 in Apache Spark by Esha
 • 4,541 views
0 votes
1 answer

Companion objects in Scala

When a singleton object is named the ...READ MORE

Feb 24, 2019 in Apache Spark by Uma
 • 640 views
0 votes
1 answer

Spark SQL in databricks

In sparkSql, we can use CASE when ...READ MORE

Feb 24, 2019 in Apache Spark by Rishi
 • 2,098 views
0 votes
1 answer

Not able to preserve shuffle files in Spark

You lose the files because by default, ...READ MORE

Feb 24, 2019 in Apache Spark by Rana
 • 1,250 views
0 votes
1 answer

Why is Spark map output compressed?

Spark thinks that it is a good ...READ MORE

Feb 24, 2019 in Apache Spark by Wasim
 • 886 views
0 votes
1 answer

How do spark extra listeners work?

Yes. You can use extra listeners by setting ...READ MORE

Feb 24, 2019 in Apache Spark by Rishi
 • 2,645 views
+1 vote
1 answer

Facing out-of-memory errors in Spark driver

I am guessing that the configuration set ...READ MORE

Feb 23, 2019 in Apache Spark by Rishab
 • 1,560 views
0 votes
1 answer

Increase number of cores in Spark

Now that the job is already running, ...READ MORE

Feb 23, 2019 in Apache Spark by Reshma
 • 1,825 views
0 votes
1 answer

Loading Spark properties dynamically

First, create an empty conf using this ...READ MORE

Feb 22, 2019 in Apache Spark by Mansoor
 • 1,274 views
0 votes
0 answers

Why doesn't my Spark Yarn client runs on all available worker machines?

I am running an application on Spark ...READ MORE

Feb 22, 2019 in Apache Spark by Uzair Ahmad
edited Feb 22, 2019 by Omkar • 7,728 views
0 votes
1 answer

Apache Spark, usage of yield.

Yield is used in sequence comprehensions. It is ...READ MORE

Feb 22, 2019 in Apache Spark by Saruj
 • 2,754 views
0 votes
1 answer

Installing Spark on Ubuntu

Hey. Follow these steps to install Spark ...READ MORE

Feb 20, 2019 in Apache Spark by Omkar
 • 69,230 points • 1,595 views
0 votes
1 answer

Passing condition dynamically to Spark application.

You can try this: d.filter(col("value").isin(desiredThings: _*)) and if you ...READ MORE

Feb 19, 2019 in Apache Spark by Omkar
 • 69,230 points • 8,414 views
0 votes
1 answer

How to select all columns with group by?

You can use the following to print ...READ MORE

Feb 19, 2019 in Apache Spark by Omkar
 • 69,230 points • 13,462 views
0 votes
1 answer

Parquet to ORC format in Spark

I appreciate that you want to try ...READ MORE

Feb 15, 2019 in Apache Spark by Anjali
 • 2,107 views
0 votes
1 answer

How can I remove headers from dataframe?

You can use filter to do this. ...READ MORE

Feb 15, 2019 in Apache Spark by Aryan
 • 19,642 views
0 votes
1 answer

Spark context (sc) not found

Maybe the hadoop service didn't start properly. Try ...READ MORE

Feb 14, 2019 in Apache Spark by John
 • 1,638 views
0 votes
1 answer

where can i get spark-terasort.jar and not .scala file, to do spark terasort in windows.

Hi! I found 2 links on github where ...READ MORE

Feb 13, 2019 in Apache Spark by Omkar
 • 69,230 points • 1,156 views
0 votes
1 answer

Multidimensional Array in Scala

Multidimensional array is an array which store ...READ MORE

Feb 11, 2019 in Apache Spark by Omkar
 • 69,230 points • 1,667 views
0 votes
1 answer

Error using double map.

You have forgotten to mention the case ...READ MORE

Feb 11, 2019 in Apache Spark by Omkar
 • 69,230 points • 448 views
0 votes
1 answer

Query regarding a spark split logic

First, import the data in Spark and ...READ MORE

Feb 9, 2019 in Apache Spark by Omkar
 • 69,230 points • 392 views
+1 vote
1 answer

Spark interview

Preparing for an interview? We have something ...READ MORE

Feb 7, 2019 in Apache Spark by Edureka
 • 2,960 points • 612 views
0 votes
1 answer

Error reading avro dataset in spark

For avro, you need to download and ...READ MORE

Feb 4, 2019 in Apache Spark by Omkar
 • 69,230 points • 1,942 views
0 votes
1 answer

Error while using Spark SQL filter API

You have to use "===" instead of ...READ MORE

Feb 4, 2019 in Apache Spark by Omkar
 • 69,230 points • 569 views
0 votes
1 answer

Invalid syntax in spark

There's a problem with your syntax. There ...READ MORE

Jan 31, 2019 in Apache Spark by Omkar
 • 69,230 points • 1,847 views
0 votes
1 answer

Sliding function in spark

The sliding function is used when you ...READ MORE

Jan 29, 2019 in Apache Spark by Omkar
 • 69,230 points • 2,487 views
–1 vote
1 answer

Deciding number of spark context objects

How many spark context objects you should ...READ MORE

Jan 16, 2019 in Apache Spark by Omkar
 • 69,230 points • 496 views
0 votes
1 answer

Spark and Scale Auxiliary constructor doubt

println("Slayer") is an anonymous block and gets ...READ MORE

Jan 8, 2019 in Apache Spark by Omkar
 • 69,230 points • 539 views
0 votes
1 answer

Is there an API for implementing graphs in Spark?

GraphX is the Spark API for graphs and ...READ MORE

Jan 5, 2019 in Apache Spark by Frankie
 • 9,830 points • 507 views
0 votes
1 answer

What is Executor Memory in a Spark application?

Every spark application has same fixed heap ...READ MORE

Jan 5, 2019 in Apache Spark by Frankie
 • 9,830 points • 6,244 views
–1 vote
1 answer

Not able to use sc in spark shell

Seems like master and worker are not ...READ MORE

Jan 3, 2019 in Apache Spark by Omkar
 • 69,230 points • 1,432 views
0 votes
1 answer

Unresolved dependency issue on sbt package command

Check if you are able to access ...READ MORE

Jan 3, 2019 in Apache Spark by Omkar
 • 69,230 points • 2,458 views
0 votes
1 answer

How to get ID of a map task in Spark?

you can access task information using TaskContext: import org.apache.spark.TaskContext sc.parallelize(Seq[Int](), ...READ MORE

Nov 20, 2018 in Apache Spark by Frankie
 • 9,830 points • 3,110 views
0 votes
1 answer

How to open/stream .zip files through Spark?

You can try and check this below ...READ MORE

Nov 20, 2018 in Apache Spark by Frankie
 • 9,830 points • 2,265 views
0 votes
1 answer

Is 'sparkline' a method?

I suggest you to check 2 things That jquery.sparkline.js is actually ...READ MORE

Nov 9, 2018 in Apache Spark by Frankie
 • 9,830 points • 1,019 views
0 votes
1 answer

Filter, Option or FlatMap in spark

If, for option 2, you mean have ...READ MORE

Nov 9, 2018 in Apache Spark by Frankie
 • 9,830 points • 2,478 views
0 votes
1 answer

Internal work of Spark

Spark revolves around the concept of a ...READ MORE

Oct 11, 2018 in Apache Spark by nitinrawat895
 • 11,380 points • 775 views
0 votes
1 answer

Spark - repartition() vs coalesce()

It avoids a full shuffle. If it's ...READ MORE

Oct 11, 2018 in Apache Spark by nitinrawat895
 • 11,380 points • 7,839 views
0 votes
1 answer

How can I minimize data transfers when working with Spark?

Minimizing data transfers and avoiding shuffling helps ...READ MORE

Sep 19, 2018 in Apache Spark by zombie
 • 3,790 points • 2,676 views
0 votes
1 answer

In what kind of use cases has Spark outperformed Hadoop in processing?

I can list some but there can ...READ MORE

Sep 19, 2018 in Apache Spark by zombie
 • 3,790 points • 914 views
0 votes
1 answer

What happens to RDD when one of the nodes goes down?

Whenever a node goes down, Spark knows ...READ MORE

Sep 3, 2018 in Apache Spark by nitinrawat895
 • 11,380 points • 1,611 views
Page:

Most popular tags

"PMP®","PMI®", "PMI-ACP®" and "PMBOK®" are registered marks of the Project Management Institute, Inc. MongoDB®, Mongo and the leaf logo are the registered trademarks of MongoDB, Inc.