Most viewed questions in Apache Spark

0 votes
1 answer

When is speculation enabled in Spark?

Speculation is enabled when a fraction of ...READ MORE

Mar 12, 2019 in Apache Spark by veer
 • 944 views
0 votes
1 answer

Prevent immediate shutdown of Streaming context after JVM shutdown

In technical terms, you want to gracefully shut down the ...READ MORE

Mar 19, 2019 in Apache Spark by Jai
 • 943 views
0 votes
1 answer

Different types of identifiers in Scala?

Hey, We have four kinds of identifiers in ...READ MORE

Jul 26, 2019 in Apache Spark by Gitika
 • 65,910 points • 942 views
0 votes
0 answers

What does the below code print? [closed]

What does the below code print? val AgeDs ...READ MORE

Nov 25, 2020 in Apache Spark by ritu
 • 960 points
closed Nov 25, 2020 by Gitika • 931 views
0 votes
1 answer

How to change default Spark dashboard port?

You can change it dynamically while using ...READ MORE

Mar 6, 2019 in Apache Spark by Rohit
 • 929 views
0 votes
1 answer

How can I run a blacklisted task?

When a task results in too many ...READ MORE

Mar 12, 2019 in Apache Spark by Raj
 • 919 views
0 votes
1 answer

From the below code. what is the most appropriate next step in ML process?

Hi@ritu, The most appropriate step according to me ...READ MORE

Nov 25, 2020 in Apache Spark by MD
 • 95,440 points • 915 views
0 votes
1 answer

How to change block manager port for driver only?

Yes, it is possible to change the ...READ MORE

Mar 10, 2019 in Apache Spark by Siri
 • 915 views
+1 vote
0 answers

How to combine a nested json file, which is being partitioned on the basis of source tags, and has varying internal structure, into a single json file; ( differently sourced Tag and varying structure)

Source tags are different: {  x : [         { ...READ MORE

Oct 11, 2019 in Apache Spark by anonymous
 • 160 points • 911 views
0 votes
1 answer

How to disable credentials for services in Spark?

Disabling this feature will compromise the security ...READ MORE

Mar 15, 2019 in Apache Spark by Karan
 • 909 views
0 votes
1 answer

In what kind of use cases has Spark outperformed Hadoop in processing?

I can list some but there can ...READ MORE

Sep 19, 2018 in Apache Spark by zombie
 • 3,790 points • 908 views
0 votes
1 answer

Can I set different protocol for SSL in Spark?

There is no protocol set by default. ...READ MORE

Mar 15, 2019 in Apache Spark by Karan
 • 901 views
+1 vote
0 answers

Difference Between rdd dataframe dataset [closed]

Sep 13, 2019 in Apache Spark by Rajesh pagadala
closed Sep 13, 2019 by Omkar • 885 views
0 votes
1 answer

How to access variables in s string interpolation in Scala?

Hey, You can use below code to access variables ...READ MORE

Jul 31, 2019 in Apache Spark by Gitika
 • 65,910 points • 884 views
0 votes
1 answer

Why is Spark map output compressed?

Spark thinks that it is a good ...READ MORE

Feb 24, 2019 in Apache Spark by Wasim
 • 877 views
0 votes
1 answer

what is an anonymous function Scala?

Hi, An anonymous function in Scala is a ...READ MORE

Jul 29, 2019 in Apache Spark by Gitika
 • 65,910 points • 866 views
0 votes
1 answer

Can we change the path where the Hive data is stored in HDFS?

Yes, you can but it has to ...READ MORE

Jul 14, 2019 in Apache Spark by Yogi
 • 865 views
0 votes
1 answer

How the if-else statement is different for Scala and Java?

Hey, There is one main difference. This statement ...READ MORE

Jul 29, 2019 in Apache Spark by Gitika
 • 65,910 points • 862 views
0 votes
1 answer

How to reverse a Scala list?

Hi, This reverses the order of elements in ...READ MORE

Aug 1, 2019 in Apache Spark by Gitika
 • 65,910 points • 856 views
0 votes
1 answer

Scala: Loading a csv file

Refer to the below command: val input_df = ...READ MORE

Jul 31, 2019 in Apache Spark by Emma
 • 856 views
0 votes
1 answer

How to enable SASL authentication after Spark authentication?

You can do this by setting the ...READ MORE

Mar 13, 2019 in Apache Spark by Venu
 • 856 views
0 votes
1 answer

after installing hadoop 3.0.1 I can's access spark shell or hive shell.

Hi@abdul, Hadoop 3.0.1 has lots of new features. ...READ MORE

Jun 16, 2020 in Apache Spark by MD
 • 95,440 points • 853 views
0 votes
1 answer

Load CSV file data inot MySQL

If you are trying to load file ...READ MORE

Jul 25, 2019 in Apache Spark by Sita
 • 853 views
0 votes
1 answer

Parquet File

Parquet is a columnar format file supported ...READ MORE

Jun 4, 2018 in Apache Spark by Data_Nerd
 • 2,390 points • 851 views
0 votes
1 answer

How to search for a Key in a Scala Map?

Hey, The Map.contains() method will tell you if ...READ MORE

Jul 29, 2019 in Apache Spark by Gitika
 • 65,910 points • 849 views
0 votes
1 answer

What is a lens in Scala?

Hi, A lens is an abstraction from functional ...READ MORE

Jul 26, 2019 in Apache Spark by Gitika
 • 65,910 points • 847 views
0 votes
1 answer

Convert the given Spar rdd object to Spark DataFrame.

You can create a DataFrame from the ...READ MORE

Jun 6, 2018 in Apache Spark by Shubham
 • 13,490 points • 844 views
0 votes
0 answers

Load data to mysql from local

I'm trying to load data to mysql ...READ MORE

Jul 23, 2019 in Apache Spark by Yogi
 • 843 views
0 votes
2 answers

map() and flatmap()

map(): Return a new distributed dataset formed by ...READ MORE

Jul 4, 2018 in Apache Spark by zombie
 • 3,790 points • 841 views
0 votes
1 answer

Error Loading data to mysql in Spark

You have to use sqoop to export data ...READ MORE

Jul 11, 2019 in Apache Spark by Jishan
 • 839 views
0 votes
1 answer

13)Refer the input and identify the output if the below code is run

Option c)  Run time error - A READ MORE

Nov 25, 2020 in Apache Spark by Gitika
 • 65,910 points • 837 views
0 votes
1 answer

Monitoring Spark application

Spark-submit jobs are also run from client/edge ...READ MORE

Aug 9, 2019 in Apache Spark by Umesh
 • 832 views
0 votes
1 answer

How to make Spark skip node locality?

If you set the node wait time ...READ MORE

Mar 11, 2019 in Apache Spark by Raj
 • 830 views
0 votes
1 answer

Spark logs not overwriting

Spark does not allow you to overwrite ...READ MORE

Mar 6, 2019 in Apache Spark by Rohit
 • 822 views
0 votes
1 answer

Close file after writing write-ahead on driver

You can change the property to close ...READ MORE

Mar 19, 2019 in Apache Spark by Jai
 • 819 views
0 votes
1 answer

How to retain Spark jar and app jar after staging?

By default, Spark jar, app jar, and ...READ MORE

Mar 27, 2019 in Apache Spark by Ginni
 • 818 views
0 votes
1 answer

What is Case Class in Scala?

Hey, Scala case classes let us model immutable data. ...READ MORE

Jul 29, 2019 in Apache Spark by Gitika
 • 65,910 points • 816 views
0 votes
1 answer

Starting Spark in Windows

Run below commands spark-class org.apache.spark.deploy.master.Master spark-class org.apache.spark.deploy.worker.Worker spark://192.168.254.1:7077 NOTE: The ...READ MORE

May 22, 2019 in Apache Spark by Reshma
 • 809 views
0 votes
1 answer

How to disable automatic remove of application of failures?

Yes, you have read it right. The ...READ MORE

Mar 25, 2019 in Apache Spark by Hari
 • 805 views
0 votes
1 answer

Set Spark executable for R scripts

To change the default executable, assign the ...READ MORE

Mar 19, 2019 in Apache Spark by Jai
 • 802 views
0 votes
1 answer

From the following graph code ,which code snippet will return the no.of flight routes?

Hey, @Ritu, I am getting error in your ...READ MORE

Nov 25, 2020 in Apache Spark by Gitika
 • 65,910 points • 798 views
0 votes
1 answer

Spark Machine Learning pipeline works fine in Spark 1.6, but it gives error when executed on Spark 2.x?

You need to change the following: val pipeline ...READ MORE

May 31, 2018 in Apache Spark by Shubham
 • 13,490 points • 797 views
0 votes
1 answer

Setting directory for Spark recovery

You can set the directory to store ...READ MORE

Mar 25, 2019 in Apache Spark by Hari
 • 796 views
0 votes
1 answer

How to set stage abort in Spark?

You can do it as follows. Use ...READ MORE

Mar 12, 2019 in Apache Spark by veer
 • 795 views
0 votes
1 answer

How to use nested function in Scala?

Hey, With Scala, we can define a Scala ...READ MORE

Jul 26, 2019 in Apache Spark by Gitika
 • 65,910 points
edited Jun 1, 2023 by Srinath • 792 views
0 votes
1 answer

How to check if user has permission in Web UI?

You can implement this as follows: First, add ...READ MORE

Mar 14, 2019 in Apache Spark by Raj
 • 792 views
0 votes
1 answer

Spark Driver roles

A Spark driver (aka an application’s driver ...READ MORE

Jun 21, 2018 in Apache Spark by Ashish
 • 2,650 points • 792 views
0 votes
1 answer

Which Scala library is used for functional programming?

Hi, Scala library has purely functional data structures ...READ MORE

Jul 24, 2019 in Apache Spark by Gitika
 • 65,910 points • 791 views
0 votes
1 answer

Not able to clone Hadoop configuration.

Run the following command in Spark shell ...READ MORE

Mar 10, 2019 in Apache Spark by Siri
 • 786 views
0 votes
1 answer

Need to disable unpersist in Spark

You can dynamically change this function by ...READ MORE

Mar 19, 2019 in Apache Spark by Jai
 • 784 views
Page:

Most popular tags

"PMP®","PMI®", "PMI-ACP®" and "PMBOK®" are registered marks of the Project Management Institute, Inc. MongoDB®, Mongo and the leaf logo are the registered trademarks of MongoDB, Inc.