Trending questions in Apache Spark

0 votes
1 answer

Spark:error:throws stack overflow when union a lot.

Hey, Use SparkContext.union(...) instead to union many RDDs at once You ...READ MORE

Jul 31 in Apache Spark by Gitika
• 25,360 points
131 views
0 votes
1 answer

Spark: Dataframe vs Dataset

Recently, there are two new data abstractions ...READ MORE

Jul 29 in Apache Spark by Jackie
211 views
0 votes
1 answer

In how many modes Apache spark can run?

Hey, You can launch spark application in four ...READ MORE

Aug 2 in Apache Spark by Gitika
• 25,360 points
33 views
0 votes
1 answer

Join in RDD using keys

Suppose you have two dataset results( id, ...READ MORE

Aug 2 in Apache Spark by Trisha
37 views
0 votes
1 answer

What is the use of App class in Scala?

Hi, Scala provides a helper class, called App, that ...READ MORE

Jul 31 in Apache Spark by Gitika
• 25,360 points
113 views
0 votes
1 answer

What is Hive on Spark?

Hi, Hive contains significant support for Apache Spark, ...READ MORE

Aug 2 in Apache Spark by Gitika
• 25,360 points
28 views
0 votes
1 answer

"main" java.lang.NoClassDefFoundError: org/apache/hadoop/fs/FSDataInputStream

1. We will check whether master and ...READ MORE

Jul 29 in Apache Spark by Yogi
191 views
0 votes
1 answer

Spark + Hive connectivity

The problem is probably with the command. ...READ MORE

Aug 1 in Apache Spark by Rishni
63 views
0 votes
1 answer

How to use uniform list in Scala?

Hey, The method List.fill() creates a list and ...READ MORE

Jul 31 in Apache Spark by Gitika
• 25,360 points
90 views
0 votes
1 answer

Read multiple xml files in Spark

You can do this using globbing. See ...READ MORE

Jul 25 in Apache Spark by Jack
353 views
0 votes
1 answer

PySpark not starting: No active sparkcontext

Seems like Spark hadoop daemons are not ...READ MORE

Jul 30 in Apache Spark by Jishan
118 views
0 votes
1 answer

What are these in scala : _* & @_*

As is widely used, and has different ...READ MORE

Jul 31 in Apache Spark by Turic
66 views
0 votes
1 answer
0 votes
1 answer

How to reverse a Scala list?

Hi, This reverses the order of elements in ...READ MORE

Jul 31 in Apache Spark by Gitika
• 25,360 points
37 views
0 votes
1 answer

How to create singleton classes in Scala?

Hey, Scala introduces a new object keyword, which is used ...READ MORE

Jul 31 in Apache Spark by Gitika
• 25,360 points
48 views
0 votes
1 answer

What is 'TRAITS' in Scala

Hi, Traits are basically Scala's workaround for the ...READ MORE

Jul 31 in Apache Spark by Gitika
• 25,360 points
43 views
0 votes
1 answer

How to access variables in s string interpolation in Scala?

Hey, You can use below code to access variables ...READ MORE

Jul 31 in Apache Spark by Gitika
• 25,360 points
37 views
0 votes
1 answer

How shallow copy carry out using Scala?

Hey, Scala uses the method copy() to carry ...READ MORE

Jul 31 in Apache Spark by Gitika
• 25,360 points
28 views
0 votes
1 answer

Removing the header of a text file in SparkRDD

1) First we loaded the data to ...READ MORE

Jul 31 in Apache Spark by Namitha
39 views
0 votes
1 answer

How do find Max and Min values in a set in Scala?

Hey, Here is the example of which will return ...READ MORE

Jul 30 in Apache Spark by Gitika
• 25,360 points
67 views
0 votes
1 answer

Scala: Loading a csv file

Refer to the below command: val input_df = ...READ MORE

Jul 31 in Apache Spark by Emma
34 views
0 votes
0 answers

How to define SparkConf?

Can anyone explain how to define SparkConf? READ MORE

Aug 1 in Apache Spark by Danish
21 views
0 votes
1 answer

How to work with multidimensional arrays in Scala?

Hi, Here is an example you can follow: scala> ...READ MORE

Jul 30 in Apache Spark by Gitika
• 25,360 points
61 views
0 votes
1 answer

Unable to run the java- spark standalone program

Though there is nothing wrong with the ...READ MORE

Jul 30 in Apache Spark by Lohit
64 views
0 votes
1 answer

Error : split value is not a member of org.apache.spark.sql.Row

spark.read.csv is used when loading into a ...READ MORE

Jul 22 in Apache Spark by Firoz
398 views
0 votes
1 answer

Date formats : how to cast string to date?

Try this, it should work: > from pyspark.sql.functions ...READ MORE

Jul 29 in Apache Spark by Niall
94 views
0 votes
1 answer

How to concatenate sets in Scala?

Hey, Yes, there are two ways of doing ...READ MORE

Jul 30 in Apache Spark by Gitika
• 25,360 points
37 views
0 votes
1 answer

How to declare sets with value in Scala?

Hey, You can follow this code: scala> var s=Set(1,4,4,3) s: ...READ MORE

Jul 30 in Apache Spark by Gitika
• 25,360 points
35 views
0 votes
1 answer

How to format a string in Scala?

Hey, To format a string, use the .format ...READ MORE

Jul 30 in Apache Spark by Gitika
• 25,360 points
30 views
0 votes
1 answer

How to find values common to two sets in Scala

Hey, There are two ways to find the ...READ MORE

Jul 30 in Apache Spark by Gitika
• 25,360 points
31 views
0 votes
1 answer

How 'Scala Higher Order function' works in Scala?

Hey, Scala allows the definition of a higher-order function. These ...READ MORE

Jul 30 in Apache Spark by Gitika
• 25,360 points
27 views
0 votes
1 answer

Unable to use ml library in pyspark

The error message you have shared with ...READ MORE

Jul 30 in Apache Spark by Karan
27 views
0 votes
1 answer

What is the differentiate between equals() and == in Scala. Is the latter the same as == in Java?

Hey, When we try to compare two instances ...READ MORE

Jul 29 in Apache Spark by Gitika
• 25,360 points
57 views
0 votes
1 answer

Scala: 30: error: value partitions is not a member of String

Try this code: val rdd= sc.textFile (“file.txt”, 5) rdd.partitions.size Output ...READ MORE

Jul 29 in Apache Spark by Nijit
53 views
–2 votes
1 answer

What is the difference in Java’s “If..Else” and Scala’s “If..Else”? [closed]

Hey, Java’s “If. Else”: In Java, “If. Else” is a statement, ...READ MORE

Jul 31 in Apache Spark by Gitika
• 25,360 points
45 views
0 votes
1 answer

How to concatenate Maps in Scala?

Hey, You can concatenate/join two Maps in more than ...READ MORE

Jul 29 in Apache Spark by Gitika
• 25,360 points

edited Jul 29 by Gitika 31 views
0 votes
1 answer

How to search for a Key in a Scala Map?

Hey, The Map.contains() method will tell you if ...READ MORE

Jul 29 in Apache Spark by Gitika
• 25,360 points
30 views
0 votes
1 answer

How to declare a Empty Scala Map?

Hi, You can either declare an empty Scala ...READ MORE

Jul 29 in Apache Spark by Gitika
• 25,360 points

edited Jul 29 by Gitika 30 views
0 votes
1 answer

Spark, Scala: Load custom delimited file

You can load a DAT file into ...READ MORE

Jul 16 in Apache Spark by Shri
599 views
0 votes
1 answer

load/save text file in spark

The reason you are able to load ...READ MORE

Jul 22 in Apache Spark by Giri
332 views
0 votes
1 answer

What is Lazy val in Scala?

Hey, Scala executes a val when we define ...READ MORE

Jul 29 in Apache Spark by Gitika
• 25,360 points
30 views
0 votes
1 answer

what is an anonymous function Scala?

Hi, An anonymous function in Scala is a ...READ MORE

Jul 29 in Apache Spark by Gitika
• 25,360 points
29 views
0 votes
1 answer

How the if-else statement is different for Scala and Java?

Hey, There is one main difference. This statement ...READ MORE

Jul 29 in Apache Spark by Gitika
• 25,360 points
27 views
0 votes
1 answer

What is Case Class in Scala?

Hey, Scala case classes let us model immutable data. ...READ MORE

Jul 29 in Apache Spark by Gitika
• 25,360 points
28 views
0 votes
1 answer

How to fit multiple assignment statements into one statement in Scala?

Hi, You can assign a Tuple to a ...READ MORE

Jul 29 in Apache Spark by Gitika
• 25,360 points
20 views
0 votes
1 answer

Copy file from local to hdfs from the spark job in yarn mode

Refer to the below code: import org.apache.hadoop.conf.Configuration import org.apache.hadoop.fs.FileSystem import ...READ MORE

Jul 24 in Apache Spark by Yogi
232 views
0 votes
1 answer

Consider for-comprehensions in Scala. What are they syntactic sugars for?

Hi, A for-comprehension is one way to carry ...READ MORE

Jul 29 in Apache Spark by Gitika
• 25,360 points
15 views
+1 vote
1 answer

How to install Scala Build Tool (SBT) on ubuntu?

Hey, To install SBT on Ubuntu first you need ...READ MORE

Jul 23 in Apache Spark by Gitika
• 25,360 points
171 views
0 votes
1 answer

How to append a list in Scala?

Hey, For this purpose, we use the single ...READ MORE

Jul 26 in Apache Spark by Gitika
• 25,360 points
80 views
0 votes
1 answer

How to call the Debug Mode in PySpark?

As far as I understand your intentions ...READ MORE

Jul 26 in Apache Spark by ravikiran
• 4,560 points
80 views