Trending questions in Apache Spark

+1 vote
1 answer

By default how many partitions are created in RDD in Apache spark?

Well, it depends on the block of ...READ MORE

Aug 2, 2019 in Apache Spark by Gitika
 • 65,910 points • 2,548 views
0 votes
1 answer

Scala: error: value unary_+ is not a member of (Int, Int)

All prefix operators' symbols are predefined: +, -, ...READ MORE

Jul 22, 2019 in Apache Spark by karan
 • 2,946 views
0 votes
1 answer

How to start spark history server?

Hey, You can use this command to start​ ...READ MORE

Jul 25, 2019 in Apache Spark by Gitika
 • 65,910 points • 2,770 views
+1 vote
0 answers

What is the use case of map and flatMap? [closed]

What is the major use case for ...READ MORE

Aug 25, 2019 in Apache Spark by anonymous
 • 130 points
closed Aug 26, 2019 by Omkar • 1,421 views
0 votes
1 answer

Scala: Add user input to array

You can try this:  object printarray { ...READ MORE

Jun 19, 2019 in Apache Spark by Dinesha
 • 4,270 views
0 votes
1 answer

Unable to use ml library in pyspark

The error message you have shared with ...READ MORE

Jul 30, 2019 in Apache Spark by Karan
 • 2,483 views
0 votes
1 answer

Error : split value is not a member of org.apache.spark.sql.Row

spark.read.csv is used when loading into a ...READ MORE

Jul 22, 2019 in Apache Spark by Firoz
 • 2,802 views
0 votes
1 answer

How to create paired RDD using subString method in Spark?

Hi, If you have a file with id ...READ MORE

Aug 2, 2019 in Apache Spark by Gitika
 • 65,910 points • 2,325 views
0 votes
1 answer

How to concatenate Maps in Scala?

Hey, You can concatenate/join two Maps in more than ...READ MORE

Jul 29, 2019 in Apache Spark by Gitika
 • 65,910 points
edited Jul 29, 2019 by Gitika • 2,469 views
0 votes
1 answer

What are these in scala : _* & @_*

As is widely used, and has different ...READ MORE

Jul 31, 2019 in Apache Spark by Turic
 • 2,306 views
0 votes
1 answer

Spark: Read from Hive, store in HDFS

Below is an example of reading data ...READ MORE

Jul 26, 2019 in Apache Spark by Lohit
 • 2,478 views
0 votes
1 answer

How SparkSQL is different from HQL and SQL?

Hi, SparkSQL is a special component on the ...READ MORE

Jul 3, 2019 in Apache Spark by Gitika
 • 65,910 points • 3,447 views
0 votes
1 answer

What is the difference between persist() and cache() in apache spark?

Hi, persist () allows the user to specify ...READ MORE

Jul 3, 2019 in Apache Spark by Gitika
 • 65,910 points • 3,335 views
0 votes
1 answer

Create dataframe for Avro file

Yes, we can work with Avro files ...READ MORE

Jul 22, 2019 in Apache Spark by Rishi
 • 2,499 views
0 votes
1 answer

RDD word count with line numbers

df = spark.createDataFrame([("A", 2000), ("A", 2002), ("A", ...READ MORE

Jul 25, 2019 in Apache Spark by Siri
 • 2,332 views
0 votes
3 answers

I don't understand the reason behind Spark RDD being immutable.

There are few reasons for keeping RDD ...READ MORE

Apr 18, 2019 in Apache Spark by santlal561987@gmail.com
 • 12,195 views
0 votes
1 answer

Spark + Hive connectivity

The problem is probably with the command. ...READ MORE

Aug 1, 2019 in Apache Spark by Rishni
 • 1,918 views
0 votes
1 answer

How to create dataframe for the comma delimited file?

 Refer to the below command used: val df ...READ MORE

Jul 5, 2019 in Apache Spark by karan
 • 2,974 views
0 votes
1 answer

How to declare a Empty Scala Map?

Hi, You can either declare an empty Scala ...READ MORE

Jul 29, 2019 in Apache Spark by Gitika
 • 65,910 points
edited Jul 29, 2019 by Gitika • 1,845 views
0 votes
1 answer

error: reassingment to val

Hi, This error will only generate when you ...READ MORE

Jul 5, 2019 in Apache Spark by Gitika
 • 65,910 points • 2,915 views
0 votes
1 answer

Average function is not commutative and associative?

Hey, I guess the only problem with the ...READ MORE

Jul 23, 2019 in Apache Spark by Gitika
 • 65,910 points • 2,132 views
0 votes
1 answer

Scala: save filtered data row by row using saveAsTextFile

Try this code, it worked for me: val ...READ MORE

Aug 2, 2019 in Apache Spark by Karan
 • 1,614 views
0 votes
1 answer

error:error: only classes can have declared but undefined members.

Hi, This happens in Scala whenever you won't ...READ MORE

Jul 24, 2019 in Apache Spark by Gitika
 • 65,910 points • 1,925 views
0 votes
1 answer

How to create singleton classes in Scala?

Hey, Scala introduces a new object keyword, which is used ...READ MORE

Jul 31, 2019 in Apache Spark by Gitika
 • 65,910 points • 1,566 views
–2 votes
1 answer

What is the difference in Java’s “If..Else” and Scala’s “If..Else”? [closed]

Hey, Java’s “If. Else”: In Java, “If. Else” is a statement, ...READ MORE

Jul 31, 2019 in Apache Spark by Gitika
 • 65,910 points • 1,614 views
0 votes
1 answer

What is Piping in Spark?

Hi, Spark provides a pipe() method on RDDs. ...READ MORE

Jul 3, 2019 in Apache Spark by Gitika
 • 65,910 points • 2,702 views
0 votes
0 answers

WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable [closed]

Hi All I am running Scala program on ...READ MORE

May 5, 2019 in Apache Spark by Vishal
closed May 6, 2019 by Omkar • 5,271 views
0 votes
1 answer

How to handle data shuffle in Spark

Hi, You can do it using map partition ...READ MORE

Aug 6, 2019 in Apache Spark by Gitika
 • 65,910 points • 1,160 views
0 votes
1 answer

In how many modes Apache spark can run?

Hey, You can launch spark application in four ...READ MORE

Aug 2, 2019 in Apache Spark by Gitika
 • 65,910 points • 1,331 views
0 votes
1 answer

Scala: org.apache.poi.openxml4j.exceptions.InvalidFormatException: Your InputStream was neither an OLE2 stream, nor an OOXML stream

Try executing the below code, def readExcel(file: String): ...READ MORE

Jul 30, 2019 in Apache Spark by Raman
 • 1,461 views
0 votes
1 answer

Passing condition dynamically to Spark application.

You can try this: d.filter(col("value").isin(desiredThings: _*)) and if you ...READ MORE

Feb 19, 2019 in Apache Spark by Omkar
 • 69,210 points • 8,392 views
0 votes
1 answer

Load custom delimited file in Spark

Refer to the following code: val sqlContext = ...READ MORE

Jul 24, 2019 in Apache Spark by Ritu
 • 1,694 views
0 votes
1 answer

Explain the for loop for printing the Map values in Scala in Apache Spark?

Hey, You can see this following code to ...READ MORE

Jul 23, 2019 in Apache Spark by Gitika
 • 65,910 points • 1,766 views
0 votes
1 answer

Spark comparing two big data files using scala

Try this and see if this does ...READ MORE

Apr 2, 2019 in Apache Spark by Omkar
 • 69,210 points • 6,658 views
0 votes
1 answer

How to find values common to two sets in Scala

Hey, There are two ways to find the ...READ MORE

Jul 31, 2019 in Apache Spark by Gitika
 • 65,910 points • 1,401 views
0 votes
1 answer

How to load data of .csv file in MySQL Database Table?

You can do it using a code ...READ MORE

Jul 22, 2019 in Apache Spark by Vishwa
 • 1,668 views
0 votes
1 answer

How to use uniform list in Scala?

Hey, The method List.fill() creates a list and ...READ MORE

Aug 1, 2019 in Apache Spark by Gitika
 • 65,910 points • 1,253 views
0 votes
1 answer

How to concatenate sets in Scala?

Hey, Yes, there are two ways of doing ...READ MORE

Jul 31, 2019 in Apache Spark by Gitika
 • 65,910 points • 1,282 views
0 votes
1 answer

Spark Streaming Pyspark code not working

The address you are using in the ...READ MORE

Jul 11, 2019 in Apache Spark by Shir
 • 2,112 views
0 votes
1 answer

Query regarding Appending " to a string in Scala

You can perform this task in two ...READ MORE

Jul 10, 2019 in Apache Spark by Esha
 • 2,140 views
0 votes
1 answer

Spark Submit: class does not exists

In the command, you have mentioned the ...READ MORE

Jul 26, 2019 in Apache Spark by Jimmy
 • 1,436 views
0 votes
1 answer

Monitoring Spark application

Spark-submit jobs are also run from client/edge ...READ MORE

Aug 9, 2019 in Apache Spark by Umesh
 • 824 views
+1 vote
1 answer

How to install Scala Build Tool (SBT) on ubuntu?

Hey, To install SBT on Ubuntu first you need ...READ MORE

Jul 23, 2019 in Apache Spark by Gitika
 • 65,910 points • 1,447 views
0 votes
1 answer

What is Spark Driver?

Hi, Spark Driver is the program that runs ...READ MORE

Jul 5, 2019 in Apache Spark by Gitika
 • 65,910 points • 2,277 views
0 votes
1 answer

Spark Processing Internals

Spark uses a master/slave architecture. As you ...READ MORE

Jul 15, 2019 in Apache Spark by Jimmy
edited Jun 9, 2020 by MD • 1,804 views
0 votes
1 answer

Spark to check if a particular string exists in a file

You can use this: lines = sc.textFile(“hdfs://path/to/file/filename.txt”); def isFound(line): if ...READ MORE

Mar 15, 2019 in Apache Spark by Raj
 • 7,064 views
0 votes
1 answer

How to print string text in scala?

Hi, You can see this example to see ...READ MORE

Jul 5, 2019 in Apache Spark by Gitika
 • 65,910 points • 2,208 views
0 votes
1 answer

Spark to Hive Table creation

There's an easier way to achieve your ...READ MORE

Jul 23, 2019 in Apache Spark by Dinesh
 • 1,433 views
+1 vote
1 answer

Scala: CSV file to Save data into HBase

Check the reference code mentioned below: def main(args: ...READ MORE

Jul 25, 2019 in Apache Spark by Hari
 • 1,288 views
0 votes
1 answer

Unable to run the java- spark standalone program

Though there is nothing wrong with the ...READ MORE

Jul 30, 2019 in Apache Spark by Lohit
 • 1,112 views
Page:

Most popular tags

"PMP®","PMI®", "PMI-ACP®" and "PMBOK®" are registered marks of the Project Management Institute, Inc. MongoDB®, Mongo and the leaf logo are the registered trademarks of MongoDB, Inc.