Most voted questions in Apache Spark

0 votes
1 answer

error: reassingment to val

Hi, This error will only generate when you ...READ MORE

Jul 4 in Apache Spark by Gitika
• 25,340 points
18 views
0 votes
1 answer

What does reduce action do in Spark?

Hey, Reduce action converts an RDD to a ...READ MORE

Jul 4 in Apache Spark by Gitika
• 25,340 points
23 views
0 votes
1 answer

What is RDD Lineage in Spark?

Hey, Lineage is an RDD process to reconstruct ...READ MORE

Jul 4 in Apache Spark by Gitika
• 25,340 points
82 views
0 votes
1 answer

What is SparkCore functionalities?

Hey, Spark Core is a base engine of ...READ MORE

Jul 4 in Apache Spark by Gitika
• 25,340 points
15 views
0 votes
1 answer

Error: value textfile is not a member of org.apache.spark.SparkContext

Hi, Regarding this error, you just need to change ...READ MORE

Jul 4 in Apache Spark by Gitika
• 25,340 points
231 views
0 votes
1 answer

error: identified expected but integer literal found.

Hi, You can resolve this error with a ...READ MORE

Jul 4 in Apache Spark by Gitika
• 25,340 points
101 views
0 votes
1 answer

How to create RDD from an external file source in scala?

Hi, To create an RDD from external file ...READ MORE

Jul 3 in Apache Spark by Gitika
• 25,340 points
72 views
0 votes
0 answers

How to create RDD from existing RDD in scala?

Can anyone suggest how to create RDD ...READ MORE

Jul 3 in Apache Spark by Nihal
21 views
0 votes
1 answer

How to create RDD from parallelized collection in scala?

Hi, You can check this example in your ...READ MORE

Jul 3 in Apache Spark by Gitika
• 25,340 points
60 views
0 votes
1 answer

error: identifier expected but ']' found.

Hi, You can try this remove brackets from ...READ MORE

Jul 3 in Apache Spark by Gitika
• 25,340 points
105 views
0 votes
1 answer

What is Action in Spark?

Hi, Actions are RDD’s operation, that value returns ...READ MORE

Jul 3 in Apache Spark by Gitika
• 25,340 points
27 views
0 votes
1 answer

What is Map and flatMap in Spark?

Hi, The map is a specific line or ...READ MORE

Jul 3 in Apache Spark by Gitika
• 25,340 points
194 views
0 votes
1 answer

Why Partitions are immutable in Spark?

Hi, Every transformation generates a new partition. Partitions ...READ MORE

Jul 3 in Apache Spark by Gitika
• 25,340 points
54 views
0 votes
1 answer

How SparkSQL is different from HQL and SQL?

Hi, SparkSQL is a special component on the ...READ MORE

Jul 3 in Apache Spark by Gitika
• 25,340 points
125 views
0 votes
1 answer

What is Piping in Spark?

Hi, Spark provides a pipe() method on RDDs. ...READ MORE

Jul 3 in Apache Spark by Gitika
• 25,340 points
44 views
0 votes
1 answer

What is the difference between persist() and cache() in apache spark?

Hi, persist () allows the user to specify ...READ MORE

Jul 3 in Apache Spark by Gitika
• 25,340 points
411 views
0 votes
1 answer

What is meant by Transformation? Give some examples.

Hi, The transformations are the functions that are ...READ MORE

Jul 3 in Apache Spark by Gitika
• 25,340 points
19 views
0 votes
1 answer

When we create an RDD, does it bring the data and load it into the memory?

Hey, No, an RDD is made up of ...READ MORE

Jul 3 in Apache Spark by Gitika
• 25,340 points
30 views
0 votes
1 answer

What is sparkContext?

Hi, Spark Context is the entry point to ...READ MORE

Jul 3 in Apache Spark by Gitika
• 25,340 points
194 views
0 votes
1 answer

What is reduce() action in Spark?

Hey, It takes a function that operates on two ...READ MORE

Jul 2 in Apache Spark by Gitika
• 25,340 points
130 views
0 votes
1 answer

What is persist() in Spark?

Hi, Spark’s RDDs are by default recomputed each ...READ MORE

Jul 2 in Apache Spark by Gitika
• 25,340 points
45 views
0 votes
1 answer

What is a Parquet file in Spark?

Hey, Parquet is a columnar format file supported ...READ MORE

Jul 2 in Apache Spark by Gitika
• 25,340 points
61 views
0 votes
1 answer

How to execute a function in apache-scala?

Hi, Here is a simple example of how ...READ MORE

Jul 1 in Apache Spark by Gitika
• 25,340 points
45 views
0 votes
1 answer

How to calculate the result of formula with Scala?

Hi, You can use a simple mathematical calculation ...READ MORE

Jul 1 in Apache Spark by Gitika
• 25,340 points
48 views
0 votes
1 answer

By which components spark ecosystem libraries are composed of?

Hi, Spark ecosystem libraries are composed of various ...READ MORE

Jul 1 in Apache Spark by Gitika
• 25,340 points
36 views
0 votes
1 answer

What is polyglot in spark?

Hi, Spark provides a high-level API in Java, ...READ MORE

Jul 1 in Apache Spark by Gitika
• 25,340 points
183 views
0 votes
1 answer

What is RDD in Apache spark?

Hi, RDD in spark stands for REsilient distributed ...READ MORE

Jul 1 in Apache Spark by Gitika
• 25,340 points
89 views
0 votes
1 answer

Doubt in display(id, name, salary) before display function

The statement display(id, name, salary) is written before the display function ...READ MORE

Jun 19 in Apache Spark by Ritu
28 views
0 votes
1 answer

Scala pass input data as arguments

Please refer to the below code as ...READ MORE

Jun 19 in Apache Spark by Lisa
43 views
0 votes
1 answer

Scala: Add user input to array

You can try this:  object printarray { ...READ MORE

Jun 19 in Apache Spark by Dinesha
35 views
0 votes
1 answer

Spark CLI issue

For spark.read.textFile we need spark-2.x. Please try ...READ MORE

Jun 19 in Apache Spark by Maahi
23 views
0 votes
1 answer

Spark foldbykey doubt

Please have a look below for your ...READ MORE

Jun 19 in Apache Spark by Tina
60 views
0 votes
0 answers

_spark_metadata/0 doesn't exist while Compacting batch 9 Structured streaming error

We have Streaming Application implemented using Spark ...READ MORE

May 31 in Apache Spark by AzimKangda
• 120 points
228 views
0 votes
1 answer

Starting Spark Scala console

To get command prompt for Scala open ...READ MORE

May 24 in Apache Spark by Cassy
33 views
0 votes
1 answer
0 votes
1 answer

Difference between RDD as val and var

Variable declaration can be done in two ...READ MORE

May 23 in Apache Spark by Arun
112 views
0 votes
1 answer

Error while reading multiline Json

peopleDF: org.apache.spark.sql.DataFrame = [_corrupt_record: string] The above that ...READ MORE

May 23 in Apache Spark by Conny
212 views
0 votes
1 answer

Copy all files from local (Windows) to HDFS with Scala code

Please try the following Scala code: import org.apache.hadoop.conf.Configuration import ...READ MORE

May 22 in Apache Spark by Karan
373 views
0 votes
1 answer

Starting Spark in Windows

Run below commands spark-class org.apache.spark.deploy.master.Master spark-class org.apache.spark.deploy.worker.Worker spark://192.168.254.1:7077 NOTE: The ...READ MORE

May 22 in Apache Spark by Reshma
28 views
0 votes
1 answer

Spark: Saving file csv

 If you need a single output file ...READ MORE

May 22 in Apache Spark by Rishi
73 views
0 votes
0 answers

WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable [closed]

Hi All I am running Scala program on ...READ MORE

May 5 in Apache Spark by Vishal

closed May 6 by Omkar 271 views
0 votes
1 answer

How to find the number of null contain in dataframe?

Hey there! You can use the select method of the ...READ MORE

May 3 in Apache Spark by Omkar
• 67,660 points
247 views
0 votes
1 answer

How can we use spark shell for scala without cluster?

You can run the Spark shell for ...READ MORE

Apr 28 in Apache Spark by Giri
63 views
0 votes
1 answer

Spark comparing two big data files using scala

Try this and see if this does ...READ MORE

Apr 2 in Apache Spark by Omkar
• 67,660 points
614 views
0 votes
1 answer

Spark Yarn: Changing maximum number of time to submit application

By default, the maximum number of times ...READ MORE

Mar 28 in Apache Spark by Raj
199 views
0 votes
1 answer

Set Library to launch Yarn master

You can make use of Special Library path to ...READ MORE

Mar 28 in Apache Spark by Raj
22 views
0 votes
1 answer

How to set extra JVM options for Spark application?

You cans set extra JVM options that ...READ MORE

Mar 28 in Apache Spark by Raj
511 views
0 votes
1 answer

Thread to use Yarn application master is limited

This is because the maximum number of ...READ MORE

Mar 28 in Apache Spark by Raj
59 views
0 votes
1 answer

How to use Spark jars for Yarn distribution?

First, store upload this archive to hdfs and ...READ MORE

Mar 28 in Apache Spark by Raj
234 views
0 votes
1 answer

Changing Yarn queue in Spark application

To change the default queue to which ...READ MORE

Mar 28 in Apache Spark by Raj
405 views