Trending questions in Apache Spark

0 votes
1 answer

What is RDD Lineage in Spark?

Hey, Lineage is an RDD process to reconstruct ...READ MORE

Jul 4 in Apache Spark by Gitika
• 25,340 points
61 views
0 votes
1 answer

error: identified expected but integer literal found.

Hi, You can resolve this error with a ...READ MORE

Jul 4 in Apache Spark by Gitika
• 25,340 points
61 views
0 votes
1 answer

How SparkSQL is different from HQL and SQL?

Hi, SparkSQL is a special component on the ...READ MORE

Jul 3 in Apache Spark by Gitika
• 25,340 points
90 views
0 votes
1 answer

error: reassingment to val

Hi, This error will only generate when you ...READ MORE

Jul 4 in Apache Spark by Gitika
• 25,340 points
17 views
0 votes
1 answer

What is polyglot in spark?

Hi, Spark provides a high-level API in Java, ...READ MORE

Jul 1 in Apache Spark by Gitika
• 25,340 points
148 views
0 votes
1 answer

How to create RDD from an external file source in scala?

Hi, To create an RDD from external file ...READ MORE

Jul 3 in Apache Spark by Gitika
• 25,340 points
54 views
0 votes
1 answer

How to create RDD from parallelized collection in scala?

Hi, You can check this example in your ...READ MORE

Jul 3 in Apache Spark by Gitika
• 25,340 points
53 views
0 votes
1 answer

error: identifier expected but ']' found.

Hi, You can try this remove brackets from ...READ MORE

Jul 3 in Apache Spark by Gitika
• 25,340 points
63 views
0 votes
1 answer

What does reduce action do in Spark?

Hey, Reduce action converts an RDD to a ...READ MORE

Jul 4 in Apache Spark by Gitika
• 25,340 points
20 views
0 votes
1 answer

What is SparkCore functionalities?

Hey, Spark Core is a base engine of ...READ MORE

Jul 4 in Apache Spark by Gitika
• 25,340 points
14 views
0 votes
0 answers

How to create RDD as string file?

Can anyone suggest how to create RDD ...READ MORE

Jul 4 in Apache Spark by anand
26 views
0 votes
1 answer

Why Partitions are immutable in Spark?

Hi, Every transformation generates a new partition. Partitions ...READ MORE

Jul 3 in Apache Spark by Gitika
• 25,340 points
41 views
0 votes
1 answer

What is reduce() action in Spark?

Hey, It takes a function that operates on two ...READ MORE

Jul 2 in Apache Spark by Gitika
• 25,340 points
81 views
0 votes
1 answer

What is Piping in Spark?

Hi, Spark provides a pipe() method on RDDs. ...READ MORE

Jul 3 in Apache Spark by Gitika
• 25,340 points
32 views
0 votes
1 answer

When we create an RDD, does it bring the data and load it into the memory?

Hey, No, an RDD is made up of ...READ MORE

Jul 3 in Apache Spark by Gitika
• 25,340 points
28 views
0 votes
1 answer

What is Action in Spark?

Hi, Actions are RDD’s operation, that value returns ...READ MORE

Jul 3 in Apache Spark by Gitika
• 25,340 points
23 views
0 votes
1 answer

What is meant by Transformation? Give some examples.

Hi, The transformations are the functions that are ...READ MORE

Jul 3 in Apache Spark by Gitika
• 25,340 points
17 views
0 votes
1 answer

What is a Parquet file in Spark?

Hey, Parquet is a columnar format file supported ...READ MORE

Jul 2 in Apache Spark by Gitika
• 25,340 points
50 views
0 votes
1 answer

What is RDD in Apache spark?

Hi, RDD in spark stands for REsilient distributed ...READ MORE

Jul 1 in Apache Spark by Gitika
• 25,340 points
79 views
0 votes
1 answer

What is persist() in Spark?

Hi, Spark’s RDDs are by default recomputed each ...READ MORE

Jul 2 in Apache Spark by Gitika
• 25,340 points
40 views
0 votes
0 answers

How to create RDD from existing RDD in scala?

Can anyone suggest how to create RDD ...READ MORE

Jul 3 in Apache Spark by Nihal
17 views
0 votes
1 answer

How to calculate the result of formula with Scala?

Hi, You can use a simple mathematical calculation ...READ MORE

Jul 1 in Apache Spark by Gitika
• 25,340 points
45 views
0 votes
1 answer

How to execute a function in apache-scala?

Hi, Here is a simple example of how ...READ MORE

Jul 1 in Apache Spark by Gitika
• 25,340 points
43 views
0 votes
1 answer

By which components spark ecosystem libraries are composed of?

Hi, Spark ecosystem libraries are composed of various ...READ MORE

Jul 1 in Apache Spark by Gitika
• 25,340 points
33 views
0 votes
1 answer

Spark foldbykey doubt

Please have a look below for your ...READ MORE

Jun 19 in Apache Spark by Tina
44 views
0 votes
1 answer

Scala pass input data as arguments

Please refer to the below code as ...READ MORE

Jun 19 in Apache Spark by Lisa
37 views
0 votes
1 answer

Scala: Add user input to array

You can try this:  object printarray { ...READ MORE

Jun 19 in Apache Spark by Dinesha
28 views
0 votes
1 answer

Doubt in display(id, name, salary) before display function

The statement display(id, name, salary) is written before the display function ...READ MORE

Jun 19 in Apache Spark by Ritu
26 views
0 votes
1 answer

Spark CLI issue

For spark.read.textFile we need spark-2.x. Please try ...READ MORE

Jun 19 in Apache Spark by Maahi
18 views
0 votes
2 answers

map() vs flatMap() in Spark

Spark map function expresses a one-to-one transformation. ...READ MORE

Jun 17 in Apache Spark by vishal
• 160 points
2,031 views
0 votes
7 answers

How to print the contents of RDD in Apache Spark?

Simple and easy: line.foreach(println) READ MORE

Dec 10, 2018 in Apache Spark by Kuber
10,943 views
0 votes
1 answer
0 votes
0 answers

_spark_metadata/0 doesn't exist while Compacting batch 9 Structured streaming error

We have Streaming Application implemented using Spark ...READ MORE

May 31 in Apache Spark by AzimKangda
• 120 points
199 views
0 votes
1 answer

How do I get number of columns in each line from a delimited file??

Instead of spliting on '\n'. You should ...READ MORE

Aug 7 in Apache Spark by ashish
461 views
0 votes
1 answer

Copy all files from local (Windows) to HDFS with Scala code

Please try the following Scala code: import org.apache.hadoop.conf.Configuration import ...READ MORE

May 22 in Apache Spark by Karan
269 views
0 votes
1 answer

Error while reading multiline Json

peopleDF: org.apache.spark.sql.DataFrame = [_corrupt_record: string] The above that ...READ MORE

May 23 in Apache Spark by Conny
139 views
0 votes
1 answer

Difference between RDD as val and var

Variable declaration can be done in two ...READ MORE

May 23 in Apache Spark by Arun
79 views
0 votes
1 answer

Starting Spark Scala console

To get command prompt for Scala open ...READ MORE

May 24 in Apache Spark by Cassy
29 views
0 votes
1 answer

Spark: Saving file csv

 If you need a single output file ...READ MORE

May 22 in Apache Spark by Rishi
46 views
0 votes
1 answer

Starting Spark in Windows

Run below commands spark-class org.apache.spark.deploy.master.Master spark-class org.apache.spark.deploy.worker.Worker spark://192.168.254.1:7077 NOTE: The ...READ MORE

May 22 in Apache Spark by Reshma
26 views
0 votes
0 answers

WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable [closed]

Hi All I am running Scala program on ...READ MORE

May 5 in Apache Spark by Vishal

closed May 6 by Omkar 191 views
0 votes
1 answer

How to find the number of null contain in dataframe?

Hey there! You can use the select method of the ...READ MORE

May 3 in Apache Spark by Omkar
• 67,620 points
218 views
0 votes
1 answer

How can we use spark shell for scala without cluster?

You can run the Spark shell for ...READ MORE

Apr 28 in Apache Spark by Giri
58 views
0 votes
1 answer

Spark comparing two big data files using scala

Try this and see if this does ...READ MORE

Apr 2 in Apache Spark by Omkar
• 67,620 points
428 views
0 votes
1 answer

How to increase worker timeout in Spark application?

By default, the timeout is set to ...READ MORE

Mar 25 in Apache Spark by Hari
507 views
0 votes
1 answer

How to set extra JVM options for Spark application?

You cans set extra JVM options that ...READ MORE

Mar 28 in Apache Spark by Raj
384 views
0 votes
1 answer

How to store files in executor's working directory?

You have to specify a comma-separated list ...READ MORE

Mar 28 in Apache Spark by Raj
338 views
0 votes
1 answer

Changing Yarn queue in Spark application

To change the default queue to which ...READ MORE

Mar 28 in Apache Spark by Raj
315 views
0 votes
1 answer

what are the spark real time issues ?

Some of the issues I have faced ...READ MORE

Mar 18 in Apache Spark by Sharman
667 views
0 votes
1 answer

How to use Spark jars for Yarn distribution?

First, store upload this archive to hdfs and ...READ MORE

Mar 28 in Apache Spark by Raj
180 views