Trending questions in Apache Spark

0 votes
1 answer

load/save in spark

The reason why you are able to ...READ MORE

Jul 5, 2019 in Apache Spark by Firoz
696 views
0 votes
1 answer

Can we change the path where the Hive data is stored in HDFS?

Yes, you can but it has to ...READ MORE

Jul 14, 2019 in Apache Spark by Yogi
292 views
0 votes
1 answer

What is Piping in Spark?

Hi, Spark provides a pipe() method on RDDs. ...READ MORE

Jul 3, 2019 in Apache Spark by Gitika
• 65,870 points
749 views
0 votes
1 answer

How to run spark in Standalone client mode?

Hi, These are the steps to run spark in ...READ MORE

Jul 5, 2019 in Apache Spark by Gitika
• 65,870 points
626 views
0 votes
1 answer

What is polyglot in spark?

Hi, Spark provides a high-level API in Java, ...READ MORE

Jul 1, 2019 in Apache Spark by Gitika
• 65,870 points
784 views
0 votes
1 answer

How to create RDD from parallelized collection in scala?

Hi, You can check this example in your ...READ MORE

Jul 3, 2019 in Apache Spark by Gitika
• 65,870 points
687 views
0 votes
1 answer

How to print string text in scala?

Hi, You can see this example to see ...READ MORE

Jul 5, 2019 in Apache Spark by Gitika
• 65,870 points
580 views
0 votes
1 answer

How to create RDD from an external file source in scala?

Hi, To create an RDD from external file ...READ MORE

Jul 3, 2019 in Apache Spark by Gitika
• 65,870 points
621 views
0 votes
1 answer

Error Loading data to mysql in Spark

You have to use sqoop to export data ...READ MORE

Jul 11, 2019 in Apache Spark by Jishan
286 views
0 votes
1 answer

Is fetching data from apache flume webcrawling?

Web crawling is a program or automated ...READ MORE

Jul 11, 2019 in Apache Spark by Esha
258 views
0 votes
1 answer

How to use yield keyword in scala and why it is used instead of println?

Hi, The yield keyword is used because the ...READ MORE

Jul 5, 2019 in Apache Spark by Gitika
• 65,870 points
504 views
0 votes
1 answer

What is Lazy evaluated in Spark?

Hi, If you execute a bunch of programs, ...READ MORE

Jul 5, 2019 in Apache Spark by Gitika
• 65,870 points
515 views
0 votes
1 answer

Scala: Add user input to array

You can try this:  object printarray { ...READ MORE

Jun 19, 2019 in Apache Spark by Dinesha
1,179 views
0 votes
1 answer

Scala join comma delimited file as tables

Dataframe creation commands:​ Now we will register them ...READ MORE

Jul 9, 2019 in Apache Spark by Suraj
235 views
0 votes
1 answer

Do real-time data processing is possible with Spark SQL?

Hey, Real-time data processing is not possible directly ...READ MORE

Jul 5, 2019 in Apache Spark by Gitika
• 65,870 points
423 views
0 votes
1 answer

Working of map function on data

The map function creates an array of ...READ MORE

Jul 11, 2019 in Apache Spark by Krish
151 views
0 votes
1 answer

error: reassingment to val

Hi, This error will only generate when you ...READ MORE

Jul 4, 2019 in Apache Spark by Gitika
• 65,870 points
361 views
0 votes
1 answer

When we create an RDD, does it bring the data and load it into the memory?

Hey, No, an RDD is made up of ...READ MORE

Jul 3, 2019 in Apache Spark by Gitika
• 65,870 points
405 views
0 votes
1 answer

What is a Parquet file in Spark?

Hey, Parquet is a columnar format file supported ...READ MORE

Jul 2, 2019 in Apache Spark by Gitika
• 65,870 points
437 views
0 votes
1 answer

Why Partitions are immutable in Spark?

Hi, Every transformation generates a new partition. Partitions ...READ MORE

Jul 3, 2019 in Apache Spark by Gitika
• 65,870 points
363 views
0 votes
1 answer

Spark foldbykey doubt

Please have a look below for your ...READ MORE

Jun 19, 2019 in Apache Spark by Tina
924 views
0 votes
1 answer

What is RDD in Apache spark?

Hi, RDD in spark stands for REsilient distributed ...READ MORE

Jul 1, 2019 in Apache Spark by Gitika
• 65,870 points
394 views
0 votes
1 answer

How can we optimize and minimize the memory when work with scala use case?

Hi, There is a term in Scala that is ...READ MORE

Jul 5, 2019 in Apache Spark by Gitika
• 65,870 points
178 views
0 votes
1 answer

How to assign block expression in scala?

Hi, You can follow this example to know ...READ MORE

Jul 5, 2019 in Apache Spark by Gitika
• 65,870 points
166 views
0 votes
0 answers

How to create RDD as string file?

Can anyone suggest how to create RDD ...READ MORE

Jul 4, 2019 in Apache Spark by anand
237 views
0 votes
1 answer

How to calculate the result of formula with Scala?

Hi, You can use a simple mathematical calculation ...READ MORE

Jul 1, 2019 in Apache Spark by Gitika
• 65,870 points
315 views
0 votes
0 answers

WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable [closed]

Hi All I am running Scala program on ...READ MORE

May 5, 2019 in Apache Spark by Vishal

closed May 6, 2019 by Omkar 2,829 views
0 votes
1 answer

How can you use "for" statement in scala to print list from collection?

Hi, You can use for loop in scala using ...READ MORE

Jul 5, 2019 in Apache Spark by Gitika
• 65,870 points
133 views
0 votes
1 answer

When we create an RDD, does it bring the data and load it into the memory?

Hi, No. An RDD is made up of ...READ MORE

Jul 5, 2019 in Apache Spark by Gitika
• 65,870 points
140 views
0 votes
1 answer

How will you explain yield keyword in Scala?

Hi, Yield keyword can be used either before ...READ MORE

Jul 5, 2019 in Apache Spark by Gitika
• 65,870 points
139 views
0 votes
1 answer

How to print loop with condition in scala?

Hi, Yes, in scala there is a guard condition where ...READ MORE

Jul 5, 2019 in Apache Spark by Gitika
• 65,870 points
115 views
0 votes
1 answer

How to implement two level loop in scala?

Hi, You can use two level loops using the ...READ MORE

Jul 5, 2019 in Apache Spark by Gitika
• 65,870 points
112 views
0 votes
0 answers

How can we optimize and minimize the memory when work with scala use case?

When we calculate some use case with ...READ MORE

Jul 5, 2019 in Apache Spark by nilam
153 views
0 votes
1 answer

Is it mandatory to start Hadoop to run spark application?

Hi, No, not mandatory, but there is no ...READ MORE

Jul 5, 2019 in Apache Spark by Gitika
• 65,870 points
125 views
0 votes
1 answer

Copy all files from local (Windows) to HDFS with Scala code

Please try the following Scala code: import org.apache.hadoop.conf.Configuration import ...READ MORE

May 22, 2019 in Apache Spark by Karan
1,990 views
0 votes
1 answer

What is SparkCore functionalities?

Hey, Spark Core is a base engine of ...READ MORE

Jul 4, 2019 in Apache Spark by Gitika
• 65,870 points
131 views
0 votes
1 answer

What does reduce action do in Spark?

Hey, Reduce action converts an RDD to a ...READ MORE

Jul 4, 2019 in Apache Spark by Gitika
• 65,870 points
117 views
0 votes
1 answer

What is meant by Transformation? Give some examples.

Hi, The transformations are the functions that are ...READ MORE

Jul 3, 2019 in Apache Spark by Gitika
• 65,870 points
118 views
0 votes
1 answer

What is persist() in Spark?

Hi, Spark’s RDDs are by default recomputed each ...READ MORE

Jul 2, 2019 in Apache Spark by Gitika
• 65,870 points
146 views
0 votes
1 answer

By which components spark ecosystem libraries are composed of?

Hi, Spark ecosystem libraries are composed of various ...READ MORE

Jul 1, 2019 in Apache Spark by Gitika
• 65,870 points
118 views
0 votes
1 answer

Scala pass input data as arguments

Please refer to the below code as ...READ MORE

Jun 19, 2019 in Apache Spark by Lisa
614 views
0 votes
1 answer

Spark comparing two big data files using scala

Try this and see if this does ...READ MORE

Apr 2, 2019 in Apache Spark by Omkar
• 69,110 points
4,050 views
0 votes
1 answer

How to increase worker timeout in Spark application?

By default, the timeout is set to ...READ MORE

Mar 25, 2019 in Apache Spark by Hari
4,223 views
0 votes
1 answer

Spark: Saving file csv

 If you need a single output file ...READ MORE

May 22, 2019 in Apache Spark by Rishi
1,687 views
0 votes
1 answer

Doubt in display(id, name, salary) before display function

The statement display(id, name, salary) is written before the display function ...READ MORE

Jun 19, 2019 in Apache Spark by Ritu
108 views
0 votes
1 answer

Spark CLI issue

For spark.read.textFile we need spark-2.x. Please try ...READ MORE

Jun 19, 2019 in Apache Spark by Maahi
94 views
0 votes
1 answer

Error while reading multiline Json

peopleDF: org.apache.spark.sql.DataFrame = [_corrupt_record: string] The above that ...READ MORE

May 23, 2019 in Apache Spark by Conny
1,216 views
0 votes
1 answer

Passing condition dynamically to Spark application.

You can try this: d.filter(col("value").isin(desiredThings: _*)) and if you ...READ MORE

Feb 19, 2019 in Apache Spark by Omkar
• 69,110 points
5,048 views
0 votes
1 answer

How to find the number of null contain in dataframe?

Hey there! You can use the select method of the ...READ MORE

May 3, 2019 in Apache Spark by Omkar
• 69,110 points
1,743 views
0 votes
1 answer

Difference between RDD as val and var

Variable declaration can be done in two ...READ MORE

May 23, 2019 in Apache Spark by Arun
811 views