Latest questions in Apache Spark

0 votes
1 answer

org.apache.spark.sql.AnalysisException: cannot resolve "`id`" given input columns

I have used a header-less csv file ...READ MORE

Jul 14, 2019 in Apache Spark by Puneet
 • 17,442 views
0 votes
1 answer

Difference between cogroup and full outer join in spark

Please go through the below explanation : Full ...READ MORE

Jul 14, 2019 in Apache Spark by Kiran
 • 9,433 views
+1 vote
1 answer

How do I turn off INFO Logging in Spark?

Hi, You need to edit one property in ...READ MORE

Jul 12, 2019 in Apache Spark by ravikiran
 • 4,620 points
edited Dec 20, 2020 by MD • 5,672 views
0 votes
1 answer

Working of map function on data

The map function creates an array of ...READ MORE

Jul 11, 2019 in Apache Spark by Krish
 • 698 views
0 votes
1 answer

Spark error: Caused by: java.lang.RuntimeException: The root scratch dir: /tmp/hive on HDFS should be writable.

Give  read-write permissions to  C:\tmp\hive folder Cd to winutils bin folder ...READ MORE

Jul 11, 2019 in Apache Spark by Rajiv
 • 7,044 views
0 votes
1 answer

Is fetching data from apache flume webcrawling?

Web crawling is a program or automated ...READ MORE

Jul 11, 2019 in Apache Spark by Esha
 • 743 views
0 votes
1 answer

Why do we use sc.parallelize?

Spark revolves around the concept of a ...READ MORE

Jul 11, 2019 in Apache Spark by Suman
 • 13,125 views
0 votes
1 answer

Error Loading data to mysql in Spark

You have to use sqoop to export data ...READ MORE

Jul 11, 2019 in Apache Spark by Jishan
 • 840 views
0 votes
1 answer

Spark Streaming Pyspark code not working

The address you are using in the ...READ MORE

Jul 11, 2019 in Apache Spark by Shir
 • 2,117 views
+1 vote
1 answer

How to add package com.databricks.spark.avro in spark?

Start spark shell using below line of ...READ MORE

Jul 10, 2019 in Apache Spark by Jishnu
 • 5,592 views
0 votes
1 answer

Query regarding Appending " to a string in Scala

You can perform this task in two ...READ MORE

Jul 10, 2019 in Apache Spark by Esha
 • 2,150 views
0 votes
1 answer

org.apache.spark.sql.AnalysisException: cannot resolve given input columns

The string Productivity has to be enclosed between single ...READ MORE

Jul 10, 2019 in Apache Spark by Tina
 • 42,079 views
0 votes
1 answer

Query regarding Operator Overloading in Scala

All prefix operators' symbols are predefined: +, -, ...READ MORE

Jul 10, 2019 in Apache Spark by Karan
 • 1,371 views
0 votes
2 answers

Error : split value is not a member of org.apache.spark.sql.Row

var d=rdd2col.rdd.map(x=>x.split(",")) or val names=rd ...READ MORE

Aug 5, 2020 in Apache Spark by Ramkumar Ramasamy.
 • 11,156 views
0 votes
1 answer

Scala join comma delimited file as tables

Dataframe creation commands:​ Now we will register them ...READ MORE

Jul 9, 2019 in Apache Spark by Suraj
 • 752 views
0 votes
1 answer

How to use yield keyword in scala and why it is used instead of println?

Hi, The yield keyword is used because the ...READ MORE

Jul 6, 2019 in Apache Spark by Gitika
 • 65,910 points • 1,696 views
0 votes
1 answer

How to print loop with condition in scala?

Hi, Yes, in scala there is a guard condition where ...READ MORE

Jul 6, 2019 in Apache Spark by Gitika
 • 65,910 points • 405 views
0 votes
1 answer

How to implement two level loop in scala?

Hi, You can use two level loops using the ...READ MORE

Jul 5, 2019 in Apache Spark by Gitika
 • 65,910 points • 598 views
0 votes
1 answer

How can you use "for" statement in scala to print list from collection?

Hi, You can use for loop in scala using ...READ MORE

Jul 5, 2019 in Apache Spark by Gitika
 • 65,910 points • 518 views
0 votes
1 answer

How can we iterate any function using "foreach" function in scala?

Hi, Yes, "foreach" function you use because it will ...READ MORE

Jul 5, 2019 in Apache Spark by Gitika
 • 65,910 points • 1,375 views
0 votes
1 answer

How to print string text in scala?

Hi, You can see this example to see ...READ MORE

Jul 5, 2019 in Apache Spark by Gitika
 • 65,910 points • 2,221 views
0 votes
1 answer

How can we optimize and minimize the memory when work with scala use case?

Hi, There is a term in Scala that is ...READ MORE

Jul 5, 2019 in Apache Spark by Gitika
 • 65,910 points • 702 views
0 votes
1 answer

Which syntax to use to take the sum of list of collection in scala?

Hi, You can see this example to get ...READ MORE

Jul 5, 2019 in Apache Spark by Gitika
 • 65,910 points • 1,556 views
0 votes
1 answer

How to assign block expression in scala?

Hi, You can follow this example to know ...READ MORE

Jul 5, 2019 in Apache Spark by Gitika
 • 65,910 points • 690 views
0 votes
1 answer

load/save in spark

The reason why you are able to ...READ MORE

Jul 5, 2019 in Apache Spark by Firoz
 • 1,351 views
0 votes
1 answer

How to create dataframe for the comma delimited file?

 Refer to the below command used: val df ...READ MORE

Jul 5, 2019 in Apache Spark by karan
 • 2,991 views
0 votes
1 answer

Spark Installation problem

After downloading Spark, you need to set ...READ MORE

Jul 5, 2019 in Apache Spark by Rishi
 • 4,096 views
0 votes
1 answer

When we create an RDD, does it bring the data and load it into the memory?

Hi, No. An RDD is made up of ...READ MORE

Jul 5, 2019 in Apache Spark by Gitika
 • 65,910 points • 598 views
0 votes
1 answer

How to create scala project in intellij?

You have to install Intellij with scala plugin. ...READ MORE

Jul 5, 2019 in Apache Spark by Jimmy
 • 2,067 views
0 votes
1 answer

Is it mandatory to start Hadoop to run spark application?

Hi, No, not mandatory, but there is no ...READ MORE

Jul 5, 2019 in Apache Spark by Gitika
 • 65,910 points • 758 views
0 votes
1 answer

What is Lazy evaluated in Spark?

Hi, If you execute a bunch of programs, ...READ MORE

Jul 5, 2019 in Apache Spark by Gitika
 • 65,910 points • 1,111 views
0 votes
1 answer

Which File System is supported by Apache Spark?

Hi, Apache Spark is an advanced data processing ...READ MORE

Jul 5, 2019 in Apache Spark by Gitika
 • 65,910 points • 5,784 views
0 votes
1 answer

How will you explain yield keyword in Scala?

Hi, Yield keyword can be used either before ...READ MORE

Jul 5, 2019 in Apache Spark by Gitika
 • 65,910 points • 591 views
0 votes
1 answer

Do real-time data processing is possible with Spark SQL?

Hey, Real-time data processing is not possible directly ...READ MORE

Jul 5, 2019 in Apache Spark by Gitika
 • 65,910 points • 1,219 views
0 votes
1 answer

What is Spark Driver?

Hi, Spark Driver is the program that runs ...READ MORE

Jul 5, 2019 in Apache Spark by Gitika
 • 65,910 points • 2,281 views
0 votes
1 answer

How to run spark in Standalone client mode?

Hi, These are the steps to run spark in ...READ MORE

Jul 5, 2019 in Apache Spark by Gitika
 • 65,910 points • 1,483 views
0 votes
0 answers

How to create RDD as string file?

Can anyone suggest how to create RDD ...READ MORE

Jul 5, 2019 in Apache Spark by anand
 • 667 views
0 votes
1 answer

error: reassingment to val

Hi, This error will only generate when you ...READ MORE

Jul 5, 2019 in Apache Spark by Gitika
 • 65,910 points • 2,932 views
0 votes
1 answer

What does reduce action do in Spark?

Hey, Reduce action converts an RDD to a ...READ MORE

Jul 4, 2019 in Apache Spark by Gitika
 • 65,910 points • 468 views
0 votes
1 answer

What is RDD Lineage in Spark?

Hey, Lineage is an RDD process to reconstruct ...READ MORE

Jul 4, 2019 in Apache Spark by Gitika
 • 65,910 points • 4,059 views
0 votes
1 answer

What is SparkCore functionalities?

Hey, Spark Core is a base engine of ...READ MORE

Jul 4, 2019 in Apache Spark by Gitika
 • 65,910 points • 719 views
+1 vote
1 answer

Error: value textfile is not a member of org.apache.spark.SparkContext

Hi, Regarding this error, you just need to change ...READ MORE

Jul 4, 2019 in Apache Spark by Gitika
 • 65,910 points • 3,955 views
0 votes
1 answer

error: identified expected but integer literal found.

Hi, You can resolve this error with a ...READ MORE

Jul 4, 2019 in Apache Spark by Gitika
 • 65,910 points • 5,565 views
0 votes
1 answer

How to create RDD from an external file source in scala?

Hi, To create an RDD from external file ...READ MORE

Jul 4, 2019 in Apache Spark by Gitika
 • 65,910 points • 1,548 views
0 votes
1 answer

How to create RDD from existing RDD in scala?

scala> val rdd1 = sc.parallelize(List(1,2,3,4,5))                           -  Creating ...READ MORE

Feb 29, 2020 in Apache Spark by anonymous
 • 1,220 views
0 votes
1 answer

How to create RDD from parallelized collection in scala?

Hi, You can check this example in your ...READ MORE

Jul 4, 2019 in Apache Spark by Gitika
 • 65,910 points • 1,347 views
0 votes
1 answer

error: identifier expected but ']' found.

Hi, You can try this remove brackets from ...READ MORE

Jul 3, 2019 in Apache Spark by Gitika
 • 65,910 points • 5,094 views
0 votes
1 answer

What is Action in Spark?

Hi, Actions are RDD’s operation, that value returns ...READ MORE

Jul 3, 2019 in Apache Spark by Gitika
 • 65,910 points • 11,327 views
0 votes
1 answer

What is Map and flatMap in Spark?

Hi, The map is a specific line or ...READ MORE

Jul 3, 2019 in Apache Spark by Gitika
 • 65,910 points • 1,891 views
0 votes
1 answer

Why Partitions are immutable in Spark?

Hi, Every transformation generates a new partition. Partitions ...READ MORE

Jul 3, 2019 in Apache Spark by Gitika
 • 65,910 points • 1,571 views
Page:

Most popular tags

"PMP®","PMI®", "PMI-ACP®" and "PMBOK®" are registered marks of the Project Management Institute, Inc. MongoDB®, Mongo and the leaf logo are the registered trademarks of MongoDB, Inc.