Most voted questions in Apache Spark

0 votes
1 answer

How to check if a particular keyword exists in Apache Spark?

Hey, You can try this code to get ...READ MORE

Jul 23, 2019 in Apache Spark by Gitika
• 65,910 points
4,266 views
0 votes
1 answer

How to compute the square root of sum of squares of numbers?

Hey, You need to follow some steps to complete ...READ MORE

Jul 23, 2019 in Apache Spark by Gitika
• 65,910 points
5,447 views
0 votes
1 answer

Average function is not commutative and associative?

Hey, I guess the only problem with the ...READ MORE

Jul 23, 2019 in Apache Spark by Gitika
• 65,910 points
2,095 views
0 votes
1 answer

Scala: when to use x(2) and x._2?

In the above statement, x(2) is specifying an array ...READ MORE

Jul 22, 2019 in Apache Spark by Yogi
1,078 views
0 votes
1 answer

Scala: error: value unary_+ is not a member of (Int, Int)

All prefix operators' symbols are predefined: +, -, ...READ MORE

Jul 22, 2019 in Apache Spark by karan
2,876 views
0 votes
1 answer

Load .xlsx files to hive tables with spark scala

This should work: def readExcel(file: String): DataFrame = ...READ MORE

Jul 22, 2019 in Apache Spark by Kishan
4,035 views
0 votes
1 answer

Code to compute average in Apache Spark?

Hi, You can compute the average using this ...READ MORE

Jul 22, 2019 in Apache Spark by Gitika
• 65,910 points
956 views
0 votes
1 answer

How fault tolerance is achieved in Apache Spark?

Hey, In Apache Spark, the data storage model is ...READ MORE

Jul 22, 2019 in Apache Spark by Gitika
• 65,910 points
8,134 views
0 votes
1 answer

How to remove the elements with a key present in any other RDD?

Hey, You can use the subtractByKey () function to ...READ MORE

Jul 22, 2019 in Apache Spark by Gitika
• 65,910 points
3,760 views
0 votes
1 answer

Error : split value is not a member of org.apache.spark.sql.Row

spark.read.csv is used when loading into a ...READ MORE

Jul 22, 2019 in Apache Spark by Firoz
2,787 views
0 votes
1 answer

load/save text file in spark

The reason you are able to load ...READ MORE

Jul 22, 2019 in Apache Spark by Giri
3,179 views
0 votes
1 answer

Create dataframe for Avro file

Yes, we can work with Avro files ...READ MORE

Jul 22, 2019 in Apache Spark by Rishi
2,487 views
0 votes
1 answer

How to load data of .csv file in MySQL Database Table?

You can do it using a code ...READ MORE

Jul 22, 2019 in Apache Spark by Vishwa
1,656 views
0 votes
1 answer

Spark Null Pointer Exception.

I used Spark 1.5.2 with Hadoop 2.6 ...READ MORE

Jul 19, 2019 in Apache Spark by ravikiran
• 4,620 points
5,648 views
0 votes
1 answer

Spark on Yarn

If you just want to get your ...READ MORE

Jul 18, 2019 in Apache Spark by ravikiran
• 4,620 points
602 views
0 votes
1 answer

Pyspark is taking default path

The HDFS path for MyLab is /user/edureka_id. ...READ MORE

Jul 16, 2019 in Apache Spark by Khushi
1,195 views
0 votes
1 answer

Spark, Scala: Load custom delimited file

You can load a DAT file into ...READ MORE

Jul 16, 2019 in Apache Spark by Shri
8,905 views
0 votes
1 answer

Spark Processing Internals

Spark uses a master/slave architecture. As you ...READ MORE

Jul 15, 2019 in Apache Spark by Jimmy

edited Jun 9, 2020 by MD 1,794 views
0 votes
1 answer

Spark-shell not working

First, reboot the system. And after reboot, ...READ MORE

Jul 15, 2019 in Apache Spark by Mahesh
4,406 views
0 votes
1 answer

Spark memory processing on a not temporary table

Temporary table is more like an index ...READ MORE

Jul 14, 2019 in Apache Spark by Suri
1,280 views
0 votes
1 answer

Can we change the path where the Hive data is stored in HDFS?

Yes, you can but it has to ...READ MORE

Jul 14, 2019 in Apache Spark by Yogi
847 views
0 votes
1 answer

What does the command df.registerTempTable() do?

df.registerTempTable(“airports”) This command is used to register ...READ MORE

Jul 14, 2019 in Apache Spark by James
7,830 views
0 votes
1 answer

Spark: How can i create temp views in user defined database instead of default database?

You can try the below code: df.registerTempTable(“airports”) sqlContext.sql(" create ...READ MORE

Jul 14, 2019 in Apache Spark by Ishan
4,110 views
0 votes
1 answer

org.apache.spark.sql.AnalysisException: cannot resolve "`id`" given input columns

I have used a header-less csv file ...READ MORE

Jul 14, 2019 in Apache Spark by Puneet
17,410 views
0 votes
1 answer

Difference between cogroup and full outer join in spark

Please go through the below explanation : Full ...READ MORE

Jul 14, 2019 in Apache Spark by Kiran
9,373 views
0 votes
1 answer

Working of map function on data

The map function creates an array of ...READ MORE

Jul 11, 2019 in Apache Spark by Krish
691 views
0 votes
1 answer

Spark error: Caused by: java.lang.RuntimeException: The root scratch dir: /tmp/hive on HDFS should be writable.

Give  read-write permissions to  C:\tmp\hive folder Cd to winutils bin folder ...READ MORE

Jul 11, 2019 in Apache Spark by Rajiv
6,964 views
0 votes
1 answer

Is fetching data from apache flume webcrawling?

Web crawling is a program or automated ...READ MORE

Jul 11, 2019 in Apache Spark by Esha
728 views
0 votes
1 answer

Why do we use sc.parallelize?

Spark revolves around the concept of a ...READ MORE

Jul 11, 2019 in Apache Spark by Suman
13,082 views
0 votes
1 answer

Error Loading data to mysql in Spark

You have to use sqoop to export data ...READ MORE

Jul 11, 2019 in Apache Spark by Jishan
801 views
0 votes
1 answer

Spark Streaming Pyspark code not working

The address you are using in the ...READ MORE

Jul 11, 2019 in Apache Spark by Shir
2,092 views
0 votes
1 answer

Query regarding Appending " to a string in Scala

You can perform this task in two ...READ MORE

Jul 10, 2019 in Apache Spark by Esha
2,117 views
0 votes
1 answer

org.apache.spark.sql.AnalysisException: cannot resolve given input columns

The string Productivity has to be enclosed between single ...READ MORE

Jul 10, 2019 in Apache Spark by Tina
41,982 views
0 votes
1 answer

Query regarding Operator Overloading in Scala

All prefix operators' symbols are predefined: +, -, ...READ MORE

Jul 10, 2019 in Apache Spark by Karan
1,333 views
0 votes
2 answers

Error : split value is not a member of org.apache.spark.sql.Row

var d=rdd2col.rdd.map(x=>x.split(",")) or val names=rd ...READ MORE

Aug 5, 2020 in Apache Spark by Ramkumar Ramasamy.
11,058 views
0 votes
1 answer

Scala join comma delimited file as tables

Dataframe creation commands:​ Now we will register them ...READ MORE

Jul 9, 2019 in Apache Spark by Suraj
731 views
0 votes
1 answer

How to use yield keyword in scala and why it is used instead of println?

Hi, The yield keyword is used because the ...READ MORE

Jul 6, 2019 in Apache Spark by Gitika
• 65,910 points
1,679 views
0 votes
1 answer

How to print loop with condition in scala?

Hi, Yes, in scala there is a guard condition where ...READ MORE

Jul 6, 2019 in Apache Spark by Gitika
• 65,910 points
394 views
0 votes
1 answer

How to implement two level loop in scala?

Hi, You can use two level loops using the ...READ MORE

Jul 5, 2019 in Apache Spark by Gitika
• 65,910 points
576 views
0 votes
1 answer

How can you use "for" statement in scala to print list from collection?

Hi, You can use for loop in scala using ...READ MORE

Jul 5, 2019 in Apache Spark by Gitika
• 65,910 points
489 views
0 votes
1 answer

How can we iterate any function using "foreach" function in scala?

Hi, Yes, "foreach" function you use because it will ...READ MORE

Jul 5, 2019 in Apache Spark by Gitika
• 65,910 points
1,329 views
0 votes
1 answer

How to print string text in scala?

Hi, You can see this example to see ...READ MORE

Jul 5, 2019 in Apache Spark by Gitika
• 65,910 points
2,173 views
0 votes
1 answer

How can we optimize and minimize the memory when work with scala use case?

Hi, There is a term in Scala that is ...READ MORE

Jul 5, 2019 in Apache Spark by Gitika
• 65,910 points
680 views
0 votes
1 answer

Which syntax to use to take the sum of list of collection in scala?

Hi, You can see this example to get ...READ MORE

Jul 5, 2019 in Apache Spark by Gitika
• 65,910 points
1,519 views
0 votes
1 answer

How to assign block expression in scala?

Hi, You can follow this example to know ...READ MORE

Jul 5, 2019 in Apache Spark by Gitika
• 65,910 points
673 views
0 votes
1 answer

load/save in spark

The reason why you are able to ...READ MORE

Jul 5, 2019 in Apache Spark by Firoz
1,324 views
0 votes
1 answer

How to create dataframe for the comma delimited file?

 Refer to the below command used: val df ...READ MORE

Jul 5, 2019 in Apache Spark by karan
2,944 views
0 votes
1 answer

Spark Installation problem

After downloading Spark, you need to set ...READ MORE

Jul 5, 2019 in Apache Spark by Rishi
4,043 views
0 votes
1 answer

When we create an RDD, does it bring the data and load it into the memory?

Hi, No. An RDD is made up of ...READ MORE

Jul 5, 2019 in Apache Spark by Gitika
• 65,910 points
577 views
0 votes
1 answer

How to create scala project in intellij?

You have to install Intellij with scala plugin. ...READ MORE

Jul 5, 2019 in Apache Spark by Jimmy
2,034 views