What's the difference between 'filter' and 'where' in Spark SQL?

0 votes
I have a file which contains employee data and I want to filter out the results using Spark SQL. So I've tried filter as well as where clause and I found they both works same.

example:
val items =  List(1, 2, 3)
 

using filter
employees.filter($"emp_id".isin(items:_*)).show
 

using where
employees.where($"emp_id".isin(items:_*)).show
 

Got the same result in both the cases.
Can anyone tell me why am I getting the same result using filter and where?
May 23, 2018 in Apache Spark by kurt_cobain
• 9,290 points

recategorized May 23, 2018 by kurt_cobain 9,508 views

1 answer to this question.

0 votes
Both 'filter' and 'where' in Spark SQL gives same result. There is no difference between the two.

// The following are equivalent:
employee.filter($"age" > 15)
employee.where($"age" > 15)
employees.filter($"emp_id".isin(items:_*)).show
employees.where($"emp_id".isin(items:_*)).show

 
It's just filter is simply the standard Scala name for such a function, and where is for people who prefer SQL.
answered May 23, 2018 by nitinrawat895
• 10,840 points

Related Questions In Apache Spark

0 votes
1 answer

What is the difference between persist() and cache() in apache spark?

Hi, persist () allows the user to specify ...READ MORE

answered Jul 3, 2019 in Apache Spark by Gitika
• 25,460 points
745 views
0 votes
1 answer

Difference between cogroup and full outer join in spark

Please go through the below explanation : Full ...READ MORE

answered Jul 13, 2019 in Apache Spark by Kiran
1,787 views
0 votes
0 answers

Difference between map() and mapPartitions() function in Spark??

Hi everyone, Can someone tell me, what is ...READ MORE

2 days ago in Apache Spark by akhtar
• 220 points
14 views
0 votes
1 answer

where can i get spark-terasort.jar and not .scala file, to do spark terasort in windows.

Hi! I found 2 links on github where ...READ MORE

answered Feb 13, 2019 in Apache Spark by Omkar
• 68,860 points
182 views
0 votes
1 answer

SQLInterpreter in Spark

SQL Interpreter & Optimizer handles the functional ...READ MORE

answered Jun 7, 2018 in Apache Spark by kurt_cobain
• 9,290 points
79 views
0 votes
1 answer

Functions of Spark SQL?

Spark SQL is capable of: Loading data from ...READ MORE

answered Sep 3, 2018 in Apache Spark by nitinrawat895
• 10,840 points
133 views
+1 vote
1 answer
0 votes
1 answer

Writing File into HDFS using spark scala

The reason you are not able to ...READ MORE

answered Apr 5, 2018 in Big Data Hadoop by kurt_cobain
• 9,290 points
7,120 views
+1 vote
3 answers

What is the difference between rdd and dataframes in Apache Spark ?

Comparison between Spark RDD vs DataFrame 1. Release ...READ MORE

answered Aug 27, 2018 in Apache Spark by shams
• 3,580 points
19,586 views
0 votes
1 answer

What is the difference between Apache Spark SQLContext vs HiveContext?

Spark 2.0+ Spark 2.0 provides native window functions ...READ MORE

answered May 25, 2018 in Apache Spark by nitinrawat895
• 10,840 points
2,344 views