How to groupBy count then filter on count in Scala

0 votes

I'm using spark 2.1, I was trying to use the groupBy on the "count" column i have. It throws an exception.

Code: 

df.groupBy("travel").count()
  .filter("count >= 1000")
  .show()

java.lang.RuntimeException: [1.15] failure: ``('' expected but `>=' found count >= 1000
Apr 19, 2018 in Big Data Hadoop by Shubham
• 13,490 points
29,623 views

1 answer to this question.

0 votes

I think the exception is caused because you used the keyword Count.

Now when you use the filter function, in the background it's actually SQL code running. So count being a keyword in SQL is misinterpreted here.

You can either specify it as a column by using $ sign

df.groupBy("travel").count()
  .filter($"count >= 1000")
  .show()

Alternatively, you can use the rename function also

df.groupBy("travel").count().withColumnRenamed("count", "x")
  .filter("x >= 1000")
  .show()


Hope this helps!!

If you need to learn more about Scala, It's recommended to join Scala Certification course today.

Thank you!

answered Apr 19, 2018 by kurt_cobain
• 9,390 points

Related Questions In Big Data Hadoop

0 votes
1 answer

How to count lines in a file on hdfs command?

Use the below commands: Total number of files: hadoop ...READ MORE

answered Aug 10, 2018 in Big Data Hadoop by Neha
• 6,300 points
28,675 views
+1 vote
0 answers

How to set up Hadoop cluster on Mac in intelliJ IDEA

I have Installed hadoop using brew and ...READ MORE

Jul 25, 2018 in Big Data Hadoop by Neha
• 6,300 points
1,053 views
+1 vote
1 answer

How to count number of rows in alias in PIG?

COUNT is part of pig LOGS= LOAD 'log'; LOGS_GROUP= ...READ MORE

answered Oct 15, 2018 in Big Data Hadoop by Omkar
• 69,220 points
2,772 views
0 votes
3 answers

Spark Scala: How to list all folders in directory

val spark = SparkSession.builder().appName("Demo").getOrCreate() val path = new ...READ MORE

answered Dec 5, 2018 in Big Data Hadoop by Mark
17,416 views
0 votes
1 answer

Writing File into HDFS using spark scala

The reason you are not able to ...READ MORE

answered Apr 6, 2018 in Big Data Hadoop by kurt_cobain
• 9,390 points
17,278 views
0 votes
1 answer

Changing Column position in spark dataframe

Yes, you can reorder the dataframe elements. You need ...READ MORE

answered Apr 19, 2018 in Apache Spark by Ashish
• 2,650 points
13,708 views
+5 votes
11 answers

Concatenate columns in apache spark dataframe

its late but this how you can ...READ MORE

answered Mar 21, 2019 in Apache Spark by anonymous
72,294 views
0 votes
3 answers

How to transpose Spark DataFrame?

Please check the below mentioned links for ...READ MORE

answered Jan 1, 2019 in Apache Spark by anonymous
19,848 views
0 votes
1 answer

How to start working on Hadoop?

Ok, so basically, you are looking forward ...READ MORE

answered Mar 30, 2018 in Big Data Hadoop by kurt_cobain
• 9,390 points
933 views
webinar REGISTER FOR FREE WEBINAR X
REGISTER NOW
webinar_success Thank you for registering Join Edureka Meetup community for 100+ Free Webinars each month JOIN MEETUP GROUP