Filtering a row in Spark DataFrame based on matching values from a list

0 votes

Announcement! Career Guide 2019 is out now. Explore careers to become a Big Data Developer or Architect!

I am filtering the Spark DataFrame using filter:

var notFollowingList=List(9.8,7,6,3,1)
df.filter(col("uid”).isin(notFollowingList))

But I get an error saying:

Unsupported literal type classscala.collection.immutable.$colon$colon 

Can anyone help me in resolving the error?

Jun 5, 2018 in Apache Spark by code799
48,070 views

2 answers to this question.

+1 vote

Use the function as following:

var notFollowingList=List(9.8,7,6,3,1)
df.filter(col("uid").isin(notFollowingList:_*))
answered Jun 5, 2018 by Shubham
• 13,380 points
0 votes

You need isInCollection

answered 1 day ago by anonymous

Related Questions In Apache Spark

+1 vote
1 answer

getting null values in spark dataframe while reading data from hbase

Can you share the screenshots for the ...READ MORE

answered Jul 31, 2018 in Apache Spark by kurt_cobain
• 9,310 points
646 views
0 votes
7 answers

How to replace null values in Spark DataFrame?

in spark 2.x you can directly use ...READ MORE

answered 3 days ago in Apache Spark by gaurav
35,396 views
+1 vote
1 answer

How to read a data from text file in Spark?

Hey, You can try this: from pyspark import SparkContext SparkContext.stop(sc) sc ...READ MORE

answered Aug 6, 2019 in Apache Spark by Gitika
• 26,270 points
1,380 views
+1 vote
1 answer
+1 vote
1 answer
0 votes
3 answers

How to connect Spark to a remote Hive server?

JDBC is not required here. Create a hive ...READ MORE

answered Mar 8, 2019 in Big Data Hadoop by Vijay Dixon
• 190 points
2,264 views
0 votes
3 answers

How to transpose Spark DataFrame?

Please check the below mentioned links for ...READ MORE

answered Dec 31, 2018 in Apache Spark by anonymous
9,435 views
0 votes
1 answer

Different Spark Ecosystem

Spark has various components: Spark SQL (Shark)- for ...READ MORE

answered Jun 4, 2018 in Apache Spark by kurt_cobain
• 9,310 points
116 views
0 votes
2 answers

In a Spark DataFrame how can I flatten the struct?

// Collect data from input avro file ...READ MORE

answered Jul 4, 2019 in Apache Spark by Dhara dhruve
2,007 views
+1 vote
1 answer

How can I write a text file in HDFS not from an RDD, in Spark program?

Yes, you can go ahead and write ...READ MORE

answered May 29, 2018 in Apache Spark by Shubham
• 13,380 points
2,585 views