Filtering a row in Spark DataFrame based on matching values from a list

Question

Announcement! Career Guide 2019 is out now. Explore careers to become a Big Data Developer or Architect!

I am filtering the Spark DataFrame using filter:

var notFollowingList=List(9.8,7,6,3,1)
df.filter(col("uid”).isin(notFollowingList))

But I get an error saying:

Unsupported literal type classscala.collection.immutable.$colon$colon 

Can anyone help me in resolving the error?

Answer 1

Use the function as following:

var notFollowingList=List(9.8,7,6,3,1)
df.filter(col("uid").isin(notFollowingList:_*))

You can even check out the details of a successful Spark developers with the Pyspark training course.

Answer 2

You need isInCollection

Answer 3

Is it possible to filter Spark DataFrames to return all rows where a , How can I return only the rows of a Spark DataFrame where the values for a column are within a specified list? Here's my Python pandas way of  How can I return only the rows of a Spark DataFrame where the values for a column are within a specified list? Here's my Python pandas way of doing this operation: df_start = df[df['name'].isin(['App Opened', 'App Launched'])].copy() I saw this SO scala implementation and tried several permutations, but couldn't get it to work.