Cannot resolve Error In Spark when filter records with two where condition

Question

SPARK 1.6, SCALA, MAVEN

i have created a dataframe from RDD and trying to filter out all records where cola= null or empty string and colb = 2 or 3.

i tried something like this.

df.filter($"cola in(null or '') where colb ='01' & '05'")

but getting cannot resolve error. May someone please help, am i missing anything. kindly suggest

Comments

Can you post the error logs? Cannot resolve could be for multiple reasons.

Answer 1

Try

df.where($"cola".isNotNull && $"cola" =!= "" && !$"colb".isin(2,3))

your syntax is incorect.