Question

How to find the number of null contain in dataframe

Answer 1

Hey there!

You can use the select method of the dataframe to filter out the values.

df.select([count(when(isnull(c), c)).alias(c) for c in df.columns]).show()

This will display a table with column names and the number of Null values in each column.

If you want to check Null values for a column, then you can use the below code:

df.where(df.col("<Enter column name here>").isNull).count()

Comments

I am getting an error with this command and it says "illegal start of simple expresssion". Please help.

df.select([count(when(isnull(c), c)).alias(c) for c in df.columns]).show()