How to replace null values in Spark DataFrame

Question

Announcement! Career Guide 2019 is out now. Explore careers to become a Big Data Developer or Architect!

I want to remove null values from a csv file. So tried the following things.

val df = sqlContext.read.format("com.databricks.spark.csv").option("header", "true").load("/usr/local/spark/cars.csv")

After loading the file it looks like as shown below. Now, I want to remove null values.

So, I do this :

df.na.fill("e",Seq("blank"))
But the null values didn't change.Can anyone help me?

Answer 1

This is basically very simple. You'll need to create a new DataFrame. I'm using the DataFrame df that you have defined earlier.

val newDf = df.na.fill("e",Seq("blank"))

DataFrames are immutable structures. Each time you perform a transformation which you need to store, you'll need to affect the transformed DataFrame to a new value.

You can even check out the details of a successful Spark developers with the Pyspark training course.

Answer 2

val map = Map("comment" -> "a", "blank" -> "a2")

df.na.fill(map).show()

Answer 3

df1 = df.na().fill("e",Seq("blank"));

Answer 4

String[] colNames = {"NameOfColumn"}
dataframe = dataframe.na.fill("ValueToBeFilled", colNames)

Answer 5

def isEvenOption(n: Integer): Option[Boolean] = {
  val num = Option(n).getOrElse(return None)
  Some(num % 2 == 0)
}

val isEvenOptionUdf = udf[Option[Boolean], Integer](isEvenOption)

Source: Dealing with null in Spark

Comments

For ,we have to use, drop()

DF.na.drop()
.show(false)

drop() will remove all the null from the DF

Answer 6

Hi i hope this will help for you.

option("nullValue","defaultvalue")

val df = sqlContext.read.format("com.databricks.spark.csv").option("nullValue","defaultvalue").option("header", "true").load("/usr/local/spark/cars.csv"

Comments

Is a closed parenthesis missing at the end of the command?

---

Sir, Can you please explain this code?

Answer 7

in spark 2.x you can directly use df.dropna()  you can drop null from dataframe

Answer 8

Hi,

In Spark, fill() function of DataFrameNaFunctions class is used to replace NULL values on the DataFrame column with either zero(0), empty string, space, or any constant literal values.

//Replace all integer and long columns
df.na.fill(0)
    .show(false)

//Replace with specific columns
df.na.fill(0,Array("population"))
  .show(false)