2677/use-length-function-in-substring-in-spark
I'm using spark 2.1.
Using a length function inside a substring for a Dataframe is giving me an error (mismatch).
val SSDF = testDF.withColumn("newcol", substring($"col", 1, length($"col")-1))
You can use the function expr
val data = List("..", "...", "...") val df = sparkContext.parallelize(data).toDF("value") val result = df.withColumn("cutted", expr("substring(value, 1, length(value)-1)")) result.show(false)
This might help
You can try this
val substrDF =testDF.withColumn("newcol", $"col".substr(lit(1), length($"col")-1))
You have passed the wrong parameters. Here is the right syntax:
substring(str: Column, pos: Int, len: Int): Column
You can also use UDF
testDF.withColumn("newcol", regexp_replace($"name", ".$" , "") ).show
val coder: (Int => String) = v ...READ MORE
I can list some but there can ...READ MORE
Seems like master and worker are not ...READ MORE
The sliding function is used when you ...READ MORE
Instead of spliting on '\n'. You should ...READ MORE
Hey there! You can use the select method of the ...READ MORE
JDBC is not required here. Create a hive ...READ MORE
Please check the below mentioned links for ...READ MORE
DataFrames and SparkSQL performed almost about the ...READ MORE
As parquet is a column based storage ...READ MORE
OR
At least 1 upper-case and 1 lower-case letter
Minimum 8 characters and Maximum 50 characters
Already have an account? Sign in.