Error split value is not a member of org apache spark sql Row

Question

Hi,

Error : split value is not a member of org.apache.spark.sql.Row

I am working on eclipse EdurekaVM, when adding split(",") i am getting this error, please help.

val rdd2col = spark.read.csv("file:///home/edureka/rk/2Cols.csv")
val names = rdd2col.map(line => line.split(",")(0))

Answer 1

spark.read.csv is used when loading into a dataframe. The error message that you are getting states that the 'split' function that you are using is not a member of spark.sql. It is searching split in spark.sql because of the loading technique.

Please refer to the below commands for your requirement:

val rdd2col = sc.textFile("file:///home/edureka/Documents/datasets/emp.csv")
val names = rdd2col.map(line => line.split(",")(1))
names.collect()

Comments

What does (1) mean in map(line => line.split(",")(1))?

---

It indicates the index value. It means the above code splits the data first and return only those values which are in index 1. You can change it with -1, +1 etc. according to your need.

Answer 2

var d=rdd2col.rdd.map(x=>x.split(","))

or

val names=rdd2col.select("name").map(line => line.getString(0)).collect.toList

Comments

Hey,

Thank you for your contribution to the Edureka Community.

Register/Sign up on the community to gain points for further contributions. You may ask questions, answer, upvote, and downvote an answer. Each of these would fetch you points and you could be among the top contributors and win exciting merchandise from Edureka.

Cheers!