How to index one csv file with no header after converting the csv to a dataframe i need to name the columns in order to normalize in minmaxScaler

0 votes
Sep 9, 2020 in Apache Spark by Manas
• 120 points
371 views

1 answer to this question.

0 votes

Hi@Manas,

You can read your dataset from CSV file to Dataframe and set header value to false. So it will create a data frame with the index value.

df = spark.read.format("csv").option("header", "false").load("csvfile.csv")

After that, you can replace the index value with column name.

val df2 = df.withColumnRenamed(0,"DateOfBirth")
           .withColumnRenamed(1,"salary")
df2.printSchema()
answered Sep 10, 2020 by MD
• 95,060 points

Related Questions In Apache Spark

0 votes
2 answers

In a Spark DataFrame how can I flatten the struct?

// Collect data from input avro file ...READ MORE

answered Jul 4, 2019 in Apache Spark by Dhara dhruve
3,449 views
+1 vote
2 answers
0 votes
1 answer

How to remove the elements with a key present in any other RDD?

Hey, You can use the subtractByKey () function to ...READ MORE

answered Jul 22, 2019 in Apache Spark by Gitika
• 65,870 points
1,294 views
+1 vote
1 answer
0 votes
1 answer

if i want to see my public key after running cat <path> command in gitbash but saying no such file or directory.

Hey, @KK, You can fix this issue may be ...READ MORE

answered May 26, 2020 in Apache Spark by Gitika
• 65,870 points
151 views
0 votes
1 answer
+1 vote
1 answer

How can I write a text file in HDFS not from an RDD, in Spark program?

Yes, you can go ahead and write ...READ MORE

answered May 29, 2018 in Apache Spark by Shubham
• 13,480 points
4,802 views
0 votes
1 answer
+2 votes
14 answers

How to create new column with function in Spark Dataframe?

val coder: (Int => String) = v ...READ MORE

answered Apr 4, 2019 in Apache Spark by anonymous

edited Apr 5, 2019 by Omkar 64,908 views
0 votes
1 answer

How to parse a textFile to csv in pyspark?

Hi, Use this below given code, it will ...READ MORE

answered Apr 13, 2020 in Apache Spark by MD
• 95,060 points
887 views