Reading a text file through spark data frame

+1 vote

Hi team,

val df = sc.textFile("HDFS://nameservice1/user/edureka_168049/Structure_IT/samplefile.txt")
df.show()

the above is not working and when checking my NameNode it is saying security is off and safe mode is off.

even the below is also not working

val df = spark.read
.format("csv")
.option("header","true")
.option("inferSchema","true") 
.option("nullValue","NA")
.option("timestampFormat","yyyy-MM-dd"T"HH:mm:ss")
.option("mode","failfast")
.load("hdfs://nameservice1/user/edureka_168049/Structure_IT/sparkfile.csv")
Jul 24, 2019 in Apache Spark by Kiran
4,596 views

1 answer to this question.

0 votes

Try this:

val df = sc.textFile("HDFS://nameservice1/user/edureka_168049/Structure_IT/samplefile.txt")
df.collect()
val df = spark.read.option("header","true").option("inferSchema","true").csv("hdfs://localhost :9000/airports.csv")
df.show()
answered Jul 24, 2019 by Suri

Related Questions In Apache Spark

+1 vote
1 answer

How to read a data from text file in Spark?

Hey, You can try this: from pyspark import SparkContext SparkContext.stop(sc) sc ...READ MORE

answered Aug 6, 2019 in Apache Spark by Gitika
• 25,440 points
817 views
+1 vote
1 answer

How can I write a text file in HDFS not from an RDD, in Spark program?

Yes, you can go ahead and write ...READ MORE

answered May 29, 2018 in Apache Spark by Shubham
• 13,370 points
1,977 views
+1 vote
1 answer

Scala: Convert text file data into ORC format using data frame

Converting text file to Orc: Using Spark, the ...READ MORE

answered Aug 1, 2019 in Apache Spark by Esha
434 views
+1 vote
1 answer

getting null values in spark dataframe while reading data from hbase

Can you share the screenshots for the ...READ MORE

answered Jul 31, 2018 in Apache Spark by kurt_cobain
• 9,290 points
537 views
+1 vote
1 answer
0 votes
1 answer

How to find the number of null contain in dataframe?

Hey there! You can use the select method of the ...READ MORE

answered May 3, 2019 in Apache Spark by Omkar
• 68,840 points
346 views
+2 votes
4 answers

use length function in substring in spark

You can use the function expr val data ...READ MORE

answered May 3, 2018 in Apache Spark by kurt_cobain
• 9,290 points
18,463 views
0 votes
3 answers

How to connect Spark to a remote Hive server?

JDBC is not required here. Create a hive ...READ MORE

answered Mar 8, 2019 in Big Data Hadoop by Vijay Dixon
• 190 points
1,837 views
0 votes
1 answer

What is a Parquet file in Spark?

Hey, Parquet is a columnar format file supported ...READ MORE

answered Jul 2, 2019 in Apache Spark by Gitika
• 25,440 points
96 views
0 votes
1 answer

How to open/stream .zip files through Spark?

You can try and check this below ...READ MORE

answered Nov 20, 2018 in Apache Spark by Frankie
• 9,810 points
538 views