Reading a text file through spark data frame

+1 vote

Hi team,

val df = sc.textFile("HDFS://nameservice1/user/edureka_168049/Structure_IT/samplefile.txt")
df.show()

the above is not working and when checking my NameNode it is saying security is off and safe mode is off.

even the below is also not working

val df = spark.read
.format("csv")
.option("header","true")
.option("inferSchema","true") 
.option("nullValue","NA")
.option("timestampFormat","yyyy-MM-dd"T"HH:mm:ss")
.option("mode","failfast")
.load("hdfs://nameservice1/user/edureka_168049/Structure_IT/sparkfile.csv")
Jul 24, 2019 in Apache Spark by Kiran
13,271 views

1 answer to this question.

0 votes

Try this:

val df = sc.textFile("HDFS://nameservice1/user/edureka_168049/Structure_IT/samplefile.txt")
df.collect()
val df = spark.read.option("header","true").option("inferSchema","true").csv("hdfs://localhost :9000/airports.csv")
df.show()
answered Jul 24, 2019 by Suri

Related Questions In Apache Spark

+1 vote
1 answer

How to read a data from text file in Spark?

Hey, You can try this: from pyspark import SparkContext SparkContext.stop(sc) sc ...READ MORE

answered Aug 6, 2019 in Apache Spark by Gitika
• 33,110 points
2,151 views
+1 vote
1 answer

How can I write a text file in HDFS not from an RDD, in Spark program?

Yes, you can go ahead and write ...READ MORE

answered May 29, 2018 in Apache Spark by Shubham
• 13,380 points
3,582 views
+1 vote
1 answer

Scala: Convert text file data into ORC format using data frame

Converting text file to Orc: Using Spark, the ...READ MORE

answered Aug 1, 2019 in Apache Spark by Esha
974 views
+1 vote
1 answer

getting null values in spark dataframe while reading data from hbase

Can you share the screenshots for the ...READ MORE

answered Jul 31, 2018 in Apache Spark by kurt_cobain
• 9,310 points
800 views
+1 vote
2 answers
0 votes
1 answer

How to find the number of null contain in dataframe?

Hey there! You can use the select method of the ...READ MORE

answered May 3, 2019 in Apache Spark by Omkar
• 69,040 points
635 views
+2 votes
4 answers

use length function in substring in spark

You can use the function expr val data ...READ MORE

answered May 3, 2018 in Apache Spark by kurt_cobain
• 9,310 points
27,397 views
0 votes
3 answers

How to connect Spark to a remote Hive server?

JDBC is not required here. Create a hive ...READ MORE

answered Mar 8, 2019 in Big Data Hadoop by Vijay Dixon
• 190 points
3,181 views
0 votes
1 answer

What is a Parquet file in Spark?

Hey, Parquet is a columnar format file supported ...READ MORE

answered Jul 2, 2019 in Apache Spark by Gitika
• 33,110 points
241 views
0 votes
1 answer

How to open/stream .zip files through Spark?

You can try and check this below ...READ MORE

answered Nov 20, 2018 in Apache Spark by Frankie
• 9,810 points
885 views