Reading a text file through spark data frame

+1 vote

Hi team,

val df = sc.textFile("HDFS://nameservice1/user/edureka_168049/Structure_IT/samplefile.txt")
df.show()

the above is not working and when checking my NameNode it is saying security is off and safe mode is off.

even the below is also not working

val df = spark.read
.format("csv")
.option("header","true")
.option("inferSchema","true") 
.option("nullValue","NA")
.option("timestampFormat","yyyy-MM-dd"T"HH:mm:ss")
.option("mode","failfast")
.load("hdfs://nameservice1/user/edureka_168049/Structure_IT/sparkfile.csv")
Jul 24, 2019 in Apache Spark by Kiran
26,457 views

1 answer to this question.

0 votes

Try this:

val df = sc.textFile("HDFS://nameservice1/user/edureka_168049/Structure_IT/samplefile.txt")
df.collect()
val df = spark.read.option("header","true").option("inferSchema","true").csv("hdfs://localhost :9000/airports.csv")
df.show()
To know more about SQL, It's recommended to join PySpark course today.
answered Jul 24, 2019 by Suri

Related Questions In Apache Spark

+1 vote
1 answer

How to read a data from text file in Spark?

Hey, You can try this: from pyspark import SparkContext SparkContext.stop(sc) sc ...READ MORE

answered Aug 6, 2019 in Apache Spark by Gitika
• 65,770 points
5,043 views
+1 vote
1 answer

How can I write a text file in HDFS not from an RDD, in Spark program?

Yes, you can go ahead and write ...READ MORE

answered May 29, 2018 in Apache Spark by Shubham
• 13,490 points
8,494 views
+1 vote
1 answer

Scala: Convert text file data into ORC format using data frame

Converting text file to Orc: Using Spark, the ...READ MORE

answered Aug 1, 2019 in Apache Spark by Esha
3,641 views
+1 vote
1 answer

getting null values in spark dataframe while reading data from hbase

Can you share the screenshots for the ...READ MORE

answered Jul 31, 2018 in Apache Spark by kurt_cobain
• 9,350 points
2,337 views
+1 vote
2 answers
0 votes
1 answer

How to find the number of null contain in dataframe?

Hey there! You can use the select method of the ...READ MORE

answered May 3, 2019 in Apache Spark by Omkar
• 69,220 points
5,136 views
+2 votes
4 answers

use length function in substring in spark

You can use the function expr val data ...READ MORE

answered May 3, 2018 in Apache Spark by kurt_cobain
• 9,350 points
43,054 views
0 votes
3 answers

How to connect Spark to a remote Hive server?

JDBC is not required here. Create a hive ...READ MORE

answered Mar 8, 2019 in Big Data Hadoop by Vijay Dixon
• 190 points
12,816 views
0 votes
1 answer

What is a Parquet file in Spark?

Hey, Parquet is a columnar format file supported ...READ MORE

answered Jul 2, 2019 in Apache Spark by Gitika
• 65,770 points
1,389 views
0 votes
1 answer

How to open/stream .zip files through Spark?

You can try and check this below ...READ MORE

answered Nov 20, 2018 in Apache Spark by Frankie
• 9,830 points
2,517 views
webinar REGISTER FOR FREE WEBINAR X
REGISTER NOW
webinar_success Thank you for registering Join Edureka Meetup community for 100+ Free Webinars each month JOIN MEETUP GROUP