Reading a text file through spark data frame

0 votes

Hi team,

val df = sc.textFile("HDFS://nameservice1/user/edureka_168049/Structure_IT/samplefile.txt")
df.show()

the above is not working and when checking my NameNode it is saying security is off and safe mode is off.

even the below is also not working

val df = spark.read
.format("csv")
.option("header","true")
.option("inferSchema","true") 
.option("nullValue","NA")
.option("timestampFormat","yyyy-MM-dd"T"HH:mm:ss")
.option("mode","failfast")
.load("hdfs://nameservice1/user/edureka_168049/Structure_IT/sparkfile.csv")
Jul 24 in Apache Spark by Kiran
906 views

1 answer to this question.

0 votes

Try this:

val df = sc.textFile("HDFS://nameservice1/user/edureka_168049/Structure_IT/samplefile.txt")
df.collect()
val df = spark.read.option("header","true").option("inferSchema","true").csv("hdfs://localhost :9000/airports.csv")
df.show()
answered Jul 24 by Suri

Related Questions In Apache Spark

0 votes
1 answer

How to read a data from text file in Spark?

Hey, You can try this: from pyspark import SparkContext SparkContext.stop(sc) sc ...READ MORE

answered Aug 6 in Apache Spark by Gitika
• 25,340 points
159 views
0 votes
1 answer

How can I write a text file in HDFS not from an RDD, in Spark program?

Yes, you can go ahead and write ...READ MORE

answered May 29, 2018 in Apache Spark by Shubham
• 13,300 points
1,397 views
0 votes
1 answer

Scala: Convert text file data into ORC format using data frame

Converting text file to Orc: Using Spark, the ...READ MORE

answered Aug 1 in Apache Spark by Esha
135 views
+1 vote
1 answer

getting null values in spark dataframe while reading data from hbase

Can you share the screenshots for the ...READ MORE

answered Jul 31, 2018 in Apache Spark by kurt_cobain
• 9,260 points
416 views
0 votes
1 answer
0 votes
1 answer

How to find the number of null contain in dataframe?

Hey there! You can use the select method of the ...READ MORE

answered May 3 in Apache Spark by Omkar
• 67,600 points
228 views
+2 votes
4 answers

use length function in substring in spark

You can use the function expr val data ...READ MORE

answered May 3, 2018 in Apache Spark by kurt_cobain
• 9,260 points
14,282 views
0 votes
3 answers

How to connect Spark to a remote Hive server?

JDBC is not required here. Create a hive ...READ MORE

answered Mar 8 in Big Data Hadoop by Vijay Dixon
• 180 points
1,415 views
0 votes
1 answer

What is a Parquet file in Spark?

Hey, Parquet is a columnar format file supported ...READ MORE

answered Jul 2 in Apache Spark by Gitika
• 25,340 points
53 views
0 votes
1 answer

How to open/stream .zip files through Spark?

You can try and check this below ...READ MORE

answered Nov 20, 2018 in Apache Spark by Frankie
• 9,810 points
418 views