sparkstream.textfilstreaming(localpathdirectory). I am getting empty results

+1 vote

I have given the local path for spark text file streaming. But it is giving empty results even after adding new files. The path exists and it is not open also. Can you please tell how to rectify this.

Below is the code:

val pathFile:Path=new Path(
  "file:\\\\\\D:\\sparklearning\\inputFileStreaming\\"
)
println(pathFile.toString())
val linesDStream = streamingContext.textFileStream(pathFile.toString)
linesDStream.foreachRDD(x=>println(x.toString()))
Nov 7, 2019 in Apache Spark by c.kothamasu
• 160 points
133 views

2 answers to this question.

+1 vote
Hi, according to the documentation, you have to mention the path of hdfs. So, start the Spark job, copy the file form your local file system to hdfs and then use Spark text file streaming.
answered Nov 7, 2019 by Ryan
+1 vote
Hey @c.kothamasu

You should copy your file to HDFS because Spark process only those files that are copied into HDFS while the program is running.
answered Nov 7, 2019 by Manas

Related Questions In Apache Spark

0 votes
1 answer

I am not able to run the apache spark program in mac oc

Hi@Srinath, It seems you didn't set Hadoop for ...READ MORE

answered Sep 21 in Apache Spark by MD
• 65,200 points
41 views
0 votes
1 answer

Can I read a CSV represented as a string into Apache Spark?

You can use the following command. This ...READ MORE

answered May 3, 2018 in Apache Spark by kurt_cobain
• 9,320 points
246 views
0 votes
1 answer

Getting error while connecting zookeeper in Kafka - Spark Streaming integration

I guess you need provide this kafka.bootstrap.servers ...READ MORE

answered May 24, 2018 in Apache Spark by Shubham
• 13,450 points
1,334 views
0 votes
2 answers

In a Spark DataFrame how can I flatten the struct?

// Collect data from input avro file ...READ MORE

answered Jul 4, 2019 in Apache Spark by Dhara dhruve
3,059 views
+1 vote
2 answers
+1 vote
1 answer

Hadoop Mapreduce word count Program

Firstly you need to understand the concept ...READ MORE

answered Mar 16, 2018 in Data Analytics by nitinrawat895
• 10,950 points
6,070 views
0 votes
1 answer

hadoop.mapred vs hadoop.mapreduce?

org.apache.hadoop.mapred is the Old API  org.apache.hadoop.mapreduce is the ...READ MORE

answered Mar 16, 2018 in Data Analytics by nitinrawat895
• 10,950 points
925 views
+1 vote
11 answers

hadoop fs -put command?

put syntax: put <localSrc> <dest> copy syntax: copyF ...READ MORE

answered Dec 7, 2018 in Big Data Hadoop by Aditya
38,997 views
0 votes
1 answer

How do I access the Map Task ID in Spark?

You can access task information using TaskContext: import org.apache.spark.TaskContext sc.parallelize(Seq[Int](), ...READ MORE

answered Jul 23, 2019 in Apache Spark by ravikiran
• 4,600 points
247 views