sparkstream textfilstreaming localpathdirectory I am getting empty results

+1 vote

I have given the local path for spark text file streaming. But it is giving empty results even after adding new files. The path exists and it is not open also. Can you please tell how to rectify this.

Below is the code:

val pathFile:Path=new Path(
  "file:\\\\\\D:\\sparklearning\\inputFileStreaming\\"
)
println(pathFile.toString())
val linesDStream = streamingContext.textFileStream(pathFile.toString)
linesDStream.foreachRDD(x=>println(x.toString()))
Nov 7, 2019 in Apache Spark by c.kothamasu
• 160 points
172 views

2 answers to this question.

+1 vote
Hi, according to the documentation, you have to mention the path of hdfs. So, start the Spark job, copy the file form your local file system to hdfs and then use Spark text file streaming.
answered Nov 7, 2019 by Ryan
+1 vote
Hey @c.kothamasu

You should copy your file to HDFS because Spark process only those files that are copied into HDFS while the program is running.
answered Nov 7, 2019 by Manas

Related Questions In Apache Spark

0 votes
1 answer

I am not able to run the apache spark program in mac oc

Hi@Srinath, It seems you didn't set Hadoop for ...READ MORE

answered Sep 21, 2020 in Apache Spark by MD
• 95,180 points
193 views
0 votes
1 answer

Can I read a CSV represented as a string into Apache Spark?

You can use the following command. This ...READ MORE

answered May 3, 2018 in Apache Spark by kurt_cobain
• 9,390 points
645 views
0 votes
1 answer

Getting error while connecting zookeeper in Kafka - Spark Streaming integration

I guess you need provide this kafka.bootstrap.servers ...READ MORE

answered May 24, 2018 in Apache Spark by Shubham
• 13,480 points
1,664 views
0 votes
2 answers

In a Spark DataFrame how can I flatten the struct?

// Collect data from input avro file ...READ MORE

answered Jul 4, 2019 in Apache Spark by Dhara dhruve
3,609 views
+1 vote
2 answers
+1 vote
1 answer

Hadoop Mapreduce word count Program

Firstly you need to understand the concept ...READ MORE

answered Mar 16, 2018 in Data Analytics by nitinrawat895
• 11,380 points
7,208 views
0 votes
1 answer

hadoop.mapred vs hadoop.mapreduce?

org.apache.hadoop.mapred is the Old API  org.apache.hadoop.mapreduce is the ...READ MORE

answered Mar 16, 2018 in Data Analytics by nitinrawat895
• 11,380 points
1,169 views
+2 votes
11 answers

hadoop fs -put command?

Hi, You can create one directory in HDFS ...READ MORE

answered Mar 16, 2018 in Big Data Hadoop by nitinrawat895
• 11,380 points
52,785 views
0 votes
1 answer

How do I access the Map Task ID in Spark?

You can access task information using TaskContext: import org.apache.spark.TaskContext sc.parallelize(Seq[Int](), ...READ MORE

answered Jul 23, 2019 in Apache Spark by ravikiran
• 4,620 points
404 views