sparkstream.textfilstreaming(localpathdirectory). I am getting empty results

0 votes

I have given the local path for spark text file streaming. But it is giving empty results even after adding new files. The path exists and it is not open also. Can you please tell how to rectify this.

Below is the code:

val pathFile:Path=new Path(
  "file:\\\\\\D:\\sparklearning\\inputFileStreaming\\"
)
println(pathFile.toString())
val linesDStream = streamingContext.textFileStream(pathFile.toString)
linesDStream.foreachRDD(x=>println(x.toString()))
Nov 7 in Apache Spark by c.kothamasu
• 140 points
36 views

2 answers to this question.

0 votes
Hi, according to the documentation, you have to mention the path of hdfs. So, start the Spark job, copy the file form your local file system to hdfs and then use Spark text file streaming.
answered Nov 7 by Ryan
0 votes
Hey @c.kothamasu

You should copy your file to HDFS because Spark process only those files that are copied into HDFS while the program is running.
answered Nov 7 by Manas

Related Questions In Apache Spark

0 votes
1 answer

Can I read a CSV represented as a string into Apache Spark?

You can use the following command. This ...READ MORE

answered May 3, 2018 in Apache Spark by kurt_cobain
• 9,260 points
69 views
0 votes
1 answer

Getting error while connecting zookeeper in Kafka - Spark Streaming integration

I guess you need provide this kafka.bootstrap.servers ...READ MORE

answered May 24, 2018 in Apache Spark by Shubham
• 13,310 points
702 views
0 votes
2 answers

In a Spark DataFrame how can I flatten the struct?

// Collect data from input avro file ...READ MORE

answered Jul 4 in Apache Spark by Dhara dhruve
1,214 views
0 votes
1 answer

How can I compare the elements of the RDD using MapReduce?

You have to use the comparison operator ...READ MORE

answered May 24, 2018 in Apache Spark by Shubham
• 13,310 points
456 views
0 votes
1 answer
0 votes
1 answer

Hadoop Mapreduce word count Program

Firstly you need to understand the concept ...READ MORE

answered Mar 16, 2018 in Data Analytics by nitinrawat895
• 10,730 points
3,370 views
0 votes
1 answer

hadoop.mapred vs hadoop.mapreduce?

org.apache.hadoop.mapred is the Old API  org.apache.hadoop.mapreduce is the ...READ MORE

answered Mar 16, 2018 in Data Analytics by nitinrawat895
• 10,730 points
405 views
0 votes
10 answers

hadoop fs -put command?

put syntax: put <localSrc> <dest> copy syntax: copyFr ...READ MORE

answered Dec 7, 2018 in Big Data Hadoop by Aditya
16,730 views
0 votes
1 answer

How do I access the Map Task ID in Spark?

You can access task information using TaskContext: import org.apache.spark.TaskContext sc.parallelize(Seq[Int](), ...READ MORE

answered Jul 23 in Apache Spark by ravikiran
• 4,560 points
53 views