Prevent immediate shutdown of Streaming context after JVM shutdown

Question

I have a spark application running and am facing some issue. Whenever the JVM shuts down, the Streaming context immediately shuts down. I want the context to shut down properly. So how to disable the immediate shut down function?

Answer 1

In technical terms, you want to gracefully shut down the Streaming Context. Use the below command to do this:

val sc = new SparkContext(new SparkConf())
./bin/spark-submit <all your existing options> --spark.streaming.stopGracefullyOnShutdown=true