Retaining the batch by status APIs before Garbage Collection

Question

By default Spark Streaming UI and status APIs are retaining some amount of batch before the garbage collection happens. But I want more number of batches to be retained before garbage collection. How to do this?

Answer 1

By default, 1000 batches are retained by Spark Streaming UI and status API. To change this, you can run the below command:

val sc = new SparkContext(new SparkConf())

./bin/spark-submit <all your existing options> --spark.streaming.ui.retainedBatches=2000