Retaining the batch by status APIs before Garbage Collection

0 votes
By default Spark Streaming UI and status APIs are retaining some amount of batch before the garbage collection happens. But I want more number of batches to be retained before garbage collection. How to do this?
Mar 19 in Apache Spark by Jim
24 views

1 answer to this question.

0 votes

By default, 1000 batches are retained by Spark Streaming UI and status API. To change this, you can run the below command:

val sc = new SparkContext(new SparkConf())

./bin/spark-submit <all your existing options> --spark.streaming.ui.retainedBatches=2000
answered Mar 19 by Jai

Related Questions In Apache Spark

0 votes
1 answer

How to increase Garbage Collection speed?

The time interval between Garbage Collection is ...READ MORE

answered Mar 7 in Apache Spark by Pavitra
13 views
0 votes
1 answer

How to limit the cores being used by a cluster?

You can set the maximum number of ...READ MORE

answered Mar 11 in Apache Spark by Raj
39 views
0 votes
1 answer

Which syntax to use to take the sum of list of collection in scala?

Hi, You can see this example to get ...READ MORE

answered Jul 5 in Apache Spark by Gitika
• 25,300 points
32 views
0 votes
1 answer

Spark error: Caused by: java.lang.RuntimeException: The root scratch dir: /tmp/hive on HDFS should be writable.

Give  read-write permissions to  C:\tmp\hive folder Cd to winutils bin folder ...READ MORE

answered Jul 11 in Apache Spark by Rajiv
54 views
0 votes
1 answer
0 votes
1 answer

Hadoop Mapreduce word count Program

Firstly you need to understand the concept ...READ MORE

answered Mar 16, 2018 in Data Analytics by nitinrawat895
• 10,490 points
2,384 views
0 votes
1 answer

hadoop.mapred vs hadoop.mapreduce?

org.apache.hadoop.mapred is the Old API  org.apache.hadoop.mapreduce is the ...READ MORE

answered Mar 16, 2018 in Data Analytics by nitinrawat895
• 10,490 points
243 views
0 votes
10 answers

hadoop fs -put command?

put syntax: put <localSrc> <dest> copy syntax: copyFr ...READ MORE

answered Dec 7, 2018 in Big Data Hadoop by Aditya
12,163 views
0 votes
1 answer
0 votes
1 answer

How to change the location of Spark event logs?

You can change the location where you ...READ MORE

answered Mar 6 in Apache Spark by Rohit
98 views