Limit displaying completed applications

Question

Hi. I want to limit the number of completed applications that are shown in the UI. Right now, I see too many applications. I want to limit is to a lesser value, say 20. How can I do this?

Answer 1

By default, the number of completed applications displayed on the UI is 200 and the rest are dropped. To change this number to display only 20 completed applications, you can use the below commands:

val sc = new SparkContext(new SparkConf())

./bin/spark-submit <all your existing options> --spark.deploy.retainedApplications=20