Change number of completed drivers displayed

Question

Hi. Please help me to change the number of completed drivers to be displayed in the UI. Please tell me what command I should use to change this dynamically?

Answer 1

You can change the number of completed drivers displayed dynamically using the following command:

val sc = new SparkContext(new SparkConf())

./bin/spark-submit <all your existing options> --spark.deploy.retainedDrivers= <number of drivers to be displayed>