Monitoring Spark application

Question

Hi, in Hadoop map-reduce form edge node we had run jar file. But in spark on which node we will execute spark-submit
1) How do we monitor the spark application, is there any UI,
2) If UI is there which node of the cluster it is representing.

Answer 1

Spark-submit jobs are also run from client/edge node but its execution environment will depend on the parameters --master and --deploy-mode such as if the --deploy-mode is cluster then the driver class of the code will be executed in Spark cluster and if its client then it will get executed in the edge node itself.

1. To monitor the spark application there will a spark web UI from which you can see the spark application current status as well as history. The image attached below will help you understand how.

2. The Spark Web UI will show you all the jobs running in all worker nodes as a whole and no specific number will be mentioned as its an automated thing to choose which worker node will get assigned to which job, only the master node log has the details.