Authenticate internal connections in Spark

Question

Hi. How can I set up Spark to authenticate internal connections in an application?

Answer 1

To make Spark authenticate internal connections, you need to enable the authentication property. You can do it like this:

val sc = new SparkContext(new SparkConf())

./bin/spark-submit <all your existing options> --spark.authenticate=true