How to authenticate Spark internal connections using a secret key

Question

I have enabled Spark authentication for internal connections in Spark. Now instead of normal authentication, I want the authentication to be done using a secret key. How can I do this?

Answer 1

You need to set the secret key that you want to use for authentication. After you have enabled authentication, set the secret key as follows:

val sc = new SparkContext(new SparkConf())

./bin/spark-submit <all your existing options> --spark.authenticate.secret= <secret key>