Set Library to launch Yarn master

Question

I have a custom library that I want to use to launch Spark application master. I am running Spark application in Client mode. I can see that no library is set to be used. How can I set the custom library for the application?

Answer 1

You can make use of Special Library path to make your application use your custom library for Yarn application master. To do this, run the below command in Spark shell after running your Spark application.

val sc = new SparkContext(new SparkConf())

./bin/spark-submit <all your existing options> --spark.yarn.am.extraLibraryPath= <path to custom library>