Set archives to be extracted in executor directory

Question

Hello. I want some archives to be extracted in the working directory of each executor. I am not sure if I can just copy and paste it(tried doing it but no luck). Can someone tell me how to do it?

Answer 1

I don't think you can copy and paste the archives. You will have to set the archives to a spark Yarn property. Try this:

val sc = new SparkContext(new SparkConf())

./bin/spark-submit <all your existing options> --spark.yarn.dist.archives= <comma separated values of archives>