Set archives to be extracted in executor directory

0 votes
Hello. I want some archives to be extracted in the working directory of each executor. I am not sure if I can just copy and paste it(tried doing it but no luck). Can someone tell me how to do it?
Mar 28, 2019 in Apache Spark by Judy
224 views

1 answer to this question.

0 votes

I don't think you can copy and paste the archives. You will have to set the archives to a spark Yarn property. Try this:

val sc = new SparkContext(new SparkConf())

./bin/spark-submit <all your existing options> --spark.yarn.dist.archives= <comma separated values of archives>
answered Mar 28, 2019 by Raj

Related Questions In Apache Spark

0 votes
1 answer

How to set stage abort in Spark?

You can do it as follows. Use ...READ MORE

answered Mar 12, 2019 in Apache Spark by veer
129 views
0 votes
1 answer

How to set executors for static allocation in Spark Yarn?

Open Spark shell and run the following ...READ MORE

answered Mar 28, 2019 in Apache Spark by Raj
266 views
0 votes
1 answer

Which query to use for better performance, join in SQL or using Dataset API?

DataFrames and SparkSQL performed almost about the ...READ MORE

answered Apr 19, 2018 in Apache Spark by kurt_cobain
• 9,290 points
170 views
0 votes
1 answer

Efficient way to read specific columns from parquet file in spark

As parquet is a column based storage ...READ MORE

answered Apr 20, 2018 in Apache Spark by kurt_cobain
• 9,290 points
1,721 views
+1 vote
1 answer
+1 vote
1 answer

Hadoop Mapreduce word count Program

Firstly you need to understand the concept ...READ MORE

answered Mar 16, 2018 in Data Analytics by nitinrawat895
• 10,840 points
3,941 views
0 votes
1 answer

hadoop.mapred vs hadoop.mapreduce?

org.apache.hadoop.mapred is the Old API  org.apache.hadoop.mapreduce is the ...READ MORE

answered Mar 16, 2018 in Data Analytics by nitinrawat895
• 10,840 points
545 views
+1 vote
11 answers

hadoop fs -put command?

put syntax: put <localSrc> <dest> copy syntax: copyFr ...READ MORE

answered Dec 7, 2018 in Big Data Hadoop by Aditya
21,042 views
0 votes
1 answer

How to store files in executor's working directory?

You have to specify a comma-separated list ...READ MORE

answered Mar 28, 2019 in Apache Spark by Raj
590 views
0 votes
1 answer