Set archives to be extracted in executor directory

0 votes
Hello. I want some archives to be extracted in the working directory of each executor. I am not sure if I can just copy and paste it(tried doing it but no luck). Can someone tell me how to do it?
Mar 28 in Apache Spark by Judy
77 views

1 answer to this question.

0 votes

I don't think you can copy and paste the archives. You will have to set the archives to a spark Yarn property. Try this:

val sc = new SparkContext(new SparkConf())

./bin/spark-submit <all your existing options> --spark.yarn.dist.archives= <comma separated values of archives>
answered Mar 28 by Raj

Related Questions In Apache Spark

0 votes
1 answer

How to set stage abort in Spark?

You can do it as follows. Use ...READ MORE

answered Mar 12 in Apache Spark by veer
88 views
0 votes
1 answer

How to set executors for static allocation in Spark Yarn?

Open Spark shell and run the following ...READ MORE

answered Mar 28 in Apache Spark by Raj
151 views
0 votes
1 answer

Which query to use for better performance, join in SQL or using Dataset API?

DataFrames and SparkSQL performed almost about the ...READ MORE

answered Apr 19, 2018 in Apache Spark by kurt_cobain
• 9,240 points
107 views
0 votes
1 answer

Efficient way to read specific columns from parquet file in spark

As parquet is a column based storage ...READ MORE

answered Apr 20, 2018 in Apache Spark by kurt_cobain
• 9,240 points
1,227 views
0 votes
1 answer
0 votes
1 answer

Hadoop Mapreduce word count Program

Firstly you need to understand the concept ...READ MORE

answered Mar 16, 2018 in Data Analytics by nitinrawat895
• 10,670 points
2,736 views
0 votes
1 answer

hadoop.mapred vs hadoop.mapreduce?

org.apache.hadoop.mapred is the Old API  org.apache.hadoop.mapreduce is the ...READ MORE

answered Mar 16, 2018 in Data Analytics by nitinrawat895
• 10,670 points
289 views
0 votes
10 answers

hadoop fs -put command?

put syntax: put <localSrc> <dest> copy syntax: copyFr ...READ MORE

answered Dec 7, 2018 in Big Data Hadoop by Aditya
13,563 views
0 votes
1 answer

How to store files in executor's working directory?

You have to specify a comma-separated list ...READ MORE

answered Mar 28 in Apache Spark by Raj
303 views
0 votes
1 answer