How to increase HDFS replication level in Spark

Question

Hi guys. I need some help with Spark HDFS replication level. The level is set to 3 right now by default. I want to know how I can change this and make the replication level as 5. Please help. Thanks

Answer 1

Hi @Raunak. You can change the replication level as follows:

Open the Spark shell and run the following command:

val sc = new SparkContext(new SparkConf())

./bin/spark-submit <all your existing options> --spark.yarn.submit.file.replication=5