How to increase the amount of data to be transferred to shuffle service at the same time

Question

Facing problem with shuffle service. The data being sent to shuffle service is big. The complete data is not being sent and when the client retries, after some time I am getting fetch fail error. I was finding a solution for this and found that this can be avoided by increasing the size of data being sent at a time. So how can i do it?

Answer 1

The amount of data to be transferred at the same time is set pretty high and you wouldn't usually reach that limit unless you have a really huge dataset. I am not sure if the solution you think might work will actually work but anyway, here's how you can do it. You can change it at runtime. 

val sc = new SparkContext(new SparkConf())

./bin/spark-submit <all your existing options> --spark.shuffle.maxChunksBeingTransferred=NEW_VALUE