The default capacity of listener bus is set to 10000. Try changing it to 20000:
val sc = new SparkContext(new SparkConf())
./bin/spark-submit <all your existing options> --spark.scheduler.listenerbus.eventqueue.capacity=20000
Is it spark.scheduler.listenerbus.eventqueue.size or spark.scheduler.listenerbus.eventqueue.capacity?
I think, it is spark.scheduler.listenerbus.eventqueue.capacity. In spark documentation, there is no property available as spark.scheduler.listenerbus.eventqueue.size. You can go through the below link.
You can change the location where you ...READ MORE
To make Spark store the event logs, ...READ MORE
Probably the spill is because you have ...READ MORE
By default, the timeout is set to ...READ MORE
Instead of spliting on '\n'. You should ...READ MORE
Firstly you need to understand the concept ...READ MORE
org.apache.hadoop.mapred is the Old API
org.apache.hadoop.mapreduce is the ...READ MORE
put <localSrc> <dest>
copyF ...READ MORE
You can change the scheduling mode as ...READ MORE
When a task results in too many ...READ MORE
Already have an account? Sign in.