Getting "buffer limit exceeded" exception inside Kryo.

0 votes

Hi. I am running a Spark application and I am facing some problem in serialization. During the serialization, I am getting the following error in Kyro:

buffer limit exceeded
Mar 7 in Apache Spark by Firoz
62 views

1 answer to this question.

0 votes

Seems like the object being sent for serialization is exceeding the buffer size of Kyro. By default the maximum allowed size is 64MiB and to increase this, you can do the following:

val sc = new SparkContext(new SparkConf())
./bin/spark-submit <all your existing options> --spark.kryoserializer.buffer.max=<new size>
answered Mar 7 by Pavitra

Related Questions In Apache Spark

0 votes
1 answer

Getting error while connecting zookeeper in Kafka - Spark Streaming integration

I guess you need provide this kafka.bootstrap.servers ...READ MORE

answered May 24, 2018 in Apache Spark by Shubham
• 13,290 points
560 views
+1 vote
1 answer

getting null values in spark dataframe while reading data from hbase

Can you share the screenshots for the ...READ MORE

answered Jul 31, 2018 in Apache Spark by kurt_cobain
• 9,240 points
374 views
0 votes
1 answer

How to limit the cores being used by a cluster?

You can set the maximum number of ...READ MORE

answered Mar 11 in Apache Spark by Raj
43 views
0 votes
1 answer

Client connection getting rejected due to SASL authentication

Seems like you have set the configuration ...READ MORE

answered Mar 14 in Apache Spark by Raj
13 views
0 votes
1 answer

Limit displaying completed applications

By default, the number of completed applications ...READ MORE

answered Mar 25 in Apache Spark by hari
44 views
0 votes
1 answer

Spark Null Pointer Exception.

I used Spark 1.5.2 with Hadoop 2.6 ...READ MORE

answered Jul 19 in Apache Spark by ravikiran
• 4,560 points
85 views
0 votes
1 answer
0 votes
1 answer

Hadoop Mapreduce word count Program

Firstly you need to understand the concept ...READ MORE

answered Mar 16, 2018 in Data Analytics by nitinrawat895
• 10,670 points
2,738 views
0 votes
1 answer

hadoop.mapred vs hadoop.mapreduce?

org.apache.hadoop.mapred is the Old API  org.apache.hadoop.mapreduce is the ...READ MORE

answered Mar 16, 2018 in Data Analytics by nitinrawat895
• 10,670 points
289 views
0 votes
10 answers

hadoop fs -put command?

put syntax: put <localSrc> <dest> copy syntax: copyFr ...READ MORE

answered Dec 7, 2018 in Big Data Hadoop by Aditya
13,567 views