Getting "buffer limit exceeded" exception inside Kryo.

0 votes

Hi. I am running a Spark application and I am facing some problem in serialization. During the serialization, I am getting the following error in Kyro:

buffer limit exceeded
Mar 7, 2019 in Apache Spark by Firoz
243 views

1 answer to this question.

0 votes

Seems like the object being sent for serialization is exceeding the buffer size of Kyro. By default the maximum allowed size is 64MiB and to increase this, you can do the following:

val sc = new SparkContext(new SparkConf())
./bin/spark-submit <all your existing options> --spark.kryoserializer.buffer.max=<new size>
answered Mar 7, 2019 by Pavitra

Related Questions In Apache Spark

0 votes
1 answer

Getting error while connecting zookeeper in Kafka - Spark Streaming integration

I guess you need provide this kafka.bootstrap.servers ...READ MORE

answered May 24, 2018 in Apache Spark by Shubham
• 13,380 points
1,043 views
+1 vote
1 answer

getting null values in spark dataframe while reading data from hbase

Can you share the screenshots for the ...READ MORE

answered Jul 31, 2018 in Apache Spark by kurt_cobain
• 9,310 points
725 views
0 votes
1 answer

How to limit the cores being used by a cluster?

You can set the maximum number of ...READ MORE

answered Mar 11, 2019 in Apache Spark by Raj
68 views
0 votes
1 answer

Client connection getting rejected due to SASL authentication

Seems like you have set the configuration ...READ MORE

answered Mar 14, 2019 in Apache Spark by Raj
40 views
0 votes
1 answer

Limit displaying completed applications

By default, the number of completed applications ...READ MORE

answered Mar 25, 2019 in Apache Spark by hari
64 views
+1 vote
2 answers
+1 vote
1 answer

Hadoop Mapreduce word count Program

Firstly you need to understand the concept ...READ MORE

answered Mar 16, 2018 in Data Analytics by nitinrawat895
• 10,920 points
5,039 views
0 votes
1 answer

hadoop.mapred vs hadoop.mapreduce?

org.apache.hadoop.mapred is the Old API  org.apache.hadoop.mapreduce is the ...READ MORE

answered Mar 16, 2018 in Data Analytics by nitinrawat895
• 10,920 points
717 views
+1 vote
11 answers

hadoop fs -put command?

put syntax: put <localSrc> <dest> copy syntax: copyFr ...READ MORE

answered Dec 7, 2018 in Big Data Hadoop by Aditya
29,631 views