Let’s say we would like to increase the limit t 10MB …​

Broker Configuration

Apply the new limit either by modifying the server.properties like this…​

max.message.bytes=10485760

  1. or apply it to a specific topic using

kafka-configs.sh --bootstrap-server localhost:9092 \
    --entity-type topics  \
    --entity-name thetopic \
    --alter \
    --add-config max.message.bytes=10485760

Producer Configuration for Spring Boot

We simply need to add the following line to our application.properties:

spring.kafka.producer.properties.max.request.size=spring.kafka.producer.properties.max.request.size

The following proof-of-concept demonstrates that without the property, sending a large message fails, with the property it succeeds:

KafkaSizeLimitTest.java
package com.hascode.sample;

import org.apache.kafka.clients.admin.NewTopic;
import org.springframework.boot.ApplicationRunner;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.context.annotation.Bean;
import org.springframework.kafka.core.KafkaTemplate;

@SpringBootApplication
public class KafkaSizeLimitTest {

    public static void main(String[] args) {
        SpringApplication.run(KafkaSizeLimitTest.class, args);
    }

    @Bean
    public NewTopic topic() {
        return new NewTopic("thetopic", 1, (short) 1);
    }

    @Bean
    public ApplicationRunner runner(KafkaTemplate<String, String> template) {
        return args -> template.send("thetopic", new String(new byte[1024 * 1024 * 6]));
    }

}

Typical Exceptions

Topic or Broker not configured for message size

2022-06-09 14:31:27.887 ERROR 4892 --- [ad | producer-1] o.s.k.support.LoggingProducerListener    : Exception thrown when sending a message with key='null' and payload='                                                                                                    ...' to topic thetopic:

org.apache.kafka.common.errors.RecordTooLargeException: The request included a message larger than the max message size the server will accept.

Message Producer not configured for message size

Caused by: org.springframework.kafka.KafkaException: Send failed; nested exception is org.apache.kafka.common.errors.RecordTooLargeException: The message is 6291544 bytes when serialized which is larger than 1, which is the value of the max.request.size configuration.
	at org.springframework.kafka.core.KafkaTemplate.doSend(KafkaTemplate.java:666) ~[spring-kafka-2.8.6.jar:2.8.6]
	at org.springframework.kafka.core.KafkaTemplate.send(KafkaTemplate.java:403) ~[spring-kafka-2.8.6.jar:2.8.6]
	at com.hascode.sample.KafkaSizeLimitTest.lambda$runner$0(KafkaMeasureApplication.java:26) ~[classes/:na]
	at org.springframework.boot.SpringApplication.callRunner(SpringApplication.java:762) ~[spring-boot-2.7.0.jar:2.7.0]
	... 5 common frames omitted
Caused by: org.apache.kafka.common.errors.RecordTooLargeException: The message is 6291544 bytes when serialized which is larger than 1, which is the value of the max.request.size configuration.