CodeWithYash

Intermediate · Decouple

Design Retries, Ordering & Dead Letters

Classify transient and permanent failures without building an invisible infinite retry loop.

Primary source: Spring Boot 3.5 Reference: Messaging with Apache Kafka

45 min

Lesson 3 of 4

1d streak

The Architectural Question

Retrying poison messages forever buries the partition; dropping them loses money—failure classification and dead-letter ownership are the actual design.

What this guide proves

  • Use bounded exponential backoff with jitter
  • Preserve ordering only where the domain requires it
  • Treat dead letters as an operational workflow with ownership

Error Handling in Kafka: DefaultErrorHandler & Exponential Backoff

When an exception occurs during message consumption in a @KafkaListener, Spring Kafka relies on a CommonErrorHandler. In modern Spring Kafka 3.x / Spring Boot 3.5, DefaultErrorHandler replaces deprecated SeekToCurrentErrorHandler.

DefaultErrorHandler allows configuring retry backoff, max delivery attempts, and classifying fatal versus non-fatal exceptions.

src/main/java/com/example/kafka/KafkaErrorConfig.java
package com.example.kafka;

import org.apache.kafka.common.errors.SerializationException;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.kafka.core.KafkaTemplate;
import org.springframework.kafka.listener.DeadLetterPublishingRecoverer;
import org.springframework.kafka.listener.DefaultErrorHandler;
import org.springframework.util.backoff.ExponentialBackOff;

@Configuration(proxyBeanMethods = false)
public class KafkaErrorConfig {

    @Bean
    public DefaultErrorHandler errorHandler(KafkaTemplate<String, Object> kafkaTemplate) {
        // Exponential backoff: 500ms, 1000ms, 2000ms (multiplier 2.0, max 3 attempts)
        ExponentialBackOff backOff = new ExponentialBackOff(500L, 2.0);
        backOff.setMaxElapsedTime(5000L);

        // DeadLetterPublishingRecoverer routes failed messages to topic.DLT
        DeadLetterPublishingRecoverer recoverer = new DeadLetterPublishingRecoverer(kafkaTemplate);

        DefaultErrorHandler handler = new DefaultErrorHandler(recoverer, backOff);

        // Classify non-retryable fatal exceptions (fail fast to DLT immediately!)
        handler.addNotRetryableExceptions(
            SerializationException.class,
            IllegalArgumentException.class,
            ClassCastException.class
        );

        return handler;
    }
}

Fatal vs Non-Fatal Exceptions

Never retry DeserializationException or malformed JSON payloads. Retrying a poison pill with corrupt syntax will never succeed and blocks the partition indefinitely. Always route fatal errors to DLT immediately.

Dead Letter Queues (DLT) & Header Propagation

When all retry attempts are exhausted, DeadLetterPublishingRecoverer publishes the failed message to a designated Dead Letter Topic (DLT), typically named <original-topic>.DLT. It automatically attaches debugging headers:

Kafka HeaderDescriptionValue Example
kafka_dlt-original-topicSource topic where failure occurredorders.placed.v1
kafka_dlt-original-partitionSource partition number2
kafka_dlt-original-offsetSource offset of failed message10482
kafka_dlt-exception-messageException message stringConnection refused to payment gateway
kafka_dlt-exception-stacktraceFull Java exception stacktracejava.net.ConnectException: ...

Dead Letter Reprocessing Strategies & Operational Remediation

A dead letter queue is not a trash can; it is an active operational backlog. In production, provide a dedicated consumer or operational tool to reprocess resolved messages after fixing downstream bugs or database issues.

The Secondary DLQ Poison Loop

If the DLT consumer throws an unhandled exception while reprocessing a message, it can cause an infinite loop or send the message to a DLT.DLT topic. Always catch and log terminal errors during DLT replay.

Simulating Poison Pills and Verifying DLT Routing

Verify retry backoff and dead letter publishing by injecting transient and permanent exceptions:

snippet.bash
# 1. Produce malformed message (poison pill) to Kafka topic
kcat -b localhost:9092 -t orders.placed.v1 -P <<EOF
{"corrupt_json": true
EOF

# 2. Observe consumer log:
# WARN  o.s.k.l.DefaultErrorHandler - Fatal SerializationException: Sending to DLT immediately

# 3. Consume from Dead Letter Topic to verify error headers and message body
kcat -b localhost:9092 -t orders.placed.v1.DLT -C -e -u -J | jq .

Active Recall Checkpoint

Why should deserialization errors never be retried with backoff?

Hint: Differentiate transient network errors from permanent data corruption.

Show answer & explanation

A deserialization error is caused by invalid syntax or schema incompatibility. Re-reading the identical malformed byte payload will produce the same error every time, wasting CPU and blocking the partition.

Runtime Architecture & Flow

Reliable event flow · runtime visualization

Design Retries, Ordering & Dead Letters

Local commit

Use bounded exponential backoff with jitter

Publisher

Preserve ordering only where the domain requires it

Broker

Treat dead letters as an operational workflow with ownership

Consumer

Route poison messages to a dead-letter topic and build a replay procedure.

Working Implementation

Code Walkthrough

Route poison messages to a dead-letter topic and build a replay procedure.

KafkaDlqRetryLab.java
package com.example.kafka;

import org.apache.kafka.common.errors.SerializationException;
import org.springframework.kafka.core.KafkaTemplate;
import org.springframework.kafka.listener.DeadLetterPublishingRecoverer;
import org.springframework.kafka.listener.DefaultErrorHandler;
import org.springframework.util.backoff.ExponentialBackOff;

public final class KafkaDlqRetryLab {

    private KafkaDlqRetryLab() {}

    public static DefaultErrorHandler buildResilientErrorHandler(KafkaTemplate<String, Object> template) {
        // Backoff: 500ms initial, 2.0 multiplier, max 3 attempts
        var backoff = new ExponentialBackOff(500L, 2.0);
        backoff.setMaxElapsedTime(5000L);

        var recoverer = new DeadLetterPublishingRecoverer(template);
        var handler = new DefaultErrorHandler(recoverer, backoff);

        // Fail poison pills to DLT immediately without wasting retry cycles
        handler.addNotRetryableExceptions(
            SerializationException.class,
            IllegalArgumentException.class,
            ClassCastException.class
        );

        return handler;
    }
}
DefaultErrorHandler replaces deprecated legacy SeekToCurrentErrorHandler in modern Spring Kafka.
Fatal exceptions like SerializationException bypass retries and route directly to the .DLT topic.
DeadLetterPublishingRecoverer attaches original topic, partition, offset, and exception stack trace headers.

Synthesis & Production Verification

Production Acceptance Criteria

  • Classify transient versus permanent failures in one policy object
  • Back off exponentially with jitter and cap total attempts
  • Operate dead letters as owned work with triage and replay tooling

Architectural & Technical Review

Core Interview Questions

Q01Why should DeserializationException and malformed JSON payloads never be retried with backoff?Show answer

Deserialization errors are permanent data corruption bugs (poison pills). Re-evaluating the identical malformed byte payload will never succeed, wasting CPU and blocking partition consumption. Fatal exceptions must route to the DLT immediately.

Q02What metadata headers does DeadLetterPublishingRecoverer attach when routing a message to the DLT?Show answer

It attaches original topic name, original partition number, original message offset, exception message, and the full Java exception stack trace as Kafka record headers, providing full diagnostic context for operational triage.

Q03What operational safeguard is required when building a DLT reprocessing consumer?Show answer

The DLT consumer must catch and log terminal errors to prevent infinite poison pill retry loops that would continuously bounce failing messages back into the dead letter topic.

Production Pitfalls

Critical Anti-Patterns

  • Pitfall: Retrying non-retryable poison pills with infinite backoff in Kafka listeners — Permanently stalls consumer partition progress, causing consumer lag to skyrocket across the cluster.
  • Pitfall: Discarding failed messages silently without publishing to a Dead Letter Topic — Destroys transaction auditability and loses customer orders without any operational alert.

Primary Documentation & Specifications

Lesson Primary References

Spring Boot 3.5 Reference: Messaging with Apache KafkaSpring for Apache Kafka Documentation

Supplementary Module Resources

Spring Framework task execution