CodeWithYash

Intermediate · Decouple

Publish Reliably with Kafka & the Outbox Pattern

Close the database-plus-broker dual-write gap and consume at-least-once messages safely.

Primary source: Spring Boot 3.5 Reference: Messaging with Apache Kafka

55 min

Lesson 2 of 4

1d streak

The Architectural Question

Writing the database row and publishing the event touch two systems; without an outbox, crashes between the two lose events or emit ghosts for rolled-back transactions.

What this guide proves

  • Commit business state and outbox record atomically
  • Make consumer effects idempotent
  • Choose partition keys from required ordering guarantees

Apache Kafka Producer and Consumer Architecture

Apache Kafka provides distributed, partitioned commit-log semantics for high-throughput event streaming. Spring for Apache Kafka (spring-kafka) wraps native Kafka clients with KafkaTemplate for producing and @KafkaListener for consuming.

Key Kafka architectural guarantees:

1. Partition Ordering: Messages with the same key (e.g. orderId) are guaranteed to arrive on the same partition in strict sequential order.

2. Consumer Groups: Multiple instances of a service sharing the same group.id share partition assignments, providing horizontal load balancing and fault tolerance.

src/main/java/com/example/kafka/OrderEventService.java
package com.example.kafka;

import org.springframework.kafka.annotation.KafkaListener;
import org.springframework.kafka.core.KafkaTemplate;
import org.springframework.kafka.support.KafkaHeaders;
import org.springframework.messaging.handler.annotation.Header;
import org.springframework.stereotype.Service;

@Service
public class OrderEventService {

    private final KafkaTemplate<String, OrderEvent> kafkaTemplate;

    public OrderEventService(KafkaTemplate<String, OrderEvent> kafkaTemplate) {
        this.kafkaTemplate = kafkaTemplate;
    }

    public void publishOrderPlaced(OrderEvent event) {
        // Publish using orderId as key to ensure partition affinity
        kafkaTemplate.send("orders.placed.v1", event.orderId(), event);
    }

    @KafkaListener(topics = "orders.placed.v1", groupId = "inventory-group")
    public void handleOrderPlaced(OrderEvent event, @Header(KafkaHeaders.RECEIVED_PARTITION) int partition) {
        // Process inventory allocation
    }
}

The Transactional Outbox Pattern with PostgreSQL & Kafka

Directly calling kafkaTemplate.send() inside a Spring @Transactional database method is an architectural anti-pattern known as Dual-Write. If the database transaction commits but the Kafka broker network connection times out, the event is permanently lost, leading to distributed state divergence.

The Transactional Outbox pattern guarantees zero-loss event publishing: write the event into an outbox table in PostgreSQL inside the same local relational transaction, then publish asynchronously to Kafka using Change Data Capture (Debezium) or a polling worker.

src/main/resources/db/migration/V2__outbox_table.sql
CREATE TABLE outbox_events (
    id UUID PRIMARY KEY,
    aggregate_type VARCHAR(64) NOT NULL,
    aggregate_id VARCHAR(128) NOT NULL,
    event_type VARCHAR(128) NOT NULL,
    payload JSONB NOT NULL,
    created_at TIMESTAMP WITH TIME ZONE NOT NULL,
    processed BOOLEAN DEFAULT FALSE NOT NULL
);

CREATE INDEX idx_outbox_unprocessed ON outbox_events(created_at) WHERE processed = FALSE;

Consumer Idempotency and Deduplication in Distributed Systems

Kafka provides at-least-once delivery semantics by default. Network retries, rebalances, and consumer crashes mean that your consumer will receive duplicate messages. Consumers must be designed to be completely idempotent.

src/main/java/com/example/kafka/IdempotentPaymentConsumer.java
package com.example.kafka;

import org.springframework.dao.DataIntegrityViolationException;
import org.springframework.jdbc.core.simple.JdbcClient;
import org.springframework.stereotype.Service;
import org.springframework.transaction.annotation.Transactional;

@Service
public class IdempotentPaymentConsumer {

    private final JdbcClient jdbcClient;

    public IdempotentPaymentConsumer(JdbcClient jdbcClient) {
        this.jdbcClient = jdbcClient;
    }

    @Transactional
    public boolean processPaymentOnce(String eventId, String customerId, int amountCents) {
        try {
            // Atomic deduplication using UNIQUE constraint on processed_events(event_id)
            jdbcClient.sql("INSERT INTO processed_events (event_id, processed_at) VALUES (?, NOW())")
                .param(eventId)
                .update();
        } catch (DataIntegrityViolationException ex) {
            // Already processed in earlier delivery; return without duplicating payment
            return false;
        }

        // Execute payment deduction exactly once
        return true;
    }
}

Verifying Outbox Atomicity and Duplicate Defense

Verify transactional outbox persistence and consumer deduplication using Testcontainers Kafka:

snippet.bash
# 1. Trigger order creation with outbox write
curl -X POST http://localhost:8080/api/orders -d '{"orderId":"ORD-42","amount":1999}'

# 2. Verify PostgreSQL outbox table contains pending record
psql -d app_db -c "SELECT aggregate_id, event_type, processed FROM outbox_events WHERE aggregate_id='ORD-42';"
# aggregate_id | event_type   | processed
# -------------+--------------+----------
# ORD-42       | ORDER_PLACED | f

# 3. Simulate Kafka rebalance and replay of message
# Verify processed_events table prevents duplicate balance deductions

Active Recall Checkpoint

Why does calling kafkaTemplate.send() inside @Transactional fail to guarantee event delivery?

Hint: Think about the two-phase commit fallacy in distributed systems.

Show answer & explanation

If the database commits but the Kafka producer network request fails or times out, the database changes exist without the event ever reaching Kafka.

Runtime Architecture & Flow

High-value visual model

Outbox Crash Points

The outbox pattern separates local database consistency from broker delivery while the consumer handles duplicate messages deliberately.

1. Commit state and outbox

Business row and outbox event are written in one database transaction.

2. Relay publishes

A background relay publishes committed events to Kafka.

3. Broker may redeliver

Retries and rebalances can repeat a message.

4. Consumer deduplicates

The business effect and processed-event marker commit together.

Reliable messaging crash-point sequence: Design for at-least-once delivery. Exactly-once business effects require idempotent consumers.

Working Implementation

Code Walkthrough

Publish order-created through an outbox relay and deduplicate by event ID.

KafkaOutboxLab.java
package com.example.kafka;

import org.springframework.dao.DataIntegrityViolationException;
import org.springframework.jdbc.core.simple.JdbcClient;
import org.springframework.kafka.core.KafkaTemplate;
import org.springframework.stereotype.Service;
import org.springframework.transaction.annotation.Transactional;

import java.time.Instant;
import java.util.UUID;

@Service
public class KafkaOutboxLab {

    private final JdbcClient jdbcClient;
    private final KafkaTemplate<String, String> kafkaTemplate;

    public KafkaOutboxLab(JdbcClient jdbcClient, KafkaTemplate<String, String> kafkaTemplate) {
        this.jdbcClient = jdbcClient;
        this.kafkaTemplate = kafkaTemplate;
    }

    // Transactional Outbox pattern: local DB write ensures zero lost messages
    @Transactional
    public void saveOrderAndOutbox(String orderId, String payload) {
        jdbcClient.sql("INSERT INTO orders (id, status) VALUES (?, 'PLACED')")
            .param(orderId)
            .update();

        jdbcClient.sql("""
            INSERT INTO outbox_events (id, aggregate_type, aggregate_id, event_type, payload, created_at)
            VALUES (?, 'ORDER', ?, 'ORDER_CREATED', ?, ?)
            """)
            .params(UUID.randomUUID(), orderId, payload, Instant.now())
            .update();
    }

    // Consumer idempotency: duplicate deliveries return early without duplicating state changes
    @Transactional
    public boolean consumeOrderEventIdempotently(String eventId) {
        try {
            jdbcClient.sql("INSERT INTO processed_events (event_id, processed_at) VALUES (?, ?)")
                .params(eventId, Instant.now())
                .update();
            return true;
        } catch (DataIntegrityViolationException ex) {
            // Already processed; deduplicated successfully
            return false;
        }
    }
}
Writing to an outbox table within the same transaction eliminates the Kafka dual-write dilemma.
Consumer deduplication using a unique constraint on event_id guarantees at-least-once safety.
Partition ordering is preserved by publishing with the aggregate ID as the Kafka message key.

Synthesis & Production Verification

Deliberate Failure Condition

  1. Break: Call kafkaTemplate.send() inside the order transaction
  2. Observe: The broker receives events for transactions that later roll back, or none for ones that commit
  3. Repair: Commit business state and an outbox record atomically, relay afterward, and make consumer effects idempotent by event ID.

Production Acceptance Criteria

  • Persist order state plus outbox row in one transaction
  • Relay published records to Kafka and track delivery progress
  • Replay duplicate deliveries without duplicating fulfillment

Architectural & Technical Review

Core Interview Questions

Q01What is the distributed dual-write problem when publishing to Kafka inside a Spring @Transactional method?Show answer

If the database transaction commits but the network request to Kafka times out or fails, the event is permanently lost. Conversely, if the message sends but the DB rolls back, an invalid event is published. Writing to a local outbox table within the same transaction ensures atomicity.

Q02Why does Kafka guarantee at-least-once delivery, and how do consumers achieve idempotency?Show answer

Consumer rebalances, network retries, and commit timeouts cause Kafka to deliver duplicate messages. Consumers achieve idempotency by inserting the unique message/event ID into a processed_events table with a PRIMARY KEY / UNIQUE constraint within the consumer transaction.

Q03How does Kafka preserve message ordering across consumer instances?Show answer

Kafka guarantees strict sequential ordering strictly within a single partition. Publishing messages with the aggregate ID (e.g. orderId) as the message key hashes all events for that entity to the exact same partition.

Production Pitfalls

Critical Anti-Patterns

  • Pitfall: Publishing directly to Kafka via KafkaTemplate inside a database transaction without an outbox — Broker connection drops after database commit cause permanent distributed state desynchronization.
  • Pitfall: Assuming Kafka auto-commit prevents duplicate message processing during container restarts — Auto-commit commits offsets asynchronously; pod crashes before commit cause all messages since the last commit to be re-processed.

Primary Documentation & Specifications

Lesson Primary References

Spring Boot 3.5 Reference: Messaging with Apache KafkaSpring for Apache Kafka Documentation

Supplementary Module Resources

Spring Framework task execution