CodeWithYash

Intermediate · Decouple

Production Implementation: Outbox Order Events

Commit events beside business state with the outbox pattern, consume idempotently, and prove both under failure.

Primary source: Spring Boot 3.5 Reference: Messaging with Apache Kafka

50 min

Lesson 4 of 4

1d streak

The Architectural Question

Dual writes—save to the database then publish to Kafka—are guaranteed to diverge under crashes, producing orders without events or events for orders that rolled back.

What this guide proves

  • Write the outbox row in the same transaction as business data
  • Publish via a relay and consume deduplicated on event id
  • Prove no lost and no duplicate side effects under crash

Schema Registry & Apache Avro Compatibility Rules

In large-scale enterprise messaging, publishing unstructured JSON dictionaries across microservices leads to silent breaking changes. Confluent Schema Registry with Apache Avro or Protobuf establishes strict schema contracts.

Compatibility ModeEvolution RuleBest Practice
BACKWARDConsumers with new schema can read data written with old schemaDelete fields; add optional fields with defaults
FORWARDConsumers with old schema can read data written with new schemaAdd fields; delete optional fields with defaults
FULLBoth backward and forward compatibleRecommended for decoupled microservices with rolling updates

Consumer Concurrency & Partition Alignment

A Kafka consumer group cannot scale beyond the number of partitions in a topic. If a topic has 6 partitions and your consumer factory configures concurrency = 12, 6 consumer threads will remain completely idle.

Align your topic partition count with target throughput and configure container concurrency to match partition distribution across your Kubernetes pods.

Distributed Tracing & W3C Trace Context over Kafka Headers

Spring Boot can connect Kafka observations to Micrometer Tracing when the tracing bridge and exporter are configured, but propagation is not automatic in every setup. Spring Kafka requires observation to be enabled on the KafkaTemplate and listener container; only then can configured propagators inject and extract W3C context such as traceparent.

End-to-End Production Messaging Verification Checklist

Verify production event-driven pipeline performance and partition health:

snippet.bash
# 1. Check consumer lag across consumer groups
kafka-consumer-groups.sh --bootstrap-server localhost:9092 --group inventory-group --describe
# Verify LAG is 0 under nominal traffic

# 2. Inspect active consumer partition assignments
# Verify all partitions are evenly balanced across active pods

Active Recall Checkpoint

What happens if you have 8 partitions and 10 consumer instances in the same consumer group?

Hint: Recall the rule: at most one consumer per partition per consumer group.

Show answer & explanation

8 consumers are assigned exactly 1 partition each, and 2 consumers remain idle as hot standbys with no assigned partitions.

Working Implementation

Code Walkthrough

Emit order-created events through an outbox table and make the consumer safe to run twice on every message.

MessagingCapstoneLab.java
package com.example.messaging;

import org.springframework.kafka.annotation.KafkaListener;
import org.springframework.kafka.core.KafkaTemplate;
import org.springframework.kafka.support.KafkaHeaders;
import org.springframework.messaging.handler.annotation.Header;
import org.springframework.stereotype.Service;

@Service
public class MessagingCapstoneLab {

    private final KafkaTemplate<String, String> kafkaTemplate;

    public MessagingCapstoneLab(KafkaTemplate<String, String> kafkaTemplate) {
        this.kafkaTemplate = kafkaTemplate;
    }

    public void publishEvent(String topic, String key, String payload) {
        kafkaTemplate.send(topic, key, payload);
    }

    @KafkaListener(
        topics = "orders.events.v1",
        groupId = "fulfillment-service",
        concurrency = "3"
    )
    public void handleOrderEvent(
            String payload,
            @Header(KafkaHeaders.RECEIVED_KEY) String key,
            @Header(KafkaHeaders.RECEIVED_PARTITION) int partition) {
        // Concurrency = 3 distributes partitions across 3 worker threads in this pod
    }
}
Consumer concurrency aligns worker threads directly with topic partition distribution.
Micrometer Tracing automatically propagates W3C traceparent headers across Kafka record headers.
Full capstone integrates schema compatibility, partition ordering, and dead letter remediation.

Core Spring Implementation

Outbox-backed order event flow

Persist order events in a transactional outbox, relay committed rows to Kafka, and deduplicate consumers so redelivery cannot double-create fulfillment.

Component Boundaries

  1. 01OrderService saves the order and the outbox row in one database transaction; Kafka receives nothing until that commit.
  2. 02A relay polls committed outbox rows, publishes them to orders.placed, and marks rows published.
  3. 03FulfillmentConsumer guards every side effect behind a unique-indexed processed_events check on eventId.

Required Evidence

  • Poison messages land in orders.placed.dlt via @RetryableTopic or DefaultErrorHandler with DeadLetterPublishingRecoverer, and replay requires an operator command plus an audit entry.
  • An automated test republishes one eventId twice and asserts exactly one fulfillment row.
  • kafka-consumer-groups.sh --describe --group fulfillment shows consumer lag draining to 0 within the release SLO after each deploy.

Implementation sequence

Step-by-Step Integration

Follow these structured steps to inject Core Spring functionality cleanly across your boundaries.

01

Commit the outbox row beside business state

OrderService.java

Splitting the write across broker and database either loses events on crash or publishes rolled-back work; one transaction makes both artifacts appear together or not at all.

OrderService.java
@Transactional
OrderId place(PlaceOrder command) {
    Order order = orders.save(Order.from(command));
    outbox.save(OutboxEvent.orderPlaced(order.id(), command.requestId()));
    return order.id();
}

Verify Configuration

Throw inside place() after orders.save, then confirm psql returns 0 for SELECT count(*) FROM outbox WHERE request_id = ? and 0 orders rows for the same requestId.

02

Make the consumer idempotent on eventId

FulfillmentConsumer.java

Kafka delivers at-least-once; producer retries and group rebalances replay consumed offsets, so the unique index on processed_events turns duplicate deliveries into no-ops.

FulfillmentConsumer.java
@KafkaListener(topics = "orders.placed", groupId = "fulfillment")
@Transactional
void on(OrderPlaced event) {
    if (processedEvents.existsById(event.eventId())) return;
    fulfillment.createFor(event.orderId());
    processedEvents.save(new ProcessedEvent(event.eventId()));
}

Verify Configuration

kafka-console-producer sends the same OrderPlaced payload twice; SELECT count(*) FROM fulfillments stays 1 and the second delivery returns before fulfillment.createFor runs.

Synthesis & Production Verification

Deliberate Failure Condition

  1. Break: Publish to Kafka after committing the order in two separate steps
  2. Observe: Downstream consumers miss some created orders while others exist twice
  3. Repair: The outbox makes the database the source of truth; a relay publishes what committed, and consumer-side deduplication absorbs at-least-once delivery.

Production Acceptance Criteria

  • Crash between commit and publish and lose nothing after relay catch-up
  • Replay one consumed event twice with a single applied effect
  • Monitor outbox backlog so relay failure is visible

Architectural & Technical Review

Core Interview Questions

Q01How do Confluent Schema Registry compatibility modes prevent breaking changes across microservices?Show answer

FULL compatibility ensures that services deployed on older versions can read events written by newer versions, and vice versa. It enforces adding optional fields with defaults and forbids removing required fields without deprecation cycles.

Q02What happens if consumer container concurrency exceeds the number of partitions in a Kafka topic?Show answer

Because Kafka assigns at most one consumer thread per partition per consumer group, excess consumer threads remain idle, wasting memory and thread resources without increasing throughput.

Q03How does Micrometer Tracing propagate trace context across asynchronous Kafka boundaries?Show answer

KafkaTemplate injects W3C traceparent and tracestate headers into Kafka record headers, and @KafkaListener extracts them, seamlessly linking asynchronous event processing back to the originating HTTP request trace.

Production Pitfalls

Critical Anti-Patterns

  • Pitfall: Publishing untyped JSON schemas with arbitrary field renaming across microservice boundaries — Causes silent deserialization crashes in downstream consumer services during rolling deployments.
  • Pitfall: Over-allocating consumer concurrency beyond topic partition count — Spawns idle worker threads that consume heap memory without contributing to message consumption.

Module Certification Quiz

Prove Module Mastery

Comprehensive question set covering all lessons in this module.

Take Module Quiz

Primary Documentation & Specifications

Lesson Primary References

Spring Boot 3.5 Reference: Messaging with Apache KafkaSpring for Apache Kafka Documentation

Supplementary Module Resources

Spring Framework task execution