Intermediate · Harden

Budget Timeouts, Retries, Circuit Breakers & Bulkheads

Compose resilience mechanisms around one caller deadline and idempotency contract.

Primary source: Spring Boot 3.5 Reference: Calling REST Services with RestClient

Written and reviewed by Yash Patel 55 minBoot 3.5.16 · Java 21· Reviewed 2026-09-07

Lesson 3 of 3

1d streak

The Architectural Question

Infinite default timeouts, retries without budgets, and breakers without bulkheads compound one slow dependency into a caller-wide outage.

What this guide proves

Set connect and response timeouts explicitly
Retry only bounded transient failures with jitter
Use bulkheads to protect scarce caller resources

Socket Timeout Hygiene and Thread Starvation

The single most common cause of complete Spring Boot application freezes in production is unconfigured or infinite timeouts on external HTTP clients. When a third-party partner service hangs or suffers a network degradation, your threads block indefinitely waiting on TCP read operations.

Because Tomcat by default allocates a pool of 200 worker threads, as few as 200 concurrent requests to a hung external service will exhaust every thread in the pool within seconds. Once the thread pool is depleted, health check probes fail, and Kubernetes restarts the container, causing a cascading outage.

Timeout Parameter	Recommended Default	Consequence of Missing/Infinite Value
Connect Timeout	500ms – 1000ms	Hangs on unroutable IP or firewall drop until OS TCP timeout (75s)
Read / Response Timeout	2000ms – 5000ms	Worker threads block forever if external server stalls while writing body
Connection Request Timeout	500ms	Threads block waiting for a free connection from the internal pool

Resilience4j Circuit Breaker State Transitions

A circuit breaker prevents an application from repeatedly attempting an operation that is guaranteed to fail, giving the downstream service time to recover and protecting the caller from latency degradation.

The circuit breaker operates across three primary states:

1. CLOSED: Normal state. Invocations pass through. Metrics are recorded in a sliding window (count-based or time-based).

2. OPEN: If failure rate (or slow call rate) exceeds the configured threshold (e.g. 50%), the circuit transitions to OPEN. All subsequent calls fail fast immediately with CallNotPermittedException without calling the downstream network!

3. HALF-OPEN: After a wait duration (e.g. 10s), the circuit transitions to HALF-OPEN to allow a trial batch of probe requests. If probe requests succeed, it returns to CLOSED; if they fail, it trips back to OPEN.

src/main/java/com/example/service/PricingService.javajava

package com.example.service;

import io.github.resilience4j.circuitbreaker.annotation.CircuitBreaker;
import org.springframework.stereotype.Service;

@Service
public class PricingService {

    private final PricingExternalClient externalClient;

    public PricingService(PricingExternalClient externalClient) {
        this.externalClient = externalClient;
    }

    @CircuitBreaker(name = "pricingClient", fallbackMethod = "cachedPricingFallback")
    public PriceQuote fetchQuote(String productId) {
        return externalClient.getQuote(productId);
    }

    // Fallback executed when circuit is OPEN or exception is thrown
    public PriceQuote cachedPricingFallback(String productId, Throwable ex) {
        return new PriceQuote(productId, 0, "FALLBACK_ESTIMATE");
    }
}

Retry Constraints: Idempotency and Exponential Backoff

Retries must be applied with extreme discipline. A naive retry policy that fires 3 immediate retries on every failure acts as a Denial-of-Service amplifier against an already struggling downstream service.

Use it when

Retry ONLY idempotent HTTP operations (GET, PUT, DELETE) where re-execution cannot create duplicate state mutations. Combine retries with exponential backoff and randomized jitter to desynchronize retry spikes.

Reconsider when

NEVER automatically retry non-idempotent operations (POST /orders, POST /payments) without a verified server-supported Idempotency-Key header. If a POST times out, the server may have processed the transaction successfully!

src/main/resources/application.ymlyaml

resilience4j:
  circuitbreaker:
    instances:
      pricingClient:
        slidingWindowSize: 20
        failureRateThreshold: 50
        waitDurationInOpenState: 10s
        permittedNumberOfCallsInHalfOpenState: 5
        slowCallRateThreshold: 50
        slowCallDurationThreshold: 2s
  retry:
    instances:
      pricingClient:
        maxAttempts: 3
        waitDuration: 500ms
        enableExponentialBackoff: true
        exponentialBackoffMultiplier: 2
        retryExceptions:
          - java.io.IOException
          - org.springframework.web.client.HttpServerErrorException

Fault Injection Testing with WireMock

Verify circuit breaker trips and fallback invocation by simulating 500 errors and network delays using WireMock:

verify-resilience.shbash

# 1. Inspect Resilience4j Actuator metrics
curl -s http://localhost:8080/actuator/metrics/resilience4j.circuitbreaker.state?tag=state:open
# Verify state transition to OPEN under fault injection

# 2. Observe fast failure response time (< 5ms) when circuit is OPEN
curl -w "\nTotal Time: %{time_total}s\n" http://localhost:8080/api/prices/PROD-42
# Output: {"productId":"PROD-42","status":"FALLBACK_ESTIMATE"}
# Total Time: 0.003s (Fast failure without network delay!)

Active Recall Checkpoint

Why is retrying a non-idempotent HTTP POST request without an idempotency key dangerous?

Hint: Differentiate network transport failures from remote execution status.

Show answer & explanation

If the timeout occurred while waiting for the response after the remote server successfully processed the request, retrying will execute the charge or order a second time.

Runtime Architecture & Flow

Spring MVC · runtime visualization

Budget Timeouts, Retries, Circuit Breakers & Bulkheads

Dispatcher

Set connect and response timeouts explicitly

Binding

Retry only bounded transient failures with jitter

Use case

Use bulkheads to protect scarce caller resources

Response

Fault-inject latency and errors into a catalog client and chart state transitions.

Synthesis & Production Verification

Deliberate Failure Condition

Break: Stack retry(3) on top of a 30-second timeout with no overall deadline
Observe: Worst-case request latency approaches two minutes while threads pile up behind the retries
Repair: Budget an end-to-end deadline first, give each hop a share of it, and retry only bounded transient failures on idempotent operations with jitter.

Production Acceptance Criteria

Set connect and response timeouts explicitly on every client
Retry only bounded transient failures with jitter
Bulkhead the caller and prove a failing dependency sheds load instead of pooling threads

Architectural & Technical Review

Core Interview Questions

Q01Why do unconfigured or infinite read timeouts on external HTTP clients lead to complete Tomcat thread pool starvation?Show answer

When a downstream service stalls, incoming worker threads block waiting on TCP read packets. Under moderate traffic, all 200 Tomcat worker threads become blocked on hanging sockets, rendering the application unable to process health checks or new requests.

Q02What are the three states of the Resilience4j CircuitBreaker and what triggers transitions between them?Show answer

CLOSED permits calls while recording failure rates. If failure rate exceeds the threshold (e.g. 50%), it transitions to OPEN and fails fast immediately. After a wait duration, it enters HALF-OPEN to trial probe calls before deciding to re-close or re-open.

Q03Why is automatically retrying non-idempotent HTTP POST requests hazardous without an idempotency key?Show answer

If a POST times out, the remote server may have processed the transaction successfully while only the response timed out. Retrying the POST directly creates duplicate orders, payments, or records.

Production Pitfalls

Critical Anti-Patterns

Pitfall: Relying on default infinite socket read timeouts for third-party microservice calls — Hangs worker threads during partner outages, exhausting the Tomcat thread pool in seconds.
Pitfall: Applying blind retries without exponential backoff and randomized jitter — Amplifies load against failing downstream systems, converting minor hiccups into total outages.

Module Certification Quiz

Prove Module Mastery

Comprehensive question set covering all lessons in this module.

Take Module Quiz

Primary Documentation & Specifications

Lesson Primary References

Spring Boot 3.5 Reference: Calling REST Services with RestClient Spring Framework 6.2 HTTP Interfaces & RestClient

Supplementary Module Resources

REST clients Resilience4j