Advanced · Ship

Engineer Probes, Shutdown & Rolling Deploys

Coordinate startup, readiness, in-flight work, connection pools, and termination grace periods.

Primary source: Spring Boot 3.5 Reference: Container Images & Probes

Written and reviewed by Yash Patel 45 minBoot 3.5.16 · Java 21· Reviewed 2026-09-07

Lesson 3 of 4

1d streak

The Architectural Question

Liveness probes that ping remote dependencies restart healthy pods during outages, and abrupt termination drops every in-flight request at the worst moment.

What this guide proves

Keep liveness local and conservative
Remove readiness before stopping traffic-bearing work
Close consumers and pools within the termination budget

Kubernetes Probes: Liveness vs Readiness Semantics

In Kubernetes, misconfiguring health probes is the primary cause of self-inflicted cascade outages. Spring Boot Actuator provides two dedicated, isolated probe endpoints under /actuator/health:

1. Readiness Probe (/actuator/health/readiness): Indicates whether the application is ready to accept user traffic. If readiness fails, Kubernetes immediately stops routing traffic from Service endpoints to this pod. The pod is NOT restarted.

2. Liveness Probe (/actuator/health/liveness): Indicates whether the internal JVM process is alive or permanently deadlocked. If liveness fails, Kubernetes KILLS the container and spawns a replacement pod.

Never Include External Dependencies in Liveness Probes

Never check database, Redis, or external service health inside the Liveness probe! If your PostgreSQL database suffers a temporary network hiccup, all 50 application pods would fail liveness simultaneously. Kubernetes will terminate and restart all 50 pods at once, compounding the outage into total cluster collapse.

src/main/resources/application.ymlyaml

management:
  endpoint:
    health:
      probes:
        enabled: true
  health:
    livenessstate:
      enabled: true
    readinessstate:
      enabled: true

Graceful Shutdown & Connection Draining

When Kubernetes scales down a deployment or performs a rolling update, it sends a SIGTERM signal to the pod. If an application terminates abruptly, in-flight HTTP requests and database transactions are severed, resulting in HTTP 502/504 errors for active users.

Enable Spring Boot graceful shutdown to allow in-flight requests to complete while rejecting new traffic:

src/main/resources/application.ymlyaml

server:
  shutdown: graceful

spring:
  lifecycle:
    timeout-per-shutdown-phase: 30s

Upon receiving SIGTERM: (1) Readiness probe transitions to REFUSING_TRAFFIC; (2) Kubernetes removes the pod from the Service endpoint list; (3) Embedded Tomcat stops accepting new connections; (4) Active requests are granted up to 30 seconds to finish execution before JVM exit.

Emergency Diagnostics: Thread Dumps and Heap Dumps Under Pressure

When an application experiences high CPU, thread deadlock, or memory leaks in production, engineers must capture diagnostic snapshots before restarting the process.

capture-diagnostics.shbash

# 1. Capture full JVM thread dump to diagnose thread lock contention
curl -s http://localhost:8080/actuator/threaddump > /tmp/threaddump-$(date +%s).json

# 2. Trigger memory heap dump for OutOfMemory analysis
curl -s http://localhost:8080/actuator/heapdump > /tmp/heapdump.hprof

Protect Heap Dump Endpoints

The /actuator/heapdump endpoint dumps the entire JVM memory, including unredacted tokens, database passwords, and user data. Never expose heapdump or threaddump to the public internet; restrict access to localhost or an internal management port.

Proving Probe State Transitions and Graceful Shutdown

Verify probe state transitions programmatically using Spring ApplicationAvailability:

snippet.bashbash

# 1. Check Liveness probe (reports UP if JVM is running)
curl -i http://localhost:8080/actuator/health/liveness
# HTTP/1.1 200 OK -> {"status":"UP"}

# 2. Check Readiness probe (reports UP only when fully initialized)
curl -i http://localhost:8080/actuator/health/readiness
# HTTP/1.1 200 OK -> {"status":"UP"}

# 3. Send SIGTERM and observe graceful shutdown log
kill -15 <PID>
# LOG: Commencing graceful shutdown. Waiting for active requests to complete.

Active Recall Checkpoint

Why does a temporary database outage cause readiness to fail while liveness should stay UP?

Hint: Distinguish between internal process health and readiness to serve requests.

Show answer & explanation

Readiness failing stops incoming traffic from hitting the pod until the database recovers. If liveness failed, Kubernetes would restart the pod in a crash loop, which cannot fix a database outage.

Runtime Architecture & Flow

High-value visual model

Production Readiness Control Loop

Production readiness connects deployability, health, observability, capacity, and safe recovery.

1. Health describes routeability

Expose startup, readiness, and liveness for different platform decisions.

2. Signals explain behavior

Logs, metrics, traces, and audit events answer different operational questions.

3. Pools bound concurrency

Thread, connection, and broker pools define throughput and failure pressure.

4. Rollback and drain are practiced

Recovery paths must be tested before an incident, not invented during one.

Operational control loop: A service is production-ready when failure is diagnosable and recovery is rehearsed.

Working Implementation

Code Walkthrough

Demonstrate a zero-error rollout under continuous traffic.

Java

GracefulShutdownProbesLab.javaJava

package com.example.readiness;

import org.springframework.boot.availability.ApplicationAvailability;
import org.springframework.boot.availability.AvailabilityChangeEvent;
import org.springframework.boot.availability.LivenessState;
import org.springframework.boot.availability.ReadinessState;
import org.springframework.context.ApplicationEventPublisher;
import org.springframework.stereotype.Service;

@Service
public class GracefulShutdownProbesLab {

    private final ApplicationAvailability availability;
    private final ApplicationEventPublisher publisher;

    public GracefulShutdownProbesLab(ApplicationAvailability availability, ApplicationEventPublisher publisher) {
        this.availability = availability;
        this.publisher = publisher;
    }

    public ReadinessState getReadiness() {
        return availability.getReadinessState();
    }

    public LivenessState getLiveness() {
        return availability.getLivenessState();
    }

    // Mark application unready to stop incoming traffic during graceful draining
    public void markRefusingTraffic() {
        AvailabilityChangeEvent.publish(publisher, this, ReadinessState.REFUSING_TRAFFIC);
    }

    public void markAcceptingTraffic() {
        AvailabilityChangeEvent.publish(publisher, this, ReadinessState.ACCEPTING_TRAFFIC);
    }
}

/actuator/health/readiness controls traffic routing; /actuator/health/liveness controls container restarts.

Never include external databases or network services in liveness checks to prevent cascade restarts.

server.shutdown=graceful drains in-flight HTTP requests before JVM shutdown upon receiving SIGTERM.

Synthesis & Production Verification

Production Acceptance Criteria

Keep liveness checks local and conservative
Fail readiness before stopping traffic-bearing work
Close consumers and connection pools within the termination budget

Architectural & Technical Review

Core Interview Questions

Q01What is the operational difference between /actuator/health/liveness and /actuator/health/readiness in Kubernetes?Show answer

Readiness controls whether Kubernetes routes traffic to the pod via Services; if it fails, traffic stops but the pod stays running. Liveness checks if the JVM is deadlocked; if it fails, Kubernetes kills and restarts the container.

Q02Why does checking database connectivity inside a Kubernetes liveness probe cause catastrophic cluster collapse?Show answer

If the database experiences a temporary network blip, all application pods will fail liveness simultaneously. Kubernetes will kill and restart all pods at once, compounding the outage into total cluster collapse.

Q03How does Spring Boot server.shutdown=graceful prevent HTTP 502 Bad Gateway errors during pod rolling updates?Show answer

Upon receiving SIGTERM, the readiness probe marks the pod unready to drain ingress traffic, stops accepting new connections, and gives in-flight requests a grace period (e.g. 30s) to finish processing before JVM termination.

Production Pitfalls

Critical Anti-Patterns

Pitfall: Including database or external service checks inside the Kubernetes liveness probe — Causes Kubernetes to kill and restart all application pods simultaneously during a database hiccup.
Pitfall: Exposing /actuator/heapdump or /threaddump to public ingress without authentication — Leaks the entire JVM memory heap containing plaintext passwords, JWTs, and PII to attackers.

Primary Documentation & Specifications

Lesson Primary References

Spring Boot 3.5 Reference: Container Images & Probes Kubernetes Documentation: Configure Liveness, Readiness and Startup Probes

Supplementary Module Resources

Spring Boot container images Graceful shutdown