Intermediate · Observe

Design Health, Readiness & Management Access

Use Actuator as an operational boundary, not a public debugging console.

Primary source: Spring Boot Actuator: Production-ready Features

40 min

Lesson 1 of 3

1d streak

The Architectural Question

Actuator endpoints leak configuration, environment variables, and thread dumps when exposed publicly; readiness tied to remote dependencies amplifies every outage.

What this guide proves

  • Expose only required management endpoints
  • Distinguish liveness from dependency-aware readiness
  • Secure and separate the management surface

Kubernetes Health Probes: Liveness vs Readiness

In container orchestrators like Kubernetes, automated recovery depends on health probes. A frequent architectural disaster occurs when teams treat all health checks identically, confusing Liveness with Readiness.

Architectural Boundary · Do Not Confuse

Concept A

Liveness Probe (/actuator/health/liveness): Indicates whether the JVM process is alive and internally healthy (no thread deadlocks, JVM corruption, or unrecoverable fatal states). Action on failure: KUBERNETES KILLS AND REBOOTS THE POD.

Concept B

Readiness Probe (/actuator/health/readiness): Indicates whether the application is currently prepared to accept and process user traffic (caches warmed, database connections available, queues reachable). Action on failure: KUBERNETES STOPS SENDING TRAFFIC TO THIS POD.

If a downstream database goes offline and you fail the Liveness probe, Kubernetes reboots all pods simultaneously. Pods crash-loop, overwhelm the database with startup queries upon recovery, and cause total cluster blackout.

The Golden Rule of Liveness Checks

A Liveness check must NEVER depend on external infrastructure (databases, caches, third-party APIs). If the database is down, restarting your application container will not fix the database—it will only destroy active in-flight requests and spike CPU!

Active Recall Checkpoint

What happens when an external payment provider goes down: should your app fail the Liveness probe or the Readiness probe?

Hint: Does rebooting your application container bring the payment provider back online?

Show answer & explanation

Neither should fail the Liveness probe. If payment is non-essential, both remain UP while reporting degraded metrics. If payment is strictly required for the pod to serve traffic, fail the Readiness probe so traffic routes to fallback pods without rebooting.

Implementing HealthIndicators & AvailabilityState Transitions

Spring Boot 2.3+ introduced the ApplicationAvailability API to decouple internal state transitions from legacy static indicators. Applications can emit AvailabilityChangeEvents to change state dynamically.

OrderQueueHealthIndicator.java
package com.example.health;

import org.springframework.boot.actuate.health.Health;
import org.springframework.boot.actuate.health.HealthIndicator;
import org.springframework.stereotype.Component;

@Component("orderQueue")
public class OrderQueueHealthIndicator implements HealthIndicator {

    private final QueueClient queueClient;

    public OrderQueueHealthIndicator(QueueClient queueClient) {
        this.queueClient = queueClient;
    }

    @Override
    public Health health() {
        try {
            long lag = queueClient.getConsumerLag();
            if (lag > 10_000) {
                return Health.down()
                    .withDetail("reason", "Consumer lag exceeded threshold")
                    .withDetail("currentLag", lag)
                    .build();
            }
            return Health.up()
                .withDetail("consumerLag", lag)
                .withDetail("queueDepth", queueClient.getDepth())
                .build();
        } catch (Exception ex) {
            return Health.down(ex)
                .withDetail("error", "Failed to contact message broker")
                .build();
        }
    }
}

To register indicators with specific probe groups, configure application.yml:

application.yml
management:
  endpoint:
    health:
      probes:
        enabled: true
      group:
        readiness:
          include: "readinessState,db,orderQueue"
        liveness:
          include: "livenessState" # Strictly internal JVM state!

Active Recall Checkpoint

How do you programmatically instruct Spring Boot to mark readiness as REFUSING_TRAFFIC during graceful shutdown?

Hint: Look at AvailabilityChangeEvent.publish(...).

Show answer & explanation

Call AvailabilityChangeEvent.publish(eventPublisher, this, ReadinessState.REFUSING_TRAFFIC);. Spring Boot notifies Kubernetes to stop routing traffic before active connections are closed.

Management Port Isolation & Network Security

Exposing Actuator endpoints on the same public HTTP port as business APIs is a high-risk security hazard. In production, Actuator should listen on a dedicated internal management port accessible only by cluster internal monitoring systems.

application.yml
server:
  port: 8080 # Public HTTP API traffic

management:
  server:
    port: 8081 # Internal management traffic
    address: 127.0.0.1 # Bind strictly to loopback or VPC internal interface
  endpoints:
    web:
      exposure:
        include: "health,metrics,prometheus"
      base-path: "/actuator"

Network-Level Defense in Depth

When management.server.port is configured, Spring Boot spawns a distinct embedded WebServer on port 8081 with its own DispatcherServlet and filter chain. Public load balancers listening on port 8080 physically cannot route requests to /actuator endpoints.

Active Recall Checkpoint

What physical network guarantee does setting management.server.port=8081 provide?

Hint: Can an external internet user hit port 8081 if only 8080 is exposed by the Docker container or Ingress?

Show answer & explanation

External users routing through an Ingress or load balancer mapped only to port 8080 will receive 404s for /actuator, as the management servlet lives on a completely isolated network port.

Deliberate Failure Lab: Diagnosing Cascading Liveness Failures

Let us observe the failure chain of a misconfigured health check in production:

  1. 011. A database experiences transient network degradation (e.g. 500ms spike).
  2. 022. Pod 1 evaluates /actuator/health which includes the database check in its liveness probe.
  3. 033. Pod 1 reports DOWN to Kubernetes liveness probe.
  4. 044. Kubernetes issues SIGTERM to Pod 1, killing active user transactions and restarting the container.
  5. 055. Traffic formerly handled by Pod 1 surges into Pods 2, 3, and 4.
  6. 066. The increased load further saturates the database connection pool.
  7. 077. Pods 2, 3, and 4 fail their liveness checks and are killed simultaneously.
  8. 088. The entire service enters a Death Spiral: all pods reboot continuously.

Remediation

Remove database and network dependencies from management.endpoint.health.group.liveness.include. Keep liveness bound solely to livenessState, and route external dependencies exclusively through the readiness group.

Active Recall Checkpoint

Why does moving the database check from liveness to readiness break the cascading death spiral?

Hint: What does Kubernetes do when a readiness probe fails vs a liveness probe?

Show answer & explanation

When readiness fails, Kubernetes merely pauses traffic to that pod without killing it. Once the database recovers, the pod resumes serving traffic smoothly without re-running JVM cold start or re-initializing pools.

Runtime Architecture & Flow

Structure under inspection

Design Health, Readiness & Management Access

Part 1

Expose only required management endpoints

Part 2

Distinguish liveness from dependency-aware readiness

Part 3

Secure and separate the management surface

The connectors show containment and responsibility—not time or call order.

Synthesis & Production Verification

Production Acceptance Criteria

  • Restrict endpoint exposure to an explicit allow-list
  • Add a readiness HealthIndicator for one required downstream
  • Separate the management surface from the traffic-serving port

Architectural & Technical Review

Core Interview Questions

Q01What is the critical operational difference between Kubernetes Liveness and Readiness probes?Show answer

Liveness checks if the JVM is alive (restarts container on failure); Readiness checks if the application can accept traffic (removes pod from service endpoints on failure without restarting).

Q02Why should external databases never be included in the Liveness probe group?Show answer

If the database experiences transient latency or downtime, failing the Liveness probe causes Kubernetes to reboot all application pods simultaneously, triggering a cascading restart loop.

Q03How does management port isolation protect Actuator endpoints in production?Show answer

Setting management.server.port creates a separate embedded web server on an internal port, physically isolating administrative and monitoring endpoints from public internet traffic.

Production Pitfalls

Critical Anti-Patterns

  • Pitfall: Including external databases in liveness probe groups causing restart loops — Downstream outages will reboot healthy pods repeatedly, saturating CPU and destroying in-flight requests.
  • Pitfall: Exposing actuator endpoints on the public API HTTP port — Management endpoints must be bound to a separate internal management port or restricted by firewall rules.

Primary Documentation & Specifications

Supplementary Module Resources