CodeWithYash

Advanced · Ship

Production Implementation: Immutable Deploy Pipeline

Package a minimal image, wire probes and graceful shutdown, and deliver through a pipeline whose secrets never appear in logs or layers.

Primary source: Spring Boot 3.5 Reference: Container Images & Probes

50 min

Lesson 4 of 4

1d streak

The Architectural Question

Images that bake configuration break the moment they leave one environment; missing readiness probes make every deploy an outage window.

What this guide proves

  • Bake a non-root JVM image with layered jars
  • Split liveness from readiness and prove rolling behavior
  • Inject secrets at runtime from the platform, never the image

Enterprise GitOps & Declarative Infrastructure As Code

In modern enterprise architectures, manual kubectl commands are forbidden. GitOps establishes a single source of truth: the Git repository containing declarative Kubernetes manifests synchronized by controllers like ArgoCD or Flux.

k8s-order-service.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: order-service
  labels:
    app: order-service
spec:
  replicas: 3
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0
  selector:
    matchLabels:
      app: order-service
  template:
    metadata:
      labels:
        app: order-service
    spec:
      containers:
        - name: app
          image: registry.internal/order-service:v2.1.0
          ports:
            - containerPort: 8080
          readinessProbe:
            httpGet:
              path: /actuator/health/readiness
              port: 8080
            initialDelaySeconds: 15
            periodSeconds: 5
          livenessProbe:
            httpGet:
              path: /actuator/health/liveness
              port: 8080
            initialDelaySeconds: 30
            periodSeconds: 10

Zero-Downtime Deployment: Rolling vs Blue/Green vs Canary

Achieving true zero-downtime deployments requires aligning application backward compatibility with routing strategies:

StrategyMechanicsPros & Cons
Rolling UpdateGradually replaces old pods with new pods 1-by-1Low resource cost; both versions run concurrently, requiring database backward compatibility
Blue/GreenDeploys full duplicate v2 fleet, switches router at onceInstant rollback; requires 2x compute capacity and database backward/forward compatibility
Canary ReleaseRoutes 5% of traffic to v2, monitors error rates, then scalesLimits blast radius; requires advanced ingress/service mesh (Istio, Linkerd, Envoy)

Automated Health-Driven Rollbacks via Prometheus & Argo Rollouts

Enterprise delivery pipelines must not rely on humans staring at dashboards. Automated rollbacks must trigger if the new release exhibits elevated HTTP 5xx error rates, latency regression, or container crash loops.

analysis-template.yaml
apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata:
  name: success-rate-check
spec:
  metrics:
    - name: success-rate
      interval: 1m
      successCondition: result[0] >= 0.999
      failureLimit: 2
      provider:
        prometheus:
          address: http://prometheus.monitoring:9090
          query: |
            sum(rate(http_server_requests_seconds_count{status!~"5.*",app="order-service"}[1m]))
            /
            sum(rate(http_server_requests_seconds_count{app="order-service"}[1m]))

End-to-End Production Deployment Verification Checklist

Prior to signing off on a production deployment, verify all four golden deployment signals:

snippet.bash
# 1. Verify Pod Disruption Budget (PDB) is present to prevent node drain outages
kubectl get pdb order-service-pdb

# 2. Verify rolling update maxUnavailable is 0
kubectl get deployment order-service -o jsonpath='{.spec.strategy.rollingUpdate.maxUnavailable}'
# EXPECTED: 0 (or 0%)

# 3. Simulate rolling update and observe zero HTTP request drops
hey -n 10000 -c 50 -q 50 http://api.internal/orders &
kubectl rollout restart deployment order-service
wait
# EXPECTED: 10,000 requests, 0 status 502/503/504 errors!

Active Recall Checkpoint

Why does a rolling update require strict backward-compatible database migrations?

Hint: Review expand-and-contract schema migration patterns.

Show answer & explanation

During a rolling update, pods running version N and version N+1 operate concurrently against the same database schema. If version N+1 makes a breaking schema change (like dropping or renaming a column), version N pods will immediately crash.

Runtime Architecture & Flow

High-value visual model

Readiness, Liveness and Shutdown

Safe rollout depends on separating traffic routing, restart decisions, and termination drain.

1. Startup

The pod starts, loads configuration, and warms required infrastructure.

2. Readiness passes

The platform may now route traffic to this instance.

3. Readiness fails before stop

Traffic is removed before in-flight work is drained.

4. Graceful termination

HTTP requests, consumers, and pools close within the termination budget.

Kubernetes lifecycle timeline: Readiness controls traffic. Liveness controls restart. Shutdown controls in-flight work.

Working Implementation

Core Spring Implementation

Immutable deployment with probes and graceful shutdown

Promote one immutable image through environments, gate traffic with readiness, restart with liveness, and drain in-flight requests using server.shutdown=graceful.

Component Boundaries

  1. 01CI builds and scans a non-root image once; configuration and secrets inject at runtime and never bake into layers.
  2. 02Readiness controls Service endpoints; liveness controls restarts, so a downstream outage flips readiness only.
  3. 03management.endpoint.health.probes.enabled exposes /actuator/health/readiness and /liveness outside Kubernetes, and server.shutdown=graceful drains active requests on SIGTERM.

Required Evidence

  • kubectl delete pod logs the Tomcat line "Commencing graceful shutdown. Waiting for active requests to complete" followed by "Graceful shutdown complete".
  • terminationGracePeriodSeconds exceeds the slowest in-flight request, proven by zero 5xx spikes during a rolling update.
  • Rollback pins the previous image digest, with Flyway migrations verified forward-compatible before promotion.

Implementation sequence

Step-by-Step Integration

Follow these structured steps to inject Core Spring functionality cleanly across your boundaries.

01

Bake a minimal non-root image

Dockerfile

Root-owned containers fail restricted Pod Security Standards and widen blast radius; the same digest moves unchanged across environments, or bootBuildImage produces an equivalent Paketo non-root OCI image without a Dockerfile.

Dockerfile
FROM eclipse-temurin:21-jre
WORKDIR /app
RUN addgroup --system spring && adduser --system spring --ingroup spring
COPY target/app.jar app.jar
USER spring:spring
ENTRYPOINT ["java", "-jar", "/app/app.jar"]

Verify Configuration

docker inspect --format "{{.Config.User}}" app:1.0.0 prints spring, and trivy image app:1.0.0 reports no critical CVEs in the base layer.

02

Split liveness from readiness

deployment.yaml

Probe endpoints ship disabled by default and switch on automatically when deployed to Kubernetes, or set management.endpoint.health.probes.enabled=true; binding liveness to downstream dependencies converts a cache blip into a restart storm.

deployment.yaml
readinessProbe:
  httpGet:
    path: /actuator/health/readiness
    port: 8080
livenessProbe:
  httpGet:
    path: /actuator/health/liveness
    port: 8080

Verify Configuration

kubectl get pods shows READY 1/1 only after curl localhost:8080/actuator/health/readiness returns {"status":"UP"}; stopping Redis flips readiness to HTTP 503 {"status":"OUT_OF_SERVICE"} while /actuator/health/liveness keeps returning 200.

Synthesis & Production Verification

Deliberate Failure Condition

  1. Break: Copy application-prod.yml containing credentials into the image
  2. Observe: Anyone who can pull the image owns your database, and config changes require a rebuild
  3. Repair: Immutable images stay environment-free: inject values at runtime so one artifact serves every environment safely.

Production Acceptance Criteria

  • One image boots dev and prod differing only in environment variables
  • Readiness gates traffic until the app can serve
  • SIGTERM drains in-flight requests before exit

Architectural & Technical Review

Core Interview Questions

Q01How does GitOps with declarative Kubernetes manifests prevent configuration drift across environments?Show answer

GitOps controllers like ArgoCD continuously reconcile the live cluster state against declarative manifests in Git. Any manual changes made in the cluster are automatically overwritten or flagged as out-of-sync.

Q02What are the trade-offs between RollingUpdate, Blue/Green, and Canary deployment strategies?Show answer

RollingUpdate uses minimal resources but runs mixed versions simultaneously, requiring database backward compatibility. Blue/Green offers instant rollback but requires 2x compute capacity. Canary limits blast radius by routing a small percentage of traffic first.

Q03How do automated analysis templates trigger automatic deployment rollbacks based on Prometheus metrics?Show answer

Analysis templates query Prometheus for golden signals (HTTP 5xx error rate, p99 latency) during canary rollout. If the error rate exceeds the SLO threshold, the rollout controller aborts promotion and scales old pods back up.

Production Pitfalls

Critical Anti-Patterns

  • Pitfall: Deploying breaking database schema migrations simultaneously with rolling application updates — Version N pods still running will crash immediately when version N+1 removes or alters columns.
  • Pitfall: Setting rollingUpdate maxUnavailable > 0 without adequate replica headroom — Causes capacity drops and request throttling during high-traffic midday deployments.

Module Certification Quiz

Prove Module Mastery

Comprehensive question set covering all lessons in this module.

Take Module Quiz

Primary Documentation & Specifications

Lesson Primary References

Spring Boot 3.5 Reference: Container Images & ProbesKubernetes Documentation: Configure Liveness, Readiness and Startup Probes

Supplementary Module Resources

Spring Boot container imagesGraceful shutdown