CodeWithYash

Advanced · Ship

Build a Safe CI/CD & Secrets Path

Promote one artifact through test, scan, sign, deploy, and rollback gates.

Primary source: Spring Boot 3.5 Reference: Container Images & Probes

45 min

Lesson 2 of 4

1d streak

The Architectural Question

Secrets baked into images or pipeline logs leak through registries, caches, and build history long after the commit that exposed them.

What this guide proves

  • Never place secrets in source, image layers, or logs
  • Run database migrations as an explicit compatible step
  • Record artifact provenance and deployment evidence

Twelve-Factor Secrets: Memory Volumes vs Environment Variables

Externalize credentials and secrets from source control and application images. Environment variables and mounted files have different exposure, rotation, access-control, and diagnostic trade-offs; choose between them using the deployment threat model.

Diagnostic tools and child processes may expose environment variables, depending on the platform and tooling; treat them as sensitive input and restrict access.

Environment variables and mounted files can both leak through diagnostics or overly broad permissions. Test the chosen deployment so secrets are masked and readable only by the intended process.

Use it when

Mount Kubernetes Secrets through a platform-managed, least-privilege mechanism; read-only files can make rotation and permissions explicit. Spring Boot can natively ingest directory-based configuration using spring.config.import=configtree:/etc/secrets/.

Reconsider when

Environment variables may be acceptable for sensitive values only when their exposure and rotation risks are documented; use them freely for non-sensitive settings such as SPRING_PROFILES_ACTIVE=prod.

k8s-deployment-secrets.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: catalog-service
spec:
  template:
    spec:
      securityContext:
        runAsNonRoot: true
        runAsUser: 10001
      containers:
        - name: app
          image: registry.internal/catalog-service:v1.4.2
          volumeMounts:
            - name: db-secrets
              mountPath: /etc/secrets/db
              readOnly: true
      volumes:
        - name: db-secrets
          secret:
            secretName: catalog-db-credentials

Dynamic Secret Resolution: HashiCorp Vault and AWS Secrets Manager

Spring Boot can integrate with external configuration and secret-management systems through the relevant starter and spring.config.import. The provider, client library, authentication, and cache behavior determine whether a value is written to disk or retained in memory.

src/main/resources/application.yml
spring:
  config:
    import:
      - "configtree:/etc/secrets/db/"
      - "vault://secret/catalog-service"
  cloud:
    vault:
      uri: https://vault.internal:8200
      authentication: KUBERNETES
      kubernetes:
        role: catalog-service-role

Image Immutability and Version Promotion

A golden rule of cloud delivery is image immutability. Build the container image exactly once for a given git commit SHA, and promote that identical cryptographic digest through Development, Staging, and Production environments without rebuilding.

The "latest" Tag Anti-Pattern

Never use the :latest tag in Kubernetes manifests. :latest is mutable, non-deterministic, and breaks rollout tracking and automated rollbacks.

Verifying Secret Masking in Spring Boot Actuator

Verify that injected secrets are correctly sanitized and masked from management endpoints:

verify-secrets.sh
# Query Actuator /env endpoint and verify passwords are fully sanitized
curl -s http://localhost:8080/actuator/env/spring.datasource.password
# EXPECTED: {"property":{"source":"configtree:/etc/secrets/db/","value":"******"}}

Active Recall Checkpoint

Why are volume-mounted secrets safer than OS environment variables for sensitive credentials?

Hint: Think about what happens when a diagnostic tool runs env or printenv.

Show answer & explanation

Environment variables are often dumped in error logs, crash dumps, and child processes. File mounts on tmpfs remain in RAM, scoped strictly to file permissions, and avoid accidental log leakage.

Runtime Architecture & Flow

Resolution model

Configuration resolution checklist

  1. 1

    Never place secrets in source, image layers, or logs

  2. 2

    Run database migrations as an explicit compatible step

  3. 3

    Record artifact provenance and deployment evidence

Inspect the effective application state before assuming which declaration won.

Working Implementation

Code Walkthrough

Create a pipeline that rejects a failed test or image scan before deployment.

ExternalizedSecretsLab.java
package com.example.config;

import org.springframework.boot.context.properties.ConfigurationProperties;
import org.springframework.boot.context.properties.bind.ConstructorBinding;

import java.util.Objects;

@ConfigurationProperties(prefix = "app.security")
public final class ExternalizedSecretsLab {

    private final String apiKey;
    private final String dbPassword;

    @ConstructorBinding
    public ExternalizedSecretsLab(String apiKey, String dbPassword) {
        this.apiKey = Objects.requireNonNull(apiKey, "apiKey must not be null");
        this.dbPassword = Objects.requireNonNull(dbPassword, "dbPassword must not be null");
    }

    public String getApiKey() {
        return apiKey;
    }

    public String getDbPassword() {
        return dbPassword;
    }

    // Mask secret values in diagnostic toString output
    @Override
    public String toString() {
        return "ExternalizedSecretsLab{apiKey='******', dbPassword='******'}";
    }
}
Sensitive secrets should be mounted as memory-backed files (tmpfs) rather than raw OS environment variables.
Spring Boot ingests mounted directories via spring.config.import=configtree:/etc/secrets/.
Custom toString() and Actuator sanitization prevent credential leakage in error logs and crash dumps.

Synthesis & Production Verification

Production Acceptance Criteria

  • Prove no secrets exist in source, image layers, or logs
  • Run database migrations as an explicit step compatible with both old and new code
  • Record artifact provenance and deployment evidence for each release

Architectural & Technical Review

Core Interview Questions

Q01Why are volume-mounted secrets safer than OS environment variables for sensitive credentials in Kubernetes?Show answer

Environment variables are inherited by child processes and frequently leaked in diagnostic crash dumps (hs_err_pid.log) and APM logs. Volume mounts on tmpfs remain in RAM and are restricted by strict file system permissions.

Q02How does Spring Boot dynamically resolve runtime secrets from HashiCorp Vault using spring.config.import?Show answer

Using spring.config.import=vault://secret/app, Spring Cloud Vault authenticates via Kubernetes ServiceAccount tokens at startup and pulls secrets directly into the Spring Environment without persisting them to disk.

Q03Why is the :latest container image tag prohibited in production Kubernetes delivery pipelines?Show answer

The :latest tag is mutable and non-deterministic. It prevents reproducible deployments, makes automated rollbacks unreliable, and obscures exactly which code commit is running in production.

Production Pitfalls

Critical Anti-Patterns

  • Pitfall: Injecting database passwords directly into OS environment variables — Leads to credential exposure in child processes, diagnostic crash logs, and APM error dumps.
  • Pitfall: Rebuilding container images per environment instead of promoting immutable SHA-tagged digests — Violates delivery immutability, causing untested dependency drift between staging and prod.

Primary Documentation & Specifications

Lesson Primary References

Spring Boot 3.5 Reference: Container Images & ProbesKubernetes Documentation: Configure Liveness, Readiness and Startup Probes

Supplementary Module Resources

Spring Boot container imagesGraceful shutdown