CodeWithYash

Foundation · Observe

Production Logging & Redaction

Log exceptions with stack traces and context, shape parseable messages, and keep secrets and personal data out of every line.

Primary source: Spring Boot 3.5 Reference: Logging

25 min

Lesson 2 of 2

1d streak

The Architectural Question

toString() dumps leak email addresses, reset tokens, and card numbers into files retained for months, turning a routine log into a breach exhibit.

What this guide proves

  • Pass the Throwable so stack traces survive
  • Add identifiers (user, request, entity) not object dumps
  • Redact passwords, tokens, keys, card data, and PII before shipping

Structured JSON Logging for Enterprise Observability

In cloud-native environments (Kubernetes, AWS ECS, GCP Cloud Run), human-readable multi-line log files are an anti-pattern. Log shipping agents (Fluentbit, Logstash, Vector, Promtail) must parse unstructured text with fragile regexes. Structured JSON logging emits single-line JSON objects where every field (timestamp, level, thread, logger, trace_id, custom attributes) is a discrete key-value pair.

logback-spring.xml
<?xml version="1.0" encoding="UTF-8"?>
<configuration>
    <include resource="org/springframework/boot/logging/logback/defaults.xml"/>

    <appender name="JSON_CONSOLE" class="ch.qos.logback.core.ConsoleAppender">
        <encoder class="net.logstash.logback.encoder.LogstashEncoder">
            <includeMdcKeyName>traceId</includeMdcKeyName>
            <includeMdcKeyName>spanId</includeMdcKeyName>
            <includeMdcKeyName>userId</includeMdcKeyName>
            <customFields>{"app":"order-service","env":"production"}</customFields>
        </encoder>
    </appender>

    <root level="INFO">
        <appender-ref ref="JSON_CONSOLE"/>
    </root>
</configuration>

Immediate Ingestion without Parsing

Structured JSON logs can be indexed directly by Elasticsearch, OpenSearch, BigQuery, or Grafana Loki without needing grok filters. If an unexpected newline or exception stack trace is logged, it is encapsulated inside the JSON stack_trace string field, preventing log line fragmentation.

Active Recall Checkpoint

What problem does single-line JSON logging solve when dealing with multiline Java exception stack traces in Kubernetes?

Hint: How do log shippers like Promtail or Fluentbit interpret multiline logs without JSON?

Show answer & explanation

Without JSON, each line of a stack trace is captured as a separate, orphaned log entry. In JSON format, the entire multiline stack trace is contained within a single stack_trace string attribute of one JSON record, keeping the entire error event atomic.

MDC: Correlating Requests Across Threads & Distributed Systems

When thousands of concurrent requests execute simultaneously, logs from different users interleave on the same output stream. Searching for an individual user's failure becomes impossible. SLF4J provides MDC (Mapped Diagnostic Context), backed by ThreadLocal storage, allowing you to attach contextual metadata (like traceId, userId, tenantId) to every log emitted by that thread.

CorrelationFilter.java
package com.example.common.logging;

import jakarta.servlet.FilterChain;
import jakarta.servlet.ServletException;
import jakarta.servlet.http.HttpServletRequest;
import jakarta.servlet.http.HttpServletResponse;
import org.slf4j.MDC;
import org.springframework.core.Ordered;
import org.springframework.core.annotation.Order;
import org.springframework.stereotype.Component;
import org.springframework.web.filter.OncePerRequestFilter;

import java.io.IOException;
import java.util.UUID;

@Component
@Order(Ordered.HIGHEST_PRECEDENCE)
public class CorrelationFilter extends OncePerRequestFilter {
    public static final String CORRELATION_ID_HEADER = "X-Correlation-Id";
    public static final String MDC_KEY = "correlationId";

    @Override
    protected void doFilterInternal(HttpServletRequest request,
                                    HttpServletResponse response,
                                    FilterChain filterChain) throws ServletException, IOException {
        String correlationId = request.getHeader(CORRELATION_ID_HEADER);
        if (correlationId == null || correlationId.isBlank()) {
            correlationId = UUID.randomUUID().toString();
        }

        MDC.put(MDC_KEY, correlationId);
        response.setHeader(CORRELATION_ID_HEADER, correlationId);
        try {
            filterChain.doFilter(request, response);
        } finally {
            // CRITICAL: Prevent ThreadLocal leakage across pooled worker threads:
            MDC.remove(MDC_KEY);
        }
    }
}

The ThreadLocal Memory Leak Trap

Tomcat and Jetty reuse worker threads from a thread pool. If you do not clean up MDC via MDC.remove(...) or MDC.clear() inside a finally block, subsequent requests processed by that worker thread will inherit stale correlation IDs and user identities!

Active Recall Checkpoint

Why must MDC.clear() or MDC.remove() always be called in a finally block inside servlet filters?

Hint: What happens to the thread when the HTTP request finishes?

Show answer & explanation

Servlet containers reuse worker threads across requests. Failing to clear MDC causes context leakage, meaning subsequent independent requests will log with the previous request's correlation ID or user ID.

PII Redaction & Preventing Sensitive Data Leakage

Logging sensitive Personally Identifiable Information (PII) such as credit card numbers, passwords, authorization bearer tokens, Social Security Numbers (SSN), and health records violates PCI-DSS, GDPR, HIPAA, and SOC2 compliance. Applications must redact sensitive fields before logs reach disk or external collectors.

Sensitive Pattern TypeExample Unmasked InputMasked Output FormatRegex Mask Rule
Credit Card (PAN)4111 2222 3333 44444111-**-**-4444\b(?:\d[ -]*?){13,16}\b
Bearer TokenBearer eyJhbGciOiJIUzI1Ni...Bearer [REDACTED_TOKEN]Bearer\s+[A-Za-z0-9-_=]+\.[A-Za-z0-9-_=]+\.?[A-Za-z0-9-_.+/=]*
Password in JSON{"password": "MySecretPass123!"}{"password": "******"}"password"\s*:\s*"[^"]+"
Email Addressjohn.doe@company.comj***e@company.com([a-zA-Z0-9_.+-])[a-zA-Z0-9_.+-]+@([a-zA-Z0-9-]+\.[a-zA-Z0-9-.]+)

Defense in Depth: DTO Redaction

Do not rely solely on Logback regex appenders. Implement custom toString() methods or @ToString(exclude = "password") on records and DTOs so sensitive fields are never formatted in the first place.

Active Recall Checkpoint

Why is regex-based log masking at the Logback appender level considered a secondary line of defense rather than primary?

Hint: Consider CPU overhead of regexes and chances of unexpected payload formats.

Show answer & explanation

Regex masking on high-throughput log streams incurs noticeable CPU overhead and can miss variations in encoding or formatting. The primary line of defense is keeping sensitive data out of log statements entirely by sanitizing DTO toString() representations.

Deliberate Failure Lab: Auditing & Scrubbing Forbidden Log Patterns

In continuous integration (CI) pipelines, security teams run automated log scanners that grep test output for forbidden patterns (such as unmasked Authorization headers or database connection passwords).

audit-logs.sh
#!/usr/bin/env bash
# Audit log file for leaked credentials or tokens
set -euo pipefail

FORBIDDEN_REGEX='(Bearer [a-zA-Z0-9._-]{10,}|password[[:space:]]*[:=][[:space:]]*[^[:space:]]+)'

if grep -E -n "$FORBIDDEN_REGEX" /var/log/application.log; then
    echo "CRITICAL SECURITY AUDIT FAILURE: Forbidden credential patterns found in log output!"
    exit 1
fi

echo "Log security audit PASSED: No sensitive patterns detected."

Remediating the Failure

When a pattern matches, locate the logger call, replace raw payload logging with an explicit audit event DTO that omits the secret, and verify that the test passes cleanly.

Active Recall Checkpoint

What command can you run locally to verify whether your test suite prints raw authorization headers to standard output?

Hint: Grep standard output or test log files for "Authorization: Bearer".

Show answer & explanation

Run ./mvnw test | grep -i "authorization: bearer" (or inspect target/surefire-reports/). If any matches appear, audit and sanitize controller/client logging.

Working Implementation

Code Walkthrough

Log one failure with cause chain and ids, then grep output for forbidden patterns and fix whatever matches.

RedactedJsonLoggingLab.java
package com.example.logging;

import jakarta.servlet.FilterChain;
import jakarta.servlet.ServletException;
import jakarta.servlet.http.HttpServletRequest;
import jakarta.servlet.http.HttpServletResponse;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.slf4j.MDC;
import org.springframework.web.filter.OncePerRequestFilter;
import java.io.IOException;
import java.util.UUID;

public class RedactedJsonLoggingLab extends OncePerRequestFilter {
    private static final Logger log = LoggerFactory.getLogger(RedactedJsonLoggingLab.class);
    private static final String CORRELATION_ID = "X-Correlation-Id";

    @Override
    protected void doFilterInternal(HttpServletRequest request, HttpServletResponse response,
                                                                    FilterChain filterChain) throws ServletException, IOException {
        String traceId = request.getHeader(CORRELATION_ID);
        if (traceId == null || traceId.isBlank()) {
            traceId = UUID.randomUUID().toString();
        }

        MDC.put("traceId", traceId);
        response.setHeader(CORRELATION_ID, traceId);
        try {
            log.info("Inbound request path={} method={}", request.getRequestURI(), request.getMethod());
            filterChain.doFilter(request, response);
        } finally {
            // Mandatory cleanup to prevent ThreadLocal pollution in worker pools
            MDC.remove("traceId");
        }
    }
}
MDC (Mapped Diagnostic Context) attaches correlation IDs across all logs executed on that request thread.
Always remove MDC keys inside a finally block to prevent thread pool context leakage.
Structured JSON appenders capture MDC keys as top-level JSON fields for instant indexing in Loki or Elasticsearch.

Synthesis & Production Verification

Production Acceptance Criteria

  • One ERROR line carries message, exception class, and full stack trace
  • Structured fields allow filtering by request id
  • An automated scan finds no sensitive patterns in output

Architectural & Technical Review

Core Interview Questions

Q01What advantages does structured JSON logging offer over plain text logs in cloud environments?Show answer

Emits single-line JSON records where timestamps, levels, trace IDs, and messages are discrete fields, allowing log shippers (Fluentbit, Loki, Logstash) to index them directly without fragile regex parsing.

Q02How does SLF4J MDC enable tracing requests across concurrent worker threads?Show answer

MDC stores contextual key-value pairs (like traceId, userId) in ThreadLocal storage, automatically appending them to every log statement executed by that thread.

Q03How should sensitive Personally Identifiable Information (PII) be protected in log streams?Show answer

Implement DTO sanitization in toString(), configure masking regex converters in Logback appenders, and enforce automated CI log scanning for forbidden patterns.

Production Pitfalls

Critical Anti-Patterns

  • Pitfall: Failing to clear MDC in servlet filters causing ThreadLocal memory leaks — Servlet containers reuse pooled worker threads; uncleaned MDC leaks stale user and correlation IDs to subsequent requests.
  • Pitfall: Logging raw credit card numbers or bearer tokens in request payloads — Violates PCI-DSS and GDPR; sensitive attributes must be masked or excluded before serialization.

Module Certification Quiz

Prove Module Mastery

Comprehensive question set covering all lessons in this module.

Take Module Quiz

Primary Documentation & Specifications

Lesson Primary References

Spring Boot 3.5 Reference: LoggingSLF4J 2.0 Manual & MDC Specification

Supplementary Module Resources

LoggingOWASP logging cheat sheet