CodeWithYash

Intermediate · Observe

Build a Useful Prometheus & Grafana Pipeline

Export, scrape, query, and visualize service-level indicators instead of collecting decorative counters.

Primary source: Spring Boot Actuator: Production-ready Features

40 min

Lesson 3 of 3

1d streak

The Architectural Question

Dashboards that collect everything answer nothing; useful panels derive from the rates, errors, duration, and saturation of the resource actually constraining the service.

What this guide proves

  • Measure rates, errors, duration, and saturation
  • Alert on user impact with a sustained window
  • Keep dashboards tied to an explicit operational question

Prometheus Pull Architecture & Metric Dimensionality

Prometheus operates on a pull-based model: the Prometheus server periodically scrapes an HTTP endpoint (/actuator/prometheus) exposed by the target application, pulling a text stream formatted in the Prometheus exposition standard.

prometheus-scrape-sample.txt
# HELP http_server_requests_seconds Duration of HTTP server request handling
# TYPE http_server_requests_seconds histogram
http_server_requests_seconds_bucket{exception="None",method="GET",outcome="SUCCESS",status="200",uri="/api/orders/{id}",le="0.05"} 1420.0
http_server_requests_seconds_bucket{exception="None",method="GET",outcome="SUCCESS",status="200",uri="/api/orders/{id}",le="0.1"} 1850.0
http_server_requests_seconds_bucket{exception="None",method="GET",outcome="SUCCESS",status="200",uri="/api/orders/{id}",le="+Inf"} 1900.0
http_server_requests_seconds_count{exception="None",method="GET",outcome="SUCCESS",status="200",uri="/api/orders/{id}"} 1900.0
http_server_requests_seconds_sum{exception="None",method="GET",outcome="SUCCESS",status="200",uri="/api/orders/{id}"} 95.42

The Unparameterized URI Metric Trap

The standard HTTP server instrumentation generally records a templated route such as uri="/api/orders/{id}" instead of the raw path uri="/api/orders/12345"; custom filters can accidentally reintroduce raw paths. If a developer writes a manual filter that tags metrics with raw path variables, every unique order ID spawns a new time-series, leading to catastrophic cardinality explosion.

Active Recall Checkpoint

What happens to Prometheus memory consumption when a metric tag is populated with unique customer UUIDs?

Hint: What does Prometheus create for every unique combination of tag key-value pairs?

Show answer & explanation

Prometheus stores an independent time series for every unique combination of tag values. UUIDs, emails, and timestamps create unbounded series growth, increasing memory, storage, and query costs until the monitoring system becomes unhealthy.

The Four Golden Signals: Designing Production Grafana Dashboards

The Google Site Reliability Engineering (SRE) handbook outlines the Four Golden Signals of monitoring: Latency, Traffic, Errors, and Saturation. An effective Grafana dashboard organizes these signals into a clear visual hierarchy.

Golden SignalCore Metric in Spring BootPromQL Query Example
Traffic (Throughput)http_server_requests_seconds_countsum(rate(http_server_requests_seconds_count[1m])) by (uri)
Errors (Ratio)http_server_requests_seconds_count (status 5xx)sum(rate(http_server_requests_seconds_count{status=~"5.."}[1m])) / sum(rate(http_server_requests_seconds_count[1m])) * 100
Latency (p95 / p99)http_server_requests_seconds_buckethistogram_quantile(0.95, sum(rate(http_server_requests_seconds_bucket[5m])) by (le, uri))
Saturation (Pools & CPU)hikaricp_connections_pending / system_cpu_usagehikaricp_connections_pending{pool="HikariPool-1"}

Active Recall Checkpoint

Why should SLA/SLO latency alerts be based on p95 or p99 percentiles rather than arithmetic averages (mean)?

Hint: How do a few very fast requests distort the average for users experiencing 5-second delays?

Show answer & explanation

Averages hide outliers: 99 requests taking 10ms and 1 request taking 10,000ms yield an average of only 110ms, disguising the fact that 1% of your customers suffered severe timeouts. Percentiles accurately reflect the worst-case customer experience.

Alerting Rules: Actionable Alerts vs Alert Fatigue

Alerts must be strictly actionable: if an alert triggers and there is no direct action for an on-call engineer to take, the alert should not page human beings. Rules should target user-impacting symptoms rather than internal implementation causes.

alerts.rules.yml
groups:
  - name: order-service-alerts
    rules:
      - alert: HighHttpErrorRate
        expr: |
          sum(rate(http_server_requests_seconds_count{status=~"5.."}[5m]))
          /
          sum(rate(http_server_requests_seconds_count[5m])) > 0.02
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: "HTTP 5xx error rate exceeded 2% for 2 minutes"
          runbook_url: "https://wiki.example.com/runbooks/high-error-rate"

      - alert: DatabaseConnectionPoolStarvation
        expr: hikaricp_connections_pending > 0
        for: 1m
        labels:
          severity: warning
        annotations:
          summary: "Threads are blocked waiting for database connections in HikariCP pool"
          description: "Pool is fully saturated. Check for slow queries or connection leaks."

Active Recall Checkpoint

Why is the for: 2m clause critical in Prometheus alert rules?

Hint: What happens during transient single-second network blips without the for clause?

Show answer & explanation

The for duration prevents transient temporary spikes (lasting a few seconds) from triggering false alarms. An alert only fires if the failure condition persists continuously across the specified duration.

Deliberate Failure Lab: Diagnosing Cardinality Explosions

A cardinality explosion occurs when code dynamically introduces unbounded distinct tag values to a metric.

VulnerableController.java
@GetMapping("/user/{id}")
public User getUser(@PathVariable String id) {
    // ANTI-PATTERN: Tagging counter with arbitrary user ID:
    registry.counter("api.users.lookups", "user_id", id).increment(); // EXPLOSION HAZARD!
    return userService.findById(id);
}

The Resulting Failure

If the application has 500,000 users, Micrometer generates 500,000 distinct Meter objects. Prometheus scrapes a 50MB payload every 15 seconds, garbage collection freezes the JVM, and Prometheus out-of-memory crashes.

Remediation Rule

Tags must be bounded enums or low-cardinality categories (e.g. status: SUCCESS | FAILURE, region: US_EAST | EU_WEST, tier: FREE | PREMIUM). Never use user IDs, credit card numbers, order IDs, or timestamps as tags.

Active Recall Checkpoint

What is the maximum recommended number of distinct tag combinations (cardinality) for a single metric in a production microservice?

Hint: Keep cardinality low to protect memory and scrape latency.

Show answer & explanation

There is no universal safe maximum: keep dimensions bounded, define a series/cardinality budget for the backend, and alert on growth before scrape and query performance degrades.

Runtime Architecture & Flow

Observability · runtime visualization

Build a Useful Prometheus & Grafana Pipeline

Request

Measure rates, errors, duration, and saturation

Logs

Alert on user impact with a sustained window

Metrics

Keep dashboards tied to an explicit operational question

Traces

Create a dashboard for traffic, p95 latency, error ratio, and pool saturation.

Working Implementation

Code Walkthrough

Create a dashboard for traffic, p95 latency, error ratio, and pool saturation.

PrometheusAlertsMetricsLab.java
package com.example.observability;

import io.micrometer.core.instrument.Counter;
import io.micrometer.core.instrument.MeterRegistry;
import org.springframework.stereotype.Component;

@Component
public class PrometheusAlertsMetricsLab {
    private final Counter successCounter;
    private final Counter errorCounter;

    public PrometheusAlertsMetricsLab(MeterRegistry registry) {
        this.successCounter = Counter.builder("http_orders_total")
                .tag("status", "success")
                .tag("tier", "standard")
                .description("Total successfully processed orders")
                .register(registry);

        this.errorCounter = Counter.builder("http_orders_total")
                .tag("status", "error")
                .tag("tier", "standard")
                .description("Total failed orders for error ratio alerting")
                .register(registry);
    }

    public void recordOutcome(boolean success) {
        if (success) successCounter.increment();
        else errorCounter.increment();
    }
}
Exposed on /actuator/prometheus for scraping by Prometheus or Grafana Agent.
Tags status=success/error enable computing 5xx error percentage: sum(rate(...{status="error"}[5m])) / sum(rate(...[5m])).
Never place dynamic IDs, timestamps, or tokens in metric tags to prevent Prometheus cardinality explosions.

Synthesis & Production Verification

Production Acceptance Criteria

  • Compute p95 latency from histogram buckets
  • Alert on an error ratio held above threshold for a sustained window
  • Tie every panel to one explicit operational question

Architectural & Technical Review

Core Interview Questions

Q01How does Prometheus scraping work and what causes metric cardinality explosions?Show answer

Prometheus periodically scrapes /actuator/prometheus over HTTP. Adding unbounded tag values (like user IDs, UUIDs, or raw query paths) creates millions of unique time-series, exhausting Prometheus memory.

Q02What are the Google SRE Four Golden Signals monitored on production dashboards?Show answer

Latency (time taken to serve requests), Traffic (demand/requests per second), Errors (rate of failed requests), and Saturation (fraction of system capacity/pool utilized).

Q03Why should latency SLO alerts be defined on p95 or p99 percentiles rather than averages?Show answer

Arithmetic averages conceal extreme tail latency experienced by outliers. A low average can easily disguise that 1% of customers suffer severe 10-second timeouts.

Production Pitfalls

Critical Anti-Patterns

  • Pitfall: Tagging metrics with unbounded UUIDs or timestamps causing cardinality explosions — Each unique combination of tag key-values allocates a distinct metric time-series in memory, rapidly crashing Prometheus.
  • Pitfall: Alerting on arithmetic latency averages rather than p95/p99 percentiles — Averages hide severe tail degradation; 99 fast requests mask 1 timed-out request completely.

Module Certification Quiz

Prove Module Mastery

Comprehensive question set covering all lessons in this module.

Take Module Quiz

Primary Documentation & Specifications

Lesson Primary References

Spring Boot Actuator: Production-ready FeaturesMicrometer 1.14 Documentation

Supplementary Module Resources

Actuator endpointsObservability