Foundations · Phase 1: Foundations
Availability, SLOs, SLIs & Error Budgets
Quantify uptime math (99.9% vs 99.99%), calculate serial vs parallel system reliability, and establish actionable SRE error budget burn rates.
Primary specification: Google SRE Book: Service Level Objectives
Lesson 5 of 33
1d streakThe Architectural Question
Engineers promise 99.99% availability without recognizing that chaining five 99.9% services mathematically reduces end-to-end availability below 99.5%.
Core Engineering Objectives
- Uptime translation: 99.9% (8.7 hrs downtime/yr) vs 99.99% (52.6 mins downtime/yr)
- Component composition: serial availability product vs parallel redundancy math
- SLI measurement, SLO targets, and error budget burn rate alert thresholds
1. What is it & Why it Matters
Reliability in production systems is governed by four core terms: - SLI (Service Level Indicator): The quantifiable metric you measure (e.g., % of HTTP requests returning < 500 in under 200ms). - SLO (Service Level Objective): The internal reliability target agreed by product and engineering (e.g., 99.9% availability over 30 days). - SLA (Service Level Agreement): The contract with external customers featuring financial penalties if breached. - Error Budget: 100% - SLO (e.g., for a 99.9% SLO, the error budget is 0.1% allowable failure).
The Myth of 100% Availability
2. The Mathematics of the Nines
Every additional "9" of availability cuts allowable downtime by a factor of 10 while exponentially increasing architectural complexity:
| Availability Target | Downtime per Year | Downtime per Month (30 days) | Downtime per Week |
|---|---|---|---|
| 99% (Two 9s) | 3.65 days | 7.2 hours | 1.68 hours |
| 99.9% (Three 9s) | 8.76 hours | 43.2 minutes | 10.1 minutes |
| 99.95% (Three and a half 9s) | 4.38 hours | 21.6 minutes | 5.04 minutes |
| 99.99% (Four 9s) | 52.6 minutes | 4.32 minutes | 1.01 minutes |
| 99.999% (Five 9s) | 5.26 minutes | 25.9 seconds | 6.05 seconds |
3. Serial vs. Parallel Availability Math
Architectural Boundary · Do Not Confuse
Serial Dependencies Multiply Unavailability
Parallel Redundancy Mitigates Failures
Serial availability is the product of components: A_total = A1 * A2 * A3. Three 99.9% services in series yield (0.999)^3 = 99.7% availability (downtime triples from 43 min to 130 min/month). In parallel, A_total = 1 - (1 - A1)(1 - A2). Two 99% instances in parallel deliver 1 - (0.01)^2 = 99.99% availability.
4. How Error Budgets Govern Velocity
| Error Budget Status | Burn Rate | Engineering Action | Deployment Policy |
|---|---|---|---|
| Budget Healthy (> 50% remaining) | Normal (< 1x) | Focus on rapid feature delivery | Continuous deployment allowed |
| Budget Burning (10% lost in 24h) | Elevated (2x – 5x) | Investigate telemetry regressions during work hours | Canary deployments with automated rollback |
| Budget Exhausted (100% burned) | Critical (> 14x) | All non-critical feature releases frozen | 100% engineering bandwidth diverted to reliability |
Use it when
Enforce automatic deployment freezes when error budgets are exhausted to re-align product and engineering incentives around stability.
Reconsider when
Do not page on-call engineers for slow burns; route them to standard ticket queues to prevent alert fatigue.
5. Interview Cheat Sheet: MTBF vs. MTTR
- 01Formula: Availability = MTBF / (MTBF + MTTR), where MTBF is Mean Time Between Failures and MTTR is Mean Time To Recovery.
- 02The High-Yield Lever: Doubling MTBF (preventing bugs) is extremely hard. Reducing MTTR from 60 minutes to 1 minute (via health-check ejection, feature flags, automated rollbacks) improves availability by 60x with vastly lower engineering cost.
6. Active Recall Checkpoint
Active Recall Checkpoint
If a system consists of 3 microservices in a strict synchronous serial chain, each with 99.9% availability, what is the overall availability of the system?
Hint: Multiply the individual availabilities: (0.999)^3.
Show answer & explanation
Overall availability is (0.999)^3 ≈ 0.997, or 99.7%. Despite every component maintaining "three nines", the serial chain drops overall availability to under two and a half nines, allowing over 2 hours of downtime per month.
Architectural Trade-Off Analysis
Compare latency, throughput, consistency, and operational complexity across core industry alternatives.
Architecture Decision Matrix
Multi-axis engineering evaluation and FAANG interview rules of thumb.
SQL (Postgres / MySQL)
Relational ACID Store
Structured schema, relational joins, ACID transactions, and strict foreign keys.
NoSQL (Cassandra / DynamoDB)
Distributed Non-Relational Store
Horizontal partition keys, denormalized documents/wide-columns, and tunable consistency.
| Evaluation Axis | SQL (Postgres / MySQL) | NoSQL (Cassandra / DynamoDB) | Details |
|---|---|---|---|
Latency | Moderate | Strong | |
Throughput | Moderate | Strong | |
Scalability | Moderate | Strong | |
Consistency | Strong | Moderate | |
Complexity | Strong | Moderate | |
Cost | Moderate | Strong |
When to Choose in an Architecture Interview
Choose SQL when ACID guarantees are non-negotiable (financial ledgers, payment processing, inventory checkout) or when query access patterns are unpredictable and require ad-hoc analytical joins.
Choose NoSQL when write volume exceeds single-node capacity (>20K writes/sec), data is unstructured/hierarchical (logs, telemetry, user feeds), or multi-region active-active replication is mandatory.
Never say "NoSQL is faster than SQL" without qualification. SQL with proper B-Tree indexes on indexed primary keys is lightning fast. NoSQL wins on horizontal write scaling, not inherent single-row read speed.
Production Postmortem & Verification
Production Incident Postmortem
- Trigger / Setup: A critical financial service chains 6 microservices in series, each independently rated for 99.9% availability.
- Production Symptom: Fleet encounters over 43 hours of cumulative downtime annually, violating customer SLA commitments.
- Root Cause & Fix: Serial availability multiplies: (0.999)^6 = 99.4%; decouple dependencies with asynchronous queues and read replicas.
Acceptance Criteria & Outcomes
- Calculate annual and monthly allowed downtime minutes for any given number of 9s.
- Compute overall system availability for mixed topologies containing serial and parallel redundant components.
- Formulate an error budget burn-rate policy that halts feature deployments when risk thresholds are crossed.
Architectural & Technical Interview Review
Core FAANG Interview Questions
Q01How do you justify introducing Availability, SLOs, SLIs & Error Budgets into a high-scale production architecture?Click to reveal model answer →Hide answer ↑Requirements & Design Brief
Introduce Availability, SLOs, SLIs & Error Budgets only when measured scale, data size, or failure boundaries exceed single-node capacity. State the exact bottleneck, evaluate operational overhead, and verify that the trade-offs justify the added moving parts.
Q02What is the primary trade-off and hidden operational cost of Availability, SLOs, SLIs & Error Budgets?Click to reveal model answer →Hide answer ↑Architecture Trade-offs
It improves scalability and availability, but introduces network hops, serialization overhead, and eventual consistency or coordination complexity.
Q03How do you handle catastrophic failure or dependency degradation in Availability, SLOs, SLIs & Error Budgets?Click to reveal model answer →Hide answer ↑Failure Review & Recovery
Implement connect/read timeouts, exponential backoff with jitter, and fallback defaults so downstream degradations do not cause cascading outages.
Production Pitfalls & Anti-Patterns
Critical Failure Anti-Patterns
Premature Distributed Complexity
Scattering distributed components before identifying concrete write, read, or storage limits creates accidental points of failure and operational debt.
Production Mitigation: Start with the simplest modular design that satisfies requirements, evolving it only as measured constraints dictate.
Unbounded Retries and Cascading Storms
When a dependency degrades, immediate synchronized retries multiply downstream traffic, turning a minor slowdown into a total fleet outage.
Production Mitigation: Enforce strict retry budgets, exponential backoff, and full randomized jitter on all client calls.
Neglecting Observability and Metrics
Deploying architecture without distributed tracing or p99 tail latency alerts leaves engineers blind when partial partitions occur.
Production Mitigation: Export metrics for request rate, error rate, p95/p99 duration, and saturation at every service boundary.