Designing Actionable SRE Alerting with Prometheus and Grafana

Every Site Reliability and Application Support Engineer has experienced alert fatigue: getting paged at 03:00 AM because a single node spiked to 85% CPU for two minutes, even though user-facing latency and error rates were completely unaffected.

Effective observability separates symptoms (what users experience) from causes (resource saturation on individual instances).

1. The RED and USE Frameworks#

Structure your Grafana dashboards and Prometheus recording rules around two complementary mental models:

• Page on RED metrics (e.g., HTTP 5xx error ratio exceeding SLO or p99 latency > 500ms).
• Use USE metrics on dashboards to diagnose the root cause once an incident starts.

2. Pre-Computing SLIs with Prometheus Recording Rules#

Querying raw high-cardinality histograms across millions of time series on every Grafana refresh slows down dashboards and rule evaluations. Pre-aggregate your error ratios using recording rules:

1groups:
2  - name: slo_http_recording_rules
3    interval: 30s
4    rules:
5      - record: job:http_requests_total:rate5m
6        expr: |
7          sum by (job, service) (
8            rate(http_requests_total[5m])
9          )
10
11      - record: job:http_requests_errors:ratio_rate5m
12        expr: |
13          sum by (job, service) (
14            rate(http_requests_total{status=~"5.."}[5m])
15          )
16          /
17          job:http_requests_total:rate5m

3. Multi-Window Burn-Rate Alerts#

Instead of alerting on a raw threshold, calculate how fast your service is consuming its 30-day error budget. For a 99.9% availability SLO (error budget of 0.001), a 14.4x burn rate consumes 2% of your monthly budget in just 1 hour.

Pairing a 1-hour window with a 5-minute confirmation window prevents stale alerts from staying firing after the issue has already recovered:

1groups:
2  - name: slo_burn_rate_alerts
3    rules:
4      - alert: HighErrorBudgetBurnCritical
5        expr: |
6          (
7            job:http_requests_errors:ratio_rate1h > (14.4 * 0.001)
8            and
9            job:http_requests_errors:ratio_rate5m > (14.4 * 0.001)
10          )
11        for: 2m
12        labels:
13          severity: critical
14        annotations:
15          summary: "High error budget burn on {{ $labels.service }}"
16          description: "Service {{ $labels.service }} is burning its 30-day 99.9% error budget at >14.4x rate."
17          runbook_url: "https://geenodh.me/blog/observability-stack-prometheus-grafana-alerting"

$promtool check rules prometheus-slis.rules.yaml prometheus-burn-rate-alerts.yaml
$curl -s http://localhost:9090/-/reload