Monitoring & metrics

Before you start

• EvenTier exposes Prometheus text-format metrics from two places: the web process at GET /api/metrics, and the dedicated worker on port 9464 at /metrics. Both are disabled (404) until METRICS_TOKEN is set, and then require `Authorization: Bearer <token>` — generate one with `openssl rand -hex 32`.
• Database-derived gauges (queue depth, backlog age, worker heartbeat) are identical from every replica — any one target gives the full picture. Per-process series (memory, connection pool, domain counters) differ per pod, so scrape web and worker both; the Helm chart wires this for you.
• Domain counters follow standard Prometheus semantics: per-process, reset on restart or deploy. Always consume them with rate() or increase(), never raw values.
• HTTP request metrics (rate/errors/duration per route) are deliberately not exposed by the app — take them from your ingress controller (nginx-ingress and Traefik export per-service RED metrics for free), which also sees CDN and TLS behavior the app can't.

Step by step

1. 01

   Enable the endpoint

   Set METRICS_TOKEN in your environment (or `monitoring.enabled: true` + `monitoring.metricsToken` in Helm values — the chart injects it into both processes and opens the worker's metrics port). Verify with the curl example below.

2. 02

   Point Prometheus at it

   With the Prometheus Operator, set `monitoring.serviceMonitor.enabled: true` — one ServiceMonitor covers the web Service (/api/metrics) and the worker metrics Service (:9464/metrics). Plain Prometheus uses the scrape config example below.

3. 03

   Turn on the alerts

   Set `monitoring.prometheusRule.enabled: true` for the three shipped rules: outbox backlog age (worker fell behind — warning at 60s), worker heartbeat stale (worker died — critical at 180s), and dead-lettered events (warning when any event exhausts its retries). Thresholds are chart values.

4. 04

   Import the dashboard

   Import deploy/grafana/eventier-overview.json (Grafana → Dashboards → Import) and pick your Prometheus datasource. It ships queue health, ticket throughput, notification/email volume, AI token spend, memory, pool, and DB latency panels — a starting point to copy panels from into your own dashboards.

5. 05

   Add database-side visibility

   On PostgreSQL, set `log_min_duration_statement = 500` and enable `pg_stat_statements` to catch slow queries the app-side histograms only hint at. If you run postgres_exporter, its dashboards complement this one.

Examples

curl -H "Authorization: Bearer $METRICS_TOKEN" \
  https://eventier.internal.example.com/api/metrics

# Dedicated worker (in-cluster):
curl -H "Authorization: Bearer $METRICS_TOKEN" \
  http://<release>-worker-metrics:9464/metrics

scrape_configs:
  - job_name: eventier-web
    metrics_path: /api/metrics
    authorization:
      type: Bearer
      credentials: <METRICS_TOKEN>
    static_configs:
      - targets: ["eventier.internal.example.com"]
    scheme: https
  - job_name: eventier-worker
    metrics_path: /metrics
    authorization:
      type: Bearer
      credentials: <METRICS_TOKEN>
    static_configs:
      - targets: ["<release>-worker-metrics:9464"]

# Is the worker keeping up? (the alert signal)
max(eventier_outbox_oldest_pending_age_seconds)

# Tickets created vs resolved, per hour
sum(increase(eventier_requests_created_total[1h]))
sum(increase(eventier_requests_resolved_total[1h]))

# AI token spend by feature, per day
sum by (feature) (increase(eventier_ai_tokens_total[24h]))

# DB query latency p95 (ms)
histogram_quantile(0.95,
  sum by (le) (rate(prisma_client_queries_duration_histogram_ms_bucket[5m])))

Metric reference

Gauges marked DB-derived are identical on every target; counters are per-process and reset on restart — use rate()/increase(). Prisma's built-in series (prisma_*) ride along on every scrape.

Good to know

Related