Monitoring & metrics
Watch EvenTier with Prometheus: a token-protected metrics endpoint on the web and worker processes, queue health gauges, throughput counters and a Grafana dashboard.
- Where
- GET /api/metrics (web) · :9464/metrics (worker) · deploy/grafana/
- Permission
- METRICS_TOKEN bearer token · admin of the host, cluster or server
- Updated
- October 2026
Before you start
EvenTier publishes metrics in Prometheus text format from two places: the web process at GET /api/metrics, and the dedicated worker on port 9464 at /metrics.
Both are off, answering 404, until you set METRICS_TOKEN. After that, every scrape needs Authorization: Bearer <token>.
Gauges read from the database (queue depth, backlog age, worker heartbeat) are the same on every replica, so any one target gives the full picture.
Per-process series (memory, connection pool, domain counters) differ per pod, so scrape both the web and the worker. The Helm chart sets this up for you.
On Docker Compose and the server install, the worker runs inside the app process, so GET /api/metrics is the only target.
Counters work the standard Prometheus way: they are per process and reset on restart or deploy.
Query counters with rate() or increase(); raw values are not meaningful across restarts.
The app does not expose HTTP request metrics (rate, errors and duration per route). Take them from your ingress controller.
nginx-ingress and Traefik export per-service RED (rate, errors, duration) metrics, and the ingress also sees content delivery network (CDN) and TLS (HTTPS encryption) behavior that the app cannot.
Step by step
- 01
Enable the endpoint
Create a token and set it as
METRICS_TOKENto switch the endpoints on. Check it with the curl example below.- Helm: set
monitoring.enabled: trueandmonitoring.metricsTokenin your values instead - The chart injects the token into both processes and opens the worker's metrics port
Generate and set the tokenbash openssl rand -hex 32 # then, in your environment file METRICS_TOKEN=<the generated token> - Helm: set
- 02
Point Prometheus at it
Tell Prometheus where to scrape. With the Prometheus Operator, it is one Helm value.
- One ServiceMonitor covers the web Service (
/api/metrics) and the worker metrics Service (:9464/metrics) - Plain Prometheus: use the scrape config example below
Helm valuesyaml monitoring: serviceMonitor: enabled: true - One ServiceMonitor covers the web Service (
- 03
Turn on the alerts
Switch on the included alert rules, and point probes and uptime checks at the health endpoints.
- Critical: scrape failing, database unreachable, worker heartbeat stale (at 180s)
- Warning: uncaught server errors, outbox backlog age (at 60s), dead-lettered events, failed scheduled jobs
- Thresholds are chart values
/api/health/live: the process answers/api/health/ready: the database and applied migrations; with?require=worker, also a fresh worker heartbeat
Helm valuesyaml monitoring: prometheusRule: enabled: true - 04
Import the dashboard
Load the included Grafana dashboard and choose your Prometheus data source.
- Panels: queue health, ticket throughput, notification and email volume, AI token spend, memory, pool and database latency
- Copy panels from it into your own dashboards
- Grafana → Dashboards → Import
- File: deploy/grafana/eventier-overview.json
- 05
Add database-side visibility
Turn on slow query logging in PostgreSQL to catch slow queries that the app-side histograms only hint at.
- Also enable
pg_stat_statements - If you run postgres_exporter, its dashboards complement this one
- log_min_duration_statement = 500
- Also enable
Examples
curl -H "Authorization: Bearer $METRICS_TOKEN" \
https://eventier.internal.example.com/api/metrics
# Dedicated worker (in-cluster):
curl -H "Authorization: Bearer $METRICS_TOKEN" \
http://<release>-worker-metrics:9464/metricsscrape_configs:
- job_name: eventier-web
metrics_path: /api/metrics
authorization:
type: Bearer
credentials: <METRICS_TOKEN>
static_configs:
- targets: ["eventier.internal.example.com"]
scheme: https
- job_name: eventier-worker
metrics_path: /metrics
authorization:
type: Bearer
credentials: <METRICS_TOKEN>
static_configs:
- targets: ["<release>-worker-metrics:9464"]# Is the worker keeping up? (the alert signal)
max(eventier_outbox_oldest_pending_age_seconds)
# Tickets created vs resolved, per hour
sum(increase(eventier_requests_created_total[1h]))
sum(increase(eventier_requests_resolved_total[1h]))
# AI token spend by feature, per day
sum by (feature) (increase(eventier_ai_tokens_total[24h]))
# DB query latency p95 (ms)
histogram_quantile(0.95,
sum by (le) (rate(prisma_client_queries_duration_histogram_ms_bucket[5m])))Metric reference
Gauges marked DB-derived are the same on every target. Counters are per process and reset on restart; query them with rate() or increase(). Prisma's built-in series (prisma_*) are in every scrape.
| Metric | Type | Meaning |
|---|---|---|
| eventier_outbox_oldest_pending_age_seconds | Gauge (DB-derived) | Age of the oldest outbox event waiting to be claimed; the signal that the worker has fallen behind |
| eventier_worker_last_heartbeat_age_seconds | Gauge (DB-derived) | Seconds since the worker's last heartbeat; +Inf if none ever recorded |
| eventier_outbox_events{status} | Gauge (DB-derived) | Outbox events by status: pending · processing · failed (dead letters) |
| eventier_outbox_processed_total{event_type,result} | Counter | Outbox events processed by this process, by result (succeeded or failed) |
| eventier_requests_created_total | Counter | Tickets created, counted at the point that every intake path passes through |
| eventier_requests_resolved_total | Counter | Tickets transitioned to Resolved |
| eventier_notifications_created_total{type} | Counter | In-app notification rows written |
| eventier_emails_sent_total{result} | Counter | Outbound email handoffs: sent · error |
| eventier_ai_completions_total{feature,outcome} | Counter | AI calls by feature and outcome (success · disabled · rate_limited · error) |
| eventier_ai_tokens_total{feature} | Counter | AI tokens consumed (prompt + completion) |
| process_resident_memory_bytes · nodejs_heap_* | Gauge (per-process) | Memory of each web/worker pod |
| prisma_pool_connections_* · prisma_client_queries_* | Gauge/Counter/Histogram | Prisma connection pool and query latency, per process |
Good to know
- A missing or wrong token gets 404, never 401, so scanners cannot tell the endpoint exists. If a scrape shows 404, check the token first.
- You can rotate
METRICS_TOKENat any time. Update the secret and restart; Prometheus uses the new credential on its next scrape. - If you alert on one metric only, choose
eventier_outbox_oldest_pending_age_seconds. Nearly every failure (a worker crash, database contention, a hung downstream service) shows up there first.