Skip to content

Documentation

Monitoring & metrics

Watch EvenTier with Prometheus: a token-protected metrics endpoint on the web and worker processes, queue health gauges, throughput counters and a Grafana dashboard.

Where
GET /api/metrics (web) · :9464/metrics (worker) · deploy/grafana/
Permission
METRICS_TOKEN bearer token · admin of the host, cluster or server
Updated
October 2026

Before you start

EvenTier publishes metrics in Prometheus text format from two places: the web process at GET /api/metrics, and the dedicated worker on port 9464 at /metrics.

Both are off, answering 404, until you set METRICS_TOKEN. After that, every scrape needs Authorization: Bearer <token>.

Gauges read from the database (queue depth, backlog age, worker heartbeat) are the same on every replica, so any one target gives the full picture.

Per-process series (memory, connection pool, domain counters) differ per pod, so scrape both the web and the worker. The Helm chart sets this up for you.

On Docker Compose and the server install, the worker runs inside the app process, so GET /api/metrics is the only target.

Counters work the standard Prometheus way: they are per process and reset on restart or deploy.

Query counters with rate() or increase(); raw values are not meaningful across restarts.

The app does not expose HTTP request metrics (rate, errors and duration per route). Take them from your ingress controller.

nginx-ingress and Traefik export per-service RED (rate, errors, duration) metrics, and the ingress also sees content delivery network (CDN) and TLS (HTTPS encryption) behavior that the app cannot.

Step by step

  1. 01

    Enable the endpoint

    Create a token and set it as METRICS_TOKEN to switch the endpoints on. Check it with the curl example below.

    • Helm: set monitoring.enabled: true and monitoring.metricsToken in your values instead
    • The chart injects the token into both processes and opens the worker's metrics port
    Generate and set the tokenbash
    openssl rand -hex 32
    
    # then, in your environment file
    METRICS_TOKEN=<the generated token>
  2. 02

    Point Prometheus at it

    Tell Prometheus where to scrape. With the Prometheus Operator, it is one Helm value.

    • One ServiceMonitor covers the web Service (/api/metrics) and the worker metrics Service (:9464/metrics)
    • Plain Prometheus: use the scrape config example below
    Helm valuesyaml
    monitoring:
      serviceMonitor:
        enabled: true
  3. 03

    Turn on the alerts

    Switch on the included alert rules, and point probes and uptime checks at the health endpoints.

    • Critical: scrape failing, database unreachable, worker heartbeat stale (at 180s)
    • Warning: uncaught server errors, outbox backlog age (at 60s), dead-lettered events, failed scheduled jobs
    • Thresholds are chart values
    • /api/health/live: the process answers
    • /api/health/ready: the database and applied migrations; with ?require=worker, also a fresh worker heartbeat
    Helm valuesyaml
    monitoring:
      prometheusRule:
        enabled: true
  4. 04

    Import the dashboard

    Load the included Grafana dashboard and choose your Prometheus data source.

    • Panels: queue health, ticket throughput, notification and email volume, AI token spend, memory, pool and database latency
    • Copy panels from it into your own dashboards
    1. Grafana → Dashboards → Import
    2. File: deploy/grafana/eventier-overview.json
  5. 05

    Add database-side visibility

    Turn on slow query logging in PostgreSQL to catch slow queries that the app-side histograms only hint at.

    • Also enable pg_stat_statements
    • If you run postgres_exporter, its dashboards complement this one
    1. log_min_duration_statement = 500

Examples

Verify the endpointbash
curl -H "Authorization: Bearer $METRICS_TOKEN" \
  https://eventier.internal.example.com/api/metrics

# Dedicated worker (in-cluster):
curl -H "Authorization: Bearer $METRICS_TOKEN" \
  http://<release>-worker-metrics:9464/metrics
Plain Prometheus scrape configyaml
scrape_configs:
  - job_name: eventier-web
    metrics_path: /api/metrics
    authorization:
      type: Bearer
      credentials: <METRICS_TOKEN>
    static_configs:
      - targets: ["eventier.internal.example.com"]
    scheme: https
  - job_name: eventier-worker
    metrics_path: /metrics
    authorization:
      type: Bearer
      credentials: <METRICS_TOKEN>
    static_configs:
      - targets: ["<release>-worker-metrics:9464"]
Common PromQL queriespromql
# Is the worker keeping up? (the alert signal)
max(eventier_outbox_oldest_pending_age_seconds)

# Tickets created vs resolved, per hour
sum(increase(eventier_requests_created_total[1h]))
sum(increase(eventier_requests_resolved_total[1h]))

# AI token spend by feature, per day
sum by (feature) (increase(eventier_ai_tokens_total[24h]))

# DB query latency p95 (ms)
histogram_quantile(0.95,
  sum by (le) (rate(prisma_client_queries_duration_histogram_ms_bucket[5m])))

Metric reference

Gauges marked DB-derived are the same on every target. Counters are per process and reset on restart; query them with rate() or increase(). Prisma's built-in series (prisma_*) are in every scrape.

MetricTypeMeaning
eventier_outbox_oldest_pending_age_secondsGauge (DB-derived)Age of the oldest outbox event waiting to be claimed; the signal that the worker has fallen behind
eventier_worker_last_heartbeat_age_secondsGauge (DB-derived)Seconds since the worker's last heartbeat; +Inf if none ever recorded
eventier_outbox_events{status}Gauge (DB-derived)Outbox events by status: pending · processing · failed (dead letters)
eventier_outbox_processed_total{event_type,result}CounterOutbox events processed by this process, by result (succeeded or failed)
eventier_requests_created_totalCounterTickets created, counted at the point that every intake path passes through
eventier_requests_resolved_totalCounterTickets transitioned to Resolved
eventier_notifications_created_total{type}CounterIn-app notification rows written
eventier_emails_sent_total{result}CounterOutbound email handoffs: sent · error
eventier_ai_completions_total{feature,outcome}CounterAI calls by feature and outcome (success · disabled · rate_limited · error)
eventier_ai_tokens_total{feature}CounterAI tokens consumed (prompt + completion)
process_resident_memory_bytes · nodejs_heap_*Gauge (per-process)Memory of each web/worker pod
prisma_pool_connections_* · prisma_client_queries_*Gauge/Counter/HistogramPrisma connection pool and query latency, per process

Good to know

  • A missing or wrong token gets 404, never 401, so scanners cannot tell the endpoint exists. If a scrape shows 404, check the token first.
  • You can rotate METRICS_TOKEN at any time. Update the secret and restart; Prometheus uses the new credential on its next scrape.
  • If you alert on one metric only, choose eventier_outbox_oldest_pending_age_seconds. Nearly every failure (a worker crash, database contention, a hung downstream service) shows up there first.