> ## Documentation Index > Fetch the complete documentation index at: https://omniflow-vpn.mintlify.site/llms.txt > Use this file to discover all available pages before exploring further. # Reliability and operations > Service-level indicators, dashboards and alerts, what's bounded and reconciled, graceful degradation, and capacity guidance for a single-server installation This page is the hub for running Omniflow reliably: what to measure, what to alert on, what already degrades gracefully when a dependency fails, and how to size the host it runs on. Each section links to the page that documents its subject in full rather than repeating it. ## Service-level indicators Defined per surface in [Health, metrics, and logs](https://omniflow-vpn.mintlify.site/operations/observability.md#service-level-indicators), grounded only in metrics confirmed to actually be written. Notifications has no indicator today — there is no delivery metric to build one from, stated there rather than invented here. ## Dashboards and alerts Omniflow bundles no Prometheus or Grafana of its own, the same stance it takes on a reverse proxy: two maintained, optional artifacts live under `deploy/observability/` for you to import into your own monitoring stack. * **`deploy/observability/prometheus-alerts.yml`** — every bullet from [Alerts worth having](https://omniflow-vpn.mintlify.site/operations/observability.md#alerts-worth-having) as a real `alert:`/`expr:`/`for:` rule. The `for:` durations are sensible defaults derived from `APP_MAINTENANCE_FAILURE_STREAK` and `APP_MAINTENANCE_PROBE_SECONDS`'s own defaults — tune them to match your installation's actual configuration if you've changed those. * **`deploy/observability/grafana-dashboard.json`** — HTTP request rate, error rate, and latency; fulfillment job outcomes; payment settlement classifications; webhook signature failures; dependency health; and maintenance-mode state. Every panel is built from a metric [Health, metrics, and logs](https://omniflow-vpn.mintlify.site/operations/observability.md#what-is-written) confirms is actually written — nothing here queries a dead series. Import both with `promtool` / Grafana's dashboard import, pointed at whatever scrapes `/metrics` on `api` and `worker` in your deployment. ## External side effects: retries, dead letters, reconciliation | Side effect | Retry / backoff | Dead-letter visibility | Reconciliation | | - | - | - | - | | Remnawave fulfillment | Up to 12 attempts, exponential backoff | `GET /v1/admin/jobs?state=dead_letter`, panel Jobs screen | Scheduled `reconcile` operations compare local vs. Remnawave state and record drift — [Jobs and queues](https://omniflow-vpn.mintlify.site/operations/jobs-and-queues.md#fulfillment-jobs) | | Digital-goods delivery | Same River-backed retry policy as fulfillment | Same dead-letter view | Idempotent by the `goods_deliveries` row; a gateway with no idempotency key parks for operator review rather than retrying blind — [Jobs and queues](https://omniflow-vpn.mintlify.site/operations/jobs-and-queues.md#digital-goods-delivery-jobs) | | Payment webhooks | Provider signature verified, deduplicated by event ID | `/admin` → System → Webhooks | A 1-minute reconciler polls up to 100 in-flight intents so a missed or delayed webhook still settles — [Payment providers](https://omniflow-vpn.mintlify.site/integrations/payments.md#reconciliation-when-a-webhook-never-arrives) | | Telegram sends | Retry, flood-wait handling, and delivery-failure classification (shipped in v0.4) | Not a Prometheus series — `omniflow_telegram_sends_total` is declared but never incremented | None beyond per-send retry; there is no queue to reconcile | | Transactional outbox (`order.paid`, `wallet.credited`, `addon.purchased`) | N/A | `GET /v1/admin/outbox`, panel Outbox tab | **No consumer exists today** — [Jobs and queues](https://omniflow-vpn.mintlify.site/operations/jobs-and-queues.md#the-transactional-outbox). This is the honest exception: nothing currently reads these events, so there is no external side effect for retry/DLQ/reconciliation to bound yet. Rows are swept by the retention worker regardless of whether anything consumed them, so this is a missing integration point, not an unbounded-growth risk | ## Graceful degradation | Dependency | What happens when it's unavailable | | - | - | | PostgreSQL | Watched by automatic [maintenance detection](https://omniflow-vpn.mintlify.site/operations/maintenance-mode.md#what-it-actually-watches); after a failure streak, new purchases pause while settlement, refunds, and read paths keep working | | Remnawave | Same maintenance detection; a revoked or expired panel token counts as unavailable too, since provisioning would fail anyway | | Valkey | Same maintenance detection. Until this failure streak is reached, rate limits and the single-use replay guard fail closed on their own — see [Rate limiting](https://omniflow-vpn.mintlify.site/operations/security.md#rate-limiting) | | Telegram | Retry, flood-wait handling, and bot-blocked/deactivated classification (v0.4) — but no circuit breaker and no maintenance-triggering probe. An extended Telegram outage is visible through delivery failures in the bot's own logs, not through a dashboard series | | Payment providers | Signature-verified webhook intake plus the 1-minute reconciler cover a provider that's slow or drops a webhook — but there's no circuit breaker or provider-down detection beyond that. A fully down provider just accumulates failed checkout attempts against it, visible in `omniflow_payments_total` | | MCP / AI tool servers | The one surface with an actual circuit breaker — [MCP connections](https://omniflow-vpn.mintlify.site/operations/mcp.md#when-a-server-misbehaves) — scoped to that surface only, not reused elsewhere | ## Capacity guidance Omniflow ships as a single Docker Compose stack — see [Deployment](https://omniflow-vpn.mintlify.site/operations/deployment.md#supported-topology) — and job concurrency (the `critical` queue's 16 workers) is compiled into `internal/jobs/client.go` today, not an environment-configurable knob. Capacity planning on a single host is therefore about giving PostgreSQL and Valkey enough resources, not about tuning queue concurrency. | Tier | Rough scale | Guidance | | - | - | - | | Small | A few hundred customers, low purchase volume | 2 vCPU / 4 GB RAM is enough for PostgreSQL, Valkey, and all three Go processes together. Disk headroom matters more than CPU here: budget for backup retention (`APP_BACKUP_RETENTION_DAYS`) on top of the database's own growth | | Medium | Low thousands of customers, regular campaigns | 4 vCPU / 8 GB RAM, with PostgreSQL given most of the memory (`shared_buffers` tuned to roughly a quarter of host RAM is a reasonable starting point — see PostgreSQL's own tuning guidance). Watch `omniflow_job_duration_seconds{kind="fulfillment"}` — if it climbs, the `critical` queue's fixed 16-worker concurrency is the ceiling for one worker, and another worker replica raises it | | Large | Tens of thousands of customers, frequent bulk actions and campaigns | 8 vCPU / 16 GB RAM as a starting point, and treat PostgreSQL connection limits as the thing to watch first — the pools of every `api`, `bot` and `worker` replica (`APP_DATABASE_MAX_CONNS`, see [Deployment](https://omniflow-vpn.mintlify.site/operations/deployment.md#database-connections)) must fit inside PostgreSQL's own `max_connections` together. Consider moving PostgreSQL to dedicated, faster disk before anything else, since it is the only durable store and every write goes through it | These are starting points for capacity planning, not a load-tested ceiling — no throughput benchmark ships with the repository today. Watch the SLIs above under your own traffic and adjust from there. ## Related > **[Health, metrics, and logs](https://omniflow-vpn.mintlify.site/operations/observability.md)** > > The probes, metrics, and alerts these SLIs and dashboards are built from. > **[Jobs and queues](https://omniflow-vpn.mintlify.site/operations/jobs-and-queues.md)** > > Retry policy, the dead-letter view, and the transactional outbox in full. > **[Maintenance mode](https://omniflow-vpn.mintlify.site/operations/maintenance-mode.md)** > > What automatic detection watches and what it does when a window opens. > **[Backup, restore, and disaster recovery](https://omniflow-vpn.mintlify.site/operations/backup-restore.md)** > > The restore drill, and what a host-level disaster needs beyond it. This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.