> ## Documentation Index > Fetch the complete documentation index at: https://omniflow-vpn.mintlify.site/llms.txt > Use this file to discover all available pages before exploring further. # Troubleshooting > Real symptoms, what actually causes them, and where to look in this installation Start with `/readyz` on the processes that serve it, and with the `omniflow_dependency_up` gauge. Most incidents are one dependency, and the rest of this page assumes you have already ruled that out. Two things about readiness are worth knowing before you rely on it: * The **bot** serves no HTTP at all unless `APP_BOT_METRICS_ADDR` is set, and that address is ignored when it equals `APP_BOT_HTTP_ADDR` — publishing probes and metrics on the internet-facing webhook host is refused. Out of the box the bot has no probe endpoint, so "the bot is not answering `/readyz`" usually means it was never listening. * An **API** started without `APP_DATABASE_URL` registers no dependency probes at all and answers `200 {"status":"ready","checks":[]}`. That is a green light for a process that can serve only the probes and `/metrics`. The worker registers `postgres` only. The API registers `postgres`, `valkey`, and `remnawave` when commerce is configured. See [Observability](https://omniflow-vpn.mintlify.site/operations/observability.md). ## A customer paid and has no access This is the failure Omniflow is designed hardest against, and it is almost always provisioning rather than payment. 1. Find the order. `GET /v1/admin/orders/{orderID}` shows its state and paid amount. 2. `paid` means the money is recorded and the entitlement exists. `fulfilled` means Remnawave confirmed it. 3. `GET /v1/admin/jobs?state=dead_letter` shows provisioning runs that will not retry on their own. 4. `GET /v1/admin/fulfillment/drifts` shows entitlements whose remote state does not match. To re-drive a stalled run, `POST /v1/admin/jobs/{jobID}/retry`, or from the panel `/admin/system` → Jobs. The underlying fulfillment operation carries its own idempotency key, so a retry cannot double-apply something that already succeeded. The payment is never at risk here. Settlement and the durable job commit in one transaction, so a paid order always has a job waiting for it. ## The worker exits at start-up complaining about a missing table The `migrate` service applies the committed Atlas migrations in `database/migrations`, and those include the queue's internal tables (`river_job`, `river_leader`, `river_queue`, `river_notification`, `river_migration`). If the worker fails at `client.Start` reporting one of those relations missing, the migrations have not run to the end against the database the worker is pointed at: check the `migrate` service's log and compare `APP_DATABASE_URL` across the services before assuming a connectivity problem. Until those tables exist, the worker cannot start, and a checkout that inserts a fulfillment job inside its settlement transaction fails with it. See [Jobs and queues](https://omniflow-vpn.mintlify.site/operations/jobs-and-queues.md#the-queue-schema-ships-with-the-migrations). ## A payment succeeded at the provider but the order is pending The webhook has not arrived, or it was rejected. 1. `GET /v1/admin/webhooks` lists recent intake with `signatureValid` and `status`. 2. `signatureValid: false` means the signature did not verify. Check the provider credentials, and confirm the reverse proxy is not modifying the request body — a re-encoded body invalidates every signature, because the signature is computed over the exact bytes the provider sent. 3. `status: failed` with `payment_not_found` means the webhook referenced a payment intent this installation does not have. Usually a webhook from a different environment pointed at this one. 4. If no webhook arrived at all, `POST /v1/admin/payments/{paymentID}/reconcile` polls the provider directly. The worker's `payment_reconcile` pass also re-polls up to 100 in-flight intents every minute on its own. Both paths are idempotent. A webhook arriving after a manual reconciliation is a no-op. > [!NOTE] > > The panel's own **Reconcile** and **Refund** buttons on `/admin/finance` record an > audit event for who asked and why; they do not themselves re-poll the provider or move > money. Use the `/v1/admin` routes above when you need the provider contacted. ## Provider credentials look right in the panel and payments still fail The payment adapters the API, the bot and the worker call are built from environment variables at start-up — `APP_TELEGRAM_TOKEN`, `APP_CRYPTOBOT_TOKEN`, `APP_YOOKASSA_SHOP_ID`, `APP_YOOKASSA_SECRET`. The credentials stored on the panel's provider cards are not what the running process uses. Change the variable and restart. See [Payments](https://omniflow-vpn.mintlify.site/integrations/payments.md) and [Installation settings](https://omniflow-vpn.mintlify.site/operations/installation-settings.md). ## A top-up credited the wrong amount Look at the `topup` topic or the payment events for the order. A top-up credits exactly what the provider settled: * `overpayment` — more arrived than was requested, and all of it was credited. * `underpayment` — less arrived, and that is what was credited. Both leave the order `paid`. This is intended: a wallet owes nothing beyond the money itself. If the discrepancy needs correcting, use a wallet adjustment with a reason, which is recorded as a compensating ledger entry rather than a rewrite. ## A customer is refused a top-up they think is within the limits Top-up limits come from `APP_WALLET_TOPUP_*` on the API, bot and worker, not from the top-up card in the panel. Saving new figures on that card changes what the card shows and not what the store enforces. Check the environment values and the rolling-window sum: the refusal is usually `APP_WALLET_TOPUP_WINDOW_LIMIT_MINOR` over `APP_WALLET_TOPUP_WINDOW_HOURS` rather than the single-transaction ceiling. ## A saved cart is not buying itself A settled top-up tries the cart straight away; every other credit — a referral reward, an operator correction — releases it through the worker's `commerce_maintenance` pass, once a minute. Without a running worker those carts wait, and a failing pass is logged as `periodic task failed` with the task name. Otherwise, check `carts.last_failure`: | Value | Meaning | | - | - | | `cart_insufficient_balance` | Working as intended; the balance does not cover the re-validated total | | `cart_plan_unavailable` | The plan or an add-on was archived, retired, or lost its price | | `cart_price_changed` | The order was refused at the current price | | `cart_auto_purchase_off` | The customer turned automatic purchase off | | `cart_maintenance` | The installation is in maintenance | Remember the cart is re-priced every attempt. A cart saved before a price rise will not buy at the old price, by design. ## Duplicate charges They should be impossible, and the invariants say why: * `orders (user_id, idempotency_key)` is unique, so one checkout resolves to one order. * `ledger_transactions.idempotency_key` is unique and `(reference_type, reference_id, type)` is unique, so one settlement credits once. * `entitlement_addons (order_id, addon_version_id)` is unique, so one add-on applies once. * `fulfillment_operations.idempotency_key` is unique, so one operation runs once. If you believe you are seeing a duplicate, compare `ledger_entries` for the customer. Two entries under one transaction are the two sides of one movement, not two charges. ## The operator group is silent 1. Confirm `APP_OPERATOR_CHAT_ID` is set and the bot was restarted. With it unset the bot logs `operator notifications are disabled because no group is configured` and never drains the queue — notices still accumulate as `pending` rows and are delivered once a group is configured. 2. Check `operator_topics` for a row with `status = 'failed'` and read `last_error_code`. See [Operator notifications](https://omniflow-vpn.mintlify.site/operations/operator-notifications.md). 3. Check `operator_notifications` for rows with `status = 'suppressed'` and `error_code = 'volume_cap'`, which means the cap is doing its job. 4. If only *some* topics are quiet, that may be correct. Purchases, renewals, top-ups, refunds, fulfillment failures, and admin security all have producers. The 🚨 Incidents and 💾 Backups topics are created but nothing currently queues into them. ## Sign-in says "Sign-in failed. Try again." with correct credentials That message is what the panel shows when the request did not come back with a session, and there are two reasons that have nothing to do with the password. Both look identical on screen, and both are configuration rather than credentials. Open the browser's network tab: it tells you which one you have. **The call returned 404.** The request reached the web application instead of the API. Both panels call the API on their own origin — the browser client has no base URL and there is no CORS — so something has to serve `/v1` on the web host. Behind a reverse proxy, that is the proxy's job, and the examples in `deploy/proxies` do it; a hand-written configuration that routes only the application will produce exactly this. Without a proxy, the web service needs `API_INTERNAL_URL`, and answers `502` with a problem document naming it when it is unset. See [Reverse proxy](https://omniflow-vpn.mintlify.site/operations/reverse-proxy.md). **The call returned 200 and the next request is anonymous.** The browser discarded the cookie. Over plain HTTP that means `Secure` is still on: a browser does not return a `Secure` cookie over an unencrypted connection, so set `APP_ADMIN_COOKIE_SECURE=false` and `APP_CUSTOMER_COOKIE_SECURE=false` for a local stack, and leave them on everywhere with TLS. Check the cookie's name in the response as well — it is `__Host-omniflow_admin` with `Secure` and `omniflow_admin` without, and a `__Host-` name arriving without `Secure` is rejected outright rather than merely weakened. ## An operator cannot sign in and the page says "rate limited" Sign-in rate limiting is backed by Valkey and **fails closed**: when Valkey is unreachable the limiter answers "not allowed" rather than waving requests through, so a Valkey outage blocks operator sign-in entirely. Check the `valkey` entry in the API's `/readyz` output first. A genuine limit answers `429 rate_limited`; a locked account answers `429 account_locked` after five consecutive failures, with the wait doubling up to one hour. Note that a Valkey outage does **not** open maintenance mode. Automatic detection watches `postgres` and `remnawave` only, so `valkey` appears in `/readyz` and never triggers a window on its own. ## The bot receives no updates In long-polling mode, check the bot's logs for Telegram API errors and confirm outbound HTTPS works. In webhook mode: 1. Confirm the proxy routes `POST /telegram/webhook` to the bot's listener (`APP_BOT_HTTP_ADDR`, default `:8082`) and does not rewrite the body. The endpoint caps bodies at 1 MiB. 2. Confirm `APP_TELEGRAM_WEBHOOK_SECRET` matches what was registered. An update without the secret token header is discarded before parsing, which looks identical to no traffic. The secret must match `^[A-Za-z0-9_-]{32,256}$` or the bot refuses to start. 3. `APP_TELEGRAM_WEBHOOK_URL` must be HTTPS, or the bot refuses to start. 4. Ask Telegram what it thinks: `getWebhookInfo` reports the URL it is delivering to and the last error it saw. Switching back to long polling is a supported fallback: clear `APP_TELEGRAM_WEBHOOK_URL` and restart. ## `/ops` does nothing in Telegram The command is always registered and answers **nothing at all** to an account that may not use it, so the command's existence is not confirmed to a customer. Two conditions must both hold before you get any reply: 1. `APP_OPERATOR_TELEGRAM_IDS` contains the Telegram account's numeric ID, and the bot has been restarted since. An empty list disables the surface entirely, which is the default. 2. The message is sent in a **private** chat with the bot. In a group it is ignored. If you do get a reply and it reads *"Backups are disabled"*, the surface is working and `APP_BACKUP_ENABLED` or `APP_BACKUP_ENCRYPTION_KEY` is missing. A malformed entry in `APP_OPERATOR_TELEGRAM_IDS` is a fatal configuration error, so check the bot's start-up log if the process exited immediately instead. ## Purchases are refused with a maintenance notice Check `GET /v1/admin/maintenance` or the panel's `/admin/system` → Health tab. `source` says whether it was manual (`manual`) or automatic (`database`, `remnawave`). An automatic window clears itself once the dependency has been healthy for `APP_MAINTENANCE_RECOVERY_STREAK` consecutive samples; a manual one never clears itself. A fulfillment job that meets an active window is snoozed for a minute rather than failed, and its attempt count is given back, so a maintenance hold costs no retry budget. See [Maintenance mode](https://omniflow-vpn.mintlify.site/operations/maintenance-mode.md). ## The pending outbox count only goes up It always will. `outbox_events` rows are written on `order.paid`, `wallet.credited`, and `addon.purchased`, and nothing in the product marks them published. `GET /v1/admin/outbox` and the panel's outbox tab therefore report a number that grows for the life of the installation, and `APP_RETENTION_OUTBOX_DAYS` — which only deletes rows that have been published — removes nothing. It is not a backlog, it is not a stalled dispatcher, and no action clears it. Do not alert on it. The declared `omniflow_outbox_pending_events` gauge is never written either, so there is nothing to scrape. ## Backups are failing The failure reason is on the backup row and in the log of the process that took it, as a classified code: | Code | Cause | | - | - | | `command_exit_N` | `pg_dump` or `pg_restore` exited non-zero, usually credentials or a version mismatch | | `verification_failed` | The written file could not be read back or its digest did not match; check disk health and free space | | `file_missing` | The backup directory is not writable or not mounted | | `cancelled` | The process shut down mid-backup | | `backup_failed` | Anything else the service could not classify | Scheduled backups run in the **worker**; `/ops` backups and restores run in the **bot**, so look in that process's log even though the on-screen message mentions the worker. `pg_dump` must match the server major version. The bot and worker images ship the PostgreSQL 18 client tools for exactly this reason; if you run them elsewhere, point `APP_PG_DUMP_PATH` and `APP_PG_RESTORE_PATH` at matching binaries. See [Backup and restore](https://omniflow-vpn.mintlify.site/operations/backup-restore.md). ## A support attachment uploaded from the web is gone `APP_SUPPORT_ATTACHMENT_DIR` (default `/var/lib/omniflow/support-attachments`) is read by both the API, which writes attachments, and the worker, which reclaims their files during retention. Two failure shapes follow: * The compose stack mounts no volume at that path on either service, so an attachment written by the API lives in the container's writable layer and is lost when the container is rebuilt. Mount a volume on the API if browser-uploaded attachments must survive a redeploy. * If the worker's value differs from the API's, retention deletes the rows and leaves the bytes on disk forever. Set the same path on both. Attachments in Telegram-only installations are references to files Telegram holds and are unaffected. ## Running more than one replica Any number of `api` and `worker` replicas is supported: the periodic passes — order and cart maintenance, payment reconciliation, renewals, sweeps, campaigns, retention, backups — are enqueued by River's elected leader and run once per interval however many workers are up. The `bot` is the exception: Telegram delivers updates to one consumer, and long polling refuses a second `getUpdates`. Run exactly one bot. More replicas also means more database connections. Each one opens a pool of `APP_DATABASE_MAX_CONNS` (api 20, bot 10, worker 40 by default), and the sum must stay below PostgreSQL's `max_connections`. A worker given fewer than 32 refuses to start and says how many it needs. See [Deployment](https://omniflow-vpn.mintlify.site/operations/deployment.md#database-connections). ## A process is down and nothing brought it back The bundled `compose.yaml` declares no `restart:` policy on `postgres`, `valkey`, `api`, `bot`, `worker`, or `web` — only the Traefik overlay sets one. A crashed process stays down until somebody notices. Add a restart policy for an unattended deployment. See [Deployment](https://omniflow-vpn.mintlify.site/operations/deployment.md). ## A customer has two subscriptions and expects one Concurrent subscriptions are off by default. If a customer has more than one, either the switch is on, or they were adopted from separate Remnawave users at import. See [Subscriptions](https://omniflow-vpn.mintlify.site/commerce/subscriptions.md) for how to reduce a customer to one without orphaning access. ## Traces are empty Omniflow opens spans across inbound HTTP, Telegram updates, fulfillment jobs, PostgreSQL queries, and outbound HTTP, but it never installs a tracer provider and reads no `OTEL_*` variable. With no provider installed the global one is a no-op and no span is exported, so setting collector variables alone changes nothing. Metrics and logs are the instrumentation you can rely on today. This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.