Troubleshooting

Real symptoms, what actually causes them, and where to look in this installation

Start with /readyz on the processes that serve it, and with the omniflow_dependency_up gauge. Most incidents are one dependency, and the rest of this page assumes you have already ruled that out.

Two things about readiness are worth knowing before you rely on it:

• The bot serves no HTTP at all unless APP_BOT_METRICS_ADDR is set, and that address is ignored when it equals APP_BOT_HTTP_ADDR — publishing probes and metrics on the internet-facing webhook host is refused. Out of the box the bot has no probe endpoint, so “the bot is not answering /readyz” usually means it was never listening.
• An API started without APP_DATABASE_URL registers no dependency probes at all and answers 200 {"status":"ready","checks":[]}. That is a green light for a process that can serve only the probes and /metrics.

The worker registers postgres only. The API registers postgres, valkey, and remnawave when commerce is configured. See Observability.

A customer paid and has no access

This is the failure Omniflow is designed hardest against, and it is almost always provisioning rather than payment.

1. Find the order. GET /v1/admin/orders/{orderID} shows its state and paid amount.
2. paid means the money is recorded and the entitlement exists. fulfilled means Remnawave confirmed it.
3. GET /v1/admin/jobs?state=dead_letter shows provisioning runs that will not retry on their own.
4. GET /v1/admin/fulfillment/drifts shows entitlements whose remote state does not match.

To re-drive a stalled run, POST /v1/admin/jobs/{jobID}/retry, or from the panel /admin/system → Jobs. The underlying fulfillment operation carries its own idempotency key, so a retry cannot double-apply something that already succeeded.

The payment is never at risk here. Settlement and the durable job commit in one transaction, so a paid order always has a job waiting for it.

The worker exits at start-up complaining about a missing table

The migrate service applies the committed Atlas migrations in database/migrations, and those include the queue’s internal tables (river_job, river_leader, river_queue, river_notification, river_migration). If the worker fails at client.Start reporting one of those relations missing, the migrations have not run to the end against the database the worker is pointed at: check the migrate service’s log and compare APP_DATABASE_URL across the services before assuming a connectivity problem.

Until those tables exist, the worker cannot start, and a checkout that inserts a fulfillment job inside its settlement transaction fails with it. See Jobs and queues.

A payment succeeded at the provider but the order is pending

The webhook has not arrived, or it was rejected.

1. GET /v1/admin/webhooks lists recent intake with signatureValid and status.
2. signatureValid: false means the signature did not verify. Check the provider credentials, and confirm the reverse proxy is not modifying the request body — a re-encoded body invalidates every signature, because the signature is computed over the exact bytes the provider sent.
3. status: failed with payment_not_found means the webhook referenced a payment intent this installation does not have. Usually a webhook from a different environment pointed at this one.
4. If no webhook arrived at all, POST /v1/admin/payments/{paymentID}/reconcile polls the provider directly. The worker’s payment_reconcile pass also re-polls up to 100 in-flight intents every minute on its own.

Both paths are idempotent. A webhook arriving after a manual reconciliation is a no-op.

The panel’s own Reconcile and Refund buttons on /admin/finance record an audit event for who asked and why; they do not themselves re-poll the provider or move money. Use the /v1/admin routes above when you need the provider contacted.

Provider credentials look right in the panel and payments still fail

The payment adapters the API, the bot and the worker call are built from environment variables at start-up — APP_TELEGRAM_TOKEN, APP_CRYPTOBOT_TOKEN, APP_YOOKASSA_SHOP_ID, APP_YOOKASSA_SECRET. The credentials stored on the panel’s provider cards are not what the running process uses. Change the variable and restart. See Payments and Installation settings.

A top-up credited the wrong amount

Look at the topup topic or the payment events for the order. A top-up credits exactly what the provider settled:

• overpayment — more arrived than was requested, and all of it was credited.
• underpayment — less arrived, and that is what was credited.

Both leave the order paid. This is intended: a wallet owes nothing beyond the money itself. If the discrepancy needs correcting, use a wallet adjustment with a reason, which is recorded as a compensating ledger entry rather than a rewrite.

A customer is refused a top-up they think is within the limits

Top-up limits come from APP_WALLET_TOPUP_* on the API, bot and worker, not from the top-up card in the panel. Saving new figures on that card changes what the card shows and not what the store enforces. Check the environment values and the rolling-window sum: the refusal is usually APP_WALLET_TOPUP_WINDOW_LIMIT_MINOR over APP_WALLET_TOPUP_WINDOW_HOURS rather than the single-transaction ceiling.

A saved cart is not buying itself

A settled top-up tries the cart straight away; every other credit — a referral reward, an operator correction — releases it through the worker’s commerce_maintenance pass, once a minute. Without a running worker those carts wait, and a failing pass is logged as periodic task failed with the task name. Otherwise, check carts.last_failure:

Value	Meaning
cart_insufficient_balance	Working as intended; the balance does not cover the re-validated total
cart_plan_unavailable	The plan or an add-on was archived, retired, or lost its price
cart_price_changed	The order was refused at the current price
cart_auto_purchase_off	The customer turned automatic purchase off
cart_maintenance	The installation is in maintenance

Remember the cart is re-priced every attempt. A cart saved before a price rise will not buy at the old price, by design.

Duplicate charges

They should be impossible, and the invariants say why:

• orders (user_id, idempotency_key) is unique, so one checkout resolves to one order.
• ledger_transactions.idempotency_key is unique and (reference_type, reference_id, type) is unique, so one settlement credits once.
• entitlement_addons (order_id, addon_version_id) is unique, so one add-on applies once.
• fulfillment_operations.idempotency_key is unique, so one operation runs once.

If you believe you are seeing a duplicate, compare ledger_entries for the customer. Two entries under one transaction are the two sides of one movement, not two charges.

The operator group is silent

1. Confirm APP_OPERATOR_CHAT_ID is set and the bot was restarted. With it unset the bot logs operator notifications are disabled because no group is configured and never drains the queue — notices still accumulate as pending rows and are delivered once a group is configured.
2. Check operator_topics for a row with status = 'failed' and read last_error_code. See Operator notifications.
3. Check operator_notifications for rows with status = 'suppressed' and error_code = 'volume_cap', which means the cap is doing its job.
4. If only some topics are quiet, that may be correct. Purchases, renewals, top-ups, refunds, fulfillment failures, and admin security all have producers. The 🚨 Incidents and 💾 Backups topics are created but nothing currently queues into them.

Sign-in says “Sign-in failed. Try again.” with correct credentials

That message is what the panel shows when the request did not come back with a session, and there are two reasons that have nothing to do with the password. Both look identical on screen, and both are configuration rather than credentials. Open the browser’s network tab: it tells you which one you have.

The call returned 404. The request reached the web application instead of the API. Both panels call the API on their own origin — the browser client has no base URL and there is no CORS — so something has to serve /v1 on the web host. Behind a reverse proxy, that is the proxy’s job, and the examples in deploy/proxies do it; a hand-written configuration that routes only the application will produce exactly this. Without a proxy, the web service needs API_INTERNAL_URL, and answers 502 with a problem document naming it when it is unset. See Reverse proxy.

The call returned 200 and the next request is anonymous. The browser discarded the cookie. Over plain HTTP that means Secure is still on: a browser does not return a Secure cookie over an unencrypted connection, so set APP_ADMIN_COOKIE_SECURE=false and APP_CUSTOMER_COOKIE_SECURE=false for a local stack, and leave them on everywhere with TLS. Check the cookie’s name in the response as well — it is __Host-omniflow_admin with Secure and omniflow_admin without, and a __Host- name arriving without Secure is rejected outright rather than merely weakened.

An operator cannot sign in and the page says “rate limited”

Sign-in rate limiting is backed by Valkey and fails closed: when Valkey is unreachable the limiter answers “not allowed” rather than waving requests through, so a Valkey outage blocks operator sign-in entirely. Check the valkey entry in the API’s /readyz output first.

A genuine limit answers 429 rate_limited; a locked account answers 429 account_locked after five consecutive failures, with the wait doubling up to one hour.

Note that a Valkey outage does not open maintenance mode. Automatic detection watches postgres and remnawave only, so valkey appears in /readyz and never triggers a window on its own.

The bot receives no updates

In long-polling mode, check the bot’s logs for Telegram API errors and confirm outbound HTTPS works.

In webhook mode:

1. Confirm the proxy routes POST /telegram/webhook to the bot’s listener (APP_BOT_HTTP_ADDR, default :8082) and does not rewrite the body. The endpoint caps bodies at 1 MiB.
2. Confirm APP_TELEGRAM_WEBHOOK_SECRET matches what was registered. An update without the secret token header is discarded before parsing, which looks identical to no traffic. The secret must match ^[A-Za-z0-9_-]{32,256}$ or the bot refuses to start.
3. APP_TELEGRAM_WEBHOOK_URL must be HTTPS, or the bot refuses to start.
4. Ask Telegram what it thinks: getWebhookInfo reports the URL it is delivering to and the last error it saw.

Switching back to long polling is a supported fallback: clear APP_TELEGRAM_WEBHOOK_URL and restart.

/ops does nothing in Telegram

The command is always registered and answers nothing at all to an account that may not use it, so the command’s existence is not confirmed to a customer. Two conditions must both hold before you get any reply:

1. APP_OPERATOR_TELEGRAM_IDS contains the Telegram account’s numeric ID, and the bot has been restarted since. An empty list disables the surface entirely, which is the default.
2. The message is sent in a private chat with the bot. In a group it is ignored.

If you do get a reply and it reads “Backups are disabled”, the surface is working and APP_BACKUP_ENABLED or APP_BACKUP_ENCRYPTION_KEY is missing. A malformed entry in APP_OPERATOR_TELEGRAM_IDS is a fatal configuration error, so check the bot’s start-up log if the process exited immediately instead.

Purchases are refused with a maintenance notice

Check GET /v1/admin/maintenance or the panel’s /admin/system → Health tab. source says whether it was manual (manual) or automatic (database, remnawave). An automatic window clears itself once the dependency has been healthy for APP_MAINTENANCE_RECOVERY_STREAK consecutive samples; a manual one never clears itself.

A fulfillment job that meets an active window is snoozed for a minute rather than failed, and its attempt count is given back, so a maintenance hold costs no retry budget. See Maintenance mode.

The pending outbox count only goes up

It always will. outbox_events rows are written on order.paid, wallet.credited, and addon.purchased, and nothing in the product marks them published. GET /v1/admin/outbox and the panel’s outbox tab therefore report a number that grows for the life of the installation, and APP_RETENTION_OUTBOX_DAYS — which only deletes rows that have been published — removes nothing.

It is not a backlog, it is not a stalled dispatcher, and no action clears it. Do not alert on it. The declared omniflow_outbox_pending_events gauge is never written either, so there is nothing to scrape.

Backups are failing

The failure reason is on the backup row and in the log of the process that took it, as a classified code:

Code	Cause
command_exit_N	pg_dump or pg_restore exited non-zero, usually credentials or a version mismatch
verification_failed	The written file could not be read back or its digest did not match; check disk health and free space
file_missing	The backup directory is not writable or not mounted
cancelled	The process shut down mid-backup
backup_failed	Anything else the service could not classify

Scheduled backups run in the worker; /ops backups and restores run in the bot, so look in that process’s log even though the on-screen message mentions the worker.

pg_dump must match the server major version. The bot and worker images ship the PostgreSQL 18 client tools for exactly this reason; if you run them elsewhere, point APP_PG_DUMP_PATH and APP_PG_RESTORE_PATH at matching binaries. See Backup and restore.

A support attachment uploaded from the web is gone

APP_SUPPORT_ATTACHMENT_DIR (default /var/lib/omniflow/support-attachments) is read by both the API, which writes attachments, and the worker, which reclaims their files during retention. Two failure shapes follow:

• The compose stack mounts no volume at that path on either service, so an attachment written by the API lives in the container’s writable layer and is lost when the container is rebuilt. Mount a volume on the API if browser-uploaded attachments must survive a redeploy.
• If the worker’s value differs from the API’s, retention deletes the rows and leaves the bytes on disk forever. Set the same path on both.

Attachments in Telegram-only installations are references to files Telegram holds and are unaffected.

Running more than one replica

Any number of api and worker replicas is supported: the periodic passes — order and cart maintenance, payment reconciliation, renewals, sweeps, campaigns, retention, backups — are enqueued by River’s elected leader and run once per interval however many workers are up. The bot is the exception: Telegram delivers updates to one consumer, and long polling refuses a second getUpdates. Run exactly one bot.

More replicas also means more database connections. Each one opens a pool of APP_DATABASE_MAX_CONNS (api 20, bot 10, worker 40 by default), and the sum must stay below PostgreSQL’s max_connections. A worker given fewer than 32 refuses to start and says how many it needs. See Deployment.

A process is down and nothing brought it back

The bundled compose.yaml declares no restart: policy on postgres, valkey, api, bot, worker, or web — only the Traefik overlay sets one. A crashed process stays down until somebody notices. Add a restart policy for an unattended deployment. See Deployment.

A customer has two subscriptions and expects one

Concurrent subscriptions are off by default. If a customer has more than one, either the switch is on, or they were adopted from separate Remnawave users at import. See Subscriptions for how to reduce a customer to one without orphaning access.

Traces are empty

Omniflow opens spans across inbound HTTP, Telegram updates, fulfillment jobs, PostgreSQL queries, and outbound HTTP, but it never installs a tracer provider and reads no OTEL_* variable. With no provider installed the global one is a no-op and no span is exported, so setting collector variables alone changes nothing. Metrics and logs are the instrumentation you can rely on today.