maintenance_state and it is deliberately narrow: it blocks new purchases and holds outbound provisioning, and touches nothing else. Settlement, refunds, and order history keep working throughout, because the failure it exists to prevent is charging a customer for access the installation cannot provision — not freezing an installation that is halfway through processing payments it has already accepted.
It can be moved by an operator, from the panel or the bearer API, and by automatic dependency detection running in the API process.
What it blocks, and what it deliberately does not
Exactly two actions are refused while a window is open. A third, browsing, is explicitly allowed so a customer can still reach the notice, their orders, and support.
The guard sits on order creation, in all five paths that make one, so a new purchase can never reach a payment provider. It is not a second check inside the payment layer, which is what keeps an already-created order settleable.
An order that was paid before the window opened still settles, still credits the wallet or creates its entitlement, and still qualifies a referral. Only the outbound Remnawave call waits.
A held fulfillment job is snoozed, not failed
When the worker picks up a fulfillment job during a window, it parks the operation asretrying with last_error_code = maintenance_active, decrements the attempt count back, and snoozes the job for one minute. The hold therefore costs no retry budget however long the window lasts, and the operation keeps its record, its correlation ID and its idempotency key. It runs as soon as the window closes. See Jobs and queues.
What a customer sees
In the bot, the purchase surface is replaced by a maintenance screen at its entry point — before the customer picks anything, not after they tap pay. It shows your notice in their locale, or the built-in localized copy when the notice is empty, plus the expected return time when one is set, with buttons to retry and to open support. A lookup failure never blocks a customer: maintenance is an operator signal, not a fail-closed control, so if the state cannot be read the purchase path stays open. In the customer web panel, the dashboard shows a service notice above everything else, because an incident explains the state of every subscription below it. It uses the Russian notice for a Russian-locale customer when one was supplied and the English one otherwise. A checkout or shop purchase attempted anyway answers503 with the code maintenance_active, which the app renders as “purchases are paused” rather than a generic error. See Customer web panel.
Read-only screens are never blocked. A customer in a maintenance window can still see their subscription, their orders and their wallet, which is exactly what they will want to check.
Operating it from the panel
1
Open System → Health
Sign in at
/admin and open System (permission system.read), Health tab. It refreshes every 30 seconds. The maintenance card shows an on/off badge and, while on, a source badge.2
Read the source before you touch anything
manual means somebody switched it on and only somebody can switch it off. Any other source means detection opened it and detection will close it on recovery. Showing which is in force is what stops an operator waiting for a recovery that will never come — and when an automatic window is in force, the card explains that clearing it by hand would re-enable purchases while the dependency is still down.3
Fill in the reason and both notices
With
system.write you get a Reason field, which is always required, and Notice RU and Notice EN, both required when engaging. The toggle stays disabled until all three are non-empty. A customer who reads only one language and gets a blank screen is worse served than one who gets a notice they can read.4
Flip the switch
The panel sends
PUT /v1/panel/overview/maintenance with the reason in an X-Operator-Reason header. Clearing it needs a reason alone.source: "manual", even if a dependency is currently down. A panel change must never masquerade as automatic detection, because the two are cleared by different things.
Every transition writes a maintenance_events row with the action, source, reason, actor type and actor ID, and an audit event — panel.maintenance.activated or panel.maintenance.cleared, category system. That is what lets an operator explain the outage afterwards. See Admin panel.
expectedReturnAt is accepted by the API and rendered to customers, but the panel has no control for it. Set it over the bearer API if you want a return time on the notice.Operating it from the bearer API
The/v1/admin surface is authenticated with APP_OPERATOR_TOKEN and exists for scripts and runbooks. Its rules are different from the panel’s, and the difference matters.
"active": false.
GET /v1/admin/maintenance returns the current state, source, reason, both notices, the activation time and the expected return time.
Automatic detection
Detection runs in the API process only, because that is the process already probing every dependency for its readiness endpoint. It reuses the same probe registry, so the maintenance switch and/readyz can never disagree about whether PostgreSQL is up.
- A window opens after
FAILURE_STREAKconsecutive unhealthy samples, so one timeout never closes an installation. - It clears after
RECOVERY_STREAKconsecutive healthy samples, so a flapping dependency does not reopen purchases the moment it answers once. - A window an operator opened is never cleared automatically. Recovery only clears a window detection itself opened.
What it actually watches
Detection watchespostgres, remnawave, and valkey.
A Valkey outage already breaks rate limiting and the single-use replay guard fail-closed — see Security — so a customer mid-checkout would otherwise hit inconsistent failures rather than a clear maintenance notice. Watching it closes that gap: after the same failure streak as any other dependency, a Valkey outage pauses new purchases the same way a PostgreSQL or Remnawave outage does.
When more than one watched dependency is failing at once, the source recorded is database first, then remnawave, then valkey. The database is named first because it is the one that explains the others.
The Remnawave probe is an authenticated call, so a revoked or expired panel token counts as an unhealthy dependency and will, after the streak, pause purchases. That is usually the behaviour you want — provisioning would fail anyway — but it means a token rotation done carelessly closes the shop. See Remnawave.
Watching it
Each sample setsomniflow_maintenance_mode to 1 or 0 and omniflow_dependency_up per probe, on the API process only. Alert on the gauge staying 1 for longer than a planned window: detection opening a window you did not plan is the signal, and it staying open is the incident. See Health, metrics, and logs.
Every transition also logs at warn level:
/admin reads the maintenance_events rows, so a window that opened and closed overnight is still visible in the morning.
Every transition, automatic or manual, also posts to the operator group’s incident topic. The controller runs in the api process and the group belongs to the bot, so the api queues the notice in operator_notifications and the bot delivers it; with no operator group configured the notice simply waits. See Operator notifications.
Refusals and edge cases
- Panel: a missing reason, or activation without both notices, is a validation error and the switch does not move.
- Bearer: a missing
operatorIdis400 missing_operator; a rejected state change is422 maintenance_rejected. - A purchase attempted during a window fails with
maintenance_active—503on the customer web API, the maintenance screen in the bot. - The
maintenance_staterow is created on first read, so a fresh installation reports “inactive” rather than “missing”. - The table enforces
CHECK (NOT active OR activated_at IS NOT NULL): an active window always knows when it started. - Clearing a window neither cancels, refunds nor expires anything. Held fulfillment operations simply become due again.