Operator notifications

The Telegram group, the topics the bot creates itself, what those messages deliberately omit, and the caps that keep them readable

An operator supplies one thing: a Telegram group. The bot creates and binds every topic inside it, keeps them bound, and re-creates one that was deleted.

The stream is deliberately a signal rather than a data store. A notice names the event, the amount class, and identifiers an operator can look up through the admin API — never customer text, a subscription link, a token, or a payment payload. A Telegram group is a place messages are forwarded from, screenshotted, and kept for years, so the design question was never “what would be convenient here” but “what is safe to leave in a chat indefinitely”.

Setup

1. Create the group

   Create a Telegram group and enable Topics in its settings. Topics are a supergroup feature; a plain group cannot hold them.

2. Add the bot

   Add the bot and grant it Manage Topics and Send Messages.

3. Point the bot at it

   APP_OPERATOR_CHAT_ID=-1001234567890
   

   A supergroup ID is negative and begins with -100. Forwarding a message from the group to a bot such as @userinfobot is the usual way to read it.

4. Restart the bot

   On start the notifier binds every topic, then drains the queue every ten seconds. The start-up log says which of the two happened: operator notifications are disabled because no group is configured, or nothing at all if binding succeeded.

Only the bot process delivers notices. The api and worker processes write them into the queue, so an installation running no bot still accumulates the record of what happened — it simply has nobody to tell.

The topics

Topic	Kind	Carries	Written by
💳 Purchases	purchase	Settled subscription orders	Order settlement
♻️ Renewals	renewal	Settled extensions and renewals	Order settlement
👛 Wallet top-ups	topup	Credited top-ups, including over- and underpayments	Top-up settlement
↩️ Refunds	refund	Recorded refunds	Refund recording
⚠️ Fulfillment failures	fulfillment_failure	Provisioning runs that stopped being a blip, digital-goods delivery failures, and add-ons that settled with no live entitlement	Fulfillment worker, goods worker, settlement
🚨 Incidents	incident	Payments that need a look: a duplicate, under- or overpayment, currency mismatch, or wallet shortfall that left money at the provider with the order open, and a payment that settled an order the customer had cancelled; and every maintenance transition, automatic or manual	Payment settlement, the api’s maintenance controller
💾 Backups	backup	Backup outcomes — completed with its size, or failed with the error code — and restore outcomes	The process that took or restored the backup
🔐 Admin security	security	Operator sign-ins from a new address, password changes and resets, two-factor removal, and owner grants	Operator identity service
💬 Support tickets	support	Every new support ticket, from the bot or the web panel, as its ticket and queue identifiers — never the subject or the message, which are customer text	A database trigger on support_tickets
📣 Campaign previews	campaign_test	One copy of a campaign’s message, asked for before the audience is committed to	Campaign test send
🔔 Notice previews	notice_test	One rendered copy of a transactional notice an operator is rewording	Notice test send

The last two carry something the others deliberately do not: a rendered message body. Everything else here is a set of allowlisted, non-personal fields, and a preview is by definition the copy itself. Both say plainly that nobody else received them — a preview mistaken for a real send is somebody reaching for the pause button. See Campaigns and Message wording.

The incidents topic carries payment anomalies and nothing else today. Each notice names the order, the provider, the operation, the classification, the currency and amount, and the order’s state at the time, and is deduplicated per payment intent and classification — a replayed webhook adds nothing. What each classification means and how it is resolved is in Orders and payments. Maintenance is detected in the api process, which has no Telegram client, so the api writes each transition into the same operator_notifications queue the bot drains; one transition is one message, deduplicated per direction and minute.

Backup and restore outcomes are queued the same way by the process that ran them, keyed on the backup and the outcome, so a retried job reports once. The backups table stays the record: a notice that cannot be queued is logged and never fails the backup. See Maintenance mode and Backup and restore.

The anomaly evaluator writes notices under a ninth kind, anomaly, which the database accepts but for which the bot has no named topic. Those notices do not reach the group. Review anomalies at /admin/risk instead — see Risk controls.

Configuration

Variable	Default	Purpose
APP_OPERATOR_CHAT_ID	0	The group. 0 disables delivery entirely.
APP_OPERATOR_NOTIFICATION_CAP	30	Messages sent per kind per window before the rest are suppressed. 0 removes the cap.
APP_OPERATOR_NOTIFICATION_WINDOW_MINUTES	5	The rolling window the cap counts over. A zero or unparseable value falls back to five minutes.

All three are read once at start-up. The operator_group section on /admin/settings/integrations stores the same four numbers and changes nothing — see Installation settings.

When binding fails

A topic that cannot be created is recorded as failed on operator_topics with a classified reason rather than silently doing nothing. The raw Telegram message is dropped because it can echo request content.

last_error_code	Fix
missing_manage_topics_permission	Grant the bot Manage Topics
forum_topics_disabled	Enable Topics in the group settings
chat_not_found	Wrong chat ID, or the group was upgraded to a supergroup and its ID changed
bot_not_in_group	Re-add the bot
telegram_error	Anything else; read the bot log for context

Fix the cause and restart the bot, or wait for the next notification for that kind: delivery re-attempts the binding before it sends.

Changing APP_OPERATOR_CHAT_ID rebinds everything. Rows whose stored chat no longer matches have their topic identifier cleared and their status reset to pending, so fresh topics are created in the new group rather than messages being sent into a group you have left.

A topic deleted in Telegram is re-created on the next message to it, once, and the message is then retried.

Delivery

The notifier drains up to 50 pending notices every ten seconds, in creation order, inside one transaction with FOR UPDATE SKIP LOCKED so two bots cannot deliver the same row. Each notice ends in exactly one terminal state:

Status	Meaning
pending	Queued, not yet attempted. Rows stay here indefinitely while no group is configured.
sent	Delivered, with sent_at recorded.
suppressed	The volume cap was reached; error_code = volume_cap.
failed	Telegram refused, with a classified error_code.

A failed notice is not retried. The row and its payload remain in the database, so the record of the event survives even when the message did not, but the fix for a persistent failure is to correct the group and let the next event through rather than to expect a replay.

Volume control

APP_OPERATOR_NOTIFICATION_CAP=30
APP_OPERATOR_NOTIFICATION_WINDOW_MINUTES=5

Before each send the notifier counts how many notices of that kind were sent inside the window. At or above the cap, the notice is completed as suppressed with reason volume_cap and a warning is logged with the kind and the number suppressed in that pass.

The cap is per kind, so a Remnawave outage producing thousands of fulfillment failures cannot bury the purchase stream, and suppressed notices do not themselves count towards the cap. Nothing is lost: the rows stay in the database with their payloads.

Set the cap to 0 to send everything.

A group configured for the first time after a long run drains a backlog. The queue keeps every notice written while no group existed, and the drain is bounded to 50 every ten seconds with the volume cap applied on top — so most of a large backlog is delivered as suppressed rather than as messages. That is the intended outcome; read the history in the panel rather than expecting it in the chat.

Deduplication

Every notice carries a dedupe key: the order for a purchase, the refund for a refund, the fulfillment operation for a failure, the event and account for a security notice. operator_notifications (kind, dedupe_key) is unique and the insert is ON CONFLICT DO NOTHING, so a retried webhook, a second reconciliation pass, and a job that fails repeatedly all produce one message.

The notice is queued inside the same transaction as the event it describes. A notice therefore exists exactly when the thing it reports is durable — never for an event that rolled back, and never missing for one that committed. A queue failure is logged and never fails the transaction that caused it, because a settled payment must not be undone by a chat message.

What a notice contains

The renderer prints the topic name, then only fields from a fixed allowlist, then the timestamp. Anything else a payload happens to hold is dropped rather than filtered, so a field added to a payload in future cannot leak by omission:

orderId · subscriptionId · entitlementId · provider · operation · currency
amountMinor · creditedMinor · classification · reason · source · errorCode
fileName · sizeBytes · status · event · adminUserId · method

A purchase notice therefore looks like this:

💳 Purchases
orderId: 018f0000-0000-7000-8000-000000000001
subscriptionId: 018f0000-0000-7000-8000-000000000002
operation: purchase
currency: RUB
amountMinor: 149900
at: 2026-08-11T10:15:00Z

There is no customer name, no Telegram account, no subscription link, no provider token, and no payment payload. Identifiers are included precisely because they are useless on their own: an operator looks them up through the panel or the admin API, where authorization and audit apply.

A security notice names the event, the operator account, and the method, and nothing else. An operator’s address, a session token, and a client IP stay out of the group chat; that detail lives in the audit trail, behind the permissions that govern it. See Admin panel.

Fulfillment failure notices

A failing provisioning run does not notify immediately. Ordinary backoff is expected to recover, so an operator is told only after three failed attempts, or immediately on a permanent failure. The dedupe key is the operation, so one struggling entitlement produces one message however many times it retries.

The notice carries the entitlement, the operation, the current status, and a classified error code such as remnawave_http_502 or remnawave_unavailable. It never carries the panel’s response body.

Two related cases land in the same topic:

• A digital-goods delivery that the provider could not complete.
• An add-on that settled with no live entitlement — the order keeps its paid state so the money is never silently lost, and the notice, together with the addon.no_active_entitlement audit event, tells an operator exactly what to refund or re-apply.

Related

Telegram

BotFather, webhook versus long polling, and the operator group.

Troubleshooting

What to check when the group has gone quiet.