> ## Documentation Index
> Fetch the complete documentation index at: https://omniflow-vpn.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# Backup, restore, and disaster recovery

> Encrypted PostgreSQL backups, verified restores, key handling, and the upgrade and rollback procedure

PostgreSQL is the only durable store Omniflow has. Valkey holds cache, rate limits,
locks, and disposable state, so losing it costs nothing but a warm cache. Everything on
this page therefore concerns PostgreSQL.

## What a backup is

A backup is `pg_dump --format=custom --no-owner --no-privileges --compress=6` output,
encrypted with AES-256-GCM and written to `APP_BACKUP_DIR`. Omniflow shells out to
`pg_dump` and `pg_restore` rather than reimplementing them, so a backup is always
restorable with standard PostgreSQL tooling even if Omniflow itself will not start.

Encryption is framed rather than a single sealed blob, so a multi-gigabyte dump never has
to fit in memory:

```text theme={null}
"OFBK1"          5 bytes
base nonce      12 bytes, random per file
frames                    repeated: uint32 length, AES-256-GCM ciphertext
```

Frames are 1 MiB. Each frame's nonce is the base nonce XOR its index, and each frame
authenticates its own index and whether it ends the file. A reordered, duplicated,
dropped, or truncated frame fails to open rather than producing a plausible-looking
partial dump — which is the only outcome worse than having no backup at all. A file that
is not an Omniflow backup fails on the magic bytes instead.

Files are named `omniflow-<kind>-<UTC timestamp>.dump.enc`, for example
`omniflow-scheduled-20260811T030000Z.dump.enc`. `kind` is `scheduled` or `manual` and
nothing else — the column has a check constraint, so a third kind needs a migration. The
name is unique and the file is created exclusively, so a collision fails rather than
overwrites.

## Who takes one

| Action | Runs in | Trigger |
| - | - | - |
| Scheduled backup + prune | `worker` | `APP_BACKUP_INTERVAL_HOURS`, plus one immediately at start |
| Manual backup | `bot` | `/ops` → **Back up now** |
| Restore | `bot` | `/ops` → **Restore latest…** |
| Listing | `api` | `GET /v1/admin/backups`, `GET /v1/panel/settings/backups` |

> [!NOTE]
>
> The operator panel is read-only about backups. There is no panel route and no
> `/v1/admin` route that takes or restores one; the only two write paths in the product
> are the worker's schedule and the bot's `/ops` surface. That is deliberate — a restore
> is destructive, and it lives behind an explicitly enumerated list of Telegram accounts
> rather than behind a permission that six roles could be granted.

## Enabling backups

```bash theme={null}
APP_BACKUP_ENABLED=true
APP_BACKUP_ENCRYPTION_KEY=$(openssl rand -base64 32)
APP_BACKUP_DIR=/var/lib/omniflow/backups
APP_BACKUP_INTERVAL_HOURS=24
APP_BACKUP_RETENTION_DAYS=14
```

| Variable | Default | Purpose |
| - | - | - |
| `APP_BACKUP_ENABLED` | `false` | Turns scheduled backups on in the worker and the backup actions on in the bot. |
| `APP_BACKUP_ENCRYPTION_KEY` | *(none)* | Base64 32 bytes. Required whenever backups are enabled; otherwise the process exits with `APP_BACKUP_ENCRYPTION_KEY is required when APP_BACKUP_ENABLED is true`. |
| `APP_BACKUP_DIR` | `/var/lib/omniflow/backups` | Created with mode `0700`; files are written `0600`. With several worker replicas it must be a shared volume: a scheduled backup lands on whichever replica runs the pass. |
| `APP_BACKUP_INTERVAL_HOURS` | `24` | Scheduled interval. A zero or unparseable value falls back to 24 hours. |
| `APP_BACKUP_RETENTION_DAYS` | `14` | Stamped onto each backup row as `retain_until`. |
| `APP_PG_DUMP_PATH` | `pg_dump` | Override when the client tools are not on `PATH`. |
| `APP_PG_RESTORE_PATH` | `pg_restore` | Same, for restores. |

> [!WARNING]
>
> Store `APP_BACKUP_ENCRYPTION_KEY` somewhere the database backup itself does not reach —
> a password manager, a secrets manager, or a sealed envelope. A key that only exists
> inside the thing it decrypts is not a key.
>
> It is a **different** key from `APP_DATA_ENCRYPTION_KEY`. Both must be preserved: the
> backup key to read the dump, the data key to read the encrypted contact values, TOTP
> secrets, and provider credentials inside it. A restore with the wrong data key produces
> a database that loads and whose sealed values will not open.

The bot and worker images ship the matching PostgreSQL 18 client tools. In the compose
stack both mount a shared `backup-data` volume at `/var/lib/omniflow/backups`, so
rebuilding a container never discards backups.

The database password never reaches the child process's argv, where any local user could
read it from the process list. The DSN is decomposed into `PGHOST`, `PGPORT`,
`PGDATABASE`, `PGUSER`, `PGPASSWORD`, and `PGSSLMODE` in the child's environment instead.

## Verification

A backup is recorded as `completed` only after it has been read back: the file is
decrypted end to end and its SHA-256 digest compared with the digest computed while
writing. A backup that fails either check is deleted and recorded as `failed` with a
classified error code.

This runs on every backup, not only on demand. An unverified backup is a guess.

| Error code | Cause |
| - | - |
| `command_exit_N` | `pg_dump` or `pg_restore` exited non-zero |
| `verification_failed` | The file could not be read back, or its digest did not match |
| `file_missing` | The backup directory is not writable or not mounted |
| `cancelled` | The process shut down mid-backup; `pg_dump` is cancelled with its context |
| `backup_failed` | Anything else the service could not classify |

## Operating from Telegram

Telegram carries operator notifications and backup/restore actions only. Every other
administrative control belongs in the web panel.

```bash theme={null}
APP_OPERATOR_TELEGRAM_IDS=111111111,222222222
```

An account not in that list gets **no response at all** to `/ops` — not a refusal, which
would confirm the command exists. The command is also ignored outside a private chat with
the bot.

`/ops` shows the five most recent backups with file name, status (with the error code when
one is recorded), size, and UTC timestamp, plus:

* **💾 Back up now** — takes a manual backup immediately. If one is already running it
  answers *"A backup is already running"* rather than starting a second.
* **♻️ Restore latest…** — offered only when a completed backup exists. A two-step
  confirmation: the first tap only renders the warning, and the second carries the backup
  identifier the operator saw.
* **🔄 Refresh** — re-renders the list.

The panel shows sizes and timestamps only: no connection details, no file paths, and no
encryption material.

Both actions write append-only audit events naming the Telegram account, the action, the
backup, and the reason: `backup.created` for a manual backup, then
`backup.restore_requested` and `backup.restored` around a restore. The restore's reason is
recorded as *"restore requested from Telegram by operator \<id>"*.

## Restoring

A restore verifies the backup first, then runs
`pg_restore --clean --if-exists --no-owner --no-privileges --single-transaction` with the
decrypted stream on stdin. It is deliberately blunt, which is why it insists the caller
has confirmed the destructive intent and recorded who asked.

> [!WARNING]
>
> A restore replaces current database contents. Every change made since the backup was taken
> is lost, including orders placed, payments settled, and support messages received in that
> window.

It refuses rather than half-running in these cases:

| Refusal | Cause |
| - | - |
| `backups are not configured` | `APP_BACKUP_ENABLED` is false or the key is not exactly 32 bytes |
| `a restore requires an operator and a reason` | Both are mandatory and both are stored on the `backup_restores` row |
| `backup is not restorable` | The backup's status is not `completed` |
| digest or frame failure | The file failed re-verification; nothing is run against the database |

Before restoring:

1. Stop `api`, `bot`, and `worker`. A live worker will keep provisioning against state
   that is about to be replaced.
2. Take a fresh backup of the current database, even if you believe it is broken. It is
   the only way back from a mistaken restore.
3. Confirm the target backup's status is `completed`.

After restoring:

1. Start `worker` first and watch `/readyz` and the fulfillment topic.
2. Start `api` and `bot`.
3. Reconcile with Remnawave. Provisioning that happened after the backup point is still
   live in the panel but absent from Omniflow. Scheduled drift detection reports the
   mismatches within fifteen minutes, and each one is resolvable from the drift view.
4. Re-check pending payments. A payment that settled after the backup point is picked up
   by the reconciliation poller or by a provider webhook retry. Both are idempotent, so
   nothing is charged or credited twice.

The restore runs in the **bot** process when it is driven from `/ops`, so its log is where
a failure is described even though the on-screen message mentions the worker.

### Restoring without the bot

The worker binary doubles as a decryption filter, because disaster recovery must not
depend on the rest of the stack being able to start. It needs nothing but the binary, the
key, and the file:

```bash theme={null}
# Decrypt with the key from your secret store.
APP_BACKUP_ENCRYPTION_KEY=... \
  docker compose run --rm -T worker --decrypt-backup \
  < omniflow-scheduled-20260811T030000Z.dump.enc > backup.dump

# Restore with the client tools that ship in the postgres image.
docker compose exec -T postgres pg_restore \
  --clean --if-exists --no-owner --no-privileges --single-transaction \
  --dbname "postgres://omniflow:omniflow@localhost:5432/omniflow" < backup.dump
```

`--decrypt-backup` must be the **first** argument, and the key is read from the
environment before any configuration is loaded — no database, no Valkey, and no Remnawave
credential is required. It fails with `APP_BACKUP_ENCRYPTION_KEY is required to decrypt a
backup` or `APP_BACKUP_ENCRYPTION_KEY must be base64-encoded 32 bytes`, and on a wrong key
or a damaged file it fails loudly rather than emitting partial output.

> [!NOTE]
>
> Atlas records applied migrations in its own `atlas_schema_revisions` schema rather than
> in `public`. A dump taken by Omniflow carries the whole database, so an ordinary restore
> brings that history with it. If you rebuild a database by other means, carry that schema
> across too, or the `migrate` service will try to re-apply history and the diagnostics
> bundle will report zero applied migrations.

## Retention and pruning

Each backup row is stamped with `retain_until` at creation, `APP_BACKUP_RETENTION_DAYS`
into the future. After every scheduled run the worker prunes up to 100 expired backups:
the file is deleted and the row is marked `pruned`. The row is kept, so the history of
which backups existed stays auditable after the files are gone.

A file that cannot be removed is logged and skipped, and its row keeps its status, so the
next pass tries again rather than losing track of it.

## Reviewing history

| Surface | Auth | Returns |
| - | - | - |
| `GET /v1/panel/settings/backups?limit=` | `settings.read` | `{backups, restores}` together |
| `GET /v1/admin/backups?limit=` | `APP_OPERATOR_TOKEN` bearer | Backups only; limit 1–100, default 20 |
| `/admin/settings/operations` → Backups card | `settings.read` | The same two lists, rendered |
| `/ops` in Telegram | Listed operator accounts | The five most recent |

A backup nobody has ever restored is a backup nobody knows works, which is why the
restore history sits beside the backup list rather than on a separate screen.

## Upgrading Omniflow

Migrations are forward-only and applied by the `migrate` service before any application
process starts.

1. Take a backup and confirm it is `completed`.
2. Pull the new images. Pin an explicit tag; floating tags make a rollback ambiguous.
3. Run `docker compose up -d`. The `migrate` service runs `atlas migrate apply`, and every
   other service waits for it to complete successfully.
4. Watch `/readyz` on each process.

Applying migrations twice is a no-op: Atlas records what has already run. CI proves every
supported baseline — the empty database plus each prefix of the migration history —
upgrades to head, and that re-applying head changes nothing.

> [!NOTE]
>
> The compose `migrate` service uses a hard-coded connection string and does not read
> `APP_DATABASE_URL`. If you changed the database credentials, change them in
> `compose.yaml` as well or the migrator will connect to the wrong database.

## Rolling back

Omniflow does not ship down-migrations. A schema rollback is a restore:

1. Stop `api`, `bot`, and `worker`.
2. Restore the backup taken before the upgrade.
3. Pin the previous image tags and start the stack.
4. Reconcile with Remnawave as described above.

This is why step one of an upgrade is a backup, and why the backup is verified before it
is trusted.

## Upgrading PostgreSQL

Omniflow supports PostgreSQL 18. To move between major versions:

1. Stop `api`, `bot`, and `worker`; leave `postgres` running.
2. Take a backup and verify it.
3. Stop `postgres` and start the new major version against an **empty** data directory.
4. Restore the backup.
5. Start the application processes.

Do not point a new PostgreSQL major version at an old data directory. `pg_upgrade` is the
alternative if you prefer an in-place upgrade, but the dump-and-restore path is the one
this backup format is designed for. Remember that `pg_dump` and `pg_restore` must match
the server major version; the shipped images pin the PostgreSQL 18 client tools for
exactly this reason.

## Disaster-recovery drill

Run this quarterly. An untested backup is an assumption.

```bash theme={null}
sh tools/restore-drill.sh
```

The script is the procedure rather than a description of it. It takes the most recent
`completed` backup, decrypts it through `worker --decrypt-backup`, restores it into a
throwaway PostgreSQL instance on the stack's own network, compares row counts, opens a
sealed value with `APP_DATA_ENCRYPTION_KEY`, and removes the instance again. It never
writes to the live database, and it exits non-zero when the drill fails, so a scheduled run
means something.

Pass `COMPOSE` when the stack uses override files, and `TABLES` to compare a different set:

```bash theme={null}
COMPOSE="docker compose -f compose.yaml -f compose.prod.yaml" sh tools/restore-drill.sh
```

What it reports, and how to read it:

| Verdict | Meaning |
| - | - |
| `same` | The restored table matches the live one. |
| `live moved on` | The live table has more rows than the backup. Expected on a running installation: the database kept working after the backup was taken. For an exact comparison, run the drill immediately after taking a backup. |
| `MORE THAN LIVE` | The copy has rows the live database does not. This cannot happen by drift; the backup is not of this database. Fails the drill. |
| `MISSING` | The table is absent from the copy. The schema did not come across. Fails the drill. |

The sealed-value step is the one most often skipped and the one that most often fails: a
database restored without its data-encryption key loads perfectly and has unreadable
secrets. The script opens a sealed customer-OIDC client secret in a throwaway container, so
it needs nothing installed on the host. An installation with no OIDC provider configured has
no sealed value for it to check, and it says so rather than passing quietly — check a TOTP
secret or a provider credential by hand before trusting that backup.

Record the date and the outcome.

## Beyond the database: full disaster recovery

Everything above answers "PostgreSQL is lost or corrupted." A host-level disaster is a
wider question, and the answer to most of it is: nothing here needs a special procedure,
because PostgreSQL is the only durable store — but two things outside it still need a plan.

### Losing the whole host

A `pg_dump` restore brings back every row. It does not bring back what only ever lived
outside the database:

* **`.env`**, or whatever holds it in your secrets manager.
* **`APP_DATA_ENCRYPTION_KEY`.** This is the one to get right before anything else. Every
  sealed value — TOTP secrets, OIDC client secrets, provider credentials, customer contact
  values — is permanently unopenable without it, and there is no re-encryption path. See
  [Security](https://omniflow-vpn.mintlify.site/operations/security.md#secret-sealing-and-the-write-only-credential-design).
* **`APP_BACKUP_ENCRYPTION_KEY`**, stored separately from the backups it decrypts — a backup
  that carries both is a single artefact that opens itself.
* **`APP_OPERATOR_TOKEN`**, since nothing issues a new one; a lost token means re-bootstrapping
  the bearer surface.
* **The reverse-proxy configuration and TLS material** in front of the stack, which lives in
  your own estate rather than in Omniflow. See [Reverse proxy](https://omniflow-vpn.mintlify.site/operations/reverse-proxy.md).
* **DNS**, pointed at whatever host replaces the lost one.

None of these are backed up by the backup format on this page, because none of them are rows
in PostgreSQL. Keep them wherever you keep secrets that must outlive the thing they protect —
never inside the database backup itself.

### Losing Valkey

This needs no restore at all. Valkey holds cache entries, rate-limit counters, locks, and
single-use replay tokens — nothing durable — so a lost Valkey instance is a fresh one started
empty. See [Deployment](https://omniflow-vpn.mintlify.site/operations/deployment.md#why-valkey-is-configured-with-no-persistence).

Automatic [maintenance mode](https://omniflow-vpn.mintlify.site/operations/maintenance-mode.md#what-it-actually-watches) detection
watches Valkey along with PostgreSQL and Remnawave, so an outage pauses new purchases after the
same failure streak as any other dependency, rather than customers hitting rate-limit and
replay-guard failures inconsistently mid-checkout.

### RPO and RTO

Stated from what is actually configurable, not a promised figure this page cannot keep:

* **Recovery point objective** is governed by `APP_BACKUP_INTERVAL_HOURS` — the default is 24
  hours between scheduled backups, so up to that much recent activity can be lost in the worst
  case. Shrink the interval for a tighter RPO; a manual backup from `/ops` before a risky
  operation shrinks it to zero for that moment.
* **Recovery time objective** is dominated by restore time, which scales with database size,
  plus the reconciliation window after the stack comes back — Remnawave drift resolves within
  fifteen minutes of the scheduled reconcile pass, and in-flight payments settle on the next
  reconciler tick. There is no fixed number to promise here; size your own drill's duration as
  the baseline for your installation.

## Related

> **[Deployment](https://omniflow-vpn.mintlify.site/operations/deployment.md)**
>
> Compose topology, volumes, and the upgrade path these procedures assume.

> **[Troubleshooting](https://omniflow-vpn.mintlify.site/operations/troubleshooting.md)**
>
> What each backup error code means and where to read the log.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.