> ## Documentation Index
> Fetch the complete documentation index at: https://omniflow-vpn.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# AI providers and local models

> Registering a model provider, pointing one at a local model, routing each feature to its own model and budget, and reading the state the screen shows you

Everything AI in Omniflow starts with a provider an owner has registered and
approved. This page is how that registry works: the three adapters, the fields
on the screen, how to point one at a model running on your own hardware, how a
feature is routed to a model and given a ceiling, and what each state on the
screen means.

The decisions behind the design — why nothing talks to a provider directly, why
approval is a database row, and why a model can never act on its own — are on
[AI privacy and the gateway](https://omniflow-vpn.mintlify.site/operations/ai-privacy.md) and
[record 0006](https://omniflow-vpn.mintlify.site/architecture/decision-0006-ai-is-optional.md). Read this page for
the mechanics.

> [!WARNING]
>
> **No feature invokes a provider in this build.** The registry, the credential
> sealing, the per-feature routing, the budgets, and the usage report are real
> and are what a provider will be driven through. The gateway and the feature
> packages are implemented and unit-tested, but no process mounts them: there is
> no AI endpoint on `/v1/panel`, and no screen outside these settings calls one.
> A provider registered today is configuration waiting for the surfaces, and
> registering one sends nothing anywhere.

## Where it lives

`/admin/settings/ai`. Viewing needs `settings.read`; changing anything needs
`settings.write`, which in practice means an owner. Every change is written to
the audit trail with the actor and the reason the panel collected.

## The three adapters

A provider row has a **kind**, and the kind decides which request shape
Omniflow speaks. There are three, and they are deliberately thin — each maps a
request onto that provider's completion endpoint and maps the answer back.
Redaction, budgets, retries, and limits belong to the gateway, never to an
adapter.

| Kind | Endpoint it calls | Base URL when you leave the field empty |
| - | - | - |
| `openai_compatible` | `POST {base}/chat/completions` | `https://api.openai.com/v1` |
| `anthropic` | `POST {base}/messages` | `https://api.anthropic.com/v1` |
| `gemini` | `POST {base}/models/{model}:generateContent` | `https://generativelanguage.googleapis.com/v1beta` |

Three behaviours are shared by all of them and are worth knowing before you
diagnose anything:

* **A provider's own error message is never propagated.** A non-2xx answer
  becomes `HTTP <status>` and nothing more, because a provider's error text can
  echo the prompt back and the prompt contains customer content that has no
  business in an operator's browser. The detail goes to the log.
* **The response body is bounded at 4 MiB and the request times out after 60
  seconds.** An endpoint that streams without end fails rather than exhausting
  memory.
* **A task with no token limit gets 1024.** A feature that has not been given a
  ceiling should produce a short answer rather than an expensive one; raising it
  is a deliberate act on the feature.

The Anthropic adapter pins `anthropic-version: 2023-06-01` rather than tracking
the latest version, because a provider that changes its response shape under a
running installation is a failure nobody deployed.

> [!NOTE]
>
> The Gemini endpoint takes its API key as a query parameter, which is that
> API's design rather than a choice made here. The key therefore reaches request
> logs at the provider. If that matters for your installation, prefer one of the
> other two kinds.

## Registering a provider

**1. Open the registry**

Go to `/admin/settings/ai`. The **Providers** card lists what is registered;
on a fresh installation it says that no provider is configured and every AI
feature is therefore unavailable.

**2. Fill in the identity**

**Identifier** is the stable slug a feature will name — lowercase letters,
digits, `-` and `_`, 2 to 41 characters. It is never shown to a customer and
never changes, so pick something durable (`openai-main`, `local-llama`).
**Name** is what operators read.

**3. Choose the kind and the base URL**

Select `openai_compatible`, `anthropic`, or `gemini`. Leave **Base URL**
empty to use that kind's default from the table above, or set it to reach a
different endpoint — a regional host, a gateway of your own, or a local
model.

**4. Record what the provider does with your data**

**Zero retention** and **May train on your data** are your answers, taken
from the provider's own terms. Omniflow cannot verify them and does not try.
They are shown on the card above the switch that would send data to the
provider, and they raise warnings on any feature routed to it.

**5. Add the key and approve it**

A new provider is created **disabled**, whatever else you set. Paste the API
key, save, then turn on **Approved for use** on the provider's card. A task
routed at a provider with no enabled row is refused rather than falling back
to another one.

The API key is write-only end to end. It is sealed with
`APP_DATA_ENCRYPTION_KEY` before it reaches the database, bound to that
provider's slug so a ciphertext moved between rows will not open, and never
returned by the API. The field is therefore always empty when the screen loads,
and leaving it empty keeps the stored key — the **Rotate key** button is how you
replace one.

Leaving it empty keeps the key **only while the provider still points where the
key was entered for**. A save that changes the **kind** or the **base URL**
without a new key is refused with `422 secret_required`, and the storage layer
itself drops a stored key whose provider was moved. Otherwise anyone allowed to
edit the address could point it at a server they run and collect the key on the
next request without ever being able to read it.

## Pointing at a local model

There is no separate adapter for a local model, and that is the point: anything
that implements the OpenAI `/chat/completions` shape is registered as
`openai_compatible` with its own base URL. Ollama, vLLM, LM Studio,
llama.cpp's server, and most self-hosted gateways all do.

* Set **Base URL** to the server's OpenAI-compatible root, including any path
  prefix it serves under. The adapter appends `/chat/completions` to exactly
  what you give it, and a trailing slash is trimmed.
* The **API key field is required** even when your server ignores it, because a
  provider with no key cannot be constructed. Put any non-empty placeholder
  there for a server that does not authenticate.
* Reach the server by a hostname the API container can resolve. `localhost`
  inside a container is the container, not the host — use the Compose service
  name for a model running in the same stack, or the host's LAN address.
* **Set `APP_AI_ALLOW_PRIVATE_NETWORK=true` on the API.** By default a base URL
  must be HTTPS to a public address: the request carries the provider's API key,
  and an operator-typed address that pointed inside the deployment's network
  would turn the adapter into a way to reach it. A Compose service name or a LAN
  address resolves to a private address and is refused — at save time when it
  is a literal IP or plain HTTP, and at connect time, after resolution, when it
  is a hostname. The opt-in is an environment variable rather than a panel
  switch so that whoever runs the process decides it, not whoever holds
  `settings.write`.
* The **model** is named per feature, not on the provider, and it must be the
  name your server exposes rather than a hosted vendor's.

> [!WARNING]
>
> With `APP_AI_ALLOW_PRIVATE_NETWORK` on, **every** AI provider may reach the
> API's private network, not only the local one — anyone with `settings.write`
> can then point a provider at an internal address. Turn it on only where that
> is acceptable. Without it, every connection an adapter makes, including one a
> redirect opens, is checked after name resolution and refused if it is not a
> public address.

## Routing a feature to a model

The **Features** card lists every feature an installation could have. Each is
off until you enable it and choose a provider and a model, and a feature that is
enabled without both cannot be saved — an enabled feature that would fail on
first use looks like a bug rather than a missing setting.

| Feature | What it is for |
| - | - |
| `support_summary` | Ticket-thread summaries |
| `support_reply` | Suggested replies for an operator to edit |
| `support_rewrite` | Shorter, clearer, friendlier, more formal, more technical |
| `support_translate` | Russian and English, preserving names and variables |
| `support_classify` | Suggested category, priority, tags, and sentiment |
| `marketing_draft` | News, announcement, and campaign copy |
| `risk_analysis` | Explainable abuse and fraud assessment |
| `copilot` | The permission-aware admin assistant |
| `mcp_tools` | Tool calls through registered MCP servers |

Each feature carries its own settings, because summarising a ticket and judging
whether somebody is committing fraud are not the same risk and should not share
a configuration:

| Setting | Accepted range | What it does |
| - | - | - |
| Provider and model | An approved provider; any model string | Where this feature's requests go |
| Temperature | 0 to 2 | Passed to the provider |
| Max tokens | 1 to 200000 | Ceiling on the answer; 1024 when unset |
| Timeout | 1000 to 600000 ms | How long a single request may take |
| Budget tokens / window / cost | Any non-negative value | Checked **before** the call, not measured after it |
| Keep prompts / Keep outputs | On or off | Whether the material is stored in Omniflow |
| Retention | 0 to 3650 days | How long stored material survives |

Retention is per feature because the material differs: a marketing draft is the
operator's own copy, and a support summary is somebody's complaint. Prompts are
not kept by default; outputs are.

## Ceilings, and who they stop

Budgets on a feature bound that feature. The **Usage limits** card bounds
people, which is a different question — "the installation spent its budget" and
"one operator looped the copilot" need different answers, and only the second
should stop that operator.

A limit has a scope, a window in seconds, and any combination of a request
count, a token count, and a cost ceiling in minor units:

| Scope | Bounds |
| - | - |
| `installation` | Everything, everywhere |
| `role` | Every operator holding that role |
| `operator` | One named operator |
| `feature` | One feature, regardless of who invokes it |

When several apply, the narrowest one is what the operator is told about,
because the remedy differs: waiting will not fix a limit that names you.

An unreadable meter fails closed. Spending against a budget nobody can measure
is how an installation learns its limit from an invoice.

## Reading the state on the screen

The **Spend and performance** card reports requests, tokens, mean and p95
latency, failures, and estimated cost per feature and model over a window it
prints beside the numbers. No prompt or output content is stored in that report,
so none appears in it.

Each feature shows the warnings that apply to it, and each is a fact rather than
an opinion:

| Warning | What it means |
| - | - |
| This provider may train on your data | The provider row is marked as training |
| This provider is not zero-retention | Zero retention is not set on the provider |
| No data region recorded | The provider row has no region |
| Never connection-tested | No connection test result has ever been recorded. Press **Test connection** to record one |
| The last connection test failed | The stored result is a failure, with its detail |
| Prompts are kept in Omniflow | This feature retains prompts |
| No provider and model chosen | The feature cannot run as configured |

## Testing a provider

**Test connection** makes one real request. It opens the stored credential,
sends Omniflow's own prompt — *Reply with the single word OK* — with a ceiling
of sixteen output tokens, and records what happened on the provider row. The
result is what the screen shows on the next load, so a verdict never disagrees
with itself after a refresh.

It is a real completion rather than a reachability check because the two fail
differently and only the first answers the question. An address that resolves
and a key the provider accepts are separate facts, and a test that proved only
the first would report a revoked key as healthy.

Leave **Model to test with** empty and the test uses whatever model a feature
already points at this provider — the configuration actually worth proving. A
provider nothing points at yet has no model to infer, and the request is refused
rather than guessed.

> [!NOTE]
>
> Testing a provider you have not switched on yet is allowed, and is the order
> the work happens in: paste the key, test it, then approve it. Everything that
> can send customer content still requires **Approved for use**.

Three details are deliberate:

* **It needs `settings.write`, not `settings.read`.** The test opens a sealed
  credential and spends your money. Neither is something reading a settings
  screen should be able to do.
* **It is rate limited** to thirty tests an hour per operator, which is generous
  for the paste-test-fix loop and small enough that a stuck retry cannot spend
  an afternoon's budget.
* **A test that could not be attempted stores nothing.** No key, no model, an
  unregistered slug: each is refused and the stored result is left alone,
  because recording a failure would claim something was tried. A test that ran
  and failed *is* stored, with a category rather than the provider's own words —
  a provider message can echo the prompt back, and a transport error quotes the
  address it dialled. No adapter puts a key in the URL — Gemini's travels in the
  `x-goog-api-key` header — and transport errors are stripped of the URL before
  anything logs them.

Every test is written to the audit trail as `ai.provider.tested`, beside the
`ai.provider.saved` entry from the rotation before it. The row holds only the
latest outcome, and "it failed, somebody changed the key, then it passed" is the
shape of the question people ask.

## Troubleshooting

| Symptom | Cause |
| - | - |
| The Providers card says no provider is configured | Nothing is registered, or nothing registered is approved. A row exists but **Approved for use** is off. |
| A provider was saved but shows **No key** | The key field was empty on save. It is write-only, so an empty field means "keep what is stored", and on a new row there is nothing stored. |
| A feature will not save as enabled | Enabling requires both a provider and a model. Choose both, then enable. |
| A provider will not delete | A feature still points at it. Deleting it would turn that feature off without saying so, so the foreign key refuses. Repoint or disable the feature first. |
| Rotating the key appears to do nothing | The **Rotate key** button is disabled until the field has a value, because saving an empty field is how you *keep* the current key. |
| **Test connection** says the test could not be run | Nothing was sent and the stored result is unchanged. The message says which: no key is stored, no model was given and none could be inferred, or the provider kind has no adapter in this build. |
| A test reports the provider could not be reached, with no detail | The failure was in the transport rather than in the provider's answer. The address is never quoted back, in the screen or in the log. Check the base URL and the network path from the API container. |
| The screen is not in the navigation at all | The signed-in operator lacks `settings.read`, or this process has no operations service attached — in which case the route is absent rather than forbidden, so a caller learns nothing about what is configured. |

## Related

> **[AI privacy and the gateway](https://omniflow-vpn.mintlify.site/operations/ai-privacy.md)**
>
> What is removed before anything is sent, and what AI is never allowed to do.

> **[AI governance and evaluation](https://omniflow-vpn.mintlify.site/operations/ai-governance.md)**
>
> Per-feature enablement, the data-flow inventory, retention, and the evaluation thresholds.

> **[MCP connections](https://omniflow-vpn.mintlify.site/operations/mcp.md)**
>
> Registering an MCP server, its allowlists, and its egress restrictions.

> **[Installation settings](https://omniflow-vpn.mintlify.site/operations/installation-settings.md)**
>
> The rest of the configuration screens and how secrets are sealed.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.