# AI models for apps

An app can ask a model for text: summarise a ticket, label an email, draft a reply.

URL: https://docs.qaldy.com/connectors/ai-models/

**Status: Preview.** The whole path is built and tested against a fake provider (the gate, the
budgets, the API, the SDK, the console). It has **not been run with a live Anthropic or OpenAI
key**: no test calls a paid API, and the first real call may need a fix. Say so to anyone you let
try it.

An app can ask a model for text: summarise a ticket, label an email, draft a reply. It never holds
a provider key. A workspace admin pastes **their own** key once (Settings → Models), the API seals
it to the gate (HPKE, write-only, the same way connector secrets are sealed), and the gate adds it
to the provider call, counts the tokens and keeps every app inside its monthly budget.

```
app code ── ctx.ai.complete() ──▶ runtime host ──▶ gate ──▶ provider
                                   (signs the call)   │        (api.anthropic.com,
                                                      │         api.openai.com, or the
                          1. who is calling? which app version, which models it declared
                          2. is there budget left this month? (reserve, in the API's books)
                          3. open the sealed key for this one call, add it, send
                          4. read the answer, count tokens, scrub the key, reply
                          5. settle: what it really cost → usage, audit row, events
```

The app sees text and token counts. It never sees a key, a price or another app's use.

## Connect a provider (admins)

Settings → **Models** → **Add a provider**.

| Field | Rule |
|---|---|
| Provider | **Anthropic (Claude)**, **OpenAI**, or **OpenAI-compatible API** (any service that speaks the OpenAI chat format, such as a hosted open-model service) |
| Name | How it appears on the page (2 to 60 characters) |
| API address | Only for OpenAI-compatible: `https://<public host>[/path]`. Same rules as the [HTTP connector](https://docs.qaldy.com/connectors/http/): no IP literals, no private or `.internal` names, no userinfo or query, port 443, and the API refuses a name that resolves to a private address. The key is sealed to this host |
| API key | Write-only. 16 or more characters for Anthropic and OpenAI (8 for compatible), at most 512, printable, no spaces. Anthropic keys start `sk-ant-`, OpenAI keys `sk-`. Sealed to **this connection and this host** (`ai:<connection>:<host>`) |
| Models and prices | Up to 16 models per connection. Each has an id exactly as the provider writes it and the **input and output price in US dollars per million tokens**. Qaldy multiplies tokens by these prices; what your provider invoices is what counts, so check its price list |

Up to 5 connections per workspace. A model id belongs to one connection.

- **Suggestions.** The form offers presets with starting prices. The Anthropic prices were read
  from Anthropic's price list on 6 October 2026. The OpenAI ones were **not re-checked**: confirm
  them. Qaldy does not fetch prices for you and does not bill you for model use.
- **Default model.** What an app uses when its manifest names no model. The first model of the
  first connection until an admin picks one.
- **Test the key.** The gate asks the provider for its model list with the sealed key. It makes
  no model call and spends no tokens. A provider that refuses the key marks the connection
  **Key rejected** and apps get `not_connected` until a new key is pasted. In the demo this only
  says "Sample data: nothing was tested."
- **Change.** Rename, edit models and prices, or paste a new key (it replaces the sealed one).
  The address cannot change: the key is bound to it.
- **Remove.** Deletes the sealed key. The dialog lists the apps that lose a model they use; their
  calls fail with `model_not_declared` until another connection offers it.

Everyone in the workspace can read the page (connections, models, prices). Only owners and admins
change anything or see money. A personal access token can read but never change a key or budget.

### Where it is available

| Compliance Profile | AI through your own key |
|---|---|
| `global`, `eu` | Yes. Providers follow the workspace's allowed list (a compatible API counts as an open-weight provider) |
| `kz-standard`, `kz-strict` | **Not yet.** Those profiles redact personal data and keep models in the country, and a provider key can't promise that. The page says so and the API refuses |
| A manifest with `ai.profile: "private"` | **Not offered yet** (in-region models don't exist). The builder never writes one and approval refuses it |

## Budgets

An app declares its AI use in the manifest:

```json
"ai": { "profile": "standard", "monthlyBudgetUsd": 40, "models": ["claude-haiku-5-5"] }
```

- `models` (up to 8) names the models it may call; left out, it may call the workspace default.
- `monthlyBudgetUsd` is the most it may spend in a **calendar month in the workspace's time zone**.
- An admin can override a budget per app (Models → Apps and budgets → **Change budget**), and go
  back to the manifest's. The override is an audit row.
- Because the key is the workspace's money, an app that declares AI needs an **admin** to approve
  it (a member gets `403 admin_approval_required` and an audit row, as for connectors).

How the gate keeps to it: before anything leaves for the provider it **reserves** the worst case
(the estimated input plus `maxTokens` output tokens, at the model's prices). The API checks
`spent + open reservations + this one <= budget` under a per-app lock and either holds the amount
or answers no. When the call ends the gate **settles** with the real token counts and the
reservation is replaced by the real cost. Two calls at once, or two gates, can't both spend the
last dollar. If the API can't be reached the gate **fails closed** (`503 ai_unavailable`) and
sends nothing. Money is whole micro-dollars end to end; the TypeScript and Rust cost arithmetic is
checked against shared vectors (`tests/vectors/ai-cost.json`).

A call that fails for the provider's reasons settles at what it used (often nothing). A stream the
app stops reading settles at what had been produced.

## Use it from an app

```ts
import { defineApp, AiError } from '@qaldy/sdk'
import { manifest } from './manifest' // manifest.permissions.ai = { profile, monthlyBudgetUsd, models }

export default defineApp({
  manifest,
  async fetch(request, ctx) {
    const ticket = await request.text()
    try {
      const triage = await ctx.ai.complete({
        system: 'Reply with one word: urgent, normal or spam.',
        prompt: ticket,
        maxTokens: 8,
      })
      // triage.text, triage.usage { inputTokens, outputTokens }, triage.stopReason

      const stream = await ctx.ai.stream({ prompt: `Write a two-line summary: ${ticket}` })
      let summary = ''
      for await (const text of stream) summary += text
      const whole = await stream.result // the full CompleteOutput
      return new Response(`${triage.text}\n${summary}\n${whole.usage?.outputTokens ?? 0} tokens`)
    } catch (error) {
      if (error instanceof AiError && error.code === 'budget_exceeded') {
        return new Response('This month’s AI budget is used up.', { status: 503 })
      }
      throw error
    }
  },
})
```

Input: `prompt` **or** `messages` (up to 64, the last from the user), optional `system`, `model`
(one the manifest declares), `maxTokens` (default 1024), `temperature` (0 to 2; leave it out for
models that only take their default: the current Claude models, Haiku, Sonnet and Opus 5.5, refuse
any other value, and the gate passes the provider's refusal on). A request over 256 KB is refused before any network.
**Replies are untrusted text:** a model can be fooled by what you paste into it, so don't feed a
reply straight into something that sends or deletes.

Errors are `AiError` with a `code` and plain words:

| `code` | Gate answer | Meaning |
|---|---|---|
| `not_in_manifest` | 403 | The manifest declares no `ai` |
| `model_not_declared` | 403 | The model isn't one the manifest names, or the workspace no longer offers it |
| `budget_exceeded` | 402 | The month's budget is used up (or is zero). Nothing was sent |
| `not_connected` | 503 | No working key (none added, or the provider rejected it) |
| `too_large` | 413 | Request over 256 KB |
| `invalid_input` | 400 | Bad shape (both `prompt` and `messages`, empty text, unknown field…) |
| `rate_limited` | 429 | More than 8 of this app's calls running at once |
| `upstream_error` | 502 | The provider answered with an error |
| `unavailable` | 503 | The gate couldn't reach the API's books; nothing was sent |

A stream that fails part-way throws from the iteration and rejects `result`. Tests of an app use
`@qaldy/sdk/testing` (`aiStreamResponse` and a fake `fetch`): no network, no key.

The builder may plan and write apps that use `ctx.ai` when the workspace offers a model (it is told
which). Otherwise it plans without, and says so in a step.

## What the gate guarantees

- **The key is never visible.** It is opened inside the gate for one call, added to the provider
  request, and scrubbed from the reply, from errors and from logs (a test makes the fake provider quote
  the key in an error and checks it is gone before the app sees it). It is not in any API response, audit
  row, event or the SDK.
- **Only declared models.** A call for a model the version's manifest doesn't name is refused
  before the budget is touched.
- **Fixed host.** The provider host is pinned per connection and every address it resolves to must
  be public; redirects are never followed; the app can't choose the URL. (This is the same fence
  the connector gate uses, so AI is not a way around default-deny egress.)
- **Bounded.** Requests over 256 KB, answers over 4 MiB (non-streamed), calls over 120 s and more
  than 8 running calls per app are refused or cut.
- **Nothing about content is stored.** The books hold counts and money, never a prompt or a reply.

## What gets recorded

Like every connector call, the gate reports each AI call as an event: `connector.called` for a
call that went out, or `egress.blocked` with the reason for a refusal (`model_not_declared`,
`budget_exceeded`, `not_connected`, `request_too_large`, `rate_limited`…), and the API writes an
audit row per event. When a call ends the books add an `ai.called` audit row (app, version,
model, input and output tokens, cost in micro-dollars, outcome, duration: never the content), so
Security → Audit log shows it. Adding, changing, testing and removing a connection, setting the
default model, setting an app's budget and a provider rejecting a key are audit rows too
(`ai.connection.created`, `.updated`, `.tested`, `.deleted`, `.rejected`, `ai.default_model.set`,
`ai.budget.set`).

Usage is kept per app, day and model (`ai_usage`): calls, refused calls, tokens and cost.

- **Settings → Models** shows this month's spend against budgets, a cost-per-day chart and tables
  by app and by model (7, 30 or 90 days).
- **Analytics** has a "Model use" section for the apps you can open.
- Each **app page** shows its models, budget used and calls.
- **Capacity** counts it as `ai_credits`: one credit per US cent at your prices, an estimate.
  Qaldy does not bill model use; your provider does.

## Ask about this run (Preview)

## Limits and gaps

- **Preview**: never run against a live provider (see the top).
- Text in, text out. No tool use, images, files, embeddings or function calling yet.
- Anthropic and OpenAI-style chat completions only. Other APIs need an OpenAI-compatible endpoint.
- The Kazakhstan profiles don't get it, and there are no private in-region models yet.
- Budgets are per app per calendar month. There is no per-workspace cap, no per-user cap and no
  alert at 80 percent yet; the page shows the meter and turns it amber near the limit.
- A model's price is typed by the admin, not fetched; a price that is wrong makes the budget wrong.
- Provider-side token counts are used when the provider returns them. When it doesn't (some
  compatible APIs, or a stream cut short), the gate estimates the missing count (one token per
  three bytes of text) and the audit row says `estimated`.
- The reservation before a call uses the same rough estimate for the input, which is deliberately
  high so it never undercounts a normal request.
- Streaming is a response body passed through the runtime host and the gate. It is tested against
  a fake gate and a fake provider, not on Cloudflare Workers with a live stream.
- Builds (the builder's own model spend) are a separate ledger and are not part of these budgets.