AI models for apps

Status: Preview. The whole path is built and tested against a fake provider (the gate, the budgets, the API, the SDK, the console). It has not been run with a live Anthropic or OpenAI key: no test calls a paid API, and the first real call may need a fix. Say so to anyone you let try it.

An app can ask a model for text: summarise a ticket, label an email, draft a reply. It never holds a provider key. A workspace admin pastes their own key once (Settings → Models), the API seals it to the gate (HPKE, write-only, the same way connector secrets are sealed), and the gate adds it to the provider call, counts the tokens and keeps every app inside its monthly budget.

app code ── ctx.ai.complete() ──▶ runtime host ──▶ gate ──▶ provider
                                   (signs the call)   │        (api.anthropic.com,
                                                      │         api.openai.com, or the
                          1. who is calling? which app version, which models it declared
                          2. is there budget left this month? (reserve, in the API's books)
                          3. open the sealed key for this one call, add it, send
                          4. read the answer, count tokens, scrub the key, reply
                          5. settle: what it really cost → usage, audit row, events

The app sees text and token counts. It never sees a key, a price or another app's use.

Connect a provider (admins)#

Settings → Models → Add a provider.

Up to 5 connections per workspace. A model id belongs to one connection.

• Suggestions. The form offers presets with starting prices. The Anthropic prices were read from Anthropic's price list on 6 October 2026. The OpenAI ones were not re-checked: confirm them. Qaldy does not fetch prices for you and does not bill you for model use.
• Default model. What an app uses when its manifest names no model. The first model of the first connection until an admin picks one.
• Test the key. The gate asks the provider for its model list with the sealed key. It makes no model call and spends no tokens. A provider that refuses the key marks the connection Key rejected and apps get not_connected until a new key is pasted. In the demo this only says "Sample data: nothing was tested."
• Change. Rename, edit models and prices, or paste a new key (it replaces the sealed one). The address cannot change: the key is bound to it.
• Remove. Deletes the sealed key. The dialog lists the apps that lose a model they use; their calls fail with model_not_declared until another connection offers it.

Everyone in the workspace can read the page (connections, models, prices). Only owners and admins change anything or see money. A personal access token can read but never change a key or budget.

Where it is available#

Budgets#

An app declares its AI use in the manifest:

"ai": { "profile": "standard", "monthlyBudgetUsd": 40, "models": ["claude-haiku-5-5"] }

• models (up to 8) names the models it may call; left out, it may call the workspace default.
• monthlyBudgetUsd is the most it may spend in a calendar month in the workspace's time zone.
• An admin can override a budget per app (Models → Apps and budgets → Change budget), and go back to the manifest's. The override is an audit row.
• Because the key is the workspace's money, an app that declares AI needs an admin to approve it (a member gets 403 admin_approval_required and an audit row, as for connectors).

How the gate keeps to it: before anything leaves for the provider it reserves the worst case (the estimated input plus maxTokens output tokens, at the model's prices). The API checks spent + open reservations + this one <= budget under a per-app lock and either holds the amount or answers no. When the call ends the gate settles with the real token counts and the reservation is replaced by the real cost. Two calls at once, or two gates, can't both spend the last dollar. If the API can't be reached the gate fails closed (503 ai_unavailable) and sends nothing. Money is whole micro-dollars end to end; the TypeScript and Rust cost arithmetic is checked against shared vectors (tests/vectors/ai-cost.json).

A call that fails for the provider's reasons settles at what it used (often nothing). A stream the app stops reading settles at what had been produced.

Use it from an app#

import { defineApp, AiError } from '@qaldy/sdk'
import { manifest } from './manifest' // manifest.permissions.ai = { profile, monthlyBudgetUsd, models }

export default defineApp({
  manifest,
  async fetch(request, ctx) {
    const ticket = await request.text()
    try {
      const triage = await ctx.ai.complete({
        system: 'Reply with one word: urgent, normal or spam.',
        prompt: ticket,
        maxTokens: 8,
      })
      // triage.text, triage.usage { inputTokens, outputTokens }, triage.stopReason

      const stream = await ctx.ai.stream({ prompt: `Write a two-line summary: ${ticket}` })
      let summary = ''
      for await (const text of stream) summary += text
      const whole = await stream.result // the full CompleteOutput
      return new Response(`${triage.text}\n${summary}\n${whole.usage?.outputTokens ?? 0} tokens`)
    } catch (error) {
      if (error instanceof AiError && error.code === 'budget_exceeded') {
        return new Response('This month’s AI budget is used up.', { status: 503 })
      }
      throw error
    }
  },
})

Input: prompt or messages (up to 64, the last from the user), optional system, model (one the manifest declares), maxTokens (default 1024), temperature (0 to 2; leave it out for models that only take their default: the current Claude models, Haiku, Sonnet and Opus 5.5, refuse any other value, and the gate passes the provider's refusal on). A request over 256 KB is refused before any network. Replies are untrusted text: a model can be fooled by what you paste into it, so don't feed a reply straight into something that sends or deletes.

Errors are AiError with a code and plain words:

A stream that fails part-way throws from the iteration and rejects result. Tests of an app use @qaldy/sdk/testing (aiStreamResponse and a fake fetch): no network, no key.

The builder may plan and write apps that use ctx.ai when the workspace offers a model (it is told which). Otherwise it plans without, and says so in a step.

What the gate guarantees#

• The key is never visible. It is opened inside the gate for one call, added to the provider request, and scrubbed from the reply, from errors and from logs (a test makes the fake provider quote the key in an error and checks it is gone before the app sees it). It is not in any API response, audit row, event or the SDK.
• Only declared models. A call for a model the version's manifest doesn't name is refused before the budget is touched.
• Fixed host. The provider host is pinned per connection and every address it resolves to must be public; redirects are never followed; the app can't choose the URL. (This is the same fence the connector gate uses, so AI is not a way around default-deny egress.)
• Bounded. Requests over 256 KB, answers over 4 MiB (non-streamed), calls over 120 s and more than 8 running calls per app are refused or cut.
• Nothing about content is stored. The books hold counts and money, never a prompt or a reply.

What gets recorded#

Like every connector call, the gate reports each AI call as an event: connector.called for a call that went out, or egress.blocked with the reason for a refusal (model_not_declared, budget_exceeded, not_connected, request_too_large, rate_limited…), and the API writes an audit row per event. When a call ends the books add an ai.called audit row (app, version, model, input and output tokens, cost in micro-dollars, outcome, duration: never the content), so Security → Audit log shows it. Adding, changing, testing and removing a connection, setting the default model, setting an app's budget and a provider rejecting a key are audit rows too (ai.connection.created, .updated, .tested, .deleted, .rejected, ai.default_model.set, ai.budget.set).

Usage is kept per app, day and model (ai_usage): calls, refused calls, tokens and cost.

• Settings → Models shows this month's spend against budgets, a cost-per-day chart and tables by app and by model (7, 30 or 90 days).
• Analytics has a "Model use" section for the apps you can open.
• Each app page shows its models, budget used and calls.
• Capacity counts it as ai_credits: one credit per US cent at your prices, an estimate. Qaldy does not bill model use; your provider does.

Ask about this run (Preview)#

Limits and gaps#

• Preview: never run against a live provider (see the top).
• Text in, text out. No tool use, images, files, embeddings or function calling yet.
• Anthropic and OpenAI-style chat completions only. Other APIs need an OpenAI-compatible endpoint.
• The Kazakhstan profiles don't get it, and there are no private in-region models yet.
• Budgets are per app per calendar month. There is no per-workspace cap, no per-user cap and no alert at 80 percent yet; the page shows the meter and turns it amber near the limit.
• A model's price is typed by the admin, not fetched; a price that is wrong makes the budget wrong.
• Provider-side token counts are used when the provider returns them. When it doesn't (some compatible APIs, or a stream cut short), the gate estimates the missing count (one token per three bytes of text) and the audit row says estimated.
• The reservation before a call uses the same rough estimate for the input, which is deliberately high so it never undercounts a normal request.
• Streaming is a response body passed through the runtime host and the gate. It is tested against a fake gate and a fake provider, not on Cloudflare Workers with a live stream.
• Builds (the builder's own model spend) are a separate ledger and are not part of these budgets.