When the control plane is down

What keeps working, what waits and what stops when Qaldy's API or its database is unreachable, and why an outage never widens access or switches the spending cap off.

"The control plane" is Qaldy's API and its database: sign-in, sharing, builds, billing, the audit log. The parts that serve your apps are separate: the gate in front of them and the runtime host that runs them. So a bad day for the API does not have to be a bad day for an app people are already using.

The rule behind every row#

A failure of the control plane never widens access and never switches the spending cap off. The serving side keeps what it already knew, for a bounded time, and refuses what it never knew.

What keeps working#

What	Keeps working?	How
Apps people already opened	Keeps working	People who already opened an app keep using it for up to 30 minutes: the gate remembers who may view it, and the runtime host remembers the code.
Schedules	Partly	Nothing fires while the API is down. When it returns, a schedule that missed a tick in the last 6 hours runs once, to catch up.
Webhooks	Stops	Senders get an error and retry if they can. Qaldy does not queue webhooks at the edge yet (Planned).
Signing in, and opening an app for the first time	Stops	Signing in and opening an app for the first time need the API. People already signed in to an app are not asked again.
The console	Partly	The console loads and says the control plane is unreachable; nothing can be changed until it is back.
Builds	Stops	Builds need the API and the builder. Apps already built are not affected.

Also, from the gate and the runtime host's side:

Apps people already opened. The gate remembers every "may this person open this app" answer for 30 minutes. While the API cannot be reached, someone the gate already allowed is allowed again from that copy. A person it never saw gets a calm "Temporarily unavailable" page (HTTP 503), not a 403.
Only a failure of the control plane starts this. A timeout, a connection error or a 5xx. A "denied" answer from the API is the API speaking, and a remembered copy never replaces it. A change to sharing clears the copy for that app, so revoking someone is not undone by a later outage. Access that rests on the copy lasts at most 30 minutes.
The code of an app. A bundle never changes for a version, so the runtime host keeps it in memory and in a second sealed copy in the data centre's cache. It is read from the cache only when the API cannot be asked, never instead of asking, so a deleted version cannot come back while the API answers.
Sites. Pages already served still load; a file that this data centre never served before needs the API.
The spending cap. The gate and the runtime host keep the last signed state, so apps still go to sleep at the cap with no API at all. See Spending cap.
The shield. Rate limits, bot rules and daily limits live in the gate and do not depend on the API.
Schedules catch up. Nothing fires while the API is down. When it returns, a schedule that missed a tick in the last 6 hours runs once, however many ticks it missed: an hourly job runs once, not six times. Older ticks are not made up. The run's actor is scheduler.catchup, so the audit log tells it from an ordinary tick. Over the cap, a caught-up run waits in the queue like any other.
The console asks the API every 15 seconds. After two misses in a row it shows a banner: the control plane is not answering since a given time, apps people already opened keep working, and signing in, builds and changes wait.

What stops, and what we have not built#

Webhooks. Senders get an error and retry if they can. Qaldy does not queue webhooks at the edge yet. Planned
The Backend data API is served by the API process today, so it stops with it. Serving it from an edge service is Planned.
A status page hosted away from the cell, so you can read it when the cell is down. Planned
The drill. Stopping the API on the pilot and recording what people saw. Planned