Upgrading an application to a newer platform release

Each environment of an application runs on its own platform release, and stays on it until its team moves it. A move dry-runs the application under the newer release on its zone, backs up its database, rebuilds its model for that release and migrates its database in one transaction, then deploys the same version on the release. If any step fails, the environment keeps serving on the release it had, and the team is told why. A patch release moves applications on its own once it reaches their organization; a minor or major release opens a 30- or 60-day opt-in window, after which the move is forced.

Summary#

A zone runs several platform releases at once. An application in one environment runs on one of them — its release is part of what that environment is set to run, like its version — and nothing another release does reaches it. Moving it to a newer release is an upgrade, and the upgrade is the only thing that rebuilds the application's model or migrates its database for a release.

An upgrade of one environment runs in four stages:

Dry run — on a runtime of the newer release, in the environment's zone: the version compiles under that release, and the platform's migrations of every release in between plan cleanly against the environment's database. Nothing is written.
Backup — on that runtime: the environment's database is backed up into its zone's backup store. A backup that cannot be taken stops the move before anything is migrated.
Move — on that runtime: the version's model is rebuilt for the newer release, and those migrations are applied, in release order, in one transaction on the environment's database.
Deploy — the same version is deployed on the newer release. The environment keeps serving on the release it had until the deployment converges; a deployment that fails is rolled back onto it.

Description#

The version does not change during an upgrade — only the release it runs on. Another environment of the same application keeps its own release, so production can stay on the release it has while a sandbox tries the next one.

The work runs on the zone, under the newer release's platform, not on the control plane: the control plane runs the newest release, and the release an environment moves to may not be it. The zone starts the newer release's runtimes if it does not run them yet, hands the stage to one of them, and reports how it went. The control plane records the outcome.

Who may upgrade an environment?#

The application's own team — any member of the application — and its organization's owners and admins may ask for an environment to move, exactly as they may deploy it. An environment that needs an approval cannot be moved by its team yet. A platform operator may also move a release line, or force an environment onto a release it must run on.

Which releases can I move to?#

A release is offered when it is:

offered when	refused with
newer than the release the environment runs on	"a move goes to a newer release"
on the same line (channel and target)	"is on another line"
active, with an artifact a zone can run	"names no artifact" / "is not active"
not already the environment's release	"already runs on"
the environment is not already moving	"is already moving to"
generally available, or in early access for a ring the application's organization is in	"is in early access, and … is not in a ring it has reached"

The application's Platform page lists the offered releases for each environment under Upgrade, marks a release in early access as such, and says when the zone will have to start a release first. A platform operator moving one named environment is not refused by early access.

Does a new release reach every organization at once?#

No. A platform release reaches organizations ring by ring before it is generally available, so the platform operators can prove it on a few organizations first. Who a release is offered to is its availability:

availability	offered to	what the platform does on its own
operators only — how a release is recorded	no organization; a platform operator may move one named environment to it	nothing
early access, up to a ring	the organizations in that release ring and in every ring before it	for those organizations: a patch release is followed, and a minor or major release opens a window
generally available	every organization	for every organization: a patch release is followed, and a minor or major release opens a window

A release ring is a named, ordered group of organizations that the operators define once and reuse for every release — a pilot ring at step 1, a partners ring at step 2. The operators move a release along on the Releases page of the infrastructure console:

the operators	what happens	who is told
publish a release to a ring	it is in early access for the organizations in that ring and every ring before it	the owners and admins of each organization it newly reaches, once
widen it to the next ring	it reaches the next ring in order	the owners and admins of each organization it newly reaches, once
put an organization in a ring	every release in early access that has reached the ring now reaches it	its owners and admins, once per release
take an organization out of a ring	a release that reached it only through that ring is no longer offered to it; an environment already on one stays	—
make it generally available	every organization is offered it; within the hour a patch release is followed, and a minor or major release opens its window, for every organization it newly reaches	the owners and admins of every organization it newly reaches, once
withdraw it, with a reason	back to operators only: no new move to it can be asked for, an environment already on it stays on it, a move already under way finishes, and the grace windows it opened close without forcing anything — including a window whose deadline had already failed, whose critical alert resolves and whose environment is brought back online when that failure is what took it offline	the owners and admins of every organization it had reached, with the reason; an environment brought back is told it is serving again

A rollout only widens: publishing a release to a ring it has already reached is refused, and so is widening past the last ring — make it generally available instead. Every one of these changes is written to the platform's logs with the release, the ring and who made it. Each email is recorded with how it went — sent (and to whom), nobody to tell (the organization has no owner or admin), or failed (with the address and the mail provider's words) — and the Releases page counts the notices that could not be emailed. A release withdrawn and published again tells its organizations again.

An organization's Settings page shows the rings it is in. An organization a release has not reached is not offered it, and asking for the move anyway is refused with why:

Platform release 0.11.0 is in early access, and Acme is not in a ring it has reached. It is offered to Acme when a platform operator widens it to a ring Acme is in, or makes it generally available.

A team's first deployment into an environment runs on the newest release its organization has been reached by.

How do I upgrade from the console?#

Open the application, choose the Platform tab, press Upgrade on the environment and Move to the release. The page then shows the move's stage — DryRun, BackingUp, Moving, Deploying, Succeeded or Failed — with what each stage measured (how long it took, how many migration statements were planned and applied), the backup taken before the move (its size and when) or why it could not be taken, the reason when it failed, and who was told.

How do I upgrade from a script?#

curl -X POST https://admin.<your-domain>/api/rest/v1/infra/releases/move-environment \
  -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" \
  -d '{"applicationSlug":"billing","environmentSlug":"prod","releaseVersion":"0.10.0","reason":"the autumn release"}'

The answer is the move's id. The move is asynchronous: its stages run on the zone.

What happens when a step fails?#

fails at	the environment	who is told
the dry run (the application does not compile under the release, or a migration does not plan)	stays on its release and keeps serving; nothing was written	the organization's owners and admins, with the reason
the rehearsal is red (a failing test, a migration the copy refuses, or something the application reads that the copy lacks)	stays on its release; nothing was written to its database	the application's team, with each failing test and its message
the backup (the database cannot be reached, or the zone has nowhere to store it)	stays on its release; nothing was migrated, and the failed attempt is recorded with its reason	the organization's owners and admins, with the reason
the move (a migration the database refuses)	stays on its release; the transaction is rolled back, so its database is as it was	the organization's owners and admins, with the database's reason
a platform release drops a member this installation never recorded as deprecated (the release that deprecated it was skipped)	the move goes on, but that one drop is refused: nothing is dropped and its data is kept; the deprecation is recorded at the target release, so a later release can drop it	the operators, by a critical alert that names the member and why
the deployment (the release's runtime does not serve it)	is rolled back onto the release it was served on	whoever asked, and the operators
readiness — the runtime of the release is up, but cannot load this version's model or open its database	the deployment fails before any request reaches it, and is rolled back onto the version and release it replaced, with the runtime's own words as the reason	whoever asked, and the operators
the zone never answers a stage — the backup included — within 30 minutes	stays on its release	the organization's owners and admins
a runtime of a release is sent an application whose model another release built (a route naming the wrong release, or a release changed without the move)	the runtime refuses to serve it and never rebuilds it; the refusal travels in its zone's report, so the runtime needs no access to the control plane's registry; the alert resolves once every environment it names is healthy on the release it is set to run	the operators, by a critical alert naming both releases

A forced move that fails takes the environment offline instead, with the reason, because it may not stay on the release it had to leave. Deploying a version that runs on the release brings it back.

Every stage, refusal and outcome is written to the platform's logs with its figures, under the application's name.

What happens to a workflow run that is waiting?#

A move changes the release an environment runs on, not its data. A workflow run that was waiting — for an event, a claim or a timer — is still waiting when the move converges, in the environment's own database. When what it waits for arrives, a runtime of the newer release resumes it, on the model rebuilt for that release, and it completes there.

Each run records the platform release that last moved it. A run resumed by a runtime of another release is logged with both:

Workflow run 6f1c… parked on platform release 1.4.0 is resumed on release 1.5.0, entering state Done: its placement moved to 1.5.0 while it waited, and it continues on that release's model

A run resumed on the release it waited under logs nothing.

Is the database backed up before a move?#

Yes, every time a move is made. The move migrates in one transaction, so a move that fails leaves the database exactly as it was. A move that succeeds and turns out later to have lost or changed data is different: only a backup taken before it can bring that data back — and a rehearsal's copy is dropped when the rehearsal ends, so it is no recovery point.

So after the dry run — and after the rehearsal, or the team's approval, when the move has one — a runtime of the newer release backs up the environment's database into the zone's backup store before it migrates anything:


what is backed up	the environment's own database — the one the move migrates
where it goes	the data zone's backup store when that zone has one, so a copy stays where the data is; otherwise the compute zone's, in the bucket its backup provider names (`placement-backups` when it names none). A backup that would land outside the data zone's jurisdiction warns the organization and is logged — it never blocks the move
what is recorded	a backup of the application in that environment, taken before a migration: when, its size, its SHA-256, and where it is — or why it could not be taken
how long it is kept	30 days
when it cannot be taken	the move is not made, the environment stays on its release, and the organization's owners and admins are told why; a forced move is taken offline instead, as for any stage it cannot pass

A dry run takes no backup: it writes nothing. The application's Platform page shows each move's backup, and the emails about a move that went ahead mention it. Restoring one is a platform operator's action.

Is a move rehearsed first?#

When upgrade rehearsal is included for an environment, a move is rehearsed after its dry run and before it is made. On the environment's zone, the newer release:

copies the environment's database while it keeps serving;
runs the move against the copy;
checks that every table and column the application reads is there;
runs the tests the version was deployed with, under the newer release;
drops the copy, whatever happened.

the rehearsal	what happens to the move
green	it is made — or, when the platform started the move and the application's upgrade policy is Manual, it waits for the team's approval
red	it is not made; the environment stays on its release, and each failing test is shown with its message
skipped — the database is larger than the zone copies, or the zone cannot copy it	it waits for a person to approve or decline it

A version carries its tests when it is deployed from a project whose manifest declares them, so the rehearsal runs exactly the tests the team shipped. A version deployed without tests is rehearsed without them.

Upgrade rehearsal is a platform feature the operators grant. An offering can include it, and an organization can be granted it or have it withheld, which overrides its offerings. The application's Platform page says, for each environment, whether its moves are rehearsed and why not. A move without a rehearsal is still dry-run, applied in one transaction with rollback, and checked for readiness.

How does an application take a new release?#

Its team chooses the upgrade policy on the application's Platform page:

policy	when the platform rehearses a newer release
each move waits for your approval (the default)	the move waits for the team to Approve move or Decline move
move when the rehearsal is green	a green rehearsal makes the move on its own — offered only where upgrade rehearsal is included

Approving or declining a waiting move is the team's: its organization's owners and admins, or a member of the application.

How long can an application stay on an older release?#

What the platform does on its own depends on the release's class, and it happens for an organization once the release reaches it — through its release ring, or at general availability. A release for the operators only reaches nobody, so nothing happens.

A patch release is followed: once it reaches the organization, every environment on an earlier patch of its line is moved to it, within the hour. Each environment is asked once per release; one whose move fails is not asked again every hour, and its team is told why. Because nothing asks again, that environment goes on running the earlier patch — without whatever the patch fixed — until its team moves it or a later minor or major release's window does; the operators are alerted too, so a patch stalling across many applications is visible rather than one line in a log. That alert resolves when the environment reaches the patch, however it gets there.

A minor or major release opens an opt-in window once it reaches the organization. Until the window's deadline the team moves each environment whenever it chooses. The window is:

release	window
minor	30 days
major	60 days

The platform operators can set another length for a class of release, or for one release, on the Releases page of the infrastructure console. Setting it for one release moves the deadline of every window of that release still open — to when the window opened plus the new length, never sooner than a day away — and tells each team the new date.

when	what happens	who is told
the window opens	a deadline is recorded for the environment, and shown on the application's Platform page and the organization's Environments page	the application's team and the organization's owners and admins
14 days before the deadline	a reminder, once	the same
3 days before the deadline	a reminder, once	the same
1 day before the deadline	a reminder, once	the same
the deadline passes	the environment is at risk, and the platform forces the move — rehearsed when rehearsal is included	the application's team
that move fails — the model does not compile on the release, or a stage fails — or it cannot be asked for	the environment is taken offline at once, with the reason	the organization's owners and admins, emailed the diagnostics and how to bring it back; the operators, by a critical alert
the environment reaches the release	the deadline is met, and the alert resolves	—
the hourly sweep cannot handle one application — its window cannot be opened, or its deadline cannot be handled	the sweep goes on to every other organization, and that one is left on the release it runs, never reminded and never forced; the step is tried again within the hour	the operators, by an alert naming the application and the release, which resolves when that step succeeds

Each notice is sent once and records how its email went to each recipient. An address the mail provider refuses is tried again on the next hourly sweep — only that address — for up to a day.

A window shorter than a reminder's lead has no such reminder: a 10-day window has no 14-day reminder. When the release stops reaching the organization — it is withdrawn, or the organization is taken out of its ring — its open windows close without forcing anything, and open afresh if the release reaches the organization again.

Withdrawing goes one step further than a ring change, because a withdrawn release is one nobody may move to at all: it also closes a window whose deadline had already FAILED, resolves the critical alert that deadline raised, and brings the environment back online when that deadline is what took it offline. An environment an operator took offline for some other reason stays offline — withdrawing a release says nothing about that reason. Taking an organization out of a ring closes open windows only: the release still exists for everyone else, so a failed deadline is still owed.

To bring an environment back after a failed deadline, upgrade the application locally on the new release, fix what the report names, and deploy it. Once that version serves, the platform forces the move again, and when the environment reaches the release the deadline is met and the alert resolves. The one case you need do nothing for is the release being withdrawn, above.

A platform operator can also force one environment onto a release outside any window. When that move fails before its deployment, the environment is taken offline and the operators are alerted in the same way.

Every move records what started it — its team, a patch release, a window's deadline, an operator or a rehearsal — together with the ring the release reached the organization through and the reason. The platform's logs carry the same, with the release and the placement.

An application is at risk, never a whole environment: applications in one environment may run different releases while their windows are open. The organization's Environments page shows, for each environment, how many applications run each release and how many are at risk, and can move every application in an environment to a release — each one's own move, which each application's team may make.