# Upgrading an application to a newer platform release

> Each environment of an application runs on its own platform release, and stays on it until its team moves it. A move dry-runs the application under the newer release on its zone, backs up its database, rebuilds its model for that release and migrates its database in one transaction, then deploys the same version on the release. If any step fails, the environment keeps serving on the release it had, and the team is told why. A patch release moves applications on its own once it reaches their organization; a minor or major release opens a 30- or 60-day opt-in window, after which the move is forced.

<!-- id: zones-upgrading-an-application · area: zones · stability: stable · html: https://osysharp.com/reference/zones/upgrading-an-application/ -->

## Summary        {#summary}

A zone runs several platform releases at once. An application in one environment runs on one of them — its **release is
part of what that environment is set to run**, like its version — and nothing another release does reaches it. Moving it to
a newer release is an **upgrade**, and the upgrade is the only thing that rebuilds the application's model or migrates its
database for a release.

An upgrade of one environment runs in four stages:

1. **Dry run** — on a runtime of the newer release, in the environment's zone: the version compiles under that release, and
   the platform's migrations of every release in between plan cleanly against the environment's database. Nothing is
   written.
2. **Backup** — on that runtime: the environment's database is backed up into its zone's backup store. A backup that cannot
   be taken stops the move before anything is migrated.
3. **Move** — on that runtime: the version's model is rebuilt for the newer release, and those migrations are applied, in
   release order, in **one transaction** on the environment's database.
4. **Deploy** — the same version is deployed on the newer release. The environment keeps serving on the release it had
   until the deployment converges; a deployment that fails is rolled back onto it.

## Description    {#description}

The version does not change during an upgrade — only the release it runs on. Another environment of the same application
keeps its own release, so production can stay on the release it has while a sandbox tries the next one.

The work runs **on the zone, under the newer release's platform**, not on the control plane: the control plane runs the
newest release, and the release an environment moves to may not be it. The zone starts the newer release's runtimes if it
does not run them yet, hands the stage to one of them, and reports how it went. The control plane records the outcome.

## Who may upgrade an environment? {#who}

The application's own team — any member of the application — and its organization's owners and admins may ask for an
environment to move, exactly as they may deploy it. An environment that needs an approval cannot be moved by its team yet.
A platform operator may also move a release line, or **force** an environment onto a release it must run on.

## Which releases can I move to? {#offered}

A release is offered when it is:

| offered when | refused with |
|---|---|
| newer than the release the environment runs on | "a move goes to a newer release" |
| on the same line (channel and target) | "is on another line" |
| active, with an artifact a zone can run | "names no artifact" / "is not active" |
| not already the environment's release | "already runs on" |
| the environment is not already moving | "is already moving to" |
| generally available, or in [early access](#early-access) for a ring the application's organization is in | "is in early access, and … is not in a ring it has reached" |

The application's **Platform** page lists the offered releases for each environment under **Upgrade**, marks a release in
early access as such, and says when the zone will have to start a release first. A platform operator moving one named
environment is not refused by early access.

## Does a new release reach every organization at once? {#early-access}

No. A platform release reaches organizations **ring by ring** before it is generally available, so the platform operators can
prove it on a few organizations first. Who a release is offered to is its **availability**:

| availability | offered to | what the platform does on its own |
|---|---|---|
| operators only — how a release is recorded | no organization; a platform operator may move one named environment to it | nothing |
| early access, up to a ring | the organizations in that release ring and in every ring before it | for those organizations: a patch release is followed, and a minor or major release opens a [window](#grace-window) |
| generally available | every organization | for every organization: a patch release is followed, and a minor or major release opens a [window](#grace-window) |

A **release ring** is a named, ordered group of organizations that the operators define once and reuse for every release — a
`pilot` ring at step 1, a `partners` ring at step 2. The operators move a release along on the **Releases** page of the
infrastructure console:

| the operators | what happens | who is told |
|---|---|---|
| publish a release to a ring | it is in early access for the organizations in that ring and every ring before it | the owners and admins of each organization it newly reaches, once |
| widen it to the next ring | it reaches the next ring in order | the owners and admins of each organization it newly reaches, once |
| put an organization in a ring | every release in early access that has reached the ring now reaches it | its owners and admins, once per release |
| take an organization out of a ring | a release that reached it only through that ring is no longer offered to it; an environment already on one stays | — |
| make it generally available | every organization is offered it; within the hour a patch release is followed, and a minor or major release opens its [window](#grace-window), for every organization it newly reaches | the owners and admins of every organization it newly reaches, once |
| withdraw it, with a reason | back to operators only: no new move to it can be asked for, an environment already on it stays on it, a move already under way finishes, and the grace windows it opened close without forcing anything — including a window whose deadline had already **failed**, whose critical alert resolves and whose environment is brought back online when that failure is what took it offline | the owners and admins of every organization it had reached, with the reason; an environment brought back is told it is serving again |

A rollout only widens: publishing a release to a ring it has already reached is refused, and so is widening past the last ring —
make it generally available instead. Every one of these changes is written to the platform's logs with the release, the ring and
who made it. Each email is recorded with how it went — sent (and to whom), nobody to tell (the organization has no owner or admin),
or failed (with the address and the mail provider's words) — and the Releases page counts the notices that could not be emailed. A
release withdrawn and published again tells its organizations again.

An organization's **Settings** page shows the rings it is in. An organization a release has not reached is not offered it, and
asking for the move anyway is refused with why:

> Platform release 0.11.0 is in early access, and Acme is not in a ring it has reached. It is offered to Acme when a platform
> operator widens it to a ring Acme is in, or makes it generally available.

A team's first deployment into an environment runs on the newest release its organization has been reached by.

## How do I upgrade from the console? {#console}

Open the application, choose the **Platform** tab, press **Upgrade** on the environment and **Move to** the release. The
page then shows the move's stage — `DryRun`, `BackingUp`, `Moving`, `Deploying`, `Succeeded` or `Failed` — with what each
stage measured (how long it took, how many migration statements were planned and applied), the backup taken before the move
(its size and when) or why it could not be taken, the reason when it failed, and who was told.

## How do I upgrade from a script? {#rest}

```bash title="ask for prod to move to a newer release" syntax
curl -X POST https://admin.<your-domain>/api/rest/v1/infra/releases/move-environment \
  -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" \
  -d '{"applicationSlug":"billing","environmentSlug":"prod","releaseVersion":"0.10.0","reason":"the autumn release"}'
```

The answer is the move's id. The move is asynchronous: its stages run on the zone.

## What happens when a step fails? {#failure}

| fails at | the environment | who is told |
|---|---|---|
| the dry run (the application does not compile under the release, or a migration does not plan) | stays on its release and keeps serving; nothing was written | the organization's owners and admins, with the reason |
| the rehearsal is red (a failing test, a migration the copy refuses, or something the application reads that the copy lacks) | stays on its release; nothing was written to its database | the application's team, with each failing test and its message |
| the backup (the database cannot be reached, or the zone has nowhere to store it) | stays on its release; nothing was migrated, and the failed attempt is recorded with its reason | the organization's owners and admins, with the reason |
| the move (a migration the database refuses) | stays on its release; the transaction is rolled back, so its database is as it was | the organization's owners and admins, with the database's reason |
| a platform release drops a member this installation never recorded as deprecated (the release that deprecated it was skipped) | the move goes on, but that one drop is refused: nothing is dropped and its data is kept; the deprecation is recorded at the target release, so a later release can drop it | the operators, by a critical alert that names the member and why |
| the deployment (the release's runtime does not serve it) | is rolled back onto the release it was served on | whoever asked, and the operators |
| readiness — the runtime of the release is up, but cannot load this version's model or open its database | the deployment fails before any request reaches it, and is rolled back onto the version and release it replaced, with the runtime's own words as the reason | whoever asked, and the operators |
| the zone never answers a stage — the backup included — within 30 minutes | stays on its release | the organization's owners and admins |
| a runtime of a release is sent an application whose model another release built (a route naming the wrong release, or a release changed without the move) | the runtime refuses to serve it and never rebuilds it; the refusal travels in its zone's report, so the runtime needs no access to the control plane's registry; the alert resolves once every environment it names is healthy on the release it is set to run | the operators, by a critical alert naming both releases |

A **forced** move that fails takes the environment offline instead, with the reason, because it may not stay on the release
it had to leave. Deploying a version that runs on the release brings it back.

Every stage, refusal and outcome is written to the platform's logs with its figures, under the application's name.

## What happens to a workflow run that is waiting? {#waiting-runs}

A move changes the release an environment runs on, not its data. A workflow run that was waiting — for an event, a claim or a
timer — is still waiting when the move converges, in the environment's own database. When what it waits for arrives, a runtime
of the newer release resumes it, on the model rebuilt for that release, and it completes there.

Each run records the platform release that last moved it. A run resumed by a runtime of another release is logged with both:

```text title="a waiting run resumed after its environment moved" syntax
Workflow run 6f1c… parked on platform release 1.4.0 is resumed on release 1.5.0, entering state Done: its placement moved to 1.5.0 while it waited, and it continues on that release's model
```

A run resumed on the release it waited under logs nothing.

## Is the database backed up before a move? {#backup}

Yes, every time a move is made. The move migrates in one transaction, so a move that **fails** leaves the database exactly
as it was. A move that **succeeds** and turns out later to have lost or changed data is different: only a backup taken
before it can bring that data back — and a rehearsal's copy is dropped when the rehearsal ends, so it is no recovery point.

So after the dry run — and after the rehearsal, or the team's approval, when the move has one — a runtime of the newer
release backs up the environment's database into the zone's backup store before it migrates anything:

| | |
|---|---|
| what is backed up | the environment's own database — the one the move migrates |
| where it goes | the **data zone's** backup store when that zone has one, so a copy stays where the data is; otherwise the compute zone's, in the bucket its backup provider names (`placement-backups` when it names none). A backup that would land outside the data zone's jurisdiction warns the organization and is logged — it never blocks the move |
| what is recorded | a backup of the application in that environment, taken before a migration: when, its size, its SHA-256, and where it is — or why it could not be taken |
| how long it is kept | 30 days |
| when it cannot be taken | the move is not made, the environment stays on its release, and the organization's owners and admins are told why; a **forced** move is taken offline instead, as for any stage it cannot pass |

A dry run takes no backup: it writes nothing. The application's **Platform** page shows each move's backup, and the emails
about a move that went ahead mention it. Restoring one is a platform operator's action.

## Is a move rehearsed first? {#rehearsal}

When upgrade rehearsal is included for an environment, a move is rehearsed after its dry run and before it is made. On the
environment's zone, the newer release:

1. copies the environment's database while it keeps serving;
2. runs the move against the copy;
3. checks that every table and column the application reads is there;
4. runs the tests the version was deployed with, under the newer release;
5. drops the copy, whatever happened.

| the rehearsal | what happens to the move |
|---|---|
| green | it is made — or, when the platform started the move and the application's upgrade policy is Manual, it waits for the team's approval |
| red | it is not made; the environment stays on its release, and each failing test is shown with its message |
| skipped — the database is larger than the zone copies, or the zone cannot copy it | it waits for a person to approve or decline it |

A version carries its tests when it is deployed from a project whose manifest declares them, so the rehearsal runs exactly the
tests the team shipped. A version deployed without tests is rehearsed without them.

Upgrade rehearsal is a platform feature the operators grant. An offering can include it, and an organization can be granted it
or have it withheld, which overrides its offerings. The application's **Platform** page says, for each environment, whether its
moves are rehearsed and why not. A move without a rehearsal is still dry-run, applied in one transaction with rollback, and
checked for readiness.

## How does an application take a new release? {#policy}

Its team chooses the upgrade policy on the application's **Platform** page:

| policy | when the platform rehearses a newer release |
|---|---|
| each move waits for your approval (the default) | the move waits for the team to **Approve move** or **Decline move** |
| move when the rehearsal is green | a green rehearsal makes the move on its own — offered only where upgrade rehearsal is included |

Approving or declining a waiting move is the team's: its organization's owners and admins, or a member of the application.

## How long can an application stay on an older release? {#grace-window}

What the platform does on its own depends on the release's class, and it happens for an organization **once the release reaches
it** — through its [release ring](#early-access), or at general availability. A release for the operators only reaches nobody, so
nothing happens.

A **patch** release is followed: once it reaches the organization, every environment on an earlier patch of its line is moved to
it, within the hour. Each environment is asked once per release; one whose move fails is not asked again every hour, and its
team is told why. Because nothing asks again, that environment goes on running the earlier patch — without whatever the patch
fixed — until its team moves it or a later minor or major release's window does; the operators are alerted too, so a patch
stalling across many applications is visible rather than one line in a log. That alert resolves when the environment reaches
the patch, however it gets there.

A **minor** or **major** release opens an **opt-in window** once it reaches the organization. Until the window's deadline the
team moves each environment whenever it chooses. The window is:

| release | window |
|---|---|
| minor | 30 days |
| major | 60 days |

The platform operators can set another length for a class of release, or for **one release**, on the **Releases** page of the
infrastructure console. Setting it for one release moves the deadline of every window of that release still open — to when the
window opened plus the new length, never sooner than a day away — and tells each team the new date.

| when | what happens | who is told |
|---|---|---|
| the window opens | a deadline is recorded for the environment, and shown on the application's **Platform** page and the organization's **Environments** page | the application's team and the organization's owners and admins |
| 14 days before the deadline | a reminder, once | the same |
| 3 days before the deadline | a reminder, once | the same |
| 1 day before the deadline | a reminder, once | the same |
| the deadline passes | the environment is **at risk**, and the platform **forces** the move — rehearsed when rehearsal is included | the application's team |
| that move fails — the model does not compile on the release, or a stage fails — or it cannot be asked for | the environment is taken offline at once, with the reason | the organization's owners and admins, emailed the diagnostics and how to bring it back; the operators, by a critical alert |
| the environment reaches the release | the deadline is met, and the alert resolves | — |
| the hourly sweep cannot handle one application — its window cannot be opened, or its deadline cannot be handled | the sweep goes on to every other organization, and that one is left on the release it runs, never reminded and never forced; the step is tried again within the hour | the operators, by an alert naming the application and the release, which resolves when that step succeeds |

Each notice is sent once and records how its email went to each recipient. An address the mail provider refuses is tried again
on the next hourly sweep — only that address — for up to a day.

A window shorter than a reminder's lead has no such reminder: a 10-day window has no 14-day reminder. When the release stops
reaching the organization — it is withdrawn, or the organization is taken out of its ring — its open windows close without
forcing anything, and open afresh if the release reaches the organization again.

**Withdrawing goes one step further than a ring change**, because a withdrawn release is one nobody may move to at all: it also
closes a window whose deadline had already FAILED, resolves the critical alert that deadline raised, and brings the environment
back online when that deadline is what took it offline. An environment an operator took offline for some other reason stays
offline — withdrawing a release says nothing about that reason. Taking an organization out of a ring closes open windows only:
the release still exists for everyone else, so a failed deadline is still owed.

To bring an environment back after a failed deadline, upgrade the application locally on the new release, fix what the
report names, and deploy it. Once that version serves, the platform forces the move again, and when the environment reaches the
release the deadline is met and the alert resolves. The one case you need do nothing for is the release being withdrawn, above.

A platform operator can also force one environment onto a release outside any window. When that move fails before its
deployment, the environment is taken offline and the operators are alerted in the same way.

Every move records what started it — its team, a patch release, a window's deadline, an operator or a rehearsal — together with
the ring the release reached the organization through and the reason. The platform's logs carry the same, with the release and
the placement.

An application is at risk, never a whole environment: applications in one environment may run different releases while
their windows are open. The organization's **Environments** page shows, for each environment, how many applications run
each release and how many are at risk, and can move every application in an environment to a release — each one's own
move, which each application's team may make.

## See also       {#see-also}

- [Deploying an application to a cloud platform — the zone builds it](https://osysharp.com/reference/zones/deploying-an-application/) — deploying a version, which the zone builds
- [Deploying your app to an environment](https://osysharp.com/reference/local/deploying-your-app/) — the `osy deploy` and `osy status` verbs