Pushed mail — Gmail through Pub/Sub, Microsoft 365 through Graph notifications

Summary#

Two lines turn polling into push — where the app is told to send notifications, and the door they arrive at:

app.Mail = new MailSetup { Mailboxes = [ … ], Receiver = …, PushPath = "/api/rest/v1/mail-push" };

app.Apis = [ new RestApi("MailPush") { Route = "mail-push", Auth = new ApiAuth { Anonymous = true },
  Endpoints = [ new Endpoint(MailboxPushed) { Method = HttpMethod.Post, Path = "/{provider}", Raw = true,
    Query = ["validationToken"], ContentType = "text/plain", OidcIssuer = "https://accounts.google.com" } ] } ];

Each mailbox that can push is then subscribed as soon as it is watched (WatchMailbox), renewed before it lapses, and still asked every half hour.

Signature#

// Osysharp.Mail 0.6.0 — on IMailbox, every member defaulted (a mailbox without them is polled)
bool CanPush();
MailboxSubscription Subscribe(string notifyUrl, string secret);
MailboxSubscription Renew(MailboxSubscription current, string notifyUrl, string secret);
void Unsubscribe(MailboxSubscription current);
List<MailboxNotice> ReadPush(string body, string notifyUrl);
TimeSpan PollEvery();         // when nothing pushes — a minute by default, five for IMAP
TimeSpan SafetyNetEvery();    // while pushes arrive — 30 minutes

// MailSetup
string PushPath;              // the push door's path; the provider's key is appended ("…/mail-push/gmail")

// Osysharp.Mail.Receiving 0.2.0
string MailboxPushed(string provider);   // the push door — publish it in app.Apis at PushPath

// Osysharp.Mail.Gmail 0.2.0 — GmailMailbox
string PushTopic;             // "projects/<project>/topics/<topic>"
string PushServiceAccount;    // the push subscription's service account (its OIDC token's email)
string PushAudience;          // optional — Pub/Sub's default is the push endpoint's own URL

// Osysharp.Mail.Microsoft365 0.2.0 — Microsoft365Mailbox
bool Push = true;             // false: polled

Description#

What a push does — and does not#

A notification carries no mail. It moves the mailbox's watch to Woken; five seconds later the watch fetches from its cursor exactly as a poll does, and every push that arrives in those seconds is the SAME fetch — twenty mails in a burst are one conversation with the provider. The fetch runs on the watch's own clock rather than inside the push, because a provider gives its push a few seconds to be answered (Graph: three).

A notification is also not trusted for anything but "look". What is kept is what the provider's own API returns when asked, so a forged push could at worst cause a harmless extra fetch — and forged pushes are refused anyway:

Renewal — a durable workflow, well before the lapse#

Gmail's watch lasts seven days; a Graph subscription to mail about three. The watch's workflow checks twice a day and renews a subscription past HALF its life (Gmail: users.watch again; Graph: PATCH its expiry, which keeps its clientState; a subscription Graph has lost is made again). Graph's lifecycle notices are honoured: reauthorizationRequired renews on the next fetch, subscriptionRemoved makes a new one, missed fetches. A subscription that cannot be made is said on the watch (MailboxWatch.PushProblem) and the mailbox is POLLED at its own interval until it can — it is never left unwatched.

The safety net, and the plan#

While push is live the mailbox is asked every SafetyNetEvery() — half an hour — so a notification the provider dropped makes a mail late, never lost. A mailbox without push is asked every PollEvery(). Both are computed, so the plan's minimum interval holds them at run time ([[scheduling-schedule#minimum-interval]]): on a free plan neither runs more often than every 15 minutes — which is why push is the path that keeps a free desk prompt.

What the operator sets up — Gmail#

Once per Google Cloud project, outside the app:

1. Enable the Gmail API and the Cloud Pub/Sub API.
2. A topic — projects/<project>/topics/<topic>. Grant [email protected] the role Pub/Sub Publisher on it (Gmail publishes as that account).
3. A push subscription on the topic: endpoint https://<app's host>/api/rest/v1/mail-push/gmail (the app's PushPath + /gmail), authentication enabled with a service account of your own (its email is the mailbox's PushServiceAccount); leave the audience empty (it is then the endpoint URL) or set it and give the mailbox the same PushAudience.
4. The OAuth client the inbox is connected with (ClientId/ClientSecret), its consent screen listing gmail.readonly and gmail.send. Connect the inbox from a page (Connect an account from a page — Security.OAuthConnectUrl) or set its refresh token as a secret.

Set PushTopic and PushServiceAccount on the GmailMailbox. The watch subscribes when it starts; users.watch is called with the topic and the INBOX label.

What the operator sets up — Microsoft 365#

1. An app registration in the tenant with the application permissions Mail.Read and Mail.Send (admin consented), scoped to the support mailbox by an Exchange application access policy — the same registration that reads and sends.
2. Nothing else: Graph calls the push door directly, and the door answers its validation handshake (it POSTs ?validationToken=… and expects it back as text/plain within ten seconds). The door must be reachable over public HTTPS — which a local platform is not (use the poll, or a tunnel, to try it locally).

The push door is the app's#

A package opens no routes, so the APP publishes MailboxPushed at its PushPath: open at the transport (Anonymous = true — a provider has no account here) and proven in the body, as above. The endpoint declares what the providers need and nothing more: the raw body, Graph's validationToken, a text/plain answer, and Google as the issuer whose token the platform verifies (Endpoints a pushing provider calls — Query, ContentType, OidcIssuer).

Logs an operator reads#

Every push the door could not act on, every subscription made or renewed (with its expiry), every refusal from the provider and every forged notification is logged with the mailbox's address; a push token that did not verify is logged by the platform with WHY (no token, a key Google does not publish, expired, another issuer). On the watch row: PushProblem, PushExpiresAt, LastPushAt, Pushes, Problem, LastPolledAt — what an admin page shows (Apps/harbour's Settings → Mailboxes).

Examples#

A desk that reads a Google Workspace inbox by push — the mailbox, the door, and the grants:

app Desk {
  model "model/**/*.osy";
  use Osysharp.Identity@0;
  use Osysharp.Mail@0;
  use Osysharp.Mail.Receiving@0;
  use Osysharp.Mail.Gmail@0 { egress "oauth2.googleapis.com"; egress "gmail.googleapis.com"; }
  use Osysharp.Workflow;
  use Osysharp.Storage;
  use Osysharp.Http;
}

using Osysharp.Mail;
using Osysharp.Mail.Gmail;
using Osysharp.Mail.Receiving;

[Role] enum AppRole { Authenticator, Admin }

entity Request {
  [Required, MaxLength(300)] string Subject;
  security { allow read when IsAuthenticated; }
}

public class Requests : IMailReceiver {
  public void Received(MailMessage mail, MailReceipt receipt) { new Request { Subject = mail.Subject }; }
}

app.Secrets = [ new Secret("GoogleClientSecret"), new Secret("SupportMailboxToken") ];

app.Mail = new MailSetup {
  Mailboxes = [new GmailMailbox {
    Address            = "[email protected]",
    ClientId           = "1234-abc.apps.googleusercontent.com",
    ClientSecret       = Security.IsSecretSet("GoogleClientSecret") ? Secret.GoogleClientSecret : "",
    RefreshToken       = Security.IsSecretSet("SupportMailboxToken") ? Secret.SupportMailboxToken : "",
    PushTopic          = "projects/acme-desk/topics/support-mail",
    PushServiceAccount = "[email protected]",
  }],
  Receiver = new Requests(),
  PushPath = "/api/rest/v1/mail-push",
};

app.Apis = [
  new RestApi("MailPush") {
    Route = "mail-push",
    Auth  = new ApiAuth { Anonymous = true },
    Endpoints = [
      new Endpoint(MailboxPushed) {
        Method = HttpMethod.Post, Path = "/{provider}", Raw = true,
        Query = ["validationToken"], ContentType = "text/plain", OidcIssuer = "https://accounts.google.com",
      },
    ],
  },
];

partial entity MailboxWatch { security { allow read, create when IsAuthenticated; } }
partial entity ReceivedMail { security { allow read when IsAuthenticated; } }
// Who manages the mailboxes on the kit's settings page — the app's `IsAdmin` by default; this example has no admins.
policy ManagesMailboxes => IsAuthenticated;

public MailboxWatch StartReading() { return WatchMailbox("[email protected]"); }

See also#

• Receiving mail — watched mailboxes (Gmail, Microsoft 365, IMAP) — watching a mailbox: what is kept, the receiver, retries, erasure
• Endpoints a pushing provider calls — Query, ContentType, OidcIssuer — Query, ContentType, OidcIssuer: the endpoint seams a pushing provider needs
• Connect an account from a page — Security.OAuthConnectUrl — connecting the inbox from a page, its grant kept as an app secret
• Schedule (recurring work) — the plan's minimum interval, at compile time and at run time