data sent somewhere else, as it changes (app.Exports)

app.Exports = [ new WarehouseExport("OrdersToLake") { …, Rows = Order.Select(o => new { o.Number, o.Total }) } ];

`app.Exports` sends an entity's changes to another system — a warehouse, a lake, a SaaS table — as they commit. Each entry names a sink class that implements `IExportSink`, the name the app refers to it by, its settings, and `Rows`: the entity whose changes leave and the columns that leave with them. The platform records every change, keeps the export's place, and hands the sink batches in commit order until it has confirmed them all.

Summary#

Declare what leaves, and through which sink. A sink package supplies the class; the app says where to, with which credentials, and which rows of which entity.

use Acme.Warehouse;

app.Exports = [
  new WarehouseExport("OrdersToLake") {
    Account = "acme-eu",
    Key = Secret.WarehouseKey,
    Table = "ORDERS",
    Rows = Order.Select(o => new { o.Number, o.Total, o.PlacedAt }),
  },
];

Signature#

app.Exports = [ new <SinkClass>("<Name>") { <Setting> = <value>, …, Rows = <Entity>.Select(x => new { x.<Column>, … }) }, … ];

Description#

What leaves#

Rows names one entity and the columns that leave, in the order the destination receives them. Every change to that entity after the export is declared leaves: a create or an update as the whole row — its id, then the named columns as they stood after the commit — and a delete as its id. A change that touches none of the named columns does not leave again. A row of a type that inherits from the entity is a change of it.

Values travel as plain text in the same form a data source's rows use: invariant culture, instants in UTC, decimals exactly. The app's redaction policy (app.Audit.Redact) applies: a redacted column leaves as ‹redacted›.

A change is recorded in the same transaction as the write itself, whether the write is a commit or a set-based Update, Insert or Delete, and whether or not the app keeps a data-change trail.

What a sink class is#

A class implementing IExportSink, with a parameterless constructor and settings as fields:

public interface IExportSink {
  void EnsureSchema(ExportSchema schema);   // make the destination able to receive these columns
  void WriteBatch(ExportBatch batch);       // apply one batch; applying it twice must change nothing
}

ExportSchema lists the columns, each with its type named as a query plan names it (string, decimal, Guid, DateTime, …), and the key column, Id. EnsureSchema is called before the first batch and again whenever the declared columns change. An ExportBatch has an Id and its Changes, each with its Sequence, its Op (Upsert or Delete), its Key and its Values (in the schema's column order; empty for a delete).

Delivery#

The platform delivers each export on its own: after a write that recorded a change, a background job hands the sink the next batch of changes past the export's place, oldest first, and moves the place past them once WriteBatch returns. It runs again while changes remain.

At least once. A batch can arrive twice — after a crash between the sink's write and the platform recording it, or when two deliveries overlap. The batch's Id is its first and last sequence, so the same changes always carry the same id. A sink makes a batch safe to apply twice by keeping each row's Sequence at the destination and applying a change only when its sequence is higher — a warehouse MERGE … WHEN MATCHED AND s.seq > t.seq.
In commit order. Sequences follow the order writes committed, with no gaps, so a higher sequence is always the later value of a row.
A failure keeps the batch. When WriteBatch throws, whatever the sink wrote during it is discarded, the export stays where it was, and the same batch is tried again after 5 seconds, then 10, 20 and so on, up to an hour between attempts. The export's state records the last error.
Nothing owed is purged. A trail Retention window never removes a change an export has not yet delivered.

Delivery needs the platform's background worker, which a hosted or local platform runs.

What is refused#

you write	why
an entry with no name, or the same name twice	the name is what the export's state is kept under
a class that does not implement `IExportSink`	there would be nothing for the platform to call
no `Rows`	an export says what leaves
`Rows` with a `Where`, an `OrderBy` or a `Take`	an export sends every change of the entity
a value computed from a column, or read through another row	an export sends the row's own columns as stored
an entity the app does not store	there are no changes of it to send
a column its entity's `security` block `deny read`s	an export leaves under the app's own authority, and a withheld column is withheld from it too
a `Binary`, `Json`, `RichText`, `Vector` or `TimeSpan` column	an export sends plain values

Examples#

using Osysharp.Exports;

// A sink that accepts everything — the smallest thing that is one.
class LakeExport : IExportSink {
  public string Table = "";

  public void EnsureSchema(ExportSchema schema) { }

  public void WriteBatch(ExportBatch batch) { }
}

entity Order {
  [MaxLength(20)] string Number;
  decimal Total;
}

app.Exports = [ new LakeExport("OrdersToLake") { Table = "ORDERS", Rows = Order.Select(o => new { o.Number, o.Total }) } ];