Importing data

Summary#

Loads the rows your app ships with — a catalogue, a country list, tax rates, whatever it needs to be useful on first open — from JSON files beside its source into the app on the local platform.

osy compile ships your app's code; this ships its rows. Together they make "copy this app and it works" true of both halves, instead of opening on an empty page.

Signature#

osy import [path] [--file <path>] [--replace] [--as <login>] [--password <pw> | --password-file <path> | --password-stdin] [--code <code>] [--json]

Description#

Where the files come from#

Your manifest declares them, the same way it declares source:

data "data/**/*.json";

Files load in path order, and that order matters: a row that references another resolves against rows already imported, so data/suppliers.json loads before data/products.json. Naming your files is how you control it. --file <path> imports just one file (and repeats), ignoring the glob. A path is read from where you are, then from the project; one that names no file stops the run, by name, before anything is imported.

Running it twice is the point#

Every file names the property that identifies a row, and import upserts by it. Running the same import again updates what changed and leaves the rest alone — which is what makes it usable in a build loop and in CI, rather than a one-shot that doubles your catalogue.

key also takes a LIST, which is how you seed a row that no single property identifies. A link between two things — a membership, a dependency, an assignment — is identified by both ends together and by neither alone, so one property cannot key it:

{
  "entity": "Dependency",
  "key": ["Dependent", "Prerequisite"],

  "lookups": { "Dependent": "Task[Code]", "Prerequisite": "Task[Code]" },

  "rows": [
    { "Dependent": "SHIP-2", "Prerequisite": "SHIP-1", "Kind": "Blocks" }
  ]
}

The rule is the same one as for a single key — the combination has to identify the row — so it upserts and re-runs clean. Without this, a join entity has to grow a synthetic identifier that exists only to satisfy the importer, which is a schema bent around a tool.

An empty key part is still part of the key. A row whose "Prerequisite": null (a dependency on something outside the file) is matched on re-run by its other parts and that empty one, exactly as a filled one is: two empty values count as the same value, so running the file twice still leaves one row — not SQL's "null never equals null".

--replace deletes every existing row of each imported entity first. It destroys rows your files do not mention, so it is never the default.

⚠ An upsert needs update granted, and the first run will not tell you. An entity whose security {} grants create but not update imports perfectly the first time and fails the SECOND, when the rows already exist:

✗ data/03-policies.json row 1: Update of 'ApprovalPolicy' denied — it declares a `security { }` block, and that
  block grants no `update`. An entity with a block denies every operation the block does not name.

The fix is a rule in the model, never a flag here — reference data an operator re-seeds is data an operator may edit, so say so: allow update when IsOps. ⚑ The same shape bites in the other direction: an entity that grants create but not read cannot be MATCHED, so every run creates and the second one doubles the table. Both come from the same place — an upsert is a read, then a create or an update, and a block that names only create grants one third of it.

Files a row points at#

A row can reference an image or a document by a path relative to its own data file. Those bytes travel with the import, so an imported product photo is served from the app afterwards rather than 404ing.

The envelope's "files" map names which columns are paths and where their files are stored. The stored path is the prefix plus the file's name — the folders beside the data file are where you keep it, not where the app serves it:

{ "entity": "ProductImage", "key": ["Product", "Rank"],
  "files": { "Path": "public/products" },
  "rows": [ { "Product": "kiambu-aa", "Rank": 0, "Path": "photos/kraft_bag.jpg" } ] }

That row is stored with Path = "public/products/kraft_bag.jpg", and the photo is served from that path. A file column must store under public/ — the only files a browser is served without a login — or under private/, which you ask for by name: the file is stored and served only through a link your code signs (File.SignedUrl), as a file a shop sells is ("files": { "DownloadPath": "private/downloads" }). Any other prefix is refused. A Markdown column is the exception: its file becomes the document's text and is not stored at all.

Rows that are already somewhere in a workflow#

A row whose entity a workflow tracks — an Order whose Status is OrderFlow's — is imported in one of two ways, and the file decides which:

• The row leaves the field out. It is a new row like any other: if the workflow autostarts, its run begins at Initial and that state's enter {} runs — the stock is taken, the confirmation mail goes out.
• The row states the field. It is adopted at that state: the row and a run placed at that state, as if its workflow had brought it there. No entry of the workflow runs — importing last year's delivered orders takes no stock and sends no mail.

{ "entity": "Order", "key": "Reference",
  "lookups": { "Product": "Product[Sku]" },
  "rows": [
    { "Reference": "2024-0117", "Product": "mug", "Status": "Delivered" },
    { "Reference": "2024-0118", "Product": "mug", "Status": "Shipped" },
    { "Reference": "new-one",   "Product": "mug" }
  ] }

An adopted run behaves from then on like one that arrived by transition. What the state offers is armed from the moment of the import — its slots (who may act next), its Expire clock, the workflow's Deadline — so a later event reaches it: a return raised on 2024-0117 moves it to Returned. The run's history begins with one line, Imported, reading imported at Delivered. A terminal state (Closed) is adopted as a finished run.

What is refused, with the reason, before the row is kept:

• A state the workflow does not declare — the message names the ones it does: 'Teleported' is not a state of workflow 'OrderFlow' (which owns Order.Status) — a row may be imported at Placed, Shipped, Delivered, Returned, Closed.
• A state whose arming fails for this row — an expression the state evaluates over the row (a slot's Assignee, an Expire budget) throws on the data the row carries. The row is taken back out, so it never stands in a state no run holds, and the error carries the reason.
• Moving a row that already exists. Re-importing the same file is fine — the row matches by key and its state is unchanged. Importing it again with a different state is refused: an existing row's run moves it, by an event.

An entity no workflow tracks imports exactly as before. Adoption is the import's alone: a client — a page, a data endpoint, a function — that creates a row in a chosen state is still refused at commit, because where a new row starts is its workflow's Initial.

Does a test run get these files?#

osy test never imports — a test's rows are what its [TestFixture] seeds. It does carry the data role's stored files: every file a "files" column names travels with the run and is placed in the test branch's file store at the same path the import would give it, before the first fixture runs. So a fixture that seeds the row the import would have written — new ProductImage { Product = kiambu, Path = "public/products/kraft_bag.jpg" } for the file above — finds its photo there: File.ReadAllBytes reads it, and under osy test --pixels the product card shows the picture instead of its alt text.

Only files travel, never rows, and a Markdown column's file does not (it is never in the store). osy debug-test carries them the same way. If the files add up to more than 16 MB they are left behind and the run says so — the tests still run, and a fixture row pointing at one finds nothing.

Who it imports as#

An import is an ordinary data write, so your app's security { } decides every row, on every run. --as chooses which principal the rows are written as; it never chooses whether the rules apply. There is no privileged import.

With no --as, the import acts as anonymous inside your app. Being the app's developer authorises you to call the import — it is not an identity inside the app, so it grants nothing to the rows themselves. An entity that admits anonymous writes is seeded; one that does not is refused, row by row, with the reason.

$ osy import
✗ data/notes.json row 1: Create of 'Note' denied — it declares a `security` block

That is the common case for a real app, and --as <login> is the answer: it resolves one of your app's own users and writes as them, so the rows land exactly where that user may write. <login> is whatever app.Auth binds as its login field. Add the user first — Adding an account writes a real account through the app's own declared fields, with the roles the import will need.

--as is a login, not a label: it takes that user's own password, because naming a principal must never be enough to become one. Omit --password and you are prompted; there is no waiver, in dev or anywhere else. With nobody to prompt (--json, a script), pass it as --password-file <path> (a file only you can read) or --password-stdin (piped in) — both keep it out of the process list that a --password value sits in.

If your app's sign-in asks that account for a second step — a code from an authenticator app — the import asks for it too, after the password is accepted; --code <code> passes it up front. It goes through your app's own code step, exactly as its sign-in page does.

$ osy user add [email protected] --password 's3cret' --role Admin
$ osy import --as [email protected] --password 's3cret'
✓ data/notes.json: 3 created   (ran as [email protected])

Every run reports which principal it used, so an import that seeded as an admin can never be mistaken for one that proved an ordinary user could do it. That second reading is the other reason to use --as: pointing it at a customer's own login checks that an import they will run is one they are allowed to run.

Against a deployed app#

osyrin app import is the same command against a platform you are logged in to, with the same --as, --password (or --password-file / --password-stdin) and --code.

Examples#

osy import                                  # everything the manifest's data glob matches, as anonymous
osy import --file data/products.json        # just this file
osy import --as [email protected] --password 's3cret'   # as one of the app's own users — needed for anything it gates
osy import --replace --json                 # start from scratch, machine-readable result

See also#

Running a function — run one of the app's functions, the other half of putting it in a known state.

Compiling your app — ship the code these rows belong to.

The inner loop — where this sits in the build-run-look loop.