Osy#betaRun it on Osyrin: osyrin.com

Reference / Testing

Running only the tests a change reaches, and what no test covers

osy test [path] --affected [<base>] | --changed <path> [--list] [--json] · osy test [path] --coverage [<base>] [--json]

`osy test --affected main` runs only the tests whose call graph reaches something you changed — every test, with the reason named, when the change reaches something the graph cannot place. `osy test --coverage main` says which new or changed functions no test reaches, and which ones a test reached on main and none reaches now.

stabletestingclilocal

Summary#

The compiler already knows which functions call which: a test calls Price, Price calls Discount, a page at /cart shows Price. osy test uses that graph two ways.

  • --affected <base> runs only the tests that can reach a declaration that changed since <base> — a function, a page, a handler, a test. A change the graph cannot place (an entity, the seed, the manifest …) runs every test and names the rule that made it.
  • --coverage [<base>] runs nothing and reports which functions and components the tests reach. Against a base it names the new or changed ones no test reaches, and the ones a test reached on the base and none reaches now.

Signature#

osy test [path] --affected [<base>] [--list] [--json]
osy test [path] --changed <path> [--changed <path> …] [--list] [--json]
osy test [path] --coverage [<base>] [--json]

<base> is a git commit (main, HEAD~3: everything since it, uncommitted work included) or a range (a..b). With no <base>, --affected measures from where your branch left origin/main. Put the project path before the flag: the word after --affected or --coverage is read as the base.

Description#

What "reaches" means#

A test reaches every declaration its own text names, everything those name, and so on — there is no limit on the distance. A few things count as reaching even though they are not calls by name:

  • a page by its route — Ui.Visit("/cart") reaches the page declared [Route("/cart")];
  • a handler by its event — a test that reaches Ship, which raises Shipped, reaches the [On] handler that hears Shipped;
  • a method by its name — order.Total() reaches the class that declares Total;
  • what derives from what is reached — a slot reaches its contributions, a base its derived types;
  • the fixtures a test forks from — a change to a [TestFixture] reaches the tests that name it, and the tests of fixtures built on it.

Names are matched by their simple spelling, so two unrelated functions that share a name both count. That can only select more tests, never fewer. Comment and whitespace edits are not changes.

What runs everything#

Some changes reach code no call graph shows. Any one of these runs every test, and --list names it:

rulewhat changed
schemaan entity, an enum or an interface — the shape every stored row and every caller has
seeda file of the seed role — every branch starts from it
migrationa migration
manifestapp.osy, a using or a namespace
configosyrin.json, osyrin.lock, .secrets
settingan app setting (app.X = …)
runtimea policy, a workflow, a principal, a layout, an [OnStart] function, a handler that hears every event
unplaceda theme, a store, a translation, an asset — anything else no test reaches by a call — or an edit outside every declaration
unparseda source file that does not parse
outsidea file outside the project that the project is built from — a kit it uses, or the platform itself

Prose (.md) and .gitignore reach nothing.

--list — see the selection, run nothing#

$ osy test --affected main --list
--affected main: 3 of 5 test(s) can reach the change.
  [reaches] 1 changed declaration(s) reach 3 of 5 test(s)
  · tests/shop.test.osy::the_price_is_discounted
      ← function Discount (changed, model/shop.osy:6)
  · tests/shop.test.osy::a_basket_is_priced
      ← function Discount (changed, model/shop.osy:6)
  · tests/shop.test.osy::the_cart_page_shows_the_price
      ← function Discount (changed, model/shop.osy:6)

--list composes with --filter, --file and --test, and works without them too (it then lists what a plain run would run). It needs no platform: the answer comes from the source alone, in well under a second.

Running the selection#

Without --list, osy test --affected main runs the tests it selected, and the run's last line says it was narrowed — ran 3 of 5 — FILTERED by --affected main — so a narrowed green is never read as the whole suite passing. A change that reaches no test is an answer, not an empty pass: the command says so and exits 0 without starting a platform.

--changed <path> is for a tool that already knows the change set. With no base to compare against, every declaration in a named file counts as changed.

--coverage — what no test reaches#

$ osy test --coverage main
osy test --coverage main: the tests reach 5 of 7 function(s) and component(s) — statically: a declaration counts when a test can reach it.
  ✗ 1 new or changed, and no test reaches it:
      function Refund (added, model/shop.osy:18)
  ✗ 1 reached on the base, and no test reaches it now:
      function Ship (model/shop.osy:8) — was reached by shipping_returns_one

Coverage is static: a function counts as covered when some test can reach it through the graph above, whether or not that test's path actually executes it. So "no test reaches it" is a firm finding, while "covered" means "a test could get there". With no base, --coverage lists every function and component no test reaches.

For a gate: --json#

--list --json and --coverage --json each print one JSON document. Fields are only ever added, never renamed.

{
  "kind": "affected",
  "base": "main, uncommitted work included",
  "everything": false,
  "declared": 5,
  "selected": 3,
  "reasons": [ { "rule": "reaches", "what": "1 changed declaration(s) reach 3 of 5 test(s)" } ],
  "changes": [ { "kind": "function", "name": "Discount", "file": "model/shop.osy", "line": 6, "change": "changed" } ],
  "tests": [ { "id": "tests/shop.test.osy::the_price_is_discounted", "name": "the_price_is_discounted",
               "file": "tests/shop.test.osy", "because": [ "function Discount (changed, model/shop.osy:6)" ] } ]
}
{
  "kind": "coverage",
  "base": "main, uncommitted work included",
  "method": "static",
  "declarations": 7,
  "covered": 5,
  "uncovered": [ { "kind": "function", "name": "Refund", "file": "model/shop.osy", "line": 18, "change": "added" } ],
  "lost": [ { "kind": "function", "name": "Ship", "file": "model/shop.osy", "line": 8, "tests": [ "shipping_returns_one" ] } ],
  "unreached": [ { "kind": "function", "name": "Refund", "file": "model/shop.osy", "line": 18 } ],
  "changes": [ { "kind": "function", "name": "Refund", "file": "model/shop.osy", "line": 18, "change": "added" } ],
  "notes": []
}

When everything is true, every test is in tests and reasons names each trigger. A gate that wants "new code must be tested" fails when uncovered is not empty. A gate that wants "coverage must not shrink" fails when lost is not empty.

Examples#

osy test --affected main --list          # what would run, and why — no platform needed
osy test --affected main                 # run it
osy test --affected HEAD~1 --json        # the last commit's tests, as a stream of events
osy test --changed model/orders.osy      # a tool that already knows the changed files
osy test --coverage main --json          # what this branch added that no test reaches

See also#

Running tests locally — osy test itself, and the other ways to choose what runs (--filter, --file, --test).

Fixtures: built once, cloned for every test, and built on each other — fixtures, which a test forks from and which --affected follows.

Related

Running tests locally

Runs your app's tests against a Platform on your own machine — no account, no network, no setup beyond a running local…

Running tests

Runs your app's tests against a throwaway copy of its database, reporting each test as it finishes. Your local source…

Fixtures: built once, cloned for every test, and built on each other

A [TestFixture] is built once per run, and every test that names it runs on its own copy of what it left — rows, files…

Testing (real app, real data, real rules)

A test in Osy# is not a unit test with the world mocked out. It is your app, running against its own throwaway…