Testing model calls without a model — ScriptedModel (Llm kit)

Summary#

// where the app's models are named — in a test app, or in a kit's own test project
new LlmModel { Name = "Fast", Client = new ScriptedModel { Name = "Fast" }, Model = "scripted-fast" }

// in a test
ScriptedAnswer.Stub((model, request) => Scripted.Text("A short summary.", 1000, 200));

Signature#

class ScriptedModel : ILlmClient, IDecisionModel { string Name; }

LlmResponse ScriptedAnswer(string model, LlmRequest request)            // what a scripted model answers — stub it
LlmDecision ScriptedDecision(string model, LlmDecisionRequest request)  // what a scripted decision model decides — stub it

LlmResponse Scripted.Text(string text, int inputTokens = 100, int outputTokens = 20)
LlmResponse Scripted.ToolUse(string tool, string inputJson, string id = "call-1", int inputTokens = 100, int outputTokens = 20)
LlmDecision Scripted.Decision(string valueJson, decimal confidence, int inputTokens = 100)
string      Scripted.Asked(LlmRequest request)                           // the text of the last user message

bool GenerationIsOn()                                                    // the operator's AI switch — stub it
bool EveryModelIsScripted()                                              // false — stub it to script a real app's models

Description#

What is real, and what is scripted?#

Only the provider. A call still passes the operator's switch, is routed by the app's router, reserves against its budgets, falls back when the script throws, and writes its meter rows at the app's prices. So the thing a test asserts — who served, what it cost, what was refused — is the thing the app does.

How does a script answer?#

Stub ScriptedAnswer with a lambda taking the model's Name and the LlmRequest. It returns an LlmResponse — Scripted.Text and Scripted.ToolUse build one — whose token counts are whatever the script says were used, which is what the meter then prices. A script can answer differently per model (model == "Small"), per turn (request.Messages.Count), or from what was asked (Scripted.Asked(request)).

Unscripted, a ScriptedModel fails the way a provider does, with a message saying which stub is missing — so a test that forgot to answer is told so.

My app names a real provider. How do its tests use the script?#

Say so in the test: EveryModelIsScripted.Stub(() => true);. Every model of the app — whatever its Client is, an AnthropicClient with a key the test does not have — is then answered by ScriptedAnswer under the model's own Name (and ScriptedDecision, for a Decider). Nothing goes out; the router, the budgets and the meter at the app's prices are as real as ever.

// app.osy — what the app runs on
app.Llm = new LlmSetup { Models = [ new LlmModel { Name = "Reader", Client = new AnthropicClient { ApiKey = Secret.Anthropic },
                                                   Model = "claude-haiku-4-5" } ] };

// a test
EveryModelIsScripted.Stub(() => true);
ScriptedAnswer.Stub((model, request) => Scripted.Text("Read by " + model));   // model == "Reader"

Outside a test nothing answers it but false: a stub is a test's alone.

How do I play a provider that is down?#

Throw ExternalServiceException from the script. The call moves to the router's next model, exactly as it would in production, and the failed attempt is on the meter.

And the operator's switch?#

GenerationIsOn.Stub(() => false) — every call is then refused with LlmSwitchedOff before anything is sent.

A streamed call?#

Llm.Stream over a ScriptedModel plays the scripted answer as events: its text as TextDelta, a tool call as its three events, and the closing MessageComplete with the script's counts.

Examples#

app Bakery {
  use Osysharp.Llm;
  model "model/**/*.osy";
}

using Osysharp.Llm;

app.Llm = new LlmSetup {
  Models = [
    new LlmModel { Name = "Small", Client = new ScriptedModel { Name = "Small" }, Model = "scripted-small",
                   Price = new LlmPrice { Input = 1, Output = 5 } },
    new LlmModel { Name = "Large", Client = new ScriptedModel { Name = "Large" }, Model = "scripted-large",
                   Price = new LlmPrice { Input = 10, Output = 50 } },
  ],
  Router = new SmallThenLarge(),
};

public class SmallThenLarge : ILlmRouter {
  public LlmRoutingDecision Route(LlmTaskFacts facts) {
    return new LlmRoutingDecision { RuleId = "bakery:1", Ordered = ["Small", "Large"] };
  }
}

policy ReadsModelUsage => true;

string Describe(string loaf) { return await Llm.Complete("Describe this loaf for the shop window: " + loaf); }

[Test]
void The_script_answers_and_its_counts_are_what_the_meter_prices() {
  ScriptedAnswer.Stub((model, request) => Scripted.Text("Dark rye, baked this morning.", 400, 100));
  Assert.Equal("Dark rye, baked this morning.", Describe("rye"));
  Assert.Equal(900, ModelCall.Single().CostMicros);         // 400 at $1 a million and 100 at $5
}

[Test]
void A_script_that_throws_is_a_provider_that_is_down() {
  ScriptedAnswer.Stub((model, request) => {
    if (model == "Small") { throw new ExternalServiceException("scripted-small is down"); }
    return Scripted.Text("from " + model + ": " + Scripted.Asked(request));
  });
  Assert.Equal("from Large: Describe this loaf for the shop window: rye", Describe("rye"));
  Assert.Equal(1, ModelCall.Count(c => c.Outcome == ModelCallOutcome.Failed && c.Model == "Small"));
}

[Test]
void An_unscripted_model_says_which_stub_is_missing() {
  var fault = Assert.Throws<ExternalServiceException>(() => Describe("rye"));
  Assert.Contains("ScriptedAnswer.Stub", fault.Message);
}

[Test]
void The_switch_is_a_stub_too() {
  GenerationIsOn.Stub(() => false);
  Assert.Throws<LlmSwitchedOff>(() => Describe("rye"));
}

See also#

• The Llm kit — call a model, and know what each call cost — the call path a scripted model stands in
• Model spend — budgets, the meter and the price book (Llm kit) — asserting budgets and costs
• Typed answers from a model — Llm.Extract<T> and Llm.Decide<T> (Llm kit) — scripting Llm.Extract<T> and Llm.Decide<T>
• Stubbing your own functions in a test (fault injection) — F.Stub(…) itself