The Llm kit — call a model, and know what each call cost

Summary#

use Osysharp.Llm@2; gives an app one way to call a model, whichever provider serves it:

var line = await Llm.Complete("Summarise this in one line: " + note);

The app says which models it has; a router says which serves each call; and every attempt — served, failed or refused — is a row on the meter (Model spend — budgets, the meter and the price book (Llm kit)).

Signature#

app.Llm = new LlmSetup {
  Models  = [ new LlmModel { Name = "Fast", Client = <ILlmClient>, Model = "<the provider's id>", Price = new LlmPrice { … } }, … ],
  Default = "Fast",                 // the model a call gets with no router; "" is the first
  Router  = <ILlmRouter>,           // which model serves which call, and the order they are tried in
};

Task<string>    Llm.Complete(string prompt)
Task<string>    Llm.Complete(string prompt, string model)          // one of the app's models, by Name
Task<LlmResult> Llm.Complete(string prompt, LlmCall call)           // one prompt, with the call said: scopes, limits, purpose
Task<LlmResult> Llm.Complete(LlmRequest request, LlmCall call)     // everything said
stream<LlmStreamEvent> Llm.Stream(string prompt)
stream<LlmStreamEvent> Llm.Stream(LlmRequest request, LlmCall call)

Description#

What does a call go through?#

In this order, for every call:

1. The operator's switch. With generative AI switched off for the app, the call is refused — LlmSwitchedOff — and nothing is sent (turning AI off (the runtime switch)).
2. The router. LlmSetup.Router answers an ordered list of the app's models for the facts of this call. A name it answers that the app does not declare is dropped: a router can order and exclude, never add.
3. For each model in that order: room is reserved against every budget the call falls under ([[agent-llm-spend#budgets]]) and a place among the model's concurrent calls (LlmModel.MaxConcurrent); the provider is called; what it used is settled and a meter row written.
4. A provider that fails — an ExternalServiceException: an outage, a rate limit, an overload — is recorded, and the next model on the list is asked. When the last one fails, the call fails as it did.

Why does a call answer a task?#

A model call waits on somebody else, so Llm.Complete, Llm.Extract<T> and Llm.Decide<T> each answer a task. await it where you make it and it is the plain call. Keep the task and two calls are in flight at once — each still routed, budgeted and metered on its own:

var summary = Llm.Complete("Summarise: " + note);          // started
var tags    = Llm.Extract<Tags>(note);                      // started too — both are in flight
Save(await summary, await tags);                            // wait for both

A function that awaits needs no async of its own. A call that is started and never awaited is still waited for before its request ends, and its failure fails the request; _ = Llm.Complete(…) starts one whose failure is only logged.

How do I say which models the app has?#

One LlmModel each, in app.Llm (settings a package asks its app for (app settings)). Name is the app's own word for the model — what a call asks for, what a router orders, what the meter records. Model is the provider's id, and Client the provider: a class implementing ILlmClient, from a provider package or your own.

How is the model for a call chosen?#

By the router, from facts — never by the call site naming a provider. LlmTaskFacts carries what kind of work it is (Kind), which step of a larger job (Step), how big the prompt is, what the call needs (tools, vision, structured output, a decision), a quality floor, whether it is asking again, and the most it may cost. The router answers LlmRoutingDecision: the models in order, and each one it dropped with the reason. Both are kept on the call's meter row, so "why did this go to the expensive model" has an answer.

A router must be pure — a function of the facts and its own fields — so the same facts always reach the same decision. Osysharp.LlmRouter's CostTieredRouter is one; a class of your own is another:

app Clinic {
  use Osysharp.Llm;
  model "model/**/*.osy";
}

using Osysharp.Llm;

public class ClinicRouter : ILlmRouter {
  public LlmRoutingDecision Route(LlmTaskFacts facts) {
    var decision = new LlmRoutingDecision { RuleId = "clinic:1" };
    // A call may ask for one of the two; anything else is refused, never quietly served by another.
    if (facts.RequestedModel != null && facts.RequestedModel != "Quick" && facts.RequestedModel != "Careful") {
      decision.Denial = $"The clinic has no model called '{facts.RequestedModel}'.";
      return decision;
    }
    if (facts.RequestedModel != null) { decision.Ordered.Add(facts.RequestedModel!); return decision; }
    if (facts.Kind == LlmTaskKind.Reason) { decision.Ordered.Add("Careful"); return decision; }
    decision.Ordered.Add("Quick");
    decision.Ordered.Add("Careful");            // tried when Quick's provider fails
    return decision;
  }
}

app.Llm = new LlmSetup {
  Models = [
    new LlmModel { Name = "Quick",   Client = new ScriptedModel { Name = "Quick" },   Model = "scripted-quick",
                   Price = new LlmPrice { Input = 1, Output = 5 } },
    new LlmModel { Name = "Careful", Client = new ScriptedModel { Name = "Careful" }, Model = "scripted-careful",
                   Price = new LlmPrice { Input = 10, Output = 50 } },
  ],
  Router = new ClinicRouter(),
};

/// Everybody at the clinic may read what the models cost.
policy ReadsModelUsage => true;

string Remind(string visit) { return await Llm.Complete("Write a one-line reminder for: " + visit); }

LlmResult Triage(string note) {
  return await Llm.Complete(RequestOf(note, "You sort a clinic's messages."),
                      new LlmCall { Kind = LlmTaskKind.Reason, Scopes = ["desk:front"], Purpose = "triage" });
}

(The models here are scripted — see Testing model calls without a model — ScriptedModel (Llm kit). A real app names a provider's client instead.)

With no router, a call goes to the model it asked for, or to Default, and nothing is tried after it.

What can a call say about itself?#

await Llm.Complete(prompt) says nothing and gets the defaults. await Llm.Complete(request, call) takes an LlmCall:

It answers an LlmResult: the Response, the Model that served it, CostMicros, how many Attempts it took, and Text(). A streamed call says the same on its last event: MessageComplete carries the Model, the RouteRule, the Call and the CostMicros beside the provider's token counts.

What does a refusal look like?#

A call refused before anything was sent throws a class of LlmRefused, so code can tell "we chose not to" from "they failed":

A provider's own failure is an ExternalServiceException, raised when the last candidate failed.

Does a call commit my work?#

No. The kit's rows — reservations, meter rows — are written in units of work of their own ([CommitsAlone] — a function whose writes land by themselves). A call in the middle of your function commits none of your pending changes; a call whose surrounding work then fails is still on the meter; and that is what lets a call run [[#awaitable|as a task]] beside its caller — a task may not write to its caller's pending changes, and these never do.

Who may call a model?#

The app answers three policies, each with a default: UsesModels (who may call a model at all — everyone), ReadsModelUsage (who reads the meter — nobody) and ManagesModelSpend (who sets prices and budgets — nobody). Declare one to override it: policy UsesModels => IsStaff;.

How do I write a provider?#

Implement ILlmClient: Complete(request) answers the whole response, Stream(request) the events as they arrive, ending with MessageComplete carrying the token counts. request.Model is already decided — serve it, or throw ExternalServiceException. Report the counts the provider reported: the meter prices from them. When request.OutputSchemaJson is set, answer with one text block whose text is the JSON document; request.ToolChoice says whether the model may, must or may not call a tool. A provider is called by the kit's call path — code that calls a client directly is not routed, budgeted or metered.

Examples#

The app above, driven by tests: the first model serves, the second takes over when the first is down, and a call that asks for a model the router does not admit is refused.

using Osysharp.Llm;

[Test]
void The_cheap_model_serves_and_the_call_is_on_the_meter() {
  ScriptedAnswer.Stub((model, request) => Scripted.Text("Bring Milo at 9 on Tuesday.", 1000, 200));
  Assert.Equal("Bring Milo at 9 on Tuesday.", Remind("Milo, vaccination, Tue 09:00"));

  var row = ModelCall.Single();
  Assert.Equal("Quick", row.Model);
  Assert.Equal(ModelCallOutcome.Served, row.Outcome);
  Assert.Equal(2000, row.CostMicros);            // 1000 tokens at $1 a million and 200 at $5, in millionths
  Assert.Equal("clinic:1", row.RouteRule);
}

[Test]
void When_the_first_models_provider_is_down_the_next_one_serves() {
  ScriptedAnswer.Stub((model, request) => {
    if (model == "Quick") { throw new ExternalServiceException("scripted-quick answered 529"); }
    return Scripted.Text("Bring Milo at 9 on Tuesday.");
  });
  Assert.Equal("Bring Milo at 9 on Tuesday.", Remind("Milo, vaccination, Tue 09:00"));

  var attempts = ModelCall.OrderBy(c => c.Attempt).ToList();
  Assert.Equal(ModelCallOutcome.Failed, attempts[0].Outcome);
  Assert.Equal("Careful", attempts[1].Model);
  Assert.Equal("Quick", attempts[1].FallbackFrom);
}

[Test]
void A_call_said_in_full_is_routed_by_its_kind_and_charged_to_its_scope() {
  ScriptedAnswer.Stub((model, request) => Scripted.Text("Urgent: call back today.", 500, 100));
  var result = Triage("My dog ate a whole chocolate cake an hour ago.");
  Assert.Equal("Careful", result.Model);
  Assert.Equal(10000, result.CostMicros);
  Assert.Equal(0.01m, SpentOn("desk:front"));
}

[Test]
void A_model_the_router_does_not_admit_is_refused_not_substituted() {
  ScriptedAnswer.Stub((model, request) => Scripted.Text("never"));
  Assert.Throws<LlmRouteDenied>(() => await Llm.Complete("Hello", "Enormous"));
  Assert.Equal(0, ModelCall.Count());
}

[Test]
void With_ai_switched_off_nothing_is_sent() {
  GenerationIsOn.Stub(() => false);
  Assert.Throws<LlmSwitchedOff>(() => Remind("Milo"));
  Assert.Equal(0, ModelCall.Count());
}

See also#

• Model spend — budgets, the meter and the price book (Llm kit) — budgets, the meter and the price book
• Typed answers from a model — Llm.Extract<T> and Llm.Decide<T> (Llm kit) — Llm.Extract<T> and Llm.Decide<T>
• Testing model calls without a model — ScriptedModel (Llm kit) — testing an app's model calls without a model
• settings a package asks its app for (app settings) — app.Llm, and how a setting holds an app's providers
• [CommitsAlone] — a function whose writes land by themselves — why a call commits none of the caller's work