# The Llm kit — call a model, and know what each call cost

> `Osysharp.Llm` is the road every model call takes, written in Osy#: the operator's AI switch, the app's router, a reservation against every budget the call falls under, the provider, the next model when one fails, and a meter row for each attempt. The app lists the models it has keys for in `app.Llm`; `Llm.Complete`, `Llm.Stream`, `Llm.Extract<T>` and `Llm.Decide<T>` are the calls. A provider is a class implementing `ILlmClient`, a router a class implementing `ILlmRouter`.

<!-- id: agent-llm-calls · area: agent · stability: preview · html: https://osysharp.com/reference/agent/llm-calls/ -->

## Summary        {#summary}
`use Osysharp.Llm@2;` gives an app one way to call a model, whichever provider serves it:

```osy syntax
var line = await Llm.Complete("Summarise this in one line: " + note);
```

The app says which models it has; a router says which serves each call; and every attempt — served, failed or
refused — is a row on the meter ([Model spend — budgets, the meter and the price book (Llm kit)](https://osysharp.com/reference/agent/llm-spend/)).

## Signature      {#signature}
```osy syntax
app.Llm = new LlmSetup {
  Models  = [ new LlmModel { Name = "Fast", Client = <ILlmClient>, Model = "<the provider's id>", Price = new LlmPrice { … } }, … ],
  Default = "Fast",                 // the model a call gets with no router; "" is the first
  Router  = <ILlmRouter>,           // which model serves which call, and the order they are tried in
};

Task<string>    Llm.Complete(string prompt)
Task<string>    Llm.Complete(string prompt, string model)          // one of the app's models, by Name
Task<LlmResult> Llm.Complete(string prompt, LlmCall call)           // one prompt, with the call said: scopes, limits, purpose
Task<LlmResult> Llm.Complete(LlmRequest request, LlmCall call)     // everything said
stream<LlmStreamEvent> Llm.Stream(string prompt)
stream<LlmStreamEvent> Llm.Stream(LlmRequest request, LlmCall call)
```

## Description    {#description}

### What does a call go through?   {#the-road}
In this order, for every call:

1. **The operator's switch.** With generative AI switched off for the app, the call is refused — `LlmSwitchedOff` —
   and nothing is sent ([turning AI off (the runtime switch)](https://osysharp.com/reference/agent/ai-switch/)).
2. **The router.** `LlmSetup.Router` answers an ordered list of the app's models for the facts of this call. A name it
   answers that the app does not declare is dropped: a router can order and exclude, never add.
3. **For each model in that order:** room is reserved against every budget the call falls under
   ([[agent-llm-spend#budgets]]) and a place among the model's concurrent calls (`LlmModel.MaxConcurrent`); the
   provider is called; what it used is settled and a meter row written.
4. **A provider that fails** — an `ExternalServiceException`: an outage, a rate limit, an overload — is recorded, and
   the next model on the list is asked. When the last one fails, the call fails as it did.

### Why does a call answer a task?   {#awaitable}
A model call waits on somebody else, so `Llm.Complete`, `Llm.Extract<T>` and `Llm.Decide<T>` each answer a
[task](https://osysharp.com/reference/function/async-await/). `await` it where you make it and it is the plain call. Keep the task and two calls
are in flight at once — each still routed, budgeted and metered on its own:

```osy syntax
var summary = Llm.Complete("Summarise: " + note);          // started
var tags    = Llm.Extract<Tags>(note);                      // started too — both are in flight
Save(await summary, await tags);                            // wait for both
```

A function that awaits needs no `async` of its own. A call that is started and never awaited is still waited for
before its request ends, and its failure fails the request; `_ = Llm.Complete(…)` starts one whose failure is only
logged.

### How do I say which models the app has?   {#models}
One `LlmModel` each, in `app.Llm` ([settings a package asks its app for (app settings)](https://osysharp.com/reference/config/app-settings/)). `Name` is the app's own word for the model — what a call
asks for, what a router orders, what the meter records. `Model` is the provider's id, and `Client` the provider:
a class implementing `ILlmClient`, from a provider package or your own.

### How is the model for a call chosen?   {#routing}
By the router, from **facts** — never by the call site naming a provider. `LlmTaskFacts` carries what kind of work it
is (`Kind`), which step of a larger job (`Step`), how big the prompt is, what the call needs (tools, vision,
structured output, a decision), a quality floor, whether it is asking again, and the most it may cost. The router
answers `LlmRoutingDecision`: the models in order, and each one it dropped with the reason. Both are kept on the
call's meter row, so "why did this go to the expensive model" has an answer.

A router must be **pure** — a function of the facts and its own fields — so the same facts always reach the same
decision. `Osysharp.LlmRouter`'s `CostTieredRouter` is one; a class of your own is another:

```osy title="a router of the app's own: cheap first, the strong one behind it" test app=agent-llm-calls
app Clinic {
  use Osysharp.Llm;
  model "model/**/*.osy";
}

using Osysharp.Llm;

public class ClinicRouter : ILlmRouter {
  public LlmRoutingDecision Route(LlmTaskFacts facts) {
    var decision = new LlmRoutingDecision { RuleId = "clinic:1" };
    // A call may ask for one of the two; anything else is refused, never quietly served by another.
    if (facts.RequestedModel != null && facts.RequestedModel != "Quick" && facts.RequestedModel != "Careful") {
      decision.Denial = $"The clinic has no model called '{facts.RequestedModel}'.";
      return decision;
    }
    if (facts.RequestedModel != null) { decision.Ordered.Add(facts.RequestedModel!); return decision; }
    if (facts.Kind == LlmTaskKind.Reason) { decision.Ordered.Add("Careful"); return decision; }
    decision.Ordered.Add("Quick");
    decision.Ordered.Add("Careful");            // tried when Quick's provider fails
    return decision;
  }
}

app.Llm = new LlmSetup {
  Models = [
    new LlmModel { Name = "Quick",   Client = new ScriptedModel { Name = "Quick" },   Model = "scripted-quick",
                   Price = new LlmPrice { Input = 1, Output = 5 } },
    new LlmModel { Name = "Careful", Client = new ScriptedModel { Name = "Careful" }, Model = "scripted-careful",
                   Price = new LlmPrice { Input = 10, Output = 50 } },
  ],
  Router = new ClinicRouter(),
};

/// Everybody at the clinic may read what the models cost.
policy ReadsModelUsage => true;

string Remind(string visit) { return await Llm.Complete("Write a one-line reminder for: " + visit); }

LlmResult Triage(string note) {
  return await Llm.Complete(RequestOf(note, "You sort a clinic's messages."),
                      new LlmCall { Kind = LlmTaskKind.Reason, Scopes = ["desk:front"], Purpose = "triage" });
}
```

(The models here are scripted — see [Testing model calls without a model — ScriptedModel (Llm kit)](https://osysharp.com/reference/agent/llm-scripted-model/). A real app names a provider's client instead.)

With **no router**, a call goes to the model it asked for, or to `Default`, and nothing is tried after it.

### What can a call say about itself?   {#the-call}
`await Llm.Complete(prompt)` says nothing and gets the defaults. `await Llm.Complete(request, call)` takes an `LlmCall`:

| member | what it says |
|---|---|
| `Model` | one of the app's models, by `Name` — it can only narrow what the router admits |
| `Kind`, `Step` | what kind of work, and which step of a larger job — what a router's floors are declared per |
| `MinQuality` | the lowest rank that may serve it, on the router's own scale |
| `Attempt`, `EscalatedFrom` | the call is asking again because the last answer was not good enough |
| `Scopes` | what it is charged to beside the app, the person and the model — `task:8f1c…`, `agent:Screener` |
| `Purpose` | what it is for, in a few words, kept on the meter row |
| `OnLimit`, `MaxCostMicros` | what to do when a budget has no room, and the most this one call may cost |
| `Limits` | limits the code puts on its own calls for a scope — see [[agent-llm-spend#limits]] |

It answers an `LlmResult`: the `Response`, the `Model` that served it, `CostMicros`, how many `Attempts` it took, and
`Text()`. A streamed call says the same on its last event: `MessageComplete` carries the `Model`, the `RouteRule`, the
`Call` and the `CostMicros` beside the provider's token counts.

### What does a refusal look like?   {#refusals}
A call refused **before anything was sent** throws a class of `LlmRefused`, so code can tell "we chose not to" from
"they failed":

| thrown | when |
|---|---|
| `LlmSwitchedOff` | the operator has switched generative AI off for the app |
| `LlmRouteDenied` | the router admits no model for this call — or the call asked for one it does not admit |
| `LlmBudgetExceeded` | a budget the call falls under has no room; it carries `Scope`, `Subject`, `Window`, `LeftMicros` |
| `LlmBusy` | every candidate is already serving as many calls as it allows |

A provider's own failure is an `ExternalServiceException`, raised when the last candidate failed.

### Does a call commit my work?   {#unit-of-work}
No. The kit's rows — reservations, meter rows — are written in units of work of their own
([[CommitsAlone] — a function whose writes land by themselves](https://osysharp.com/reference/function/commits-alone/)). A call in the middle of your function commits none of your pending changes; a call
whose surrounding work then fails is still on the meter; and that is what lets a call run [[#awaitable|as a task]]
beside its caller — a task may not write to its caller's pending changes, and these never do.

### Who may call a model?   {#policies}
The app answers three policies, each with a default: `UsesModels` (who may call a model at all — everyone),
`ReadsModelUsage` (who reads the meter — nobody) and `ManagesModelSpend` (who sets prices and budgets — nobody).
Declare one to override it: `policy UsesModels => IsStaff;`.

### How do I write a provider?   {#providers}
Implement `ILlmClient`: `Complete(request)` answers the whole response, `Stream(request)` the events as they arrive,
ending with `MessageComplete` carrying the token counts. `request.Model` is already decided — serve it, or throw
`ExternalServiceException`. Report the counts the provider reported: the meter prices from them. When
`request.OutputSchemaJson` is set, answer with one text block whose text is the JSON document; `request.ToolChoice`
says whether the model may, must or may not call a tool. A provider is called by the kit's call path — code that calls
a client directly is not routed, budgeted or metered.

## Examples       {#examples}
The app above, driven by tests: the first model serves, the second takes over when the first is down, and a call
that asks for a model the router does not admit is refused.

```osy title="served, fallen back, refused — and what the meter says" run app=agent-llm-calls
using Osysharp.Llm;

[Test]
void The_cheap_model_serves_and_the_call_is_on_the_meter() {
  ScriptedAnswer.Stub((model, request) => Scripted.Text("Bring Milo at 9 on Tuesday.", 1000, 200));
  Assert.Equal("Bring Milo at 9 on Tuesday.", Remind("Milo, vaccination, Tue 09:00"));

  var row = ModelCall.Single();
  Assert.Equal("Quick", row.Model);
  Assert.Equal(ModelCallOutcome.Served, row.Outcome);
  Assert.Equal(2000, row.CostMicros);            // 1000 tokens at $1 a million and 200 at $5, in millionths
  Assert.Equal("clinic:1", row.RouteRule);
}

[Test]
void When_the_first_models_provider_is_down_the_next_one_serves() {
  ScriptedAnswer.Stub((model, request) => {
    if (model == "Quick") { throw new ExternalServiceException("scripted-quick answered 529"); }
    return Scripted.Text("Bring Milo at 9 on Tuesday.");
  });
  Assert.Equal("Bring Milo at 9 on Tuesday.", Remind("Milo, vaccination, Tue 09:00"));

  var attempts = ModelCall.OrderBy(c => c.Attempt).ToList();
  Assert.Equal(ModelCallOutcome.Failed, attempts[0].Outcome);
  Assert.Equal("Careful", attempts[1].Model);
  Assert.Equal("Quick", attempts[1].FallbackFrom);
}

[Test]
void A_call_said_in_full_is_routed_by_its_kind_and_charged_to_its_scope() {
  ScriptedAnswer.Stub((model, request) => Scripted.Text("Urgent: call back today.", 500, 100));
  var result = Triage("My dog ate a whole chocolate cake an hour ago.");
  Assert.Equal("Careful", result.Model);
  Assert.Equal(10000, result.CostMicros);
  Assert.Equal(0.01m, SpentOn("desk:front"));
}

[Test]
void A_model_the_router_does_not_admit_is_refused_not_substituted() {
  ScriptedAnswer.Stub((model, request) => Scripted.Text("never"));
  Assert.Throws<LlmRouteDenied>(() => await Llm.Complete("Hello", "Enormous"));
  Assert.Equal(0, ModelCall.Count());
}

[Test]
void With_ai_switched_off_nothing_is_sent() {
  GenerationIsOn.Stub(() => false);
  Assert.Throws<LlmSwitchedOff>(() => Remind("Milo"));
  Assert.Equal(0, ModelCall.Count());
}
```

## See also       {#see-also}
- [Model spend — budgets, the meter and the price book (Llm kit)](https://osysharp.com/reference/agent/llm-spend/) — budgets, the meter and the price book
- [Typed answers from a model — Llm.Extract<T> and Llm.Decide<T> (Llm kit)](https://osysharp.com/reference/agent/llm-typed-answers/) — `Llm.Extract<T>` and `Llm.Decide<T>`
- [Testing model calls without a model — ScriptedModel (Llm kit)](https://osysharp.com/reference/agent/llm-scripted-model/) — testing an app's model calls without a model
- [settings a package asks its app for (app settings)](https://osysharp.com/reference/config/app-settings/) — `app.Llm`, and how a setting holds an app's providers
- [[CommitsAlone] — a function whose writes land by themselves](https://osysharp.com/reference/function/commits-alone/) — why a call commits none of the caller's work