You are here: Building with agents › The kits › Llm
Summary#
Osysharp.Llm is the one road every model call in an app takes: the app's router picks the model, room is reserved against every budget the call falls under, the provider is called, the next model is tried when one fails, and every attempt is written on a meter with what it cost.
You list the models your app has keys for. Then Llm.Complete answers text, Llm.Extract<T> reads a typed value out of text, and Llm.Decide<T> makes a typed decision with a confidence. Agents call models through this kit too, so one set of budgets and one meter covers everything.
| Package | Osysharp.Llm, version 2.4.0 |
| You add it with | use Osysharp.Llm; in the app's manifest. |
| Bridges that arrive when it meets another kit | [[agent-kit-privacy-llm|Privacy.Llm]] |
| Its own test app | A seed bank that reads the notes sent in with each seed lot — kits/llm/tests |
Description#
What do you get without writing anything?#
- A meter row for every attempt: the model, why that one, the tokens, the cost in millionths of a dollar.
- Hard limits: for the app, for each person, for a model, for anything you name. A call that would pass one is not sent.
- Fallback: when a provider fails, the next model the router admits is tried.
- An answer of the wrong shape is sent back to the model once before your code sees an error.
- The operator's switch: AI can be turned off while the app runs, with no redeploy.
When do you need it?#
Always: every other kit on this map sits on it. On its own, it is all you need for one answer from a model with no tools — see [[agent-guide-choosing#agent-or-call]].
What does it expose?#
The tables below are generated from the kit itself, not written by hand: osy kits Osysharp.Llm --surface lists each of
them with the code of the kit's own test app that uses it, and the tests that show it working.
Settings — one line in the app, each field with a default.
| You write | It says |
|---|---|
app.Llm = new LlmSetup { Models, Default, Router, HoldFor } | How this app calls models. |
Contracts you can implement — a class of yours that the kit then calls.
| Contract | What it is |
|---|---|
interface ILlmClient { internal Complete, internal Stream } | What a model provider implements. |
interface IDecisionModel { internal Decide } | What a typed-decision model implements — a model built to return a value of a schema given in advance, with a confidence, rather than text: a classifier, a scorer, an extractor. |
interface ILlmRouter { Route } | What a model router implements — the ONE place that decides which model serves which call. |
Policies you answer — each is a question the kit asks your app: policy Name => …;
| Policy | The question | Until the app says otherwise |
|---|---|---|
UsesModels | Who may call a model at all. | everybody |
ReadsModelUsage | Who reads the meter — every call's model, tokens and cost. | nobody |
ReadsModelUsageOf | Who reads the calls charged to particular scopes — a call at a time, by what it is charged to (ModelCall.Scopes, one scope a line). | nobody |
ManagesModelSpend | Who sets prices and budgets. | nobody |
Functions you call, those its own tests show most first.
| Function | What it does | Tests that show it |
|---|---|---|
CostMicros(LlmPricePoint price, int inputTokens, int outputTokens, int cacheReadTokens, int cacheWriteTokens) | What some tokens cost at a price, in millionths of a dollar, rounded UP — a price per million tokens in dollars is a price per token in millionths. | 63 |
Served(LlmAdmission admission, int inputTokens, int outputTokens, int cacheReadTokens, int cacheWriteTokens, DateTime started, string? note) | The provider answered: settle what was held at what it actually cost, and write the attempt's meter row. | 63 |
Failed(LlmWalk walk, LlmAdmission admission, DateTime started, string said) | The provider failed: release what was held, write the attempt's meter row, and remember it for the next candidate. | 62 |
ScriptedAnswer(string model, LlmRequest request) | What the scripted model named model answers to request. | 62 |
RequestOf(string prompt, string system = …) | A request that is one question from the user, with an optional system prompt. | 60 |
ScriptedDecision(string model, LlmDecisionRequest request) | What the scripted decision model named model decides for request. | 60 |
RemoveBudget(string scope, string subject, LlmBudgetWindow window) | Remove a budget, and what was counted against it. | 59 |
ForgetModelCallRequester(Guid requester, DateTime? before = …) | Take a person off the meter: every row of a call made for requester loses who it was for — the row, the model, the tokens and the cost stay, charged to the same app, model and named scopes as b… | 59 |
SetBudget(string scope, string subject, LlmBudgetWindow window, decimal limit, LlmOnLimit onLimit = …, decimal warnAt = …) | Set the budget for scope (and subject within it — "" for every subject, each on its own) in window to limit US dollars. | 57 |
ModelCallsInFlightFor(Guid requester) | Whether a call made for requester is still on its way: room is held for it against a budget of theirs, and it has not reported back. | 57 |
ForgetModelCallsBefore(DateTime before) | Delete the meter rows of calls made before before. | 57 |
GenerationIsOn() | Whether the operator allows this app generative AI right now. | 54 |
SetModelPrice(string model, decimal input, decimal output, decimal? cacheRead = …, decimal? cacheWrite = …, DateTime? from = …) | From from on (now, when null), a million tokens on model cost this many US dollars. | 49 |
RunScopes() | What a model call is charged to because of where it is made: in a workflow's run, the workflow — workflow:Onboarding — and that run of it — run.Onboarding:8f1c…. | 7 |
SpentAgainst(string scope, LlmBudgetWindow window = …) | What the calls that carried a limit for scope have spent against it this window, in US dollars — settled calls only. | 5 |
SpentIn(string scope, string subject, LlmBudgetWindow window) | What subject has spent in scope this window, in US dollars — settled calls only. | 4 |
SpentOn(string scope) | What the calls charged to one scope cost in all, in US dollars — SpentOn("task:" + task.Id), SpentOn("app:"). | 4 |
RunKind(string workflow) | The kind a budget for EACH run of a workflow is set on: run.Onboarding — an empty subject gives every run an allowance of its own; a run's id as the subject sets one run's. | 3 |
WorkflowScope(string workflow) | The scope every call made for a workflow is charged to, whichever run: workflow:Onboarding. | 2 |
RunScope(string workflow, Guid run) | The scope the calls of ONE run of a workflow are charged to: run.Onboarding:8f1c…. | 2 |
Events your app can hear — [On] void Heard(EventName e) { … }
| Event | When |
|---|---|
LlmBudgetNearlySpent | A budget has passed its warning share for this period. |
LlmBudgetSpent | A call was refused because a budget had no room for it. |
What to read next#
- Osysharp.Agents — an agent given a real job inside an app, and trusted with it — an agent given a real job inside an app, and trusted with it
- Osysharp.LlmRouter — the cheapest model that is good enough for each kind of work — the cheapest model that is good enough for each kind of work
- The map of the kits — which sits on which, and what each one is for — the map of the kits — which sits on which, and what each one is for
Go deeper#
- The Llm kit — call a model, and know what each call cost — the calls, the models, routing, refusals, writing a provider
- Model spend — budgets, the meter and the price book (Llm kit) — budgets, the meter and the price book
- Typed answers from a model — Llm.Extract<T> and Llm.Decide<T> (Llm kit) —
Llm.Extract<T>andLlm.Decide<T> - Testing model calls without a model — ScriptedModel (Llm kit) — testing model calls without a model
- turning AI off (the runtime switch) — turning AI off at run time