Model spend — budgets, the meter and the price book (Llm kit)

Summary#

Money in the Llm kit is counted in millionths of a US dollar, from the token counts the provider itself reports:

SetBudget("requester", "", LlmBudgetWindow.Day, 5m);      // each person: five dollars a day
SetModelPrice("Fast", 0.10m, 0.50m);                      // dollars per million tokens, from now on
decimal spent = SpentOn("task:" + task.Id.ToString());    // what one piece of work cost, read from the meter

Signature#

void    SetBudget(string scope, string subject, LlmBudgetWindow window, decimal dollars,
                  LlmOnLimit onLimit = LlmOnLimit.Stop, decimal warnAt = 0.8m)       // needs ManagesModelSpend
bool    RemoveBudget(string scope, string subject, LlmBudgetWindow window)           // needs ManagesModelSpend
void    SetModelPrice(string model, decimal input, decimal output,
                      decimal? cacheRead = null, decimal? cacheWrite = null, DateTime? from = null)   // needs ManagesModelSpend
decimal SpentIn(string scope, string subject, LlmBudgetWindow window)   // counted against a budget, this period
decimal SpentOn(string scope)                                           // "kind:subject", totalled from the meter
decimal SpentAgainst(string scope, LlmBudgetWindow window = Total)      // counted against a limit the calls carried

new LlmCall { Limits = [ new LlmLimit { Scope = "task:" + id, LimitMicros = 500000 } ] }   // a limit code puts on its own calls
int     ForgetModelCallsBefore(DateTime before)                         // needs ManagesModelSpend
int     ForgetModelCallRequester(Guid requester, DateTime? before = null)   // needs ManagesModelSpend
bool    ModelCallsInFlightFor(Guid requester)

enum LlmBudgetWindow { Day, Month, Total }       enum LlmOnLimit { Stop, StepDown }

Description#

What is a scope?#

What a call is charged to: a kind and a subject. Every call is charged to four without being asked — app (no subject), requester (the signed-in person, or the actor the code runs as), model (the model's name) — and to whatever its caller names in LlmCall.Scopes, written kind:subject: task:8f1c…, agent:Screener, lot:A1. A budget is set on a scope; a total is read for one.

What is a call made in a workflow charged to?#

To the workflow and to that run of it, with nothing said at the call: workflow:Onboarding and run.Onboarding:8f1c… (Workflow.Current (which run is this code in?)). So what a process costs is counted as it happens, a budget can hold every run of a workflow together or each run on its own, and what one run cost is read back:

SetBudget("workflow", "Onboarding", LlmBudgetWindow.Month, 200);       // every run of it together, a month
SetBudget(RunKind("Onboarding"), "", LlmBudgetWindow.Total, 2);        // each run of it, two dollars in all
var cost = SpentOn(RunScope("Onboarding", run));                       // what one run has cost

RunScopes() answers the two scopes where it is called (none outside a run); WorkflowScope(workflow), RunScope(workflow, run) and RunKind(workflow) spell them. A call that names what it is charged to itself says ChargedToItsRun = false — code that goes on, in a run of its own, with work that was started for another: the Agents kit does, and names the run the work was given in (Agents in business processes — an agent's work as a step of a workflow, told on the run (Agents.Processes kit)).

How do I set a limit?#

SetBudget(scope, subject, window, dollars). An empty subject is a budget for every subject of that kind, each with an allowance of its own — one row gives each person five dollars a day. A budget naming a subject replaces that one for it. Windows are a UTC day, a UTC month, or Total, which never starts over.

A call's room is reserved before it leaves: the most it could cost — its prompt, and the longest answer it may give, at the model's price — against every budget it falls under, all or nothing. When the provider answers, the reservation is settled at what it actually used; when it fails, the room comes back. Calls that arrive together pass the reservation one at a time, so they cannot both take the last of a budget.

What happens at the limit?#

The call is refused before anything is sent: LlmBudgetExceeded, carrying the budget's Scope, Subject, Window and LeftMicros. That is LlmOnLimit.Stop, the default.

With LlmOnLimit.StepDown — on the budget, or on the call (LlmCall.OnLimit) — the router is asked again with what is left as the call's ceiling (LlmTaskFacts.MaxCostMicros), and a cheaper model it admits serves. When it admits none, the call stops as above. Stepping down never goes below a router's own quality floor: that is the router's to decide.

A single call can carry a ceiling of its own: LlmCall.MaxCostMicros.

Can code hold its own calls to a limit?#

Yes — without anybody setting a budget. A call carries the limit itself:

await Llm.Complete(request, new LlmCall {
  Limits = [ new LlmLimit { Scope = "job:" + job.Id, LimitMicros = 500000 } ],     // half a dollar for this job, in all
});

Every call that carries a limit for the same scope is counted against it, and each one's room is reserved before it leaves, exactly as against a budget — so twelve at once cannot overrun it together. A call that does not fit is refused with the same LlmBudgetExceeded (its Scope and Subject are the scope's two halves), or steps down when the limit says OnLimit = LlmOnLimit.StepDown. The scope is one the call is charged to, like any in Scopes.

A limit binds only the calls that carry it: code holds its own work to a limit, and can never hold anybody else's — a call that names no limit for the scope is not counted and not refused. Budgets set with SetBudget apply on top, to every call. SpentAgainst("job:" + job.Id) reads what the calls have used of it — the figure as it is stored now, so asked before a call and again after it, the second answer includes the call.

What is on the meter?#

One ModelCall row per attempt, written once and never changed:

The meter never holds a prompt or an answer. Who reads it is ReadsModelUsage — one answer for the whole app — and, a call at a time, ReadsModelUsageOf(scopes), asked of that call's own Scopes: how an app of many organisations lets each one's auditor read its own calls and no other's ([[agent-roles#tenants]]).

How is a call priced?#

tokens × dollars-per-million for each of the four counts, rounded up to a millionth of a dollar. The price is the latest ModelPrice row for the model that has started (SetModelPrice), else the LlmPrice declared on the model in app.Llm. A call keeps pointing at the row that priced it, so a new price never rewrites what was spent. Cached reads and writes cost what input does unless the price says otherwise.

Who can change any of this?#

Only the kit's own functions. ModelCall, ModelPrice, LlmBudget and what each budget has counted are guarded with deny … except via: app code, a page, an import and another package can read what the app's policies let them, and write none of it. SetBudget, RemoveBudget, SetModelPrice, ForgetModelCallsBefore and ForgetModelCallRequester run only for ManagesModelSpend, who reads the meter's rows as well.

Can a person be taken off the meter?#

Yes. A row says who a call was made for (Requester, and a requester:<id> line among what it was charged to) and keeps no text of the call. ForgetModelCallRequester(person) takes them off every row — the row, the model, the tokens and the cost stay, charged to the same app, model and named scopes — and deletes what was counted against a budget of their own and a budget that names them. With before, only what is older than that moment. It is refused while a call for them is still on its way (ModelCallsInFlightFor). An app that uses Osysharp.Privacy writes none of this: Privacy for the agent family — agents, the model meter, assignments and work repositories under a person's request does it when the person asks, and when the period ends.

Anyone who may call a model can read the budgets and what each has counted — the call path reads them as the caller.

Can my app hear about a budget?#

Two events ([Event] — what happens to a record, declared once, heard with [[function-on|[On]]]): LlmBudgetNearlySpent once per period when a budget passes its warnAt share, and LlmBudgetSpent each time a call is refused for it.

Examples#

app Studio {
  use Osysharp.Llm;
  model "model/**/*.osy";
}

using Osysharp.Llm;

app.Llm = new LlmSetup {
  Models = [ new LlmModel { Name = "Writer", Client = new ScriptedModel { Name = "Writer" }, Model = "scripted-writer",
                            Price = new LlmPrice { Input = 1, Output = 5 }, MaxOutputTokens = 1000 } ],
};

// A small studio: everybody may read the meter and manage the spend.
policy ReadsModelUsage   => true;
policy ManagesModelSpend => true;

entity Heard {
  [MaxLength(200)] string What = "";
  security { allow read, create when IsAuthenticated || IsAnonymous; }
}
[On] void BudgetRanShort(LlmBudgetSpent e) { new Heard { What = $"{e.Scope} budget spent" }; }

/// A caption for one shoot — charged to the shoot.
string Caption(string shoot, string about) {
  return (await Llm.Complete(RequestOf("Caption: " + about), new LlmCall { Scopes = ["shoot:" + shoot] })).Text();
}

using Osysharp.Llm;

[Test]
void A_call_that_does_not_fit_is_refused_before_anything_is_sent() {
  // Each call reserves about 5000 millionths (up to 1000 tokens of answer at $5 a million) and then costs 2000.
  ScriptedAnswer.Stub((model, request) => Scripted.Text("Golden hour on the pier.", 1000, 200));
  SetBudget("shoot", "pier", LlmBudgetWindow.Total, 0.008m);          // 8000 millionths for this shoot
  UnitOfWork.Commit();

  Caption("pier", "two people at sunset");                           // 2000 spent
  Caption("pier", "the same, wider");                                // 2000 + ~5000 still fits: 4000 spent
  var refused = Assert.Throws<LlmBudgetExceeded>(() => Caption("pier", "one more"));
  Assert.Equal("shoot", refused.Scope);
  Assert.Equal("pier", refused.Subject);
  Assert.Equal(4000, refused.LeftMicros);

  Assert.Equal(0.004m, SpentIn("shoot", "pier", LlmBudgetWindow.Total));
  Assert.Equal(2, ModelCall.Count(c => c.Outcome == ModelCallOutcome.Served));
  Assert.Equal(1, ModelCall.Count(c => c.Outcome == ModelCallOutcome.OverBudget));
  // Another shoot has its own room.
  Caption("harbour", "boats at dawn");

  Background.Settle();
  Assert.Equal("shoot budget spent", Heard.Single().What);
}

[Test]
void A_new_price_applies_to_the_calls_after_it() {
  ScriptedAnswer.Stub((model, request) => Scripted.Text("Golden hour on the pier.", 1000, 200));
  Caption("pier", "two people at sunset");
  SetModelPrice("Writer", 2, 10);
  UnitOfWork.Commit();
  Caption("pier", "the same, wider");

  var costs = ModelCall.OrderBy(c => c.At).ToList().Select(c => c.CostMicros).ToList();
  Assert.Equal(2000, costs[0]);
  Assert.Equal(4000, costs[1]);
  Assert.Equal(0.006m, SpentOn("shoot:pier"));
}

using Osysharp.Llm;

/// A caption for one shoot, with at most `limitMicros` spent on that shoot's captions in all.
string CaptionWithin(string shoot, long limitMicros, string about) {
  return (await Llm.Complete(RequestOf("Caption: " + about), new LlmCall {
    Limits = [ new LlmLimit { Scope = "shoot:" + shoot, LimitMicros = limitMicros } ],
  })).Text();
}

[Test]
void A_limit_a_call_carries_is_reserved_against_like_a_budget() {
  ScriptedAnswer.Stub((model, request) => Scripted.Text("Golden hour on the pier.", 1000, 200));

  CaptionWithin("pier", 8000, "two people at sunset");               // 2000 spent
  CaptionWithin("pier", 8000, "the same, wider");                    // 2000 + ~5000 still fits: 4000 spent
  var refused = Assert.Throws<LlmBudgetExceeded>(() => CaptionWithin("pier", 8000, "one more"));
  Assert.Equal("shoot", refused.Scope);
  Assert.Equal("pier", refused.Subject);
  Assert.Equal(0.004m, SpentAgainst("shoot:pier"));

  // It is nobody else's limit: a call that carries none is served.
  Caption("pier", "a fourth, with no limit of its own");
}

See also#

• The Llm kit — call a model, and know what each call cost — the call path these rows are written by
• Testing model calls without a model — ScriptedModel (Llm kit) — the scripted model the examples use
• deny … except via — a row only one function may write
• [CommitsAlone] — a function whose writes land by themselves — how a meter row outlives the failure of the work around it