# Testing an agent without a model — Says, AgentScript and Trajectory (Agents.Testing kit)

> `Osysharp.Agents.Testing` scripts what the model answers, turn by turn, and reads back what the agent then did. Only the model is scripted: the loop, the agent's rights, the tools, the budget, the meter and the record are the real ones, so a test proves the tools that ran and in what order, what was refused, what the work cost and what it handed back. It costs nothing and needs no key.

<!-- id: agent-testing · area: agent · stability: preview · html: https://osysharp.com/reference/agent/testing/ -->

## Summary        {#summary}
```osy syntax
var script = AgentScript.Of(
  Says.Tool(Actions.Of(PartsFor), "{\"fault\":\"rear brake\"}"),
  Says.HandBack(new Estimate { Hours = 1 }));
ScriptedAnswer.Stub((model, request) => script.Answer(model, request));

var outcome = await Agents.Of<Estimator>().Do(new EstimateRepair(), repair);
Assert.Equal(["PartsFor"], Trajectory.Of(outcome.Task));
```

## Signature      {#signature}
```osy syntax
// one answer of the model — each also takes (inputTokens, outputTokens), which is what the meter prices
LlmResponse Says.Tool(ActionInfo action, string argumentsJson)    // it asks for a tool
LlmResponse Says.Tool(string name, string argumentsJson)          // …by a name, listed or not
LlmResponse Says.HandBack<T>(T value)                             // it hands its typed result back
LlmResponse Says.HandBackJson(string json)                        // …a result that is not the shape
LlmResponse Says.Words(string text)                               // it answers in words
LlmResponse Says.Consult(string question, string context)         // it asks the model its agent consults
LlmResponse Says.HandTo<TAgent>(SkillBase skill, string inputJson, string call = "")   // it hands a piece of work on
string      Says.Row(Guid id)                                     // …the input that names a row by its id
LlmResponse Says.Together(params LlmResponse[] asks)              // several things asked for in ONE turn

AgentScript AgentScript.Of(params LlmResponse[] answers)   .Then(answer)
AgentScript .Advises(string model, string advice)                 // what a consulted model says
AgentScript .IsDown(string model)                                 // that model fails as an unreachable provider
LlmResponse .Answer(string model, LlmRequest request)             // hand this to ScriptedAnswer.Stub

List<string> Trajectory.Of(Guid task)                             // the tools that ran, in order
bool         Trajectory.Called(Guid task, ActionInfo action)
List<string> Trajectory.Models(Guid task)                         // the model that served each call
List<string> Trajectory.Kinds(Guid task)    List<string> Trajectory.Said(Guid task)      // every line of the record
int          Trajectory.Helps(Guid task)    long Trajectory.Cost(Guid task)

// several agents at once — work that is handed on
AgentCast AgentCast.Of()   .Plays<TAgent>(script)   .PlaysAbout<TAgent>(string about, script)
LlmResponse .Answer(string model, LlmRequest request)             // hand this to ScriptedAnswer.Stub
string AgentCast.AskedAgent(LlmRequest request)                   // who a request is put to: "Checker"
string AgentCast.FirstAsked(LlmRequest request)                   // what the piece of work was given
List<string> Trajectory.Tree(Guid task)                           // "Planner · Plan rebinding: Done", "  Buyer · Price cloth: Stopped"
List<string> Trajectory.HandedOn(Guid task)   long Trajectory.TreeCost(Guid task)   List<string> Trajectory.StillGoing(Guid task)
```

## Description    {#description}

### What is scripted, and what is real?   {#real}
The app names a [`ScriptedModel`](https://osysharp.com/reference/agent/llm-scripted-model/) where a provider would go, and a test answers for it.
Everything else is the app as it runs: the agent's account and the caller's rights, the skill's check, the tools,
the limits, the meter, the receipt and the record.

### How does a script know which turn it is?   {#turns}
From the request: a script counts the assistant turns it carries, so the same script answers the same way however
calls are interleaved. A script that runs out of answers fails the call the way a provider does, and says which turn
had none.

### What do I assert?   {#assert}
What a model cannot be asked to promise. The tools that ran and their order (`Trajectory.Of`); that a tool it was not
given never ran (`Trajectory.Called`); what it cost (`Trajectory.Cost`, the receipt); how it ended (`outcome.Status`);
and the app's own rows, which the tools really changed.

### And a whole process?   {#process}
The same way, with the days moved by `TestClock.Advance` and a person simulated by doing what she would do under
`runas`. `ProcessTrail` reads back what the agents did in the run about one row — `Skills(item)`, `States(item)`,
`Ended(item)`, `Proposed(item)`, `Cost(item)` — so a test says where the process went, what was proposed on the way
and what it cost ([[agent-processes#testing]]).

### How do I script several agents at once?   {#cast}
Work that is [handed on](https://osysharp.com/reference/agent/delegation/) has more than one model answering. An `AgentCast` gives each agent its
own script and tells the requests apart by who each says it is; `PlaysAbout<T>("Crate A-2", script)` picks among
several pieces of one agent's work by a few words of its job or its brief. `Says.HandTo<T>(skill, inputJson)` is a
model handing a piece on — `Says.Row(id)` writes the input of a skill that takes a row — and `Says.Together(…)` puts
several in one turn, which is what makes them run side by side. `Trajectory.Tree(task)` reads the tree back, a line
a piece, indented under the piece that handed it on; `Trajectory.StillGoing(task)` is empty once nothing of it is
left. [[agent-delegation#examples]] is a cast of three, compiled and run.

A scripted answer runs on its own thread when several models are asked at once, so a stub that wants to note
something writes it through a function marked `[CommitsAlone]` ([[CommitsAlone] — a function whose writes land by themselves](https://osysharp.com/reference/function/commits-alone/)).

## Examples       {#examples}
```osy title="a script, the tools that ran in order, what it cost, and a model that is down" run app=agent-testing
app Workshop {
  use Osysharp.Agents;
  use Osysharp.Agents.Testing;
  use Osysharp.Llm;
  use Osysharp.Identity;
  use Osysharp.Observability;
  model "model/**/*.osy";
}

using Osysharp.Agents;
using Osysharp.Agents.Testing;
using Osysharp.Identity;
using Osysharp.Llm;

app.Llm = new LlmSetup {
  Models = [ new LlmModel { Name = "Everyday", Client = new ScriptedModel { Name = "Everyday" }, Model = "scripted-everyday",
                            Price = new LlmPrice { Input = 1, Output = 5 }, MaxOutputTokens = 1000 } ],
};
app.Agents = new AgentsSetup { Agents = [new Estimator()], TaskBudget = 0.05m };

[Role] enum ShopRole { Authenticator, Mechanic, Manager, Helper }
partial entity RoleGrant { security { allow read when IsAuthenticated; } }
policy IsMechanic => RoleGrant.Any(g => g.Member == user && g.Role == ShopRole.Mechanic);
policy IsManager  => RoleGrant.Any(g => g.Member == user && g.Role == ShopRole.Manager);
policy IsHelper   => RoleGrant.Any(g => g.Member == user && g.Role == ShopRole.Helper);

// The kit's questions, answered by this app.
policy UsesAgents       => IsMechanic || IsManager;   // gives agents work, reads the work they asked for
policy SupervisesAgents => IsManager;                 // names sponsors, pauses and stops, reads that work happened
policy AuditsAgents     => IsManager;                 // reads everything, content included
policy ReadsModelUsage   => IsManager;
policy ManagesModelSpend => IsManager;

entity Person : Principal {
  security { allow read when IsAuthenticated || IsAuthenticator; allow create when IsAuthenticator; }
}

/// One bike in for repair.
entity Repair {
  [Required, Unique, MaxLength(40)] string Ticket;
  [Required, MaxLength(200)] string Fault;
  [MaxLength(400)] string Note = "";
  /// What the customer is charged. Only a mechanic or the manager sets it.
  decimal Price;
  security {
    allow read, update when IsMechanic || IsManager || IsHelper;
    allow create when IsMechanic || IsManager;
    deny update Price when IsHelper;
  }
}

/// What an estimate says.
public class Estimate {
  public decimal Hours;
  public string Summary = "";
  public bool NeedsParts;
}

/// The parts a fault needs.
[Action("Look up the parts a fault needs", Query)]
public List<string> PartsFor(string fault) { return fault.Contains("brake") ? ["brake pads", "cable"] : []; }

/// Write a note on the repair's ticket.
[Action("Write a note on the repair")]
public string NoteRepair(Repair repair, string note) { repair.Note = note; return $"Noted on {repair.Ticket}."; }

/// Set what the customer is charged.
[Action("Set the price")]
public string SetPrice(Repair repair, decimal price) { repair.Price = price; return $"{repair.Ticket} costs {price}."; }

/// Works out how long one repair takes.
public class EstimateRepair : Skill<Repair, Estimate> {
  public override string Description => "Estimate one repair.";
  public override string Instructions => "Look up the parts first. Write the estimate as a note on the repair.";
  public override List<ActionInfo> Tools => [Actions.Of(PartsFor), Actions.Of(NoteRepair), Actions.Of(SetPrice)];
  public override string Brief(Repair r) => $"Ticket {r.Ticket}: {r.Fault}. Its id is {r.Id}.";
  public override string? Check(Estimate e) => e.Hours <= 0 ? "An estimate takes some time." : null;
}

/// Estimates the repairs that come in. It holds the Helper role and nothing more.
[Actor(Roles = [ShopRole.Helper])]
public class Estimator : Agent {
  public static Person Principal => new Person { Email = "estimator@workshop.example", DisplayName = "Estimator" };
  public override string Purpose => "You estimate bicycle repairs for a small workshop.";
  public override List<SkillBase> Skills => [new EstimateRepair()];
  public override Permits May => Permits.ChangesAlone();        // it notes and prices by itself
}

[TestFixture]
void Shop() {
  var mia = new Person { Email = "mia@workshop.example", DisplayName = "Mia" };
  var noor = new Person { Email = "noor@workshop.example", DisplayName = "Noor" };
  new RoleGrant { Member = mia, Role = ShopRole.Mechanic };
  new RoleGrant { Member = noor, Role = ShopRole.Manager };
  new Repair { Ticket = "T-7", Fault = "rear brake does not bite" };
  new AgentSeat { Agent = "Estimator", Sponsor = noor };          // the manager answers for the agent
}
principal Mia  => Person.Single(p => p.Email == "mia@workshop.example");
principal Noor => Person.Single(p => p.Email == "noor@workshop.example");

string RepairArgs(Repair r, string rest) { return "{\"repair\":\"" + r.Id.ToString() + "\"" + rest + "}"; }

[Test(Shop)]
[runas(Mia)]
void A_script_answers_turn_by_turn_and_the_trajectory_is_what_the_agent_did() {
  var repair = Repair.Single(r => r.Ticket == "T-7");
  var script = AgentScript.Of(
    Says.Tool(Actions.Of(PartsFor), "{\"fault\":\"rear brake does not bite\"}", 400, 40),
    Says.Tool(Actions.Of(NoteRepair), RepairArgs(repair, ",\"note\":\"One hour.\""), 500, 40),
    Says.HandBack(new Estimate { Hours = 1, Summary = "New pads." }, 600, 60));
  ScriptedAnswer.Stub((model, request) => script.Answer(model, request));

  var outcome = await Agents.Of<Estimator>().Do(new EstimateRepair(), repair);

  Assert.True(outcome.Done);
  Assert.Equal(["PartsFor", "NoteRepair"], Trajectory.Of(outcome.Task));
  Assert.True(Trajectory.Called(outcome.Task, Actions.Of(NoteRepair)));
  Assert.False(Trajectory.Called(outcome.Task, Actions.Of(SetPrice)));
  Assert.Equal(["Everyday", "Everyday", "Everyday"], Trajectory.Models(outcome.Task));
  // 400 + 5×40, 500 + 5×40 and 600 + 5×60 millionths of a dollar.
  Assert.Equal(2200, Trajectory.Cost(outcome.Task));
  Assert.Equal("One hour.", Repair.Single(r => r.Ticket == "T-7").Note);
}

[Test(Shop)]
[runas(Mia)]
void A_result_that_is_not_the_shape_and_a_model_that_is_down() {
  var repair = Repair.Single(r => r.Ticket == "T-7");
  var script = AgentScript.Of(
    Says.HandBackJson("{\"Hours\":\"a while\"}"),                        // not an Estimate — sent back
    Says.HandBack(new Estimate { Hours = 2, Summary = "A while." }));
  ScriptedAnswer.Stub((model, request) => script.Answer(model, request));
  var second = await Agents.Of<Estimator>().Do(new EstimateRepair(), repair);
  Assert.Equal(2, second.Result.Hours);

  var down = AgentScript.Of(Says.HandBack(new Estimate { Hours = 1 })).IsDown("Everyday");
  ScriptedAnswer.Stub((model, request) => down.Answer(model, request));
  var failed = await Agents.Of<Estimator>().Do(new EstimateRepair(), repair);
  Assert.Equal(AgentTaskStatus.Failed, failed.Status);
}
```

## See also       {#see-also}
- [What an agent does alone, what it proposes, what it never does — the gate and proposals (Agents kit)](https://osysharp.com/reference/agent/autonomy/) — `Says.Tool("why", action, arguments)`: a tool call with the model's reason, as a proposal records it
- [An agent asks a person, and waits — the person rung of the ladder of help (Agents kit)](https://osysharp.com/reference/agent/people/) — `Says.AskPerson(question, context)`: the model asks a person, and the work waits
- [Writing an agent — a class with a purpose, skills and an account of its own (Agents kit)](https://osysharp.com/reference/agent/agents/) — the agent class, its account and who is in charge of it
- [A skill — a typed job an agent can be given, its check and its tools (Agents kit)](https://osysharp.com/reference/agent/skills/) — a typed job, its check and its tools
- [Giving an agent work — Do, Ask, Stream, For and Decide (Agents kit)](https://osysharp.com/reference/agent/ask/) — giving an agent work: `Do`, `Ask`, `Stream`, `For`, `Decide`
- [The receipt and the record — what every piece of an agent's work leaves (Agents kit)](https://osysharp.com/reference/agent/task-log/) — the receipt and the record every piece of work leaves
- [Asking for help — an agent that consults a more capable model (Agents kit)](https://osysharp.com/reference/agent/help/) — asking a more capable model
- [Testing model calls without a model — ScriptedModel (Llm kit)](https://osysharp.com/reference/agent/llm-scripted-model/) — the scripted provider underneath