Testing an agent without a model — Says, AgentScript and Trajectory (Agents.Testing kit)

AgentScript.Of(Says.Tool(Actions.Of(F), json), Says.HandBack(value), Says.Words(text), Says.Consult(q, ctx)).Advises(model, text).IsDown(model) · ScriptedAnswer.Stub((m, r) => script.Answer(m, r)) · Trajectory.Of(task)

`Osysharp.Agents.Testing` scripts what the model answers, turn by turn, and reads back what the agent then did. Only the model is scripted: the loop, the agent's rights, the tools, the budget, the meter and the record are the real ones, so a test proves the tools that ran and in what order, what was refused, what the work cost and what it handed back. It costs nothing and needs no key.

Summary#

var script = AgentScript.Of(
  Says.Tool(Actions.Of(PartsFor), "{\"fault\":\"rear brake\"}"),
  Says.HandBack(new Estimate { Hours = 1 }));
ScriptedAnswer.Stub((model, request) => script.Answer(model, request));

var outcome = await Agents.Of<Estimator>().Do(new EstimateRepair(), repair);
Assert.Equal(["PartsFor"], Trajectory.Of(outcome.Task));

Signature#

// one answer of the model — each also takes (inputTokens, outputTokens), which is what the meter prices
LlmResponse Says.Tool(ActionInfo action, string argumentsJson)    // it asks for a tool
LlmResponse Says.Tool(string name, string argumentsJson)          // …by a name, listed or not
LlmResponse Says.HandBack<T>(T value)                             // it hands its typed result back
LlmResponse Says.HandBackJson(string json)                        // …a result that is not the shape
LlmResponse Says.Words(string text)                               // it answers in words
LlmResponse Says.Consult(string question, string context)         // it asks the model its agent consults
LlmResponse Says.HandTo<TAgent>(SkillBase skill, string inputJson, string call = "")   // it hands a piece of work on
string      Says.Row(Guid id)                                     // …the input that names a row by its id
LlmResponse Says.Together(params LlmResponse[] asks)              // several things asked for in ONE turn

AgentScript AgentScript.Of(params LlmResponse[] answers)   .Then(answer)
AgentScript .Advises(string model, string advice)                 // what a consulted model says
AgentScript .IsDown(string model)                                 // that model fails as an unreachable provider
LlmResponse .Answer(string model, LlmRequest request)             // hand this to ScriptedAnswer.Stub

List<string> Trajectory.Of(Guid task)                             // the tools that ran, in order
bool         Trajectory.Called(Guid task, ActionInfo action)
List<string> Trajectory.Models(Guid task)                         // the model that served each call
List<string> Trajectory.Kinds(Guid task)    List<string> Trajectory.Said(Guid task)      // every line of the record
int          Trajectory.Helps(Guid task)    long Trajectory.Cost(Guid task)

// several agents at once — work that is handed on
AgentCast AgentCast.Of()   .Plays<TAgent>(script)   .PlaysAbout<TAgent>(string about, script)
LlmResponse .Answer(string model, LlmRequest request)             // hand this to ScriptedAnswer.Stub
string AgentCast.AskedAgent(LlmRequest request)                   // who a request is put to: "Checker"
string AgentCast.FirstAsked(LlmRequest request)                   // what the piece of work was given
List<string> Trajectory.Tree(Guid task)                           // "Planner · Plan rebinding: Done", "  Buyer · Price cloth: Stopped"
List<string> Trajectory.HandedOn(Guid task)   long Trajectory.TreeCost(Guid task)   List<string> Trajectory.StillGoing(Guid task)

Description#

What is scripted, and what is real?#

The app names a ScriptedModel where a provider would go, and a test answers for it. Everything else is the app as it runs: the agent's account and the caller's rights, the skill's check, the tools, the limits, the meter, the receipt and the record.

How does a script know which turn it is?#

From the request: a script counts the assistant turns it carries, so the same script answers the same way however calls are interleaved. A script that runs out of answers fails the call the way a provider does, and says which turn had none.

What do I assert?#

What a model cannot be asked to promise. The tools that ran and their order (Trajectory.Of); that a tool it was not given never ran (Trajectory.Called); what it cost (Trajectory.Cost, the receipt); how it ended (outcome.Status); and the app's own rows, which the tools really changed.

And a whole process?#

The same way, with the days moved by TestClock.Advance and a person simulated by doing what she would do under runas. ProcessTrail reads back what the agents did in the run about one row — Skills(item), States(item), Ended(item), Proposed(item), Cost(item) — so a test says where the process went, what was proposed on the way and what it cost ([[agent-processes#testing]]).

How do I script several agents at once?#

Work that is handed on has more than one model answering. An AgentCast gives each agent its own script and tells the requests apart by who each says it is; PlaysAbout<T>("Crate A-2", script) picks among several pieces of one agent's work by a few words of its job or its brief. Says.HandTo<T>(skill, inputJson) is a model handing a piece on — Says.Row(id) writes the input of a skill that takes a row — and Says.Together(…) puts several in one turn, which is what makes them run side by side. Trajectory.Tree(task) reads the tree back, a line a piece, indented under the piece that handed it on; Trajectory.StillGoing(task) is empty once nothing of it is left. [[agent-delegation#examples]] is a cast of three, compiled and run.

A scripted answer runs on its own thread when several models are asked at once, so a stub that wants to note something writes it through a function marked [CommitsAlone] ([CommitsAlone] — a function whose writes land by themselves).

Examples#

app Workshop {
  use Osysharp.Agents;
  use Osysharp.Agents.Testing;
  use Osysharp.Llm;
  use Osysharp.Identity;
  use Osysharp.Observability;
  model "model/**/*.osy";
}

using Osysharp.Agents;
using Osysharp.Agents.Testing;
using Osysharp.Identity;
using Osysharp.Llm;

app.Llm = new LlmSetup {
  Models = [ new LlmModel { Name = "Everyday", Client = new ScriptedModel { Name = "Everyday" }, Model = "scripted-everyday",
                            Price = new LlmPrice { Input = 1, Output = 5 }, MaxOutputTokens = 1000 } ],
};
app.Agents = new AgentsSetup { Agents = [new Estimator()], TaskBudget = 0.05m };

[Role] enum ShopRole { Authenticator, Mechanic, Manager, Helper }
partial entity RoleGrant { security { allow read when IsAuthenticated; } }
policy IsMechanic => RoleGrant.Any(g => g.Member == user && g.Role == ShopRole.Mechanic);
policy IsManager  => RoleGrant.Any(g => g.Member == user && g.Role == ShopRole.Manager);
policy IsHelper   => RoleGrant.Any(g => g.Member == user && g.Role == ShopRole.Helper);

// The kit's questions, answered by this app.
policy UsesAgents       => IsMechanic || IsManager;   // gives agents work, reads the work they asked for
policy SupervisesAgents => IsManager;                 // names sponsors, pauses and stops, reads that work happened
policy AuditsAgents     => IsManager;                 // reads everything, content included
policy ReadsModelUsage   => IsManager;
policy ManagesModelSpend => IsManager;

entity Person : Principal {
  security { allow read when IsAuthenticated || IsAuthenticator; allow create when IsAuthenticator; }
}

/// One bike in for repair.
entity Repair {
  [Required, Unique, MaxLength(40)] string Ticket;
  [Required, MaxLength(200)] string Fault;
  [MaxLength(400)] string Note = "";
  /// What the customer is charged. Only a mechanic or the manager sets it.
  decimal Price;
  security {
    allow read, update when IsMechanic || IsManager || IsHelper;
    allow create when IsMechanic || IsManager;
    deny update Price when IsHelper;
  }
}

/// What an estimate says.
public class Estimate {
  public decimal Hours;
  public string Summary = "";
  public bool NeedsParts;
}

/// The parts a fault needs.
[Action("Look up the parts a fault needs", Query)]
public List<string> PartsFor(string fault) { return fault.Contains("brake") ? ["brake pads", "cable"] : []; }

/// Write a note on the repair's ticket.
[Action("Write a note on the repair")]
public string NoteRepair(Repair repair, string note) { repair.Note = note; return $"Noted on {repair.Ticket}."; }

/// Set what the customer is charged.
[Action("Set the price")]
public string SetPrice(Repair repair, decimal price) { repair.Price = price; return $"{repair.Ticket} costs {price}."; }

/// Works out how long one repair takes.
public class EstimateRepair : Skill<Repair, Estimate> {
  public override string Description => "Estimate one repair.";
  public override string Instructions => "Look up the parts first. Write the estimate as a note on the repair.";
  public override List<ActionInfo> Tools => [Actions.Of(PartsFor), Actions.Of(NoteRepair), Actions.Of(SetPrice)];
  public override string Brief(Repair r) => $"Ticket {r.Ticket}: {r.Fault}. Its id is {r.Id}.";
  public override string? Check(Estimate e) => e.Hours <= 0 ? "An estimate takes some time." : null;
}

/// Estimates the repairs that come in. It holds the Helper role and nothing more.
[Actor(Roles = [ShopRole.Helper])]
public class Estimator : Agent {
  public static Person Principal => new Person { Email = "[email protected]", DisplayName = "Estimator" };
  public override string Purpose => "You estimate bicycle repairs for a small workshop.";
  public override List<SkillBase> Skills => [new EstimateRepair()];
  public override Permits May => Permits.ChangesAlone();        // it notes and prices by itself
}

[TestFixture]
void Shop() {
  var mia = new Person { Email = "[email protected]", DisplayName = "Mia" };
  var noor = new Person { Email = "[email protected]", DisplayName = "Noor" };
  new RoleGrant { Member = mia, Role = ShopRole.Mechanic };
  new RoleGrant { Member = noor, Role = ShopRole.Manager };
  new Repair { Ticket = "T-7", Fault = "rear brake does not bite" };
  new AgentSeat { Agent = "Estimator", Sponsor = noor };          // the manager answers for the agent
}
principal Mia  => Person.Single(p => p.Email == "[email protected]");
principal Noor => Person.Single(p => p.Email == "[email protected]");

string RepairArgs(Repair r, string rest) { return "{\"repair\":\"" + r.Id.ToString() + "\"" + rest + "}"; }

[Test(Shop)]
[runas(Mia)]
void A_script_answers_turn_by_turn_and_the_trajectory_is_what_the_agent_did() {
  var repair = Repair.Single(r => r.Ticket == "T-7");
  var script = AgentScript.Of(
    Says.Tool(Actions.Of(PartsFor), "{\"fault\":\"rear brake does not bite\"}", 400, 40),
    Says.Tool(Actions.Of(NoteRepair), RepairArgs(repair, ",\"note\":\"One hour.\""), 500, 40),
    Says.HandBack(new Estimate { Hours = 1, Summary = "New pads." }, 600, 60));
  ScriptedAnswer.Stub((model, request) => script.Answer(model, request));

  var outcome = await Agents.Of<Estimator>().Do(new EstimateRepair(), repair);

  Assert.True(outcome.Done);
  Assert.Equal(["PartsFor", "NoteRepair"], Trajectory.Of(outcome.Task));
  Assert.True(Trajectory.Called(outcome.Task, Actions.Of(NoteRepair)));
  Assert.False(Trajectory.Called(outcome.Task, Actions.Of(SetPrice)));
  Assert.Equal(["Everyday", "Everyday", "Everyday"], Trajectory.Models(outcome.Task));
  // 400 + 5×40, 500 + 5×40 and 600 + 5×60 millionths of a dollar.
  Assert.Equal(2200, Trajectory.Cost(outcome.Task));
  Assert.Equal("One hour.", Repair.Single(r => r.Ticket == "T-7").Note);
}

[Test(Shop)]
[runas(Mia)]
void A_result_that_is_not_the_shape_and_a_model_that_is_down() {
  var repair = Repair.Single(r => r.Ticket == "T-7");
  var script = AgentScript.Of(
    Says.HandBackJson("{\"Hours\":\"a while\"}"),                        // not an Estimate — sent back
    Says.HandBack(new Estimate { Hours = 2, Summary = "A while." }));
  ScriptedAnswer.Stub((model, request) => script.Answer(model, request));
  var second = await Agents.Of<Estimator>().Do(new EstimateRepair(), repair);
  Assert.Equal(2, second.Result.Hours);

  var down = AgentScript.Of(Says.HandBack(new Estimate { Hours = 1 })).IsDown("Everyday");
  ScriptedAnswer.Stub((model, request) => down.Answer(model, request));
  var failed = await Agents.Of<Estimator>().Do(new EstimateRepair(), repair);
  Assert.Equal(AgentTaskStatus.Failed, failed.Status);
}