Giving an agent work — Do, Ask, Stream, For and Decide (Agents kit)

Summary#

AgentOutcome<Estimate> outcome = await Agents.Of<Estimator>().Do(new EstimateRepair(), repair, about: "Ticket " + repair.Ticket);
if (outcome.Done) { Use(outcome.Result); }

AgentReply reply = await Agents.Of<Estimator>().Ask("Which parts does a soft brake usually need?");

Signature#

class AgentRef {
  AgentRef For(P person)                                              // work for somebody other than whoever is signed in
  AgentRef On(string scope, decimal? limit = null)                    // charge the work to something of yours
  Task<AgentOutcome<TOut>> Do<TIn, TOut>(Skill<TIn, TOut> skill, TIn input, string about = "")
  AgentWork<TIn, TOut> Open<TIn, TOut>(Skill<TIn, TOut> skill, TIn input, string about = "")   // `Do` in two steps: the receipt first
  Task<AgentReply> Ask(string question)      Task<AgentReply> Ask(List<AgentTurn> turns)
  stream<AgentSignal> Stream(string question)   stream<AgentSignal> Stream(List<AgentTurn> turns)
  Task<Decision<T>> Decide<T>(string input, string question = "")
}

class AgentOutcome<T> { T Result; AgentTaskStatus Status; bool Done; Guid Task; long CostMicros; string Problem; }
class AgentReply      { string Text; AgentTaskStatus Status; bool Done; Guid Task; long CostMicros; string Problem; }
class AgentSignal     { AgentSignalKind Kind; string Text; string Tool; bool Failed; AgentReply? Reply; }
enum  AgentTaskStatus { Working, Done, Failed, Stopped, Cancelled }

class AgentWork<TIn, TOut> { Guid Task; Task<AgentOutcome<TOut>> Run(); }

void StopAgentTask(AgentTask task, string why = "")                   // call one piece of work off at its next step

Description#

What does Do answer?#

An AgentOutcome<TOut>. Done with Result when the job finished and its result passed the skill's Check; otherwise Status says how it ended and Problem says why: Failed (the model could not be reached, the result never fitted, it went in circles), Stopped (its turns or its money ran out), Cancelled (somebody called it off) — or Waiting: it stopped on a person, a proposal to decide or a question to answer, and goes on when they have (What an agent does alone, what it proposes, what it never does — the gate and proposals (Agents kit), An agent asks a person, and waits — the person rung of the ladder of help (Agents kit)). Inside a workflow Do never answers Waiting: the run waits with the work. None of those is an exception, and each leaves a receipt — outcome.Task is its id. What the tools had already changed stays changed.

How do I reach work while it is running?#

Do is Open and then Run. Open checks everything Do checks and writes the receipt — work.Task — before anything is asked of a model; Run does the work. Keep the id somewhere stored between the two, and whoever needs to can call the work off while it is under way:

var work = Agents.Of<Tracer>().Open(new TraceParcel(), order);
shipment.TracedBy = work.Task;                    // stored before the work runs
UnitOfWork.Commit();
var outcome = await work.Run();
…
StopAgentTask(AgentTask.Single(t => t.Id == shipment.TracedBy));     // from anywhere, while it runs

Work that is called off ends as Cancelled at its next step: a model call already under way is paid for, and none of the tools it asked for runs. Charging work to something of yours is [[agent-budgets#on]].

When do I use Ask, and when Stream?#

Ask is for a question answered in words. The agent has the tools of every skill it holds and answers when it is ready; Ask(turns) carries an earlier conversation. Stream is the same work for an answer somebody watches: its signals are the text as it arrives, each tool as it starts and finishes, and a last Done signal with the reply.

Whose rights does the work run with?#

The agent's, for whoever is signed in: a tool reads what both may read and changes what both may change. For(person) names somebody else — a customer the work is about, the owner of the record a process is handling. In a process nobody is signed in, so a step's agent works with its own roles alone unless it says For.

Who may name somebody else? The work would act with what THAT person may do, so a signed-in caller names themselves and nobody else — unless the app says they may: policy DirectsAgentsForOthers => IsManager;. Anybody else is refused (AgentRefused) before a receipt is written or a model is asked. A process, where nobody is signed in, names whoever its work is for.

Why do Do and Ask answer a task?#

A piece of work waits on a model, so Do, Ask and Decide<T> each answer a task. await it where you call it and it is the plain call. Keep the tasks and several pieces of work are in flight at once — each with its own receipt, its own limit and its own record:

var front = Agents.Of<Estimator>().Do(new EstimateRepair(), frontBrake);     // started
var rear  = Agents.Of<Estimator>().Do(new EstimateRepair(), rearBrake);      // started too
var hours = (await front).Result.Hours + (await rear).Result.Hours;

var both = await Task.WhenAll(                                                   // or wait for them together
  Agents.Of<Estimator>().Do(new EstimateRepair(), frontBrake),
  Agents.Of<Estimator>().Do(new EstimateRepair(), rearBrake));

Each piece started this way runs beside its caller, with its tools, in a unit of work of its own: what one piece's tools write is that piece's, and nothing the other wrote (the examples below run two at once, and wait for two together). They are separate pieces of work: nothing gathers them under one receipt or holds them to one limit. For one job whose parts share a limit and a record, let an agent hand the work on — pieces handed on in one turn are done side by side, under the piece that handed them on.

What is Decide<T>?#

One typed decision — an enum member, a number, a class — with the model's confidence in it (Typed answers from a model — Llm.Extract<T> and Llm.Decide<T> (Llm kit)). No tools and no receipt: it is one metered model call, charged to the agent. When the agent names somebody to consult and the confidence is under its ConfidenceFloor, the decision is asked again of that model (Asking for help — an agent that consults a more capable model (Agents kit)).

Examples#

app Workshop {
  use Osysharp.Agents;
  use Osysharp.Agents.Testing;
  use Osysharp.Llm;
  use Osysharp.Identity;
  use Osysharp.Observability;
  model "model/**/*.osy";
}

using Osysharp.Agents;
using Osysharp.Agents.Testing;
using Osysharp.Identity;
using Osysharp.Llm;

app.Llm = new LlmSetup {
  Models = [ new LlmModel { Name = "Everyday", Client = new ScriptedModel { Name = "Everyday" }, Model = "scripted-everyday",
                            Price = new LlmPrice { Input = 1, Output = 5 }, MaxOutputTokens = 1000 } ],
};
app.Agents = new AgentsSetup { Agents = [new Estimator()], TaskBudget = 0.05m };

[Role] enum ShopRole { Authenticator, Mechanic, Manager, Helper }
partial entity RoleGrant { security { allow read when IsAuthenticated; } }
policy IsMechanic => RoleGrant.Any(g => g.Member == user && g.Role == ShopRole.Mechanic);
policy IsManager  => RoleGrant.Any(g => g.Member == user && g.Role == ShopRole.Manager);
policy IsHelper   => RoleGrant.Any(g => g.Member == user && g.Role == ShopRole.Helper);

// The kit's questions, answered by this app.
policy UsesAgents       => IsMechanic || IsManager;   // gives agents work, reads the work they asked for
policy SupervisesAgents => IsManager;                 // names sponsors, pauses and stops, reads that work happened
policy AuditsAgents     => IsManager;                 // reads everything, content included
policy DirectsAgentsForOthers => IsManager;           // gives an agent work FOR somebody else
policy ReadsModelUsage   => IsManager;
policy ManagesModelSpend => IsManager;

entity Person : Principal {
  security { allow read when IsAuthenticated || IsAuthenticator; allow create when IsAuthenticator; }
}

/// One bike in for repair.
entity Repair {
  [Required, Unique, MaxLength(40)] string Ticket;
  [Required, MaxLength(200)] string Fault;
  [MaxLength(400)] string Note = "";
  /// What the customer is charged. Only a mechanic or the manager sets it.
  decimal Price;
  security {
    allow read, update when IsMechanic || IsManager || IsHelper;
    allow create when IsMechanic || IsManager;
    deny update Price when IsHelper;
  }
}

/// What an estimate says.
public class Estimate {
  public decimal Hours;
  public string Summary = "";
  public bool NeedsParts;
}

/// The parts a fault needs.
[Action("Look up the parts a fault needs", Query)]
public List<string> PartsFor(string fault) { return fault.Contains("brake") ? ["brake pads", "cable"] : []; }

/// Write a note on the repair's ticket.
[Action("Write a note on the repair")]
public string NoteRepair(Repair repair, string note) { repair.Note = note; return $"Noted on {repair.Ticket}."; }

/// Set what the customer is charged.
[Action("Set the price")]
public string SetPrice(Repair repair, decimal price) { repair.Price = price; return $"{repair.Ticket} costs {price}."; }

/// Works out how long one repair takes.
public class EstimateRepair : Skill<Repair, Estimate> {
  public override string Description => "Estimate one repair.";
  public override string Instructions => "Look up the parts first. Write the estimate as a note on the repair.";
  public override List<ActionInfo> Tools => [Actions.Of(PartsFor), Actions.Of(NoteRepair), Actions.Of(SetPrice)];
  public override string Brief(Repair r) => $"Ticket {r.Ticket}: {r.Fault}. Its id is {r.Id}.";
  public override string? Check(Estimate e) => e.Hours <= 0 ? "An estimate takes some time." : null;
}

/// Estimates the repairs that come in. It holds the Helper role and nothing more.
[Actor(Roles = [ShopRole.Helper])]
public class Estimator : Agent {
  public static Person Principal => new Person { Email = "[email protected]", DisplayName = "Estimator" };
  public override string Purpose => "You estimate bicycle repairs for a small workshop.";
  public override List<SkillBase> Skills => [new EstimateRepair()];
  public override Permits May => Permits.ChangesAlone();        // it notes and prices by itself
}

public AgentOutcome<Estimate> EstimateFor(Person customer, Repair repair) {
  return await Agents.Of<Estimator>().For(customer).Do(new EstimateRepair(), repair, about: "Ticket " + repair.Ticket);
}

/// The signals of a streamed answer, as words.
public List<string> Watch(string question) {
  List<string> seen = [];
  await foreach (var s in Agents.Of<Estimator>().Stream(question)) {
    if (s.Kind == AgentSignalKind.Text) { seen.Add("text:" + s.Text); }
    else if (s.Kind == AgentSignalKind.Done) { seen.Add("done"); }
    else { seen.Add(s.Kind.Name + ":" + s.Tool); }
  }
  return seen;
}

[TestFixture]
void Shop() {
  var mia = new Person { Email = "[email protected]", DisplayName = "Mia" };
  var noor = new Person { Email = "[email protected]", DisplayName = "Noor" };
  new RoleGrant { Member = mia, Role = ShopRole.Mechanic };
  new RoleGrant { Member = noor, Role = ShopRole.Manager };
  new Repair { Ticket = "T-7", Fault = "rear brake does not bite" };
  new AgentSeat { Agent = "Estimator", Sponsor = noor };          // the manager answers for the agent
}
principal Mia  => Person.Single(p => p.Email == "[email protected]");
principal Noor => Person.Single(p => p.Email == "[email protected]");

string RepairArgs(Repair r, string rest) { return "{\"repair\":\"" + r.Id.ToString() + "\"" + rest + "}"; }

[Test(Shop)]
[runas(Mia)]
void Do_answers_the_checked_result_and_what_it_cost() {
  var repair = Repair.Single(r => r.Ticket == "T-7");
  var script = AgentScript.Of(
    Says.Tool(Actions.Of(PartsFor), "{\"fault\":\"rear brake does not bite\"}"),
    Says.Tool(Actions.Of(NoteRepair), RepairArgs(repair, ",\"note\":\"New pads and a cable: one hour.\"")),
    Says.HandBack(new Estimate { Hours = 1, Summary = "New pads and a cable.", NeedsParts = true }));
  ScriptedAnswer.Stub((model, request) => script.Answer(model, request));

  var outcome = await Agents.Of<Estimator>().Do(new EstimateRepair(), repair, about: "Ticket T-7");

  Assert.True(outcome.Done);
  Assert.Equal("New pads and a cable.", outcome.Result.Summary);
  Assert.Equal(600, outcome.CostMicros);                 // three calls at 100 tokens in and 20 out
  Assert.Equal("Ticket T-7", AgentTask.Single(t => t.Id == outcome.Task).About);
}

[Test(Shop)]
[runas(Mia)]
void Two_pieces_started_side_by_side_each_run_their_tools() {
  var repair = Repair.Single(r => r.Ticket == "T-7");
  // Each piece's model asks for the parts first, and hands back once it has been told them.
  ScriptedAnswer.Stub((model, request) => {
    if (request.Messages.Count(m => m.Role == "assistant") == 0) { return Says.Tool(Actions.Of(PartsFor), "{\"fault\":\"rear brake does not bite\"}"); }
    return Says.HandBack(new Estimate { Hours = 1, Summary = "New pads and a cable." });
  });

  var first  = Agents.Of<Estimator>().Do(new EstimateRepair(), repair, about: "First look");     // started
  var second = Agents.Of<Estimator>().Do(new EstimateRepair(), repair, about: "Second look");    // started too
  var one = await first;
  var two = await second;

  // Each is a piece of work of its own, and each ran its tool.
  Assert.True(one.Done);
  Assert.True(two.Done);
  Assert.Equal(["PartsFor"], Trajectory.Of(one.Task));
  Assert.Equal(["PartsFor"], Trajectory.Of(two.Task));
  Assert.Equal(2, AgentTask.Count());
}

/// Everything a request says, in words — which repair a piece was given is in its brief.
string Said(LlmRequest request) {
  var all = "";
  foreach (var m in request.Messages) { foreach (var b in m.Content) { all = all + (b.Text ?? ""); } }
  return all;
}

[Test(Shop)]
void Two_pieces_waited_for_together_each_write_what_their_tools_write() {
  Guid first = Guid.Empty;
  runas (Mia) {
    var front = new Repair { Ticket = "T-8", Fault = "front brake squeals" };
    UnitOfWork.Commit();
    var rear = Repair.Single(r => r.Ticket == "T-7");
    // Each piece's model writes its note on the repair it was given, then hands back.
    ScriptedAnswer.Stub((model, request) => {
      var mine = Said(request).Contains("T-8") ? front : rear;      // its brief names the ticket
      if (request.Messages.Count(m => m.Role == "assistant") == 0) {
        return Says.Tool(Actions.Of(NoteRepair), "{\"repair\":\"" + mine.Id.ToString() + "\",\"note\":\"Looked at " + mine.Ticket + ".\"}");
      }
      return Says.HandBack(new Estimate { Hours = 1, Summary = "Seen." });
    });

    var both = await Task.WhenAll(
      Agents.Of<Estimator>().Do(new EstimateRepair(), front),
      Agents.Of<Estimator>().Do(new EstimateRepair(), rear));

    Assert.True(both[0].Done);
    Assert.True(both[1].Done);
    first = both[0].Task;
  }
  // Read afresh: each piece wrote its own note as the estimator, and the two notes are not each other's.
  runas (Mia) {
    Assert.Equal("Looked at T-8.", Repair.Single(r => r.Ticket == "T-8").Note);
    Assert.Equal("Looked at T-7.", Repair.Single(r => r.Ticket == "T-7").Note);
    Assert.Equal(["NoteRepair"], Trajectory.Of(first));
    Assert.Equal(2, AgentTask.Count());
  }
}

[Test(Shop)]
[runas(Mia)]
void A_model_that_cannot_be_reached_is_a_status_not_an_exception() {
  var repair = Repair.Single(r => r.Ticket == "T-7");
  var script = AgentScript.Of(Says.HandBack(new Estimate { Hours = 1 })).IsDown("Everyday");
  ScriptedAnswer.Stub((model, request) => script.Answer(model, request));

  var outcome = await Agents.Of<Estimator>().Do(new EstimateRepair(), repair);

  Assert.Equal(AgentTaskStatus.Failed, outcome.Status);
  Assert.NotEqual("", outcome.Problem);
  Assert.Equal(AgentTaskStatus.Failed, AgentTask.Single().Status);
}

[Test(Shop)]
[runas(Mia)]
void Ask_answers_in_words_and_Stream_hands_the_answer_on_as_it_arrives() {
  var script = AgentScript.Of(
    Says.Tool(Actions.Of(PartsFor), "{\"fault\":\"soft brake\"}"),
    Says.Words("Pads and a cable, usually."));
  ScriptedAnswer.Stub((model, request) => script.Answer(model, request));

  var reply = await Agents.Of<Estimator>().Ask("Which parts does a soft brake usually need?");
  Assert.True(reply.Done);
  Assert.Equal("Pads and a cable, usually.", reply.Text);

  Assert.Equal(["ToolStarted:PartsFor", "ToolFinished:PartsFor", "text:Pads and a cable, usually.", "done"],
               Watch("Which parts does a soft brake usually need?"));
}

[Test(Shop)]
void Work_for_another_person_is_theirs_on_the_receipt_and_only_the_manager_gives_it() {
  // The manager gives the estimator work for the mechanic: it is hers on the receipt.
  runas (Noor) {
    var repair = Repair.Single(r => r.Ticket == "T-7");
    ScriptedAnswer.Stub((model, request) => Says.HandBack(new Estimate { Hours = 2, Summary = "A long one." }));

    var outcome = EstimateFor(Mia, repair);

    Assert.True(outcome.Done);
    Assert.Equal(Mia.Id, AgentTask.Single().Requester!.Id);
    Assert.Equal(Noor.Id, AgentTask.Single().StartedBy!.Id);
  }
  // The mechanic cannot have it work "for the manager" — it would act with what the manager may do.
  runas (Mia) {
    var repair = Repair.Single(r => r.Ticket == "T-7");
    Assert.Throws<AgentRefused>(() => EstimateFor(Noor, repair));
    // Naming herself is always hers to do.
    Assert.True(EstimateFor(Mia, repair).Done);
  }
}

See also#

• Writing an agent — a class with a purpose, skills and an account of its own (Agents kit) — the agent class, its account and who is in charge of it
• A skill — a typed job an agent can be given, its check and its tools (Agents kit) — a typed job, its check and its tools
• The receipt and the record — what every piece of an agent's work leaves (Agents kit) — the receipt and the record every piece of work leaves
• What an agent may spend — a hard limit on every piece of work (Agents kit) — what a piece of work may spend
• Asking for help — an agent that consults a more capable model (Agents kit) — asking a more capable model
• Testing an agent without a model — Says, AgentScript and Trajectory (Agents.Testing kit) — scripting the model in a test