Testing search by meaning — TestEmbedder.Use and a deterministic embedder

Summary#

A field marked [Searchable(Memory)] is found by meaning once the app has an embedder. In a test there is no model to call — and there should not be: a real embedder costs money, needs a key, and answers slightly differently from one release to the next. TestEmbedder.Use hands the test run an embedder of your own instead, so every row the test writes is embedded the moment it is committed, and Memory.Search on the next line finds it.

[TestFixture]
void Kitchen() {
  TestEmbedder.Use(new WordsEmbedder());   // Osysharp.Memory.Testing
  new Recipe { Title = "Lentil soup", Method = "Simmer red lentils with cumin until soft." };
}

Signature#

TestEmbedder.Use(IEmbedder embedder);      // in a [Test] or [TestFixture] only

// Osysharp.Memory — what an embedder is
public interface IEmbedder {
  string Model();                                     // recorded with each vector
  List<double> Embed(string text, int dimensions);    // `dimensions` numbers, the app's vector width
}

// Osysharp.Memory.Testing — a hashed bag of words
public class WordsEmbedder : IEmbedder {
  public int MinLength = 2;                 // shorter words count for nothing
  public List<string> Ignore = [ … ];       // "the", "and", "of" … — the commonest English words
  public List<string> WordsOf(string text); // the words a text is embedded by
}

Description#

What it changes#

For the rest of the test (a fixture's call reaches every test that declares it):

• Every searchable row is embedded at commit, with your embedder, instead of by the background worker an app uses in production. A row written on one line is found on the next.
• Memory.Search embeds its query with the same embedder, so the two are comparable.
• Nothing is metered. A test embedder is not a model call: no budget is spent and no key is read.

The width is the app's own (app.Embedding.Dimensions, or the platform's default), passed to Embed as dimensions — an embedder that answers a different number of values is refused with a message saying so.

What WordsEmbedder finds#

It turns a text into its words (lower-cased, without the commonest ones), hashes each word into one of dimensions slots, and scales the result to length one. Two texts are as close as the words they share. That makes it lexical, not semantic — "invoice" is not near "bill" — so a test written against it searches with words its rows contain, and asserts the things that matter about a search: which rows are found, in which order, and which are never found (a row the searcher may not read, a draft, something erased).

To test meaning across words or languages, write an embedder that knows them — any class implementing IEmbedder will do, and a small table of "these words mean the same" is often all a test needs.

Why it is refused outside a test#

An embedder decides what a search finds. In app code, TestEmbedder.Use would let one line replace the app's real embedder for everyone. So it is a compile error anywhere but a [Test] or [TestFixture], and the platform refuses it on a run that is not a test, saying so. The app needs use Osysharp.Memory; for it to resolve at all.

Examples#

A recipe book searched by what each method says — and a private recipe its owner's friends never find:

app RecipeBox {
  use Osysharp.Memory;
  use Osysharp.Memory.Testing@0;
  model "model/**/*.osy";
}

using Osysharp.Memory;
using Osysharp.Memory.Testing;

[Principal]
entity Cook {
  [Required, Unique, MaxLength(200)] string Email;
  security { allow read when IsAuthenticated; }
}

entity Recipe {
  [Required, MaxLength(120)] string Title;
  [Searchable(Memory), MaxLength(4000)] string Method;
  Cook? KeptBy;
  security {
    allow read where KeptBy == null || KeptBy == user;
    allow create when IsAuthenticated;
  }
}

public List<string> RecipesAbout(string words) {
  List<string> titles = [];
  foreach (var hit in Memory.Search(words, of: [Recipe], limit: 5)) {
    var r = Recipe.FirstOrDefault(x => x.Id == hit.SourceEntityId);
    if (r != null && !titles.Contains(r.Title)) { titles.Add(r.Title); }
  }
  return titles;
}

[TestFixture]
void Kitchen() {
  TestEmbedder.Use(new WordsEmbedder());
  var mia = new Cook { Email = "[email protected]" };
  new Cook { Email = "[email protected]" };
  new Recipe { Title = "Lentil soup", Method = "Simmer red lentils with cumin and onion until soft, then blend." };
  new Recipe { Title = "Flatbread", Method = "Knead flour, yoghurt and salt; rest the dough, then cook in a dry pan." };
  new Recipe { Title = "Grandma's stew", Method = "Brown the beef, add lentils and red wine, simmer three hours.", KeptBy = mia };
}

principal Mia => Cook.Single(c => c.Email == "[email protected]");
principal Leo => Cook.Single(c => c.Email == "[email protected]");

[Test(Kitchen)]
[runas(Leo)]
void a_recipe_is_found_by_what_its_method_says() {
  Assert.Equal("Lentil soup", RecipesAbout("simmer red lentils cumin")[0]);
}

[Test(Kitchen)]
void a_kept_recipe_is_found_only_by_its_cook() {
  runas (Leo) { Assert.False(RecipesAbout("beef lentils red wine simmer").Contains("Grandma's stew")); }
  runas (Mia) { Assert.Equal("Grandma's stew", RecipesAbout("beef lentils red wine simmer")[0]); }
}

See also#

• using Memory (semantic search) — what the tests above call
• [Searchable] — marking a field findable
• Stubbing your own functions in a test (fault injection) — answering for your own functions in a test
• Outbound calls in a test — answering for an HTTP call (Http.Post.Stub)
• Testing model calls without a model — ScriptedModel (Llm kit) — answering for a model call