The stemming kit — an inflected word finds its other forms

Summary#

Full-text search matches words. In English that mostly works; in Swedish, a word has a dozen endings ("faktura", "fakturan", "fakturor", "fakturorna"), and in Arabic an article and a preposition join the front of it. A reader who types the form in their head misses the article that used another. Stem both sides — the text as it is written, the query as it is asked — and two forms of one word meet:

var s = new Stemmer();
s.StemText("Så betalar du dina fakturor", "sv");   // "så betal du din faktur"
s.StemText("hur betalar jag fakturan", "sv");      // "hur betal jag faktur" — both have "betal" and "faktur"

Signature#

using Osysharp.Stemming;

public class Stemmer {
  string       StemText(string text, string language);   // every word, stemmed, joined by spaces
  string       Stem(string word, string language);       // one word, lower-cased and stemmed
  List<string> WordsOf(string text);                     // the words a text is split into
  bool         Knows(string language);                   // does it stem `language`, or match word for word?
}

Description#

Which languages#

A language tag with a region ("sv-SE") is its language. A stem is a pure function of the word and its language, so a text's stems can be written once, when the text is, and searched like any other column.

How to use it in a search#

Keep a stemmed copy beside the text, mark it searchable, and stem the query in the reader's language:

1. When the text is written: row.Stems = new Stemmer().StemText(row.Text, row.Language);
2. When searching: var qs = new Stemmer().StemText(query, readerLanguage); and match r.Text.Matches(query) || r.Stems.Matches(qs).

Matching the text as written as well keeps an exact phrase, a name or a code findable whatever the stemmer does with it. The KnowledgeBase kit does exactly this ([[ui-knowledge-kit#search]]).

Examples#

A recipe site in Swedish and Arabic whose search finds a recipe by any form of its words:

app Recipes {
  use Osysharp.Memory;
  use Osysharp.Stemming@0;
  model "model/**/*.osy";
}

using Osysharp.Memory;
using Osysharp.Stemming;

entity Recipe {
  [Required, MaxLength(120)] string Title;
  [Searchable(Entity), MaxLength(4000)] string Method;
  [Searchable(Entity), MaxLength(4000)] string MethodStems = "";
  [Required, MaxLength(10)] string Language = "sv";
  security { allow read, create, update when IsAnonymous || IsAuthenticated; }
}

public Recipe AddRecipe(string title, string method, string language) {
  return new Recipe { Title = title, Method = method, Language = language,
                      MethodStems = new Stemmer().StemText(method, language) };
}

public List<string> FindRecipes(string query, string language) {
  var qs = new Stemmer().StemText(query, language);
  return Recipe.Where(r => r.Method.Matches(query) || r.MethodStems.Matches(qs)).Select(r => r.Title).ToList();
}

[Test]
void an_inflected_query_finds_the_other_forms() {
  AddRecipe("Kanelbullar", "Rulla degen, strö över kanel och grädda bullarna.", "sv");
  UnitOfWork.Commit();
  Assert.Equal("Kanelbullar", FindRecipes("bullen", "sv")[0]);
}

[Test]
void a_stem_is_the_same_on_both_sides() {
  var s = new Stemmer();
  Assert.Equal(s.Stem("البطاقات", "ar"), s.Stem("بالبطاقة", "ar"));
  Assert.True(s.Knows("sv-SE"));
  Assert.False(s.Knows("fi"));
}

See also#

• field search (Matches / TextScore / Similarity) — full-text search over a field
• The KnowledgeBase kit — a knowledge base and the help centre it is read in — a help centre that searches each article in its own language
• Which language a reader gets — and switching it — the reader's language