Memory that keeps up when things change

You might recognize this. An organization moves, it lives, and it changes its mind all the time. A vendor raises a price, a customer churns, a colleague leaves the organization or takes over an account. That is why the memory is a core building block of Temperate. A memory that misses facts, or gives an old value with full confidence, is worse than no memory at all, because no one can rely on it.

The best parts of Temperate are the ones you never see or feel. So we show them another way: by being open and transparent about our benchmarks, our progress, scores and our misses, on a public benchmark, LongMemEval [1]. On its 78 knowledge-update questions, where a fact changes and the answer must be the newer value, Temperate answers 77 of 78 correctly: 98.7%.

Over all 500 questions, in one run with one setup, Temperate answers 469 of 500, and every type above 85%. This page shows how we measure, where we stand, and what we changed to get here.

In one run of all 500 questions, on the benchmark's own judge, Temperate scores 95.80% as the average of the six types. That is 0.93 points above the best run of Mastra's Observational Memory [2], a lead that another run could change. Against that run, Temperate leads on three types, ties on one and loses on two.

From the facts it keeps, Temperate answers 77 of the 78 knowledge-update questions with the latest value. That is two questions above the best run of Mastra's Observational Memory, with gpt-5-mini, and three above its run with gemini-3-pro-preview [2].

Temperate does not keep a long transcript to read again for each question. It reads each message once, when it is written, and keeps what is worth remembering as facts: who someone is, what they use, what was agreed and what happened. The answer gets those facts, and passages from a search of the earlier chats.

The hard part is change. A fact keeps two dates: the day it holds, and the day the person said it. When a newer fact replaces an older one, the older one ends, and its value stays as what it was before. So "what is it now" and "what was it before" both have an answer.

A fact that ends by mistake is lost for every later answer; a fact that stays by mistake only stands beside the newer one. So a fact ends only when a model is at least 80% sure that the newer fact replaces it.

Next to the facts, Temperate searches the person's earlier chats, what they said and what Temperate answered, by words and by meaning. That keeps the detail nobody would call a fact, and what Temperate itself said.

LongMemEval has six types of question. On 5 October we ran all 500 with only Temperate's search of the person's earlier chats, which finds what was said by its words and by its meaning: 453 of 500. On 30 October, with a mix of setups and the latest run of each type, it answered 461 of 500. On 31 October we ran all 500 again in one run, with the facts and the search together, and every message read again: 469 of 500, and an average over the six types of 95.8%.

The leaderboard that Mastra publishes gives the six types equal weight, so a type with 30 questions counts as much as one with 133 [2]. Averaged that way, Temperate scores 95.80%: first in this table, 0.93 points above the best run of Mastra's Observational Memory. Counted per question, it is 469 of 500 against 468. That best run still answers more on reasoning about time, 127 against 124, and across several sessions, 116 against 114. The search of earlier chats alone, on 5 October, scored 92.07%.

We read every question we got wrong: what the model wrote, what we kept and what the answer saw. Most misses were ours, not the model's.

1. 1An answer in a fixed shape. The model that reads a message answers in a strict schema: every field is there, empty when the message does not say it, and a fact that links two subjects always names the other one.
2. 2An estimate is a fact. "I think I'm close to 1,300 followers now" is the best figure the person has, so it is kept, with "close to" in its sentence. A guess about what may happen still is not.
3. 3The message before. Each message is read with the person's message before it. "I'm staying on Oahu" then belongs to the birthday trip to Hawaii, and a later stay in Medellín does not end it.
4. 4What holds now replaces what held before. "Especially now that I lead a team of five engineers" is a fact of its own, also inside a sentence about something else. We took this from the state changes in Mastra's Observer [2].
5. 5Nothing read is lost on the way. Some facts the model wrote were dropped by our own checks: a note with a figure next to another fact from the same words, a goal "before the end of the year", and an older fact that "follower" never found because it said "followers".
6. 6Two dates. Each fact the answer reads says when it holds and, when that is another day, when the person said it. "Five sessions, last year", said in October, then reads as newer than "three sessions", said in May. Mastra keeps the same two dates [2].
7. 7Exact words next to meaning. The search also looks for exact words. An exact name in one message is then not lost among messages that are nearer in meaning.
8. 8Longer passages. The search now shows more of a long message, around the words that match.
9. 9The form of the answer. An answer from memory comes first and says what it rests on, with the dates, in one to four plain sentences. A count names each thing it counted. When nothing holds the answer, it says so and names the nearest thing. When the evidence holds two readings, it gives both.

1. 1The questions. The set of 500 questions with about 50 sessions each, as published [1].
2. 2A clean start. Every question gets an organization of its own in Temperate, in a database transaction that is undone afterwards. Nothing carries over from one question to the next.
3. 3As in the product. The messages go through the same code that reads a chat in Temperate, on the day of their session. Only the person's messages are read into facts.
4. 4The score. gpt-4o-2024-08-06 judges each answer with the prompts of the benchmark's own script, and says yes or no.
5. 5The models. GPT-6 Luna, a small and fast model, reads the messages; GPT-6.1 Sol answers; OpenAI's text-embedding-3-large makes the vectors of the search. In the product, Temperate picks the model for each task and changes it when a better one comes.
6. 6Checking it. The answers and verdicts of this run are public, so you can run the benchmark's own judge on them. That checks the count, not that Temperate wrote the answers: the code of this run is in a private repository, so you cannot run it again from this page. The questions and the judge prompts are public [1], and the models are named above.

1. 1We found these changes by reading our misses on this same benchmark, on all six types, so the score flatters us. We have no questions held back to test on. The examples in our rules are our own wording, not questions of the benchmark.
2. 2Longer answers give the judge more to agree with. An answer now says what it rests on, and gives two readings when it cannot decide between them, so part of the gain can come from the form and not from the memory.
3. 3A judge changes a score. On an earlier run, our own stricter judge gave 27 of 30 on preferences, where the benchmark's judge gave 29 for the same answers. Every score on this page uses the benchmark's judge.
4. 4This is one run. A model reads a little differently each time, so a question can pass in one run and fail in the next. In repeated runs of one type we see about two questions of difference, so a lead of 0.93 points is not large.
5. 5We did not run Mastra's system. Its figures are its own. Its benchmark code can retry a failed judgement once on questions it flags, and can use a clarified version of a question [3]; we do not know which of its published runs used these. Our run uses neither.
6. 6In this run the answer always gets the full set of passages from the search. In the product it gets that set for a count, a total, a list or an order. For other questions, a small model reads fewer passages first and the answer gets its notes.
7. 7Temperate keeps facts only from what the person says, not from what the assistant answered. Questions about the assistant's own words rely on the search of earlier chats.
8. 8The benchmark is one person and an assistant. At work, a team shares one memory, with rules about who may see what. We test that with cases of our own.
9. 9GPT-6.1 Sol answers our questions, where the best run of Mastra answers with gpt-5-mini. Part of the difference comes from the model that answers, not from the memory.
10. 10We did not give GPT-6.1 Sol the whole history as a baseline. The full-context row in Figure 4 uses gpt-4o. Hindsight argues that models with large context windows make this benchmark a weaker test of memory [4]. We cannot rule that out.

It is public, it is large enough to show a difference, and its questions look like the questions people ask at work: what did we agree, what is the latest price, when did that happen, what did I say about it last month. Its six types are:

1. 1Facts the person said single-session-user, 70 questions
2. 2Knowledge updates knowledge-update, 78 questions
3. 3What the assistant said single-session-assistant, 56 questions
4. 4Reasoning about time temporal-reasoning, 133 questions
5. 5Preferences single-session-preference, 30 questions
6. 6Across several sessions multi-session, 133 questions