An AI assistant’s answer from the wrong note reads just as confident as its answer from the right one, so you’ll usually find out after you’ve acted on it. The only way to know ahead of time is to test it on questions whose answers you already know, and it’s less work than you’d think.
In August, before I chose the model that would search the memory my AI agents share, I had my agents test six free models. Every model got the same 719 pages of my notes and the same 54 questions, and for each question they counted it as found when the page that answers it appeared among the first 10 results. The best model, bge-m3, found 78% of them, and the worst one didn’t get past 43%. The notes and the questions stayed fixed for every run, so that 35-point spread came from one choice alone: which model did the searching.
This post shows you how to run that test on your own notes, from writing the questions and scoring them to trying a second model against the same list and reading the numbers honestly. The commands you’ll see are for GBrain, an open-source memory for AI agents built by Garry Tan, the president and CEO of Y Combinator, and the question list works with any AI tool you’ve got.
Why can’t a public score tell you how your memory will do?
Public scores come from public test sets, and that’s where they fall short: a model that has been tuned to a test it has already seen can look better than it’ll be on yours. The team behind MTEB, a public leaderboard for the models that power search by meaning, has built a newer benchmark called RTEB for that reason. A benchmark is a fixed test that everyone runs the same way, and the people who built RTEB keep part of its test data private, so it can’t be studied for. In “Introducing RTEB: A New Standard for Retrieval Evaluation,” published in October 2025, Frank Liu, Kenneth Enevoldsen and their coauthors wrote that “A model with a significant performance drop between the open and the private datasets would suggest overfitting,” which means the model has learned the public questions and does worse on ones it hasn’t seen.
Even vendors say their published scores won’t always carry over to you. Mem0, a memory service for AI agents, publishes strong scores in its README, the introduction a project posts beside its code, and it adds that they “reflect Mem0’s managed platform, which includes proprietary optimizations not available in the open-source SDK,” the free software kit you’d download. I’d call that an honest note, and it’s the reason you’ll want to measure on your own material before you commit to any of them.
What makes a good test question?
A good test question is one you’d really ask, and there’s one note you know holds the answer. Anthropic’s guide “Demystifying evals for AI agents” sets the bar low on purpose: “20-50 simple tasks drawn from real failures is a great start.” So start with the times your AI tool has gotten something wrong or come back empty, and then add the questions you ask it most.
Two habits will make the list worth more:
Decide which of two things you’re testing, because the wording will change the answer. If you’d like to know whether your tool finds your notes the way you really ask, word the questions the way you’d type them on a busy day. If you’d rather test search by meaning on its own, the way my agents compared models, word them with little or no language taken from the note: their 54 were written that way, and keyword search, which matches the words themselves and their simple forms, such as plurals, scored zero on them. I’d start with a list that has some of each, since it’ll tell you both.
Write down the one note that answers each question before you’ve run anything, so you can’t talk yourself into a near-miss afterward.
You can copy this table into a spreadsheet or a text file as it’s written:
question | the note that answers it | first? | top five? | date
what did I quote the spring client | clients/spring-pricing | yes / no | yes / no | 2026-10-28
why did I stop using the old invoicing app | decisions/invoicing | yes / no | yes / no | 2026-10-28If your AI tool answers in a chat, ask each question and mark yes when the answer uses or cites that note, and leave the top-five column empty. If it shows a list of results, you’ll fill in both. By the end of an afternoon you’ll have a number about your own notes that no vendor can give you, and it’s one you’ll be able to check again next month.
Below the line, for free subscribers, is the version that runs itself:
How to turn the table into a test file GBrain can score, and the naming rule that decides whether a right answer counts
The commands that score your list for first place, the top three and the top five, and which numbers to read
How to build a second test memory with a different model and compare the two fairly
How to read close results, and what to do with what the list shows