How we test
The method behind every test status
Every product has its own test set, and its page says where it stands: the models and date once its test set has run, or "test pending" until then. Results are posted as each test set runs.
1. A test set for each product
Each product has a fixed test set: three to five invented documents with known issues seeded in them, so no client or real matter is ever used. Each document comes with a written list of what a correct output must contain: the required structure, the issues it must catch, and statements it must not make.
2. Run on each supported model
The test set runs on each model the product supports, such as Claude, ChatGPT, Microsoft Copilot and Gemini. Each output is scored against the written list, check by check, and a check passes only when the output meets it.
3. Results posted with the date
The result, the models, the date and any fix go on the product page and in the update log. A product that has not been run says "test pending", and stays that way until it has.
4. Retested when a model changes
The test set runs again monthly and whenever a supported model has a major release. Any drop from the previous run is reviewed by hand, fixed, and recorded with the new version.
5. Time saved, only from timed runs
Where a product states time saved, the figure comes from timed runs in its test set: a reviewer completes the test tasks with the product and without it, and the record shows the median difference, the number of runs, the model and the date. Until those runs exist, no time figure is stated.
See also the editorial policy. Spotted an error in a product? Tell us.