The one-week AI tool evaluation that beats any feature list
A five-day protocol for deciding whether a tool stays or goes, with the four numbers to record and the trap on day five.
Feature comparisons are how people end up with subscriptions they cannot justify. Every AI product's feature list is a superset of every other one's, and none of it predicts whether the tool survives contact with your work. This is the protocol we use instead. It takes five working days and produces a decision you can defend.
Before day one: write the one sentence
Complete this sentence and put it somewhere you will see it on Friday: "This tool must reduce the time I spend on ____ from ____ to ____."
If you cannot fill in the blanks, stop. You are shopping, not evaluating, and shopping always ends in a purchase. Pick at most two candidates — one mainstream option and one specialist. Three or more guarantees a shallow test of all of them.
The five days
| Day | Task | What you record |
|---|---|---|
| 1 | Set up, run the vendor's own example | Time to first useful output |
| 2 | Run your median real task | Percentage of output you kept |
| 3 | Run your worst real input | Whether it degrades gracefully or fails |
| 4 | Hand it to one colleague, no coaching | Where they got stuck |
| 5 | Price it at real volume, then decide | Cost per month at 10x today's usage |
Day 3 is the day that decides it
Everything performs on clean input. What separates tools is behaviour at the edges: the messy recording, the 90-page PDF, the ambiguous brief. Note specifically whether failures are visible or silent. A tool that says "I could not read this" is safer than one that invents a plausible answer, and you can only discover which you have by feeding it something hard.
Day 4 catches what you stopped seeing
By day four you have learned the tool's phrasing and quirks. A colleague has not. If they cannot get value in fifteen minutes without your help, the rollout will fail no matter how well it works for you — that gap is the real adoption cost.
The four numbers
Record only these. More metrics means no decision.
- Keep rate — what fraction of output you used without rewriting.
- Total task time, including editing and fixing. This is the only number the vendor cannot influence.
- Silent failure count — confidently wrong outputs that you had to catch.
- Cost at 10x your current volume, on the tier you would actually need.
A tool with a 60% keep rate and zero silent failures usually beats one with an 85% keep rate and two, because verification cost scales with distrust.
Day five: the trap
Day five is not "do I like it". It is: cancel now, or commit deliberately. Most wasted spend comes from the third option people take by default — leave the trial running and decide later. Trials convert. Set the cancellation reminder on day one, before you have any emotional investment.
Commit only if all four are true:
- Total task time went down, not just output volume up.
- Silent failures were rare and detectable.
- Your colleague got value without coaching.
- Cost at 10x volume is still worth it.
If three of four hold, run one more week on the weak dimension rather than buying and hoping.
Turning the decision into a record
Write five lines: the sentence from day zero, the four numbers, and the decision. This takes two minutes and it is what stops you re-evaluating the same tool next quarter, or being asked to justify the line item and having nothing but a memory.
Tools to run this against
- Editorial shortlists rather than 4,921 listings: meeting tools, note taking, daily planning, CRM.
- The comparison view for two candidates side by side.
- How we review AI tools, if you want to see the same discipline applied at directory scale.
By Editorial team
Weekly AI tools digest
New launches, honest pricing notes and the tools we actually kept using.
Prefer a reader? Subscribe via RSS.