My favorite AI skill is my deep research skill.
It sends the model looking for evidence, asks it to pursue unexpected connections, and turns what it finds into a report. I use it regularly. It’s my “I know kung fu” skill.
I think the lateral-thinking agent that supports the skill is especially clever.
I like it, I use it, I think it’s clever.
None of that is evidence that it comes up with a better answer than the model on its own.
I know that because I’ve tried to measure it. I’ve tried to rebuild it. I’ve tried to measure the rebuild.
I made the mistake of turning a number into a target. I’m still not sure what the numbers mean.
I wanted to rate my skills
I had the idea of trying to benchmark the various AI skill suites I’d built. Rank them against other, similar, tools.
My research suite scored 2.2. Claude Research scored 3.0. Another system, called STORM, scored 2.7.
It probably wasn’t a great benchmark. My own skill came up with the rubric, and the same model did the ratings. For my own skills it examined them in full. For the others, it scored them from the public description.
Despite all that, I set out to improve my 2.2.
The score came with some gaps already listed.
No formal research plan before a search
No independent citation check
No search for contradictory evidence
So I built them in. I wanted a score better than 3.0.
But the score was just about methods and architecture. I could add those things. It couldn’t tell me if I’d find the report more useful. I ran myself straight into Goodhart’s Law.
By the time I got around to thinking about re-running scores, the numbers had stopped meaning anything.
All the checks passed
I’m pretty rigorous when it comes to making stuff with AI.
Before I made any changes to my skill, I ran the original against a set of briefs. I wrote down what the rebuild had to preserve.
I made checks. A separate verifier skill. I had Codex review the plan that Claude wrote. It found problems, I fixed them. Then I fixed the things the fixes broke.
It wasn’t perfect. The workflow kept deciding it was finished before it had covered the whole question. It’d give itself a passing grade, and often count “couldn’t find evidence” as “the claim is false”. I changed the rules for how it searched.
When I’d measured the rebuild against a couple of planned briefs, I merged it.
It had more checks. But that wouldn’t tell me if the reports were better. So far all I’d provably done was introduce more process.
Three ways to do the job
“Claude Research” is a multi-tool methodology rather than a product. I couldn’t run a direct comparison against it.
Instead I compared three approaches to the same brief.
My original skill.
The rebuilt skill.
The bare Claude Opus model without the skill.
It all ended up in a tie. Under the pre-registered rules, no one approach could beat another. My rebuild won one brief. It lost another one.
The bare model looked just as good when we checked whether it supported its claims. But it also made many fewer claims, so it had less to check and less to get wrong. The rebuild was verbose about the gaps in its evidence, and it wrote down a lot of working links.
Those aren’t comparable.
A short report might be easier to support. A report with exposed gaps might be more useful to scrutinize, but might not be more accurate.
The price was a big difference.
The bare model runs cost $1.19 and $1.46. The rebuild run cost over $10. That needs a lot more justification than just my sense that the workflow is more sophisticated.
None of the measurements was good enough for me to rank the reports. The rebuild made some things work better. It kept its research files for provenance. But then the checker penalized those sources, because it couldn’t open them.
When models grade models, there’s lots of scope to bring in errors. The choices the models made during the grading process can be inspected. But that doesn’t mean they’re good.
One question, more disagreement
I ran a comparison using a real question.
Which professions does AI threaten most, and does public perception match the evidence.
I wrote down the criteria I wanted to judge from. Then I ran all three approaches at once. It’s not a measurement so much as a structured opinion.
The three reports all reached the same conclusion.
They got there by different routes.
My original research skill found more. On this question, the lateral thinking passes found a study about translation that the rebuild missed. It cost $7.24.
The rebuild made it much easier for me to inspect uncertain sources. It had counter-evidence, and much better connection of claim to source. It cost $11.61.
The bare model helped with orientation around a topic. But it didn’t test its own research. It cost $1.45.
I stayed especially skeptical about the rebuild. Even verifying its claims, it still described one effect as causal when the paper it was citing only showed an association. It flat out stated there were no reports on translation, a source the original had found.
So now I’d checked the individual claims. And I still didn’t have a complete search.
I had a wide range of costs, and a better disagreement.
The answer I got
For now, I ended up splitting my research suite.
The original tool is now dr-explore. It’s for discovery: finding more leads, making more tenuous or imaginative connections.
The rebuild is now dr-rigor. It’s for work that’s more likely to be directly challenged by other people. Showing the gaps and the evidence paths.
And the bare model can still be a useful, cheap place to start.
That’s a decision about how I’m going to use the tools, not whether they improve things.
I’ll use dr-explore for finding. dr-rigor for answers I can defend. And the bare model if I’m not even sure of the question.
If those habits don’t hold, then I’ll know I need to re-examine the split.
I trust my original skill enough to keep using it. And I know better than to stop checking.
Further reading:
My deep research skill suite on GitHub - which includes both original and rebuilt skills.
Actually, the shape of the work does change - on evaluating skills built for humans versus skills built for agents.
Dunlop, A. Your Claude Skills Do Nothing. Here’s A Test To Prove It. Medium, Jun 2026.
Do public SKILL.md files actually make agents better? EverMind, Aug 2026.
Curley, B. Prompting 101: 6 ways to make AI a better thinking partner. Bryant University, Oct 2026.
Article Photo by Piret Ilver on Unsplash.