Your AI governance stack only answers half the problem

You know who asked the AI. You don't know if the AI gave the right answer.

Aug 11, 2026

Enterprise AI governance is getting good at answering a question.

Should this person be allowed to generate that?

That’s identity management and role-based access controls. Audit logs. Data-retention policies. Spend limits (now that we’re realizing tokens aren’t free!). Permissions for tools and connectors. And controls over the models available, what information they can reach, and what gets recorded.

This is good. Sensible and secure.

It’s also only half the problem.

There’s another question that gets missed.

Is what gets generated acceptable?

We’re still talking about governance. But the first question was about access. The second is about conformance.

Enterprise AI stacks are much better at the first one than the second.

Anthropic has built in SSO, SCIM, role-based permissions, audit and compliance tools, data-retention controls, observability, spend management, and controls to configure how models connect to outside systems. Microsoft has a growing governance layer around Copilot. Okta is putting work into expanding identity governance so it captures agents as well as humans.

Enterprise companies know how to solve these problems.

I’ve worked at IBM and at J.P. Morgan. These are the types of companies who need to determine who someone is, what they can access, and what they did.

So it’s natural that they apply the same methods to generative AI.

Who invoked that model? Were they allowed to? What data and tools could it use? How much did it cost?

And these could all work perfectly.

The authorized employee signs into the approved AI product at their company. They have all the right permissions. The model is locked down - only the information they’re allowed to see. The interaction is logged, there are traces. Nothing leaks, no policy is breached.

The AI makes some kind of plausible artifact.

And it’s wrong.

Maybe not obviously. But wrong according to all of the organization’s own standards.

It violates accessibility. It ignores a code convention. It’s in conflict with an existing product decision. It invents a component that’s already in your design system.

Nobody catches this. It looks right, and it passed all the governance controls.

It ships.

The governance stack only solved access. It protected the company from misuse of AI.

Conformance governance would protect it from AI that makes the wrong thing.

We’re building the first one faster than the second.

Not my first governance rodeo

Before AI (and after), I’ve spent years working on design systems.

Design systems always had a governance problem - more than they had a component problem.

The component library is the simpler part. The hard questions come when hundreds or thousands of people start using it.

Introduce standards and training. Establish review procedures and processes.

And, if you’re not careful, you become the design systems police.

I wrote about that problem four years ago.

That central team discovers their job isn’t helping people make good decisions. It’s turned into catching people who make bad ones. It’s not governance, it’s enforcement. The system is a gate.

That wasn’t scalable when the people doing the work were all human.

AI’s made the same problem much larger.

A designer can create five credible options in the time it used to take them to make one. Now there are five times as many things that need checking.

Same on the engineering side. There’s much more code to review.

Did the ten pages of analysis that the product manager made this morning align with what the company knows?

The smaller the generation problem, the bigger the output problem.

And we’re only just starting to pay attention to that part of the productivity story.

We’re all striving for velocity. For me, that means a combination of speed and quality.

AI can reduce the cost of making an artifact from four hours to four minutes.

Then we have to spend thirty minutes reviewing that artifact. Another thirty to identify all the deviations. An hour of corrective work. Coordination with other people who also need to review.

The work’s moved, not disappeared.

AI without conformance means the generation cost is converted into correction cost.

And right now more of the correction remains human. So the faster generation makes things even worse.

I wrote earlier this year that execution is becoming cheap while coordination is not.

This is one of the consequences.

The bottleneck moves downstream.

System discovery isn’t the problem

In early evals work at Knapsack, we started to see a clear distinction.

A controlled evaluation. Test a coding agent doing design system tasks. A matched task set, no customer data. Give it two conditions. One where it has an MCP connection that provides guidance for that system. One where it doesn’t.

The MCP doesn’t help the agent find the system. It does that successfully, assuming it has access. It even imports the components at about the same rate.

The agent knows to look in node_modules.

But if it doesn’t have the MCP serving the context, it doesn’t know how to use that system.

Give it that context, and design fidelity improves 5-10%. Code quality improves by more than 20%.

Prompts that produced shippable code rose from 40% to over 60%.

TypeScript errors down significantly. Prop violations similarly reduced.

The cost of running any individual task was higher. But the cost per shippable output was thirty percent cheaper.

This agent didn’t have new access. It already had the components.

But now it had context about how the organization expected the components to be used. Props, variants, compositions, and the constraints around them.

The access problem was already solved. The conformance problem wasn’t.

Conformance improves the economics of usage. Even with our early, simple context provision the benefits are clear and measurable.

The organization’s intent becomes a participant in the generation.

Plausible mistakes compound

The more complex the task, the more the agents benefit from additional context.

On a multi-screen flow, attaching the MCP gave us gains that were nearly double what we saw on simple patterns.

This isn’t a neat scaling. We don’t have enough data. On some simpler template-based tasks, the additional context sees the agent second guess itself and reduces its reuse of the right components.

But high-complexity tasks, broadly, benefit the most.

As tasks get bigger, they involve more decisions. And multiple small plausible but wrong decisions start to compound into something globally wrong.

I always talked about it when it came to design systems. You can take a bunch of completely accessible components and assemble them into a really inaccessible experience.

A complete flow can violate how your company believes the experience should work.

Conformance is broader than compliance

“Conformance” isn’t just another word for regulatory compliance.

Conformance is a broader evaluation of whether an artifact satisfies the constraints your organization has already decided.

Yes, be compliant with accessibility. Dealing with private information. Legal restrictions your company needs.

But some of the constraints and context are just how the organization has chosen to operate.

Brand standards.

Design-system conventions.

Product strategy.

Architectural decisions.

Content guidelines.

Approved patterns.

These are things that teams throughout your company have learned painfully before. And they’d prefer not to learn them painfully again.

Any organization is full of these decisions.

But they’re scattered.

Design system sites. Wikis. Policy documents. An old presentation. Architecture records. Jira tickets. Multiple Slack threads. That developer’s memory. The shared knowledge of the people who were in the room when some decision was made.

AI isn’t magically better than humans at pulling all this context together when it’s time to execute.

Given incomplete organizational context, it’s probably worse. It’ll make more guesses, ask fewer questions, and generate something plausible from the context it does have.

Which is what we asked it to do.

And generic AI evaluation only gets us to a certain point.

A benchmark can tell me if Opus 5 is generally good at coding.

An eval can tell me if GPT 5.6 tends to complete some specific task successfully.

It doesn’t tell me if the code it’s producing is what IBM wants to ship. Or if the artifacts it creates follow Amazon’s design standards.

I say it a lot. AI is very good at making “plausible but wrong.”

Conformance is contextual to the environment.

But we can’t just make another gate

The default enterprise solution is probably going to be simple.

Review everything.

Create an approval process.

Put a human in the loop (...that poor human).

Build a better police force.

It won’t work.

Generative systems are supposed to dramatically increase the amount of plausible work produced. We couldn’t scale human review enough when humans were the only ones generating work. It’s even more impossible to grow proportionally with AI generated output.

Which defeats the economics of using that system to govern AI.

The best design system teams I’ve worked on weren’t successful because they were really efficient at rejecting bad work.

They made doing the right thing the path of least resistance.

Documented decisions. Context. Clear constraints.

Visibility to what already exists.

People were guided towards conformance while they worked. It wasn’t a case of coming to the end of the day and realizing they’d violated some rule they never knew about.

I’ve described this, slightly in jest, as a virtuous panopticon. Not watching everyone. Instead making the possible and the preferred more visible.

The same applies to AI. Conformance can’t come after, it needs to participate in the generation.

Guide the way, don’t gate the path

So that’s an interesting new enterprise AI infrastructure problem.

Not how to restrict what models can reach.

How do we make our organizational standards available to them while they work.

If the model is producing UI it needs to know components, accessibility requirements, voice and tone, patterns, and interaction conventions.

If it’s also writing code, it needs access to architecture decisions, security policies, dependency rules and engineering conventions.

Output needs to be evaluated against those constraints.

Block some failures before they’re made. Warn about others. Point to an alternative.

Or just make the relevant context more visible to the human-in-the-loop.

Governance can mean guidance as much as it means permission.

It’s not enough to have standards. It’s about having standards that are easy to follow.

Which, if you’ve worked in enterprise, you’ll know is very often not the case.

There’s a standard. Or six.

Policies and approved patterns.

And the people doing the work don’t know where to find it. Or how to choose which one they should use.

If it’s a human, they probably stop and ask someone.

Generative AI doesn’t stop. It produces some finished-looking artifact.

That can be a hard failure to notice.

The missing half

AI access governance needs to be sophisticated.

We need all that information about users, controls, audits, permissions and boundaries. Especially when we’re monitoring autonomous systems.

But those controls don’t tell us whether the thing those systems produced should exist.

That’s the conformance layer.

The layer that goes beyond just who can ask the question, but whether the answer that comes back is right.

Approval queues don’t work as governance at scale. They slow everyone down. They make people hate the expert teams reduced to policing them.

AI can actually give us an opportunity to make it better.

Make our standards more easily available when the work is being created.

That means encoding organizational knowledge at the right level. And evaluating our outputs directly against what the organization has already decided.

Guide people and agents in the right direction, before we have to stop them going in the wrong one.

Access protects the company from the AI. Conformance protects the company from what it asked the AI to do.

Robin Cannon

Ready for more?