The paperwork was all there
I was recently advising a small regional bank on their first AI application.
It seemed simple enough. The workflow was simple, the process was simple, and the whole thing had human and machine agents working together. They had done a lot of the right things too: diagrams, model docs, permission lists, guardrails. The paperwork was all there.
Design review is not behavior validation
But when I asked them to show me what the app actually did, end to end, for a few real workflows, it got messy fast.
They said they’d need to go look at the logs. The problem was the logs were scattered across a bunch of different systems, and none of it really lined up.
So I could review the design, but I couldn’t really validate the behavior.
When we ended the session, the PM said nobody had thought about what an examiner would need to actually prove the system worked.
Maybe. But I think the bigger issue was just that the system itself couldn’t be seen clearly enough from start to finish.
That’s not some weird one-off either.
IBM said in May that 55% of organizations are already building or running an agentic operating model. The companies that can show how their AI behaves are the ones that are going to be able to use more of it.
Why settings-checking tools miss it
Most governance tools don’t really help with that. They check settings. That works fine for cloud stuff, where settings usually tell you how things will behave.
Agents don’t work that way. The same agent can do different things with different inputs, even with the same settings. That’s where the problems are.
The proof comes from what actually happened
So the proof has to come from what actually happened.
You need a record of the workflow, the agents, the steps, and whatever evidence gets left behind. And you need a record of the controls, the policies, the checks, and how those checks map to real steps.
That’s what ties it together.
Regulators already expect this. SR 11-7 has been asking for proof of model behavior for years. The EU AI Act and NIST AI RMF point in the same direction. Docs are useful, but docs aren’t proof.
Some teams are solving this already. Most aren’t.
Could you show an examiner?
Could you show an examiner what your workflow actually did last quarter? If yes, I’d be curious how. If not, then you probably hit the same wall I did.
I’m building EvidentAI.app for this problem.