Essay · Human-centered AI
Trust Is a System
Governing autonomous agents in the enterprise
This is my own professional perspective, written to be shared publicly. It contains no proprietary information.
Why I’m writing this.
I’ve spent nearly two decades working on products where people interact with complex systems, from Windows and connected devices to identity, security, and now AI. Looking back, I’ve been answering the same question in different forms: what do people need to understand before they can trust a system to act on their behalf? As software becomes more autonomous, that question becomes more important. I believe we’re approaching it with the wrong instincts. This is the argument I keep making, written down.
The shift hiding inside the AI moment
Most enterprise AI conversations focus on capabilities: better models, better reasoning, more automation. I think the bigger shift is different. Autonomous agents now read systems, make changes, and trigger consequences at a scale and speed no human can check one item at a time. Now, enterprise software must help people supervise systems that operate themselves.
For decades, administrators provided the intent, exercised judgment, and acted while software waited to be configured. But the admin’s question has changed from “How do I operate this system?” to
“Can I trust what this system is about to do, and will I know, in time, if it goes wrong?”
That’s not a feature request. It’s a fundamentally different relationship between people and software.
An employee reports a suspicious sign-in. Before, the admin opened a console, investigated logs, correlated the signals, revoked sessions and restored access. Now, an autonomous system may have already done most or all of that. The admin’s job has changed from doing every step to understanding why the system acted, deciding if it made the right choice, and stepping in if necessary. That’s supervision.
From configuration to supervision
Administration has evolved in stages. First, we configured individual systems. Then, as systems multiplied, we managed resources at scale through consoles. As risk increased, management became governance – using policy, compliance and audit to prove that systems are safe, and operating as intended. We enriched the surface: a better dashboard, more controls, more logs, more alerts, more visibility.
But the system being governed has now started making decisions. Admins need to understand what these autonomous systems are doing, decide if or when to trust them, and step in when needed.
Most product roadmaps are still improving dashboards, consolidating portals and bolting on chat assistants while assuming a human still holds all the judgment. But judgment is now shared between human and machine. You can’t dashboard your way into trusting an agent any more than you can manage a new coworker with a settings page.
When judgment is shared, trust becomes the product
Visibility isn't supervision
Good supervision starts with visibility, but it’s not enough.
More data doesn’t create better decisions. Visibility was the foundation of management: show more, surface more, alert more. But when systems act faster than humans can read, more information adds cognitive load, not better decisions. Attention and trust become the new scarce resources.
Trust starts beneath the screen. Great UX makes complex systems easy to understand, but it can’t compensate for a system that behaves inconsistently. If different parts of the product produce different answers, people eventually stop trusting all of it.
Autonomy needs boundaries. Enterprise admins aren’t looking for AI that does everything. They’re also not trying to stop employees from using AI. They want routine work to happen automatically, and high-impact work to be supervised, based on context, risk and accountability. They’re trying to decide where autonomy belongs, and how to design systems that earn the right to exercise it.
A model for trustworthy autonomy
I think we can design for trustworthy autonomy. These five characteristics turn “trust the system” from an aspiration into a system we can build:
A coherent operating model. Each actor has a single source of truth consistently surfaced across all workflows. Coherence is an architectural requirement, not a design choice.
Identity as the control plane. Every actor, whether a human, workload, or agent, has a governable identity, an accountable owner, and defined permissions.
Design for decisions. Guide the administrator towards decisions only they can make. Everything else quietly takes care of itself.
Adaptive autonomy with boundaries. Automate low-risk, routine, tasks. Expand autonomy as systems build a track record of reliability. Reduce automation the moment that changes. Consequential actions are explainable, reversible and never happen silently.
Runtime governance. Once a system can act autonomously, what it is doing right now matters more than what it was set up to do. Measure velocity of control: how fast the system detects & contains unexpected behavior and brings the right human in.
One principle sits across all five: govern the governor. Any system that supervises autonomous systems must itself be governed, with an identity, owner, permissions, and an audit trail. If the supervisor sits outside the governance model, privileged and opaque, the promise breaks down.
There is no trust without governance
Human-centered AI in a supervision system
Human-centered AI means reserving judgement for the decisions only humans can make, while safely automating everything else. People set boundaries, supervise the system and continuously calibrate trust.
Where this goes
The number of things enterprises need to govern is about to grow much faster than the number of people available to govern them. Those things will increasingly make independent decisions.
The question is no longer whether we can automate more work. It’s whether we can build systems that people are willing to supervise, and ultimately trust. Trust isn't something you earn with a polished interface.
Trust is what you design into the relationship between people and autonomous systems
That’s the work.