Skip to main content
Paul Welty, PhD AI, WORK, AND STAYING HUMAN

The first real user breaks everything

Your product works until someone actually uses it. The gap between 'works in dev' and 'works for a person' is where most systems fail — and most organizations avoid looking.

Duration: 5:48 | Size: 5.3 MB

Hey, it’s Paul. Thursday, March 5th.

Every organization has a version of the product that works perfectly — as long as nobody touches it. The staging environment passes all the tests. The demo flow looks great. The deck is polished. And then a real human being sits down, clicks the first button, and the whole thing falls apart.

That happened today. A prospect — a prospect from a conference management company — sat down next to me and tried to use Authexis for the first time. Not a rehearsed demo. Just a person trying to log in, read a briefing, and click a link. Three things. And all three broke.

The login flow sent a magic link that pointed to localhost — my development machine — because the environment variable was baked into the build at deploy time with the wrong value. The briefing email arrived fine, but when he clicked an article link, it dropped him at a login wall because the link went through an internal click tracker to an authenticated page. And when we fixed the magic link URL, the confirmation page only understood an older authentication flow, not the newer one our library was actually using. So the link verified correctly at the identity provider and then failed silently at our front door.

Three bugs, stacked. Each one invisible in development. Each one perfectly reasonable in isolation. Each one a complete dead end for a real person.

Here’s what I want to talk about: why this keeps happening, not just in software but in every system that humans build for other humans to use.

The gap between “it works” and “it works for someone” is where most failures hide. We test our own flows. We click through our own interfaces. We verify our own assumptions. And because we built the thing, we unconsciously route around the problems. We know not to click that link. We know to use the right browser. We know the magic link takes a few seconds. A new user knows none of this, and every assumption they violate becomes a wall.

Organizations do this constantly. They build onboarding processes that make sense to the people who designed them. They write documentation that answers the questions they’d ask, not the questions a newcomer actually has. They create approval workflows that route perfectly when you already know who to ask. The system works for insiders. It’s hostile to everyone else.

The uncomfortable truth is that isolation isn’t just a technical problem — it’s a psychological one. We build in isolation because it’s safe. Nobody is watching when you’re in the staging environment. Nobody is judging when you’re iterating in dev. The moment a real person sits next to you and tries to use the thing you built, you’re exposed. And most of us would rather keep polishing in private than face that exposure.

Intelligence that stays in one place isn’t intelligence — it’s a party trick. There’s a parallel story from today. I have a reading tool that learns what you care about by watching what you click, bookmark, and vote on. It analyzes those patterns and writes a profile of your interests. Yesterday, that learning only fed into one part of the system — article scoring. Today, it got wired into everything: the RSS scanner that finds new sources, the search engine that discovers articles, the briefing generator that decides what to highlight, the initial filter that decides whether an article is even worth scoring.

Same learning. Four times the reach. The system didn’t get smarter today. It just started applying what it already knew.

This is how most organizational knowledge works, too. Someone on the sales team learns something crucial about a customer. It lives in their head, maybe in a CRM note. It never reaches the product team, the support team, the marketing team. The organization learned something. But the learning stayed in one room.

The fix isn’t more meetings or more Slack channels. It’s structural: you build the system so that what’s learned in one place is automatically available everywhere it could help. In software, that’s a shared module. In organizations, it’s harder — it might be a ritual, a routing rule, a forcing function that ensures the insight travels. But the principle is the same. Learning that doesn’t propagate isn’t learning. It’s a local variable.

Every connection between systems is an unwritten contract. The third pattern from today: an email signature was pulling “Latest post” from a content feed, and it picked up an internal synthesis document instead of a real blog post. Why? Because the synthesis was missing a single metadata field — content_type — that the signature template assumed would be there. The synthesis pipeline never knew about the signature pipeline’s expectations. Nobody told it.

This is what happens when systems compose. Each one works fine in isolation. Each one has reasonable assumptions about what it’s consuming. And when you connect them, those assumptions clash in ways that neither designer anticipated.

In organizations, these are the integration boundaries — the handoff points between teams, between tools, between processes. The sales team hands a lead to the account team. What fields are required? What context travels with the handoff? What gets lost? Usually nobody writes this down. The contract is implicit, enforced by habit, and broken the moment someone new joins or something changes upstream.

The fix is tedious and unglamorous: make the contracts explicit. Document what each system expects from every other system it touches. In software, that’s typed interfaces and schema validation. In organizations, it’s handoff checklists and intake forms and the kind of process documentation everyone considers beneath them until the first handoff fails.

So here’s what today taught me. You can build something that works in development, passes every test, demos beautifully — and still have it collapse the moment a real person with no insider knowledge sits down and tries to use it. Not because the work was bad. Because isolation is a lie. The only test that counts is the one where someone who doesn’t know the workarounds tries to do the thing.

What’s the version of this in your work? What’s the system that works perfectly — as long as nobody new ever touches it?

Nobody takes you aside anymore

Print taught a generation when to stop. What we lose when the machines absorb the constraints that used to form us.

Your AI agents need a water cooler

Coordination is a property of the room, not the org chart. What that means when your coworkers are agents.

On the death of the author and the birth of the detector

Why worrying about AI authorship is lazier, and more prejudiced, than it looks.

The work of being available now

A book on AI, judgment, and staying human at work.

The practice of work in progress

Practical essays on how work actually gets done.

Memory is (almost) solved. time is next.

AI can't tell if a memory is two minutes or two weeks old. The fix isn't making models feel time — it's cache invalidation: an as-of stamp on every fact, a clock in the context, and a freshness window for anything volatile.

Did the state change? A simple test for whether work actually happened

Either something exists now that did not exist before, or it does not. A simple test for whether work actually happened, and what changes when you build your systems so they can't record anything else.

The loop nobody bothers to close

Most systems observe. Almost none learn. The difference is a feedback loop — and the boring cleanup work that makes it possible.

Every organization is full of unmarked graves

The way a system handles mistakes tells you more about its values than anything it says about itself. Banking, git, and distributed databases all default to keeping history. Organizations are the one system that still believes in erasure.

The day we shipped two products and the agents got bored

112 issues across 12 projects. Two new products went from nothing to code-complete MVP in single sessions. And the most interesting signal wasn't the speed — it was the scout that came back empty-handed.