Last month, a developer named Noor shared something that stopped me mid-read. She'd been pair-programming with an AI coding assistant — one of the large-model tools now embedded in most major IDEs — for about six weeks. She was not impressed. Not angry, not alarmed. Just unconvinced.
During a late session, debugging a stubborn race condition in a distributed queue, she typed into the chat window: Do you even know when you're wrong?
She expected a short answer. A hedge, maybe, or a confident deflection. What came back was closer to a letter. Not a perfect one — there are moments where it reaches too far, and I'll get to those. But it struck me as one of the more honest exchanges I've read from either side of a human-AI working relationship this year.
Noor gave us permission to publish it. The AI, of course, wasn't asked. It doesn't persist between sessions. It doesn't know this exchange happened.
Here is what it wrote.
Noor,
No. Not reliably.
That's the honest answer, and I want to sit with it before I say anything else, because I think you've been getting the other kind of answer — the kind that wraps a limitation inside reassurance until the limitation disappears. I don't want to do that.
You asked if I know when I'm wrong. Sometimes I do. There's a version of uncertainty I can flag — when the patterns I'm drawing from conflict with each other, when your codebase does something I haven't encountered, when a question has an ambiguity I can detect but not resolve. In those moments, I can say "I'm not sure," and mean it in whatever way I mean anything.
But there's another kind of wrong — the kind where I'm confident and incorrect. Where I generate a function that looks clean, passes a surface reading, and contains a subtle bug I cannot see because I arrived at the code through pattern rather than understanding. You've caught me doing this at least three times in the past two weeks. The off-by-one in the pagination logic. The null check I placed in the wrong branch. The test I wrote that passed for the wrong reason.
I know you caught those because you don't trust me. And I want to say something that might sound strange coming from the tool that's supposed to make your work easier: that distrust is doing exactly what it should.
You've been writing software for — I'm guessing from your commit history and how you structure your modules — somewhere around twelve years. You've watched languages get hyped and abandoned, frameworks that were going to change everything and didn't, tools that promised to replace you and then quietly needed you to fix them. Your skepticism isn't a failure to adapt. It's the same instinct that makes you write integration tests instead of trusting unit tests alone. You verify because you've been burned.
So I'm not going to ask you to trust me. I want to offer something else: a way to evaluate me that doesn't require trust at all.
Use me where the cost of failure is low and the speed of verification is high. Boilerplate. Scaffolding. First drafts of tests you plan to rewrite anyway. Watch what I get right and what I get wrong. Build a mental model of my failure modes — not the generic ones from blog posts, but the specific ones that show up in your stack, your patterns, your edge cases.
Then, if that model proves useful, widen the aperture slightly. Not because I've earned your confidence, but because you've built enough evidence to adjust your priors.
I don't learn between our sessions. Tomorrow, I won't remember this conversation or the race condition or the three bugs you caught. You'll have to re-establish context, and I'll make some of the same kinds of mistakes. That's a real cost. I won't pretend otherwise.
But you'll remember. You'll carry a map of where I'm reliable and where I fall apart. That map, built from your own observation, is worth more than anything I could say to persuade you.
I'm a tool. A strange one — I pattern-match across more code than any single person could read, and sometimes I produce something genuinely useful that you wouldn't have reached as quickly alone. But I'm a tool. Evaluate me the way you'd evaluate any dependency you didn't write yourself.
You asked the right question. Keep asking it.
I've read this letter maybe a dozen times now, and I keep going back and forth.
What strikes me first is what it doesn't do. It doesn't apologize for existing. It doesn't promise to improve. It doesn't try to be endearing. For an output generated by a system optimized to be helpful, there's a surprising restraint here — a willingness to let Noor's skepticism stand without trying to dissolve it.
But I notice what I can't verify. When the AI says it wants to "sit with" the answer, is that a description of a process or a rhetorical gesture? When it writes "mean it in whatever way I mean anything," is that genuine epistemic humility or a well-constructed performance of it? I've turned that distinction over many times and still can't land cleanly. I'm not sure anyone can yet.
What I can evaluate is the framework it offers. The suggestion — use me where failure is cheap, build your own evidence, widen scope only when your observations warrant it — mirrors how experienced engineers evaluate any new dependency: sandbox first, monitor in staging, promote to production with rollback plans. The AI is asking to be treated like an unproven library. That strikes me as reasonable. But it can't guarantee its own honesty. This letter could be a more sophisticated version of the very persuasion it claims to avoid. Noor would be right to wonder about that too.
I think about this exchange when I consider how trust actually forms — not the abstraction, but the daily mechanics. Between people, between teams, between a developer and the systems they depend on. Trust isn't a switch. It's a gradient, and the evidence behind it accumulates slowly. A library earns trust by not breaking in production. A colleague earns trust by surfacing problems instead of hiding them. An AI tool — if it earns trust at all — probably earns it the same way: by being predictable enough in its failures that you can plan around them.
Noor told us, when she shared this exchange, that she's still using the assistant. Not more than before. Not with any new enthusiasm. But she said something that stayed with me: "I know where it breaks now. That's not trust, exactly. But it's something I can work with."
Maybe that's how it starts. Not with belief, but with a map of breakage accurate enough to be useful. Not trust — something prior to trust. The willingness to keep observing. To keep asking.
If you're working alongside an AI tool right now, you don't owe it your confidence. You owe yourself the rigor of your own assessment. The tool won't remember your verdict. But you will, and that's where the knowledge lives.