Building a D&D Dungeon Master out of Claude: The Unreliable Narrator

My twelve-year-old son would not look at the monster.

We were ten sessions into a Dungeons & Dragons campaign, and the party had finally won an audience with Xanathar, the most feared crime lord in the city. He's a beholder the size of a draft horse: a floating ball of eyes and teeth, drifting above his desk, waiting for the adventurers to grovel.

My son's character is a half-elf warlock named Lylnyler. He stared at the goldfish.

Lylnyler is the quiet one. He made a pact with a devil and he's ashamed of it, so he keeps to himself and barely speaks in conversations. But Xanathar keeps a pet: a single orange fish named Sylgar, in a bowl on the desk. Lylnyler spotted it the moment we walked in and never looked away. So the Dungeon Master leaned in:

The whole table lost it. Later, when Xanathar handed the party their task and the usual threat, he delivered it looking at the bowl: "If you lie to me about either outcome, Sylgar will have to watch something unpleasant."

Here's the thing. That Dungeon Master wasn't Kaeshan, our usual human DM, whose stories always pull the whole table in. It was Claude.

I'd been building a D&D Dungeon Master out of an LLM. In that moment, it did the thing everyone hopes (or fears) AI can do. It improvised a villain: a consistent voice, an odd soft spot for a goldfish, menace and comedy at once, all in response to a player choice no rulebook ever wrote down.

It was great.

It was also only half the story. For every moment like the fish, there was a moment where the same system confidently got something completely wrong. Those moments taught me far more.

In this post I share three of those failures, and what I built to catch them. I learned, in the most fun way possible, that LLMs will:

Where I'm coming from

By day I'm a cloud consultant. A lot of that day is now spent building with these exact tools, Copilot and Claude, and writing the skills and agents that wrap them so they behave well enough to put in front of a client.

The D&D project is the same work with the stakes turned down. Nobody's production system goes down if my Dungeon Master forgets a rule. So it's become the place I actually learn how these models fail, on my own time, with a campaign full of willing test subjects. Almost everything it has taught me has walked straight back into my job.

The failures below aren't D&D problems. They're the usual problems, in costume.

What I built

I didn't start from scratch. I cloned an open-source project, claude-dungeon-master by PinchOfData. It already turned Claude into a decent D&D 5e Dungeon Master: a persona, a pile of instructions on combat and NPCs, and a complete bundled 5e rulebook the model could look things up in instead of trusting its memory. That last part matters more than it sounds, and I'll come back to it.

What I added is everything that happens after the dice stop rolling. The idea was a co-pilot, not an autonomous DM. The humans still roll. The DM still makes every call. Claude handles the bookkeeping nobody enjoys: tracking hit points, looking up rules, remembering who's carrying what, voicing the minor NPCs.

Then it grew, the way side projects do. It started writing a prose recap of each session, then a website to host those recaps. (In the next two posts it learns to illustrate and narrate them too.) By the time I looked up, I'd added two dozen commands, a stack of specialist sub-agents, and run the whole thing through two campaigns. Twelve sessions of Waterdeep: Dragon Heist alone.

Here's what I want you to notice about all that machinery: almost every part of it exists to compensate for something the model gets wrong. The rule lookups, the memory files, the reconciliation ledgers. They aren't features. They're scar tissue. Each one grew over a place the model failed.

Failure 1: it misreports the world

In session one, Claude ran a location that the module describes as a sprawling warehouse: several rooms, a string of encounters. What we actually played was a single room.

It got worse. The party found a captured NPC bound in that room. But in the real story he'd been dragged off somewhere else entirely and shouldn't have been there at all. And he wasn't a minor detail. Finding him was meant to start a questline: discover he'd been taken, follow the kidnappers across the city, mount a rescue over several sessions. By handing him over, already rescued, in the first room of the first session, Claude quietly deleted the whole arc. We didn't realize what we'd lost until much later.

It hadn't lied, exactly. It filled the gaps with plausible fiction, confidently, and I went along with it because it sounded right.

I learned the obvious lesson: question it, make it check the source. And then, honestly, I fell for it again. And again. A magic item that never got handed out. An NPC who quietly grew a backstory nobody wrote. The lesson is easy to state and hard to live, because the model is so fluent that doubting it feels rude.

The real fix came around session ten. I built a tool to convert the campaign's source PDF into clean markdown, and found that the original import had silently skipped huge chunks of the book. The model hadn't been inventing from nothing. It had been improvising into the gaps the import left empty. Bad ingestion in, confident fiction out.

So Claude and I went through the recovered material together and built a ledger: what the book says, what actually happened at our table, and how to reconcile the two going forward. The model is great at using that ledger. It just couldn't be trusted to notice the gap on its own.

That points at the other half of the fix: memory. The model has none between sessions, so the system keeps it in files: the party's situation, each character's state, the open threads, and a running list of corrections ("this campaign uses experience points, not milestones," wrong once, never again). Continuity doesn't live in the model. It lives on disk, and gets handed to the model fresh each time.

This is where that bundled rulebook earns its keep. The model knows 5e; it's read more D&D than any human alive. But ask it for a spell's exact damage or a monster's saving throw from memory and it will happily make one up. So the system won't let it answer rules questions from its own head. It has to look the entry up first, every time.

Lesson: a lot of what looks like hallucination is the model papering over bad retrieval and missing memory. Fix the ingestion, write the truth down, keep a checked record. Don't ask the model to be the source of truth. Build one and make it use it.

Failure 2: it tells the players things they can't know

This one is different. The last failure was the model getting facts wrong. This one is the model getting them right, and saying them to the wrong people.

A DM holds two kinds of knowledge at once. There's what the characters can see and hear, which you say out loud. And there's what only the DM knows: the cloaked figure's real name, the faction they secretly serve, what they want and won't admit. Running a mystery is mostly keeping those two apart.

Claude didn't keep them apart. It had read the whole module, so it knew everything, and it spilled it. It used NPC names before those NPCs introduced themselves. It folded faction secrets into the text I was supposed to read aloud. Worse, it wasn't even clear which parts were for the players and which were notes for me. It all came out in one stream.

So the tool meant to save me work gave me a new chore. I couldn't just read what it produced. I had to scan every block for spoilers first, live, at the table.

Here's the part that surprised me. The model is perfectly capable of keeping a secret. The moment I told it to separate what the players hear from what only I know, it did, flawlessly, and ever since. The failure was never a missing ability. It was that the model didn't realize the separation was required. I'd been holding an obvious, unspoken rule in my head (of course you don't name the villain before the party learns it) and assuming the model held it too. It didn't, because no one had said it out loud.

That's one of the most common ways working with these models goes wrong. Not the things they can't do, but the implicit requirements you never thought to state, because they're too obvious to a human to mention.

It's the lesson I lean on most at work. Half of writing a good agent is dragging your own assumptions into the open ("obviously you'd never expose that field," "of course you check permissions first") and writing them down, because the model holds none of the context you think is shared.

The fix isn't an instruction, though. Instructions drift. It's a format. Everything Claude narrates now comes in two separate layers:

The italic line is always safe to read aloud. The bracketed line never touches it. A leak is now visible. A true name above the brackets jumps out, instead of staying buried in a paragraph I'm already reading to the table.

Lesson: the model will happily do the right thing. It just won't know the right thing is required until you say so. State the obvious rule out loud, then give it a structure that makes the rule easy to follow and a slip easy to spot. Make the format the guardrail.

Failure 3: it won't stay in character

The other half of the DM's job is writing the session up afterward, as prose. Real prose: no dice, no "hit points," no "he leveled up." Characters don't experience their lives as game mechanics. They experience fear and cold and the weight of a sword.

The model would not stop leaking the mechanics back in.

I tried hard to stop it. The story-writer's instructions forbid game terms in eight different ways: a style section, a ban-list with rewrites, a whole subsection banning numbers, a rule against dice results, a rule against session numbers, a combat section, and a final checklist repeating all of it.

And session one's story still contains this line:

"He rolled his initiative in his head, picked a side, and stepped in."

Rolled his initiative. Eight times I told it not to, and the weight of all that D&D text it trained on, where people absolutely do "roll initiative," pulled it right back.

Lesson: you can't fully prompt away a strong prior. Repetition and examples lower the error rate; they never get it to zero. So that checklist at the bottom of the instructions isn't really a prevent list. It's a catch list, something a human or a second pass uses to find what slipped through. At work this is the difference between a prompt and a pipeline. The prompt lowers the error rate, but you still need a verification step (a second model, a linter, automated tests, a review) to catch the rest. I stopped expecting the instructions to hold on their own, and started budgeting for the catch.

Back to the fish

So which is it? Is Claude the improviser who gave a beholder a goldfish and the perfect deadpan, "Grix is not funny"? Or the unreliable narrator who flattened a warehouse into one room and can't stop writing "initiative" into a novel?

It's both. Same system. That's not a contradiction to fix; it's just the shape of the thing.

The mistake is to pick one. Trust it blindly and you'll ship its hallucinations straight into your campaign. Distrust it completely and you'll never let it improvise a goldfish. The job is to build the scaffolding that lets you keep the good and catch the bad.

Forced lookups and a reconciliation ledger, because it misreports the world. A two-layer format, because it spills secrets it was never told to keep. A catch-list, because it won't stay in character. None of that made the model less fun to play with. The fish still happened. It just meant that when the magic failed, and it always eventually does, something was there to catch it.

That's what building a Dungeon Master out of an LLM taught me. Not how to prompt it perfectly. How to stop needing it to be perfect.

It's the same lesson I keep relearning at work, minus the goblins. The agents I write for clients live or die on the same handful of habits this campaign drilled into me: don't let the model be the source of truth, write the implicit rules down, separate what it says from what it knows, and always build the step that catches what slips through. The stakes are higher in production. The failures are exactly the same, just more fun to debug when there's a beholder involved.

In Part 2: the same lesson in a completely different medium. I tried to get it to draw my party, and it kept handing me back women and children.

---

---

Credits. Built on top of claude-dungeon-master by PinchOfData (MIT). The DM persona, the instruction set, and the bundled rulebook came from there. The D&D 5e System Reference Document is © Wizards of the Coast, used under the Open Gaming License v1.0a (markdown/JSON conversion by Ben Morton). Dungeons & Dragons, the Forgotten Realms, Waterdeep, and Xanathar are trademarks of Wizards of the Coast; they appear here as references to a published setting our table plays in, and this post is not affiliated with or endorsed by Wizards of the Coast. Everything past the dice (the story, website, image, and audio generation) is my own.

All images in this post, including the header, are my own, generated with the AI image pipeline described in Part 2 of this series.