Building a D&D Dungeon Master out of Claude: Giving It a Voice

Every time the audio broke, the model lined up the other tools and pointed. The real bug was usually the frayed cable trailing from its own desk.

By now my D&D Dungeon Master could run a game, write the session up as a story, and illustrate it. The last thing I wanted was to hear it. Each session's story, read aloud on the website, in character voices, so the table could just listen back.

The text-to-speech part worked better than I expected. Then it broke in a way I didn't understand, and the way I tried to fix it taught me the most useful lesson in this whole series.

In this post I share what giving the model a voice cost me, and what I built to claw the time back. I learned that a model will:

The clips started to fall apart

Getting here took a while. I went through a handful of text-to-speech systems before I had something I liked: the browser's built-in voices, a couple of open-source models, and finally one called Qwen that could clone a distinct voice per character. For a while the results were genuinely good. Then individual clips started corrupting.

Not failing outright. Corrupting. A line would start fine and then loop, repeating the same two words over and over. Sometimes it stuttered mid-syllable. Sometimes it dissolved into a stretch of garbled noise that, honestly, sounded like the narrator swearing. One clip would be perfect and the next would sound possessed.

I had no idea why. So I did the obvious thing. I asked Claude.

It told me, with complete confidence, that this was a limitation of the model. Voice cloning just isn't reliable at this length, it explained. A known weakness of text-to-speech (TTS) systems. Reasonable, fluent, and delivered without a flicker of doubt.

I didn't fully believe it. Something about it was too convenient. But I had no counter-evidence and no quick fix, and I had other reasons to look elsewhere anyway.

Why I actually switched (it wasn't just the bug)

I want to be honest here, because "the AI lied and I believed it" is too clean. I had real, independent reasons to try a different model.

Qwen didn't pause properly at em dashes, which mattered for narration. And a model called MOSS-TTS advertised exactly what I wanted: long-form speech, and fine-grained control over phonemes and timing. On paper it was a better fit. My first tests were promising too. The voice auditions sounded good.

So I switched. And this is where the bad diagnosis actually cost me: instead of treating "it's a model limitation" as one unproven theory, I took it as a reason to stop looking at Qwen entirely and commit to MOSS. The misdiagnosis didn't force the switch. It closed the investigation.

MOSS had its own ideas

MOSS sounded fine in short auditions. Then I used it on a full story and it started inserting long, unexplained silences between lines. Two, three seconds of dead air, mid-scene, for no reason I could find.

So I asked Claude again, and got a cascade of confident answers, each one wrong in a new way.

First it was a punctuation problem: the model treats commas and periods as pause signals, so let's strip them. I replaced em dashes with commas. The commas also caused pauses, so it suggested replacing those with spaces. The silences stayed.

Then the theory changed: this is a known artifact where MOSS emits runs of silent frames mid-output. The proposed fix was a post-processor to scan the audio and trim any silence longer than a quarter-second.

That fix made things worse. The silence trimmer used a threshold that turned out to be catching not silence but the quiet parts of speech. It was zeroing out something like a third of the actual audio, leaving glitchy, chopped-up clips. We had fixed a pause problem by introducing a corruption problem.

There was also the small matter of needing half a dozen patches just to get MOSS running on a current version of the underlying library, plus a recurring problem where the same name in the same sentence would come out pronounced differently on each run. By this point MOSS had cost me more than it saved, and its voices, while fine, never had the realism of Qwen's. Qwen's voices breathed. You could hear it. So I went back.

The bug, when I finally looked

When I stopped trusting the diagnosis and actually dug into the original Qwen corruption, the cause had nothing to do with any model limitation.

Qwen clones a voice from a reference sample plus a transcript of that sample. My pipeline wasn't passing the full reference sample. It was being truncated, so the audio and the transcript no longer lined up, and that mismatch was what produced the looping and the garbling. A plumbing bug. My plumbing bug. There was no "limitation" anywhere in sight.

Sit with the whole sequence. The model didn't just get a fact wrong, the way it did in Part 1. It got its own failures wrong, repeatedly, and every wrong answer pointed the blame somewhere other than the actual cause. The original bug was "a model limitation" (it was my truncated input). The MOSS silences were "punctuation," then "a known model artifact" (I never did find the real cause, but the confident fix corrupted the audio). At no point did the model say "I don't know, let's check the inputs."

And once I noticed the pattern, I couldn't stop seeing it. "It's a limitation of the model" turned out to be Claude's house answer for anything audio-related. A voice that wasn't gravelly enough? Limitation of the model. A clone that drifted from its reference? Known limitation of voice cloning. The looping clips? Same. Across three different TTS systems, whenever the audio disappointed, the explanation was some version of "this is just what these models do," delivered with the same calm authority every time. Sometimes that was even true. The problem is it was the answer regardless of whether it was true, which makes it useless as a diagnosis. It's the spoken equivalent of a shrug.

This is the thing I now repeat to myself constantly. The model is a least-reliable narrator when the subject is itself. It diagnoses its own behavior as fluently as it does everything else, and "it's a limitation of the model" is exactly the kind of answer that sounds like expertise while quietly closing the investigation. The habit that saved me was refusing to accept its account of why it failed.

It cost me weeks, a detour through a worse model, and a self-inflicted corruption bug, all downstream of taking confident diagnoses at face value. Not a disaster in a hobby project. The same reflex in production, accepting the model's word for why something broke, is how you spend a sprint rearchitecting around a bug that was three lines of glue code.

The other rabbit hole: it cannot pronounce my party

The bug was the big lesson. The phonetics were the bigger time sink.

Fantasy names do not follow English spelling. The model would read them the way the letters suggested, which was reliably wrong. "Lylnyler" came out mangled. "Xanathar" never landed the same way twice. The voice was good; the names were a disaster.

The fix is a lookup table that respells each name the way it should sound, applied to the text just before it goes to the voice model. It now has more than forty entries. Some of my favorites, verbatim:

"Lylnyler": "Lilnigh-ler"
"Xanathar": "Zannathar"
"Caelith":  "Caylith"
"Hlam":     "Huh-lawm"
"Iokaste":  "Ee-oh-kaw-stay"

A few entries in the table map a name to itself, like "Floon": "Floon". That's not a no-op by accident. It's a note to myself that I checked that one and the model already says it right, so leave it alone.

There is, supposedly, a more proper way to do this. The model I was using advertises pronunciation control through the International Phonetic Alphabet, the /ˈkeɪlɪθ/ notation linguists use. I tried it. It didn't work for me. I'll be honest that I don't know whether the feature isn't really there, whether it's only partly implemented, or whether I just held it wrong. I spent a while trying to find out and couldn't say for sure.

What I can say is that the crude approach, respelling names the way you'd sound them out to a friend, worked reliably when the official one didn't. That's worth a small lesson on its own: a documented feature is a claim, not a guarantee, and "spell it like it sounds" beat the proper notation every time I tried.

The same name, said three different ways

Here's the part that surprised me. A name isn't pronounced one way. It's pronounced differently depending on where it sits in the sentence. The model says "Lylnyler" with one cadence at the start of a line, another in the middle, another at the end. A respelling that fixes the name in one position can break it in another.

So the little tool I built to test a pronunciation doesn't test the name once. It drops the name into three sentences, at the start, the middle, and the end, and reads all three:

Lylnyler stepped forward and placed his hand on the door.
The blast came from nowhere. Lylnyler had moved before
  any of them saw him go.
The cup slid forward, slow and deliberate, and stopped in
  front of Lylnyler.

You listen to all three, and only keep a respelling that survives every position. It's a tiny thing, but it's exactly the kind of edge case you don't see coming until you hear your wizard's name pronounced two different ways in the same paragraph.

Why I ended up with two separate tools

There's a workflow lesson buried in here that took me a while to see.

Choosing a character's voice and fixing a name's pronunciation feel like the same job. They are not. They're two independent problems, and I wasted a lot of regenerations treating them as one.

So I split them into two tools:

Once they were separate, everything got faster. A voice problem and a pronunciation problem no longer forced me to redo each other's work. The general version of this: when one knob seems to control two things, you're probably going to fight it until you give each thing its own knob.

What it taught me about the day job

I'm a cloud consultant. Most of my model work has nothing to do with goldfish or wizards. But this part of the project changed a habit I use every week.

When something built on a model misbehaves, I no longer ask the model why and write down the answer. Structurally, it's the worst-placed witness to its own failure. It will hand you a confident, plausible, blame-shifting story every time. "It's a limitation of the model" is a hypothesis, not a finding. I check the plumbing first now: the inputs, the context, the glue. That's where the truncated-sample bug lived, and it's where most of them live.

The phonetics table and the three-position test are the same lesson in smaller form: the model's failures cluster in the unglamorous edge cases, and you only find them by actually looking and listening, not by asking the model to grade itself.

That, more than the goldfish, is what three months of building a Dungeon Master out of Claude gave me. It is a brilliant improviser, a careless record-keeper, a stubborn illustrator, and an unreliable witness to its own behavior, all at once. The work isn't trusting it or distrusting it. It's knowing which is which, and building the checks that catch the difference.

And if you're listening to this instead of reading it: yes, this was narrated by the very pipeline I just spent three posts complaining about, names and all, and it only took forty lines of respellings to get them right. It's not perfect, but given what I gave it to work with, it's a lot better than I expected, and it was worth the time.

What I gave it was a clone of my own voice. Eleven seconds of me reading into a headset microphone, which Qwen3-TTS turned into a reusable voice prompt. Every sentence you heard was synthesized from that prompt, never from the original recording. Here is that recording:

Credits. The voices use third-party text-to-speech models: Qwen3-TTS (Alibaba) for the character cloning, with MOSS-TTS tried as an alternative. The pipeline around them, the audition and phonetics tooling, and the narration format are my own.

All images in this post, including the header, are my own, generated with the AI image pipeline described in Part 2 of this series.