Keeping the person in the story

An invitation to carry on

The conversation design starts with a small expectation: stay with what the person has just said.

A useful follow-up picks up a detail. It asks one thing. It leaves enough room for an answer. It also needs to respect a topic someone has declined and recognise that a hesitation might be thinking time.

These choices sound obvious in writing. In voice, their timing changes how the whole exchange feels. The same question can feel attentive or hurried depending on when it arrives.

The player is where a family meets the result, so it gets the same care. The waveform is decoded from the recording itself, or not drawn at all: inventing the shape of someone’s voice would be exactly the dishonesty the product exists to avoid. The rail is a real slider, and the transcript follows the voice, so any moment is one tap away. The sample below runs in a port of Tellby’s own player.

Listen to the moment you’re judging

I built a conversation linter so I could talk to the real Tellby agent and see, turn by turn, whether it was having the conversation I designed. It builds the session from the same prompt code production uses, records both voices on one clock, and judges each exchange as it ends.

Each rule is a small JSON file: what it evaluates, the eval it belongs to, the prompt constraint or research behind it, and the thresholds. Some checks are measured in code from the audio: the gap before Tellby replies, how long it spoke, and whether it talked over the storyteller. Others need interpretation: whether a follow-up builds on the story, asks two things at once or strays toward a do-not-ask topic. JEV answers focused questions for those, and code decides which condition matched.

Judgements print as small thermal receipts beside the conversation. Opening one traces it back through rule, evidence, JEV’s answer and severity, and replays the audio behind it. I label whether JEV got it right, and a Compare view groups sessions by prompt version, so a prompt change shows up as a change in pauses, overlaps and failures, not a feeling.

The linter is a development tool. Its 18 starter rules carry thresholds meant to be tuned by ear, and the live-session path still needs validating in a real spoken session. The figure below replays its scripted demo session, with a do-not-ask slip, a double question and a reply that talks over June.

A polished sentence still needs a source

Tellby also turns sessions into material a family can read. The design question changes here: has the writing kept faith with what was actually said?

The organiser can review a memory card before the family sees it. That approval boundary is enforced in the database. For generated letters, a support check compares each sentence with its cited facts and flags anything the stories do not support.

I built a letter evaluation studio to examine the check itself. A sentence sits beside its facts, and “Hear her say it” plays the storyteller saying the words behind them. I label it fine, adds something, or contradicts a fact. Running the judges sends every labelled sentence through the production check and JEV as a second judge, and only the disagreements print. When both judges disagree with me, that is the first place to look for a wrong label.

That matters because a checker can also be wrong. A missed invention changes the family’s account. An unnecessary flag gives the organiser another thing to untangle. When I call a flagged sentence fine, the studio asks what it adds that I am happy with, so the report can say which kinds of harmless detail the check over-flags.

Measuring the checker changed the decision

The evaluations showed a trade-off I could easily have missed by looking at one accuracy score. A lenient judge agreed with most labels, but missed problems. Stricter judges caught more errors and also flagged harmless added detail.

So I set the bar before running any candidate: nothing missed, every seeded error caught, and fewer fine sentences flagged. The seeded test plants a changed number, year, name or relationship into sentences I had labelled fine, and counts how many the check notices.

The comparison led me to change the judge used for the sentence support check. On the small labelled set, the replacement reduced unnecessary flags and gave steadier results across repeat runs. A model with near-identical results cost five times as much, so I did not choose it. The replacement still flags too much harmless detail. That remains work to do.

This is the kind of product building I care about: shaping the experience, then making its quality visible enough to improve it. The warm part of Tellby is the conversation. The careful part is keeping a path back to the person who told the story.

The comparison was run on 7 October against a small 58-sentence labelled set. It measured false flags, missed problems and repeatability separately; the replacement still needs further validation, and contradiction misses on real labelled sentences remain insufficiently measured.