Making conversation evaluation inspectable

The linter computes concrete measures in code, including response gap, overlap, reply length, held pauses and word counts. Rules cover behaviours such as asking one question at a time, allowing a storyteller to finish, grounding follow-ups in what was said, and respecting topics they asked not to discuss. This separation keeps measurable timing and text counts distinct from interpretive criteria.

Each rule describes the criterion, the evidence to consider and its pass, warning or fail thresholds. Jev receives the relevant exchange state and the questions for the current turn in one request. When a rule warns or fails, a second request can localise the phrase in the reply. The interface then assembles the displayed reason from the code measure, Jev answer and matched condition; it does not present that text as a verbatim model explanation.

The evaluation record preserves the state and questions sent to Jev alongside its untouched response and the normalised result. A person can inspect the rule, evidence, threshold and outcome together. The queue serialises evaluations, waits for delayed retries to settle and applies results only to the session that queued them, so a late response cannot be attached to a different session.

The starter thresholds are working values, not a universal definition of a good conversation. Reviewers can label judgements and compare prompt versions; agreement is an early signal, while the documented promotion bar calls for a held-out set. Jev’s Noul answers do not include a separate confidence value. Timing can also be affected by residual echo, and the live-session path has been exercised with a simulated Tellby peer rather than a real spoken session.

The evaluation walkthrough on the Margin case study runs simplified rules in the browser, using synthetic examples or text a visitor types. It shows how a criterion can connect a visible exchange to a finding and a route. It does not call Jev, use real storyteller data or demonstrate evaluator accuracy. The point is to make the evaluation method legible before anyone treats its outputs as evidence about a real conversation.

Richard Simms is Principal Product Designer at SEEK, where he leads design for the Career Discovery Agent and Career Feed. He also builds Sentiuma, a personal AI knowledge infrastructure layer.