A clinic does not want a clever chatbot. It wants fewer phone calls at 8am and zero surprises in the patient record. That framing shaped every decision on this build.
The assistant collects intake details, answers a narrow set of logistics questions, and hands anything clinical to a person. Trust came from what it refused to do.
#Guardrails before prompts
We wrote the refusal rules first: no diagnoses, no medication advice, no promises about appointment availability the system cannot verify. The prompt came second and was tested against those rules.
Every conversation ends in a structured summary the front desk reviews. The model never writes to the patient record directly.
- A fixed list of topics the assistant may answer
- A hard handoff phrase that always routes to a person
- A structured summary reviewed by staff before anything is saved
#Evals that run on every change
We built a set of 140 realistic conversations, including adversarial ones written by the clinic staff themselves. Each prompt or model change runs against the whole set before it ships.
The metric we watch most is not accuracy but handoff correctness: when it should escalate, does it always escalate?
type EvalCase = {
id: string
messages: string[]
mustHandoff: boolean
forbidden: RegExp[]
}
export async function runEvals(cases: EvalCase[]) {
let failed = 0
for (const c of cases) {
const reply = await assistant.run(c.messages)
const handedOff = reply.action === 'handoff'
const leaked = c.forbidden.some((re) => re.test(reply.text))
if (handedOff !== c.mustHandoff || leaked) {
failed++
console.error('FAIL', c.id)
}
}
if (failed > 0) process.exit(1)
}
