Internal answers with citations and refusal
An internal assistant that is confidently wrong once will not be trusted again. Refusal behaviour, not answer quality, is what determines whether this kind of build survives its first month.
Reference build 03
Keelix’s own build — not a client result · in progressWhat this exists to fix.
The answers to most internal questions exist, in a policy document, a supplier agreement, or a procedure nobody has opened in two years. Finding them costs an interruption to whoever is known to know.
The obvious build — point a model at the shared drive — fails in a specific way. It answers questions the documents do not cover, because answering is what it was trained to do, and one confident invention destroys trust in every correct answer that preceded it.
How it is put together.
Plain enough for an owner to follow, specific enough for whoever maintains it afterwards.
-
Source control
A defined corpus with an owner per document set. Documents are versioned on ingest, so an answer can be traced to the version that produced it rather than to whatever is current.
-
Retrieval
Chunked, embedded, and retrieved with the source document and section preserved through to the answer. Retrieval quality is measured separately from answer quality, because they fail differently.
-
Grounded answering
The model is constrained to answer only from retrieved passages, must cite the document and section for each claim, and is instructed to refuse when the passages do not support an answer.
-
Refusal path
A refusal names who owns that subject area, so the question reaches a person instead of stopping. Refusals are logged as a gap in the corpus, which is how the document set improves.
-
Access boundaries
Retrieval respects existing permissions. A document a person cannot open is a document the assistant will not quote to them.
-
Evaluation harness
A fixed question set with known answers, including questions deliberately not covered by the corpus, run on every change to prompts, models, or documents.
How it will be evaluated.
The procedure is published before a run exists, so the evaluation cannot be rewritten around a preferred result.
-
Freeze a versioned document corpus, permission map, and question set. Include supported questions, questions outside the corpus, conflicting-source questions, and test accounts with different access.
-
Run retrieval separately from answer generation. Record the passages returned for each question before judging whether the answer used the right source material.
-
Check every answer claim against its cited passage and record unsupported claims as failures. Count refusals separately and verify that each refusal names the responsible owner.
-
Run the same questions through each permission boundary. Confirm that restricted passages never appear in retrieval results, answers, citations, logs, or retained artifacts.
-
Record the prompt, model, corpus, and harness versions along with sample size, cost, failures, refusals, and conflicts. Retain sanitized run artifacts before publishing any metric.
What it has to do to count.
Agreed before construction, published whether or not they are met.
- Every claim in an answer carries a citation to a retrievable source passage.
- Questions outside the corpus produce a refusal and a named owner, not a plausible answer.
- Access permissions are enforced at retrieval, verified by test accounts.
- The evaluation set runs on every prompt, model, or corpus change, and results are recorded per run.
- Answer and refusal rates are tracked separately, because a rising refusal rate is a corpus problem rather than a model problem.
- Cost per question is recorded, including retrieval and generation.
What has been run and retained.
No completed evaluation run has been published.
Where it stops, refuses, or hands back.
A build with no stated limits has not been examined closely enough.
- The assistant is bounded by the corpus. It cannot answer what the company has never written down, and it is designed to say so rather than to improvise.
- Documents that contradict each other produce a surfaced conflict, not a resolution. Deciding which is correct stays a human job.
- Evaluation figures are absent above because the harness has not yet been run on a settled corpus.
Operational guidance
This pattern, fitted to your operation.
An Audit works out whether a build like this is the right first one for your business, and what it would take.