← back to dashboard

Methodology & limits

EvalGate's whole pitch is a number you can trust — so the scoring has to be auditable. Nothing here uses an LLM to judge another LLM. Every metric is a transparent function of strings and tokens, and the run is reproducible.

The fairness contract

The regression compares two configurations that differ in exactly two things — chunking and the prompt. Everything else is held constant:

Same synthetic corpus and the same gold questions.
Same generator: a local qwen2.5:7b at temperature 0 (deterministic), reached through Ollama — no API, no quota, no network variance.
Same retriever: nomic-embed-text cosine similarity.
Every embedding and generation is cached by a hash of its input, so a re-run reproduces byte-identical results.

Because only chunking + prompt change, the score delta is attributable to that change — that is what makes a before/after number meaningful.

The three signals

1 · Retrieval recall

For an answerable question the gold article is known. Recall is a hit if any retrieved chunk comes from that article.

retrieval_hit = gold_doc ∈ { chunk.doc : chunk ∈ top-k }

2 · Answer correctness

Case-insensitive containment: the answer is correct if any accepted gold string appears in it. No-answer questions are correct only when the model refuses.

correct = ∃ a ∈ accept : a ⊆ answer (lowercased) no_answer → correct = refusal(answer)

3 · Groundedness

Split the answer into sentences. For each sentence, take its content tokens (stopwords dropped, numbers kept) and measure how many appear in the retrieved context. A sentence below the floor is an unsupported claim — a hallucination candidate. An invented number or name simply isn't in the context, so its coverage collapses and it is flagged.

coverage(sentence) = |content_tokens ∩ context_tokens| / |content_tokens| grounded(sentence) = coverage ≥ FLOOR (FLOOR = 0.70) groundedness(answer) = mean coverage over sentences

How the flags are defined

Hallucination — an answer that is unsupported (a sentence below the floor) and wrong. Grounding catches it at inference time, no gold needed.
Grounded-but-wrong — an answer that passes grounding (it does cite retrieved context) but is still wrong, because the retriever surfaced the wrong fact. Grounding cannot catch this; only the measured gold set does.
Review queue — anything a human should double-check, using only signals available in production: an unsupported sentence, or a confidence (min of top similarity and groundedness) below 0.50. It never peeks at the gold label — that would be cheating.

Honest limits

Coverage is a lexical proxy for faithfulness. It catches invented facts well (a hallucinated number/name isn't in the context). It can be fooled by a paraphrase that reuses context words, and it can under-count a correct answer phrased with synonyms — which is exactly why some correct answers land in the review queue rather than being trusted blindly. A stronger check would add an NLI or entailment model; we trade that for determinism and auditability.

Containment scoring under-counts paraphrase. If a model answers "three seats" where the gold string is "3 seats", containment can miss it. We keep the accept-list explicit and visible so you can see every judgement.

Doc-level retrieval recall saturates on a small KB. On this 18-article corpus recall is already 100% for both configs; the failures show up downstream, in how the retrieved context supports the answer. On a large corpus the same harness surfaces retrieval misses directly.

Synthetic corpus. "Nimbus" is a fictional product invented for a self-contained demo. Point EvalGate at your real KB and gold set and the method is identical.

Published misses

The dashboard shows every question, including the ones the harness got wrong or flagged — filter the results table by Wrong, Hallucination or Review queue. Nothing is hidden; that's the point.

← back to dashboard