1. Why RAG matters

A research group might need to find a study’s inclusion criteria, compare annotation protocols, or recover an exact software setting from a long methods supplement. The answer may depend on material that is specialized, recently updated, or absent from a model’s training data. RAG provides a way to retrieve that material and attach identifiable evidence to the answer. [1]

For example, consider these possible applications:

• Research literature: compare how studies defined an outcome, while preserving population, measurement, and study-design differences.
• Human annotation: answer questions about the applicable guideline version, record disagreements, and connect a label to the supporting passage.
• Scientific software: find a documented option or workflow in the correct release of a package.
• Drug-discovery evidence: locate reported assay conditions and uncertainty statements for later expert review; a retrieved passage alone does not establish that a compound will work in another setting.

The central statistical problem is broader than generating fluent text: did the system retrieve the right evidence, interpret it correctly, and communicate what remains uncertain?

2. The workflow: from documents to a supported answer

The system combines collection preparation before questions arrive with an online question-answering workflow:

The first stage usually happens before a question arrives. Updating the collection requires updating the index and preserving version information. Passage boundaries should keep relevant conditions and exceptions together: separating an estimate from its population or assumptions can change its meaning.

For a question \(q\) and an eligible document passage \(d\), a dense retriever can score separately encoded representations as

\[ \begin{aligned} s(q,d)&=e_q(q)^\top e_d(d),\\ \mathcal R_k(q)&=\operatorname{TopK}_{d\in\mathcal D_q}s(q,d). \end{aligned} \]

Here \(\mathcal D_q\) is the permitted collection after applying the relevant access and metadata restrictions. Dense Passage Retrieval provides a dual-encoder example of this approach. [2] A dot product or cosine similarity is a ranking score, not a calibrated probability that the passage supports the eventual answer.

After reranking, let \(C(q)\) be the context passed to the language model. A typical autoregressive generator uses

\[ p_\theta(y\mid q,C(q)) =\prod_{t=1}^{T}p_\theta(y_t\mid y_{<t},q,C(q)). \]

This describes a distribution over answer tokens conditional on context. It does not imply that a likely answer is factually correct. Nor is a longer context automatically better: Lost in the Middle found that evidence position affected performance in its evaluated models and tasks. [3] Context selection and ordering therefore belong in an evaluation plan.

3. A concrete example: which annotation guideline applies?

Suppose the question is “How are annotation disagreements resolved in Project Cedar?” The miniature collection below is invented for this tutorial:

• C1 — Cedar, current guide. Project Cedar annotation disagreements are resolved by an independent third reviewer. Each case initially receives two labels.
• C0 — Cedar, superseded draft. Project Cedar annotation disagreements are resolved by majority vote after a team discussion.
• M1 — Maple, current guide. Project Maple annotation disagreements are resolved by majority vote among three annotators.
• C2 — Cedar, current documentation note. Project Cedar source passages retain document identifiers and section numbers to support citations.

The following computation uses TF-IDF cosine similarity in base R. It is a small lexical-retrieval demonstration, rather than a trained dense retriever or a complete RAG application. Inverse-document-frequency weights are computed from these four documents, and the query uses the same vocabulary and weights.

Figure 1: Query-to-passage TF-IDF cosine similarities in the four-document fictional collection. Blue passages meet the specified project and current-version filters; gray passages do not. A high lexical similarity does not establish that a passage is eligible or that it supports a claim.

Without metadata restrictions, C0 receives the highest score. With the question restricted to current Cedar documentation, C1 ranks first. A suitable answer would be:

In the current Project Cedar guide, an independent third reviewer resolves annotation disagreements. C1

This sentence is a manually written example of the desired output. The citation points to the passage supporting the claim. It would be incorrect to mix in the Maple procedure or present the superseded draft as current.

Now ask “How much was the third reviewer paid?” None of these passages answers that question. A useful system should report the missing evidence or request another source. The top-ranked passage still exists, so returning the best match alone cannot solve answerability.

# Fictional documents; this demonstrates retrieval, not an LLM call.
docs <- data.frame(
  id = c('C1', 'C0', 'M1', 'C2'),
  project = c('Cedar', 'Cedar', 'Maple', 'Cedar'),
  current = c(TRUE, FALSE, TRUE, TRUE),
  text = c(
    paste('Project Cedar annotation disagreements are resolved by an independent',
          'third reviewer. Each case initially receives two labels.'),
    paste('Project Cedar annotation disagreements are resolved by majority vote',
          'after a team discussion.'),
    paste('Project Maple annotation disagreements are resolved by majority vote',
          'among three annotators.'),
    paste('Project Cedar source passages retain document identifiers and section',
          'numbers to support citations.')))
query <- 'How are annotation disagreements resolved in Project Cedar?'
stop_words <- c('a', 'an', 'the', 'in', 'by', 'are', 'how', 'and', 'to', 'of')
tokenize <- function(text) {
  words <- strsplit(gsub('[^a-z0-9 ]', ' ', tolower(text)), '\\s+')[[1]]
  words[nzchar(words) & !words %in% stop_words]
}
doc_tokens <- lapply(docs$text, tokenize)
vocab <- sort(unique(unlist(doc_tokens)))
counts <- function(words) as.numeric(table(factor(words, levels = vocab)))
term_matrix <- t(vapply(doc_tokens, counts, numeric(length(vocab))))
idf <- log((nrow(docs) + 1) / (colSums(term_matrix > 0) + 1)) + 1
tfidf <- sweep(term_matrix, 2, idf, '*')
unit_rows <- function(M) M / pmax(sqrt(rowSums(M^2)), .Machine$double.eps)
doc_vectors <- unit_rows(tfidf)
query_vector <- unit_rows(matrix(counts(tokenize(query)) * idf, nrow = 1))
docs$score <- drop(doc_vectors %*% t(query_vector))

# Project and version filters apply before selecting top-k passages.
docs$eligible <- docs$project == 'Cedar' & docs$current
ranked <- docs[order(-docs$score), ]
retrieved <- head(subset(ranked, eligible & score > 0), 2)
stopifnot(retrieved$id[1] == 'C1', all(retrieved$eligible),
          all(docs$score >= 0), all(docs$score <= 1 + 1e-12))
retrieved[, c('id', 'score', 'text')]

4. Evaluate retrieval, answers, and citations separately

An aggregate answer score can hide different failures. A retriever can miss the relevant passage; a generator can misread a correct passage; a citation can point to a source that does not support the attached claim. ALCE explicitly evaluates generated answers and citation quality, while RAGAs provides automated evaluation dimensions for RAG pipelines. [4] [5]

For a query with a nonempty annotated relevant set \(G_q\), passage-level recall is

\[ \operatorname{Recall@}k(q) =\frac{|G_q\cap\mathcal R_k(q)|}{|G_q|}. \]

Questions with no supporting passage should form a separate answerability category, rather than receiving a recall score with a zero denominator. Define relevance granularity and treatment of duplicate passages before comparison.

Let \(A_\tau(q)\) indicate that a system answers at threshold \(\tau\), and let \(L(q)\in\{0,1\}\) record an incorrect answer. Then

\[ \begin{aligned} \operatorname{AnswerRate}(\tau)&=\mathbb E[A_\tau(q)],\\ \operatorname{SelectiveRisk}(\tau) &=\frac{\mathbb E[A_\tau(q)L(q)]}{\mathbb E[A_\tau(q)]}, \end{aligned} \]

when the denominator is positive. Reporting both prevents an almost-always-abstaining system from appearing useful merely because its answered questions have few errors.

Use held-out questions covering answerable, unanswerable, conflicting-source, and version-sensitive cases. Keep related questions or passages from the same document together when making splits. Automated judges are useful measurements, but should be checked against human annotations on the intended domain. Define the rubric, measure disagreement, and adjudicate ambiguous cases rather than assuming an evaluator’s score is ground truth.

5. Uncertainty quantification: what exactly is uncertain?

For a RAG answer, at least three uncertainties are distinct: whether the collection contains sufficient evidence, whether retrieval found it, and whether the generated claims are supported by it. A confident-sounding answer or high retrieval similarity does not separately quantify these uncertainties.

Here is a deliberately limited statistical task: classify one claim together with its question and retrieved evidence as supported (S), contradicted (C), or insufficient evidence (I). Write the label set as

\[ \mathcal Y=\{\mathrm S,\mathrm C,\mathrm I\}. \]

A fixed verification model produces label scores \(\widehat p_y(X)\), where \(X\) includes the claim and evidence. These scores need not already be calibrated probabilities. With \(m\) held-out, human-labeled calibration cases, define

\[ \begin{aligned} r_i&=1-\widehat p_{Y_i}(X_i),\\ j&=\lceil(m+1)(1-\alpha)\rceil,\\ \widehat q&=r_{(j)},\\ C_\alpha(X)&=\{y\in\mathcal Y: 1-\widehat p_y(X)\leq\widehat q\}. \end{aligned} \]

Use \(\widehat q=\infty\) if \(j>m\). With a scoring procedure fixed independently of the calibration set and exchangeable calibration and future labeled cases, split conformal prediction provides the marginal guarantee [6]

\[ \Pr\{Y_{\mathrm{new}}\in C_\alpha(X_{\mathrm{new}})\} \geq 1-\alpha. \]

The output is a set of verification labels, not a probability that an entire generated answer is true. A singleton containing “supported” and a set containing both “supported” and “insufficient evidence” communicate different uncertainty. Empty sets can also occur; they do not justify acceptance.

A calibration example

The simulation below generates a scalar feature, a three-category label, and a fixed overconfident classifier. It uses 199 calibration cases and 2,000 independent test cases from the same synthetic population. The two panels show empirical label coverage and average prediction-set size as the desired coverage increases.

Figure 2: A single synthetic split-conformal experiment. The upper panel compares empirical test coverage with the requested marginal coverage; the lower panel shows average label-set size. Both calibration randomness and finite test sampling affect the observed coverage. These are verification-label sets, not confidence scores for RAG answers.

At a 90% target, this particular run covers 89.5% of test labels and returns 1.22 labels per case on average.

The guarantee averages over calibration and future cases. It does not guarantee 90% correctness among cases accepted because their set is the singleton “supported”; that conditional error rate needs its own evaluation or risk-control procedure. Nor does per-claim coverage give simultaneous coverage for all claims in an answer. Changing the corpus, generator, verification model, annotation rubric, or query population can change the score distribution and undermine exchangeability.

Conformal Language Modeling studies a different problem: calibrated sets of generated responses and correctness of response components. [7] Extending uncertainty control across retrieval, generation, and human verification is a research problem, rather than an automatic consequence of adding conformal calibration to one component.

R code: synthetic conformal verification example

# A synthetic three-label verification problem; no actual LLM is evaluated.
set.seed(20260926)
label_names <- c('Supported', 'Contradicted', 'Insufficient evidence')
softmax <- function(Z) {
  W <- exp(Z - apply(Z, 1, max))
  W / rowSums(W)
}
simulate_cases <- function(n) {
  x <- runif(n, -2, 2)
  logits <- cbind(2 * x, -2 * x, 0.5 - x^2)
  true_prob <- softmax(logits)
  label <- vapply(seq_len(n), function(i)
    sample.int(3, 1, prob = true_prob[i, ]), integer(1))
  # Fixed, deliberately overconfident classifier, independent of calibration.
  predicted_prob <- softmax(1.6 * logits)
  list(label = label, prob = predicted_prob)
}
cal <- simulate_cases(199)
test <- simulate_cases(2000)
scores <- 1 - cal$prob[cbind(seq_along(cal$label), cal$label)]
alphas <- c(0.30, 0.20, 0.10, 0.05)
uq_results <- do.call(rbind, lapply(alphas, function(alpha) {
  m <- length(scores)
  k <- ceiling((m + 1) * (1 - alpha))
  cutoff <- if (k <= m) sort(scores)[k] else Inf
  included <- 1 - test$prob <= cutoff
  covered <- included[cbind(seq_along(test$label), test$label)]
  data.frame(target = 1 - alpha, cutoff = cutoff,
             coverage = mean(covered), size = mean(rowSums(included)),
             singleton = mean(rowSums(included) == 1))
}))
uq90 <- uq_results[which.min(abs(uq_results$target - 0.90)), ]
stopifnot(all(abs(rowSums(cal$prob) - 1) < 1e-12),
          all(diff(uq_results$cutoff) >= 0),
          all(diff(uq_results$size) >= 0),
          all(uq_results$size >= 0 & uq_results$size <= 3))
uq_results

6. Agents and research directions

An agent can make retrieval a sequence of decisions: decompose a question, inspect evidence, reformulate a search, compare sources, and stop when it has enough support—or cannot obtain it. Self-RAG is a concrete research example that learns retrieval and critique behaviors through reflection tokens. [8] Adding an agent loop alone does not establish reliability; each extra step changes cost, latency, and the opportunities for error.

Several questions connect this area to statistics and human annotation:

The allocation questions also connect to optimal experimental design: a limited annotation budget can be directed toward the questions and failure modes that are most informative. Evidence-oriented agents can likewise support drug-discovery research workflows when their outputs remain traceable to the underlying studies.

7. A practical starting plan

References and further reading

1. Lewis, P., Perez, E., Piktus, A. et al. (2020). Retrieval-augmented generation for knowledge-intensive NLP tasks. NeurIPS 2020.
2. Karpukhin, V., Oguz, B., Min, S. et al. (2020). Dense passage retrieval for open-domain question answering. EMNLP, 6769–6781.
3. Liu, N. F., Lin, K., Hewitt, J. et al. (2023 preprint). Lost in the middle: How language models use long contexts. arXiv:2307.03172.
4. Gao, T., Yen, H., Yu, J. & Chen, D. (2023). Enabling large language models to generate text with citations. EMNLP, 6465–6488.
5. Es, S., James, J., Espinosa Anke, L. & Schockaert, S. (2024). RAGAs: Automated evaluation of retrieval augmented generation. EACL: System Demonstrations, 150–158.
6. Angelopoulos, A. N. & Bates, S. (2021; revised 2022). A gentle introduction to conformal prediction and distribution-free uncertainty quantification. arXiv:2107.07511.
7. Quach, V., Fisch, A., Schuster, T. et al. (2024). Conformal language modeling. ICLR 2024.
8. Asai, A., Wu, Z., Wang, Y., Sil, A. & Hajishirzi, H. (2024). Self-RAG: Learning to retrieve, generate, and critique through self-reflection. ICLR 2024.

Notes and reproducibility

The document collection is fictional, and both numerical figures are reproducible teaching examples. They do not measure the performance of a deployed RAG system. The calibration example uses synthetic features and labels, without a real retriever, LLM, or human annotation dataset. The fixed seed makes the illustration reproducible, not universally representative.

The R Markdown source reproduces both numerical figures and the displayed examples. The code uses base R and ggplot2; it needs no API key because it does not run a language model.

The three AI-generated illustrations provide conceptual context. The research directions and starting plan describe questions to investigate; the toy examples do not solve those research problems.