01 · BITEM at the NTCIR-19 R2C2 Task: Predicting Confidence from Agentic RAG Pipeline Signals

BITEM 在 NTCIR-19 R2C2 任务：从智能体式 检索增强生成（RAG）管道信号预测置信度

Research record	Details
Authors	Julien Knafou, Luc Mottin, Alexandre Flament, Paul van Rijen, Esteban Gaillac, Patrick Ruch
Published	2026-09-29
Sources	arxiv
Focus	agentic RAG, confidence estimation, multi-pass retrieval, evidence orchestration 智能体式 检索增强生成（RAG）, 置信度估计, 多轮检索, 证据编排

English

Relevant for evaluation and orchestration aspects of the thesis (confidence signals, multi-pass pooling, HMR/accHMR), but it does not address core thesis topics (graph memory, entity resolution, KG-based multi-hop). Useful for targeted methods and metrics rather than deep technical guidance.

中文

对论文的评估与编排维度（置信度信号、多轮汇聚、HMR/accHMR）具有参考价值，但不涉及课题核心（图记忆、实体消歧、基于知识图谱的多跳推理）。适合针对性阅读以采纳方法与度量，而非深度技术细读。

Research synopsis

English

This paper describes the BITEM entry to the NTCIR-19 R2C2 task, which uses a single agentic pipeline where a model searches, reads, and records evidence over a movie corpus while an orchestrator maintains a record and applies rules to decide what to submit. Claims are admitted only after an entailment cascade checks cited passages; answers are released when sufficient checked evidence exists. Each question is executed multiple times with later runs retrieving from corpora stripped of earlier-pass material. The orchestrator computes answer confidence from recorded pipeline signals (not from model self‑ratings). Handcrafted rules on those signals improve accuracy and a proposed accHMR metric combines accuracy with Human-Model Reliability.

中文

本文介绍了 BITEM 团队在 NTCIR-19 R2C2 任务中的参赛做法：采用单一的智能体式管道，模型在电影语料上搜索、阅读并记录证据，而一个编排器维护记录并按规则决定提交内容。主张在引证段落通过蕴涵级联检查后才被接纳；当存在足够的经检查证据时才释放答案。每个问题运行多次，后续轮次从剔除前轮已见内容的语料中检索。编排器根据记录的管道信号（而非模型自评分）计算置信度，基于这些信号的手工规则可提高准确率，且文中提出 accHMR 指标将准确率与 Human-Model Reliability 结合评估。

Thesis relevance

English

Overlap: both examine agentic RAG pipelines, multi-pass retrieval, and use of pipeline signals for downstream decisions (confidence/answering). Differences: this work focuses on confidence estimation from orchestration signals in a contest setting and uses rule-based postprocessing; it does not build or evaluate persistent structured graph memory, entity resolution, or KG-based multi-hop path reuse. Complementarity: the paper’s methods for extracting and aggregating pipeline-level signals, its use of multi-pass retrieval and pooling, and metrics like HMR/accHMR could inform evaluation and verifiability components of the thesis.

中文

重合点：两者都研究智能体式 RAG 管道、多轮检索，以及利用管道信号支持后续决策（如置信度或答复）。差异：该工作侧重于在竞赛场景中从编排信号进行置信度估计并采用基于规则的后处理；其并未构建或评估持久的结构化 图记忆、实体消歧或基于知识图谱的多跳路径重用。互补性：论文中提取与聚合管道级信号、多轮检索与汇聚的做法以及 HMR/accHMR 等指标，可为本课题在可验证性和评估维度提供有价值的参考。

Writing and related work

English

Emphasize the role of an external orchestrator that computes confidence from recorded signals rather than model self-assessment; note the pragmatic benefit of simple rules applied to pipeline traces. The paper motivates including orchestration and signal logging in related-system writeups to enable post-hoc verification and confidence calibration.

中文

强调外部编排器通过管道记录信号而非模型自评分来计算置信度的作用；注意将简单规则应用于管道轨迹带来的务实收益。该论文支持在相关系统描述中加入编排与信号日志，以便事后可验证性和置信度校准。

Method and evaluation

English

Consider instrumenting your agentic system to record the same set of pipeline signals (retrieval provenance, entailment checks, passage reuse across passes) so you can test rule-based vs learned confidence predictors. Use multi-pass retrieval with exclusion/pooling as an experimental variable to study its effect on structured graph accumulation and on metrics such as HMR and the proposed accHMR. Compare hand-crafted rule thresholds against a learned scalar predictor trained on the recorded signals.

中文

考虑为你的智能体式系统记录与论文相似的一组管道信号（检索来源、蕴涵检查、跨轮次段落重用），以便比较基于规则与学习型置信度预测器。将多轮检索（带排除/汇聚）作为实验变量，以研究其对结构化 图记忆 累积的影响及对 HMR 与提出的 accHMR 等指标的效应。将手工规则阈值与基于记录信号训练的学习型标量预测器进行比较。

Future directions

English

Train a model to predict confidence from pipeline signals instead of handcrafting rules, and evaluate whether learned predictors better align confidence with correctness. Test whether similar orchestration signals can be derived from a system that populates and consults structured graph memory and whether those signals improve verifiability and stopping criteria.

中文

用模型基于管道信号学习预测置信度以替代手工规则，评估学习型预测器是否更好地将置信度与正确性对齐。测试能否从填充并查询结构化 图记忆 的系统中提取类似的编排信号，以及这些信号是否能增强可验证性和停止判据。

---