01 · MemPilot: Orchestrating On-Demand Multimodal Memory Curation for LLM Agents

MemPilot：为 LLM 智能体协调按需多模态记忆策划

Research record	Details
Authors	Haozhen Zhang, Haodong Yue, Quanyu Long, Jianzhu Bao, Qingyuan Liu, Tao Feng, Bohan Liu, Weida Liang, Wenya Wang
Published	2026-10-05
Sources	arxiv
Focus	on-demand memory curation, multimodal agent memory, reinforcement-learning policy, performance–cost–latency trade-offs 按需记忆策划、多模态 智能体记忆、强化学习策略、性能-成本-延迟权衡

English

The paper introduces RL-based, multi-objective techniques for on-demand multimodal memory curation (objective-wise advantage decoupling; prefix-based marginal utility estimation) that are methodologically novel and potentially applicable to controlling graph-memory construction and runtime cost. These methods could materially inform the thesis’ design choices for when to create or update structured memory under latency/cost constraints.

中文

该论文提出了用于按需多模态记忆策划的强化学习与多目标优化技术（objective-wise advantage decoupling；prefix-based marginal utility estimation），方法学上具有新意，可用于在延迟/成本约束下控制图记忆的构建与更新，对论文的设计选择具有实用参考价值。

Research synopsis

English

The paper presents MemPilot, a framework for on-demand curation of multimodal agent memory that aims to balance performance, computation cost, and latency. Rather than building query-agnostic memory offline, MemPilot trains a multi-step LLM policy via reinforcement learning to decide at runtime whether to retrieve from precurated memory or to invoke query-specific curation of raw multimodal history using heterogeneous LLMs and VLMs. The policy controls evidence amount, curation instructions, model selection, and visual access. To optimize competing objectives the authors adapt objective-wise advantage decoupling and introduce prefix-based marginal utility estimation. Experiments on five multimodal agent-memory benchmarks report improved trade-offs over baselines.

中文

本文提出 MemPilot 框架，用于按需策划多模态 智能体记忆，以在性能、计算成本与延迟之间取得权衡。MemPilot 不再离线构建与查询无关的记忆，而是通过强化学习训练多步 LLM 策略，在运行时决定是从已预处理的记忆检索，还是调用异构 LLM 与 VLMs 针对原始多模态历史进行查询专用策划。该策略控制证据量、策划指令、模型选择与视觉访问。为优化互相竞争的目标，作者采用了 objective-wise advantage decoupling 并引入 prefix-based marginal utility estimation。五个多模态智能体记忆基准上的实验显示比基线具有更好的权衡表现。

Thesis relevance

English

Overlap: MemPilot addresses runtime-adaptive, query-aware memory curation and explicit trade-offs among performance, cost, and latency, which is relevant to managing when and how to construct or access long-term memory in the thesis. Differences: MemPilot focuses on multimodal, query-specific curation governed by an RL policy and does not describe constructing or reusing a persistent structured graph memory, triple extraction, or entity-resolution mechanisms for multi-hop path reuse. Complementarity: the RL-driven curation and preference-sweep evaluation methods could be adapted to gate when to extract/store triples into a query-aware graph memory and to control computation budget during graph updates.

中文

重合点：MemPilot 研究运行时自适应、查询感知的记忆策划以及性能、成本与延迟之间的权衡，这与论文在管理何时及如何构建或访问长期记忆的关注点相关。差异：MemPilot 以多模态且由强化学习策略驱动的查询专用策划为核心，并未描述如何构建或重用持久的结构化图记忆、三元组抽取或为多跳路径重用做实体消歧。互补性：其 RL 驱动的策划策略和偏好扫描评估方法可用于决定何时将信息抽取为三元组并写入查询感知图记忆，以及在图更新过程中控制计算预算。

Writing and related work

English

Position MemPilot in related work as an instance of adaptive agent memory systems that prioritize runtime decision-making and cost-aware control. Cite its use of objective-wise advantage decoupling and prefix-based marginal utility estimation when discussing optimization techniques for multi-objective memory management.

中文

将 MemPilot 归入自适应智能体记忆系统相关工作，强调其以运行时决策和成本感知控制为核心。讨论多目标记忆管理优化技术时，应引用其 objective-wise advantage decoupling 与 prefix-based marginal utility estimation 的做法。

Method and evaluation

English

Consider adapting MemPilot’s RL policy to decide (a) whether to extract triples and persist them into the graph memory at query time and (b) how much multimodal evidence to include per triple. Apply objective-wise advantage decoupling to jointly optimize accuracy, latency, and memory growth, and use prefix-based marginal utility estimation to attribute benefit across multi-step curation actions. Empirically compare preference sweeps (performance–cost–latency frontiers) and include MultiHop-RAG to measure effects on multi-hop retrieval, bridge-entity identification, and end-to-end latency.

中文

可将 MemPilot 的强化学习策略改造为在查询时决定（a）是否提取三元组并将其持久化到图记忆中，及（b）每个三元组应包含多少多模态证据。采用 objective-wise advantage decoupling 来联合优化准确性、延迟与记忆增长，并用 prefix-based marginal utility estimation 为多步策划动作进行效用归因。实验上应做偏好扫描（性能-成本-延迟前沿）并将 MultiHop-RAG 纳入比较，以测量对多跳检索、桥接实体识别与端到端延迟的影响。

Future directions

English

Extend MemPilot to control structured graph operations: add policy actions for triple extraction granularity, entity-resolution invocation, and confidence-weighted graph updates. Incorporate multimodal signals into triple extraction and evaluate whether on-demand curation policies improve graph coherence and multi-hop QA accuracy on benchmarks like MultiHop-RAG.

中文

将 MemPilot 扩展为可控制结构化图操作的策略：新增关于三元组抽取粒度、实体消歧调用与基于置信度的图更新的行动。将多模态信号纳入三元组抽取，并评估按需策划策略是否能提高图一致性及在 MultiHop-RAG 等基准上的多跳问答准确率。

---