Understanding the agent training loop through Reward, State, and Credit
Agent training can be unfolded through three questions — Reward, State, and Credit: how to define the objective using environment results, how to make the training data cover the states the current policy actually visits, and how to attribute a delayed outcome to specific actions. Together, the three determine whether the training loop can produce a reliable, sustainable learning signal.
Imagine a coding agent receives a task: fix an intermittent bug.
It searches the code 12 times, reads 8 files, proposes 3 hypotheses, modifies 2 implementations, runs 6 rounds of tests, and finally makes all hidden tests pass.
From the deployment side, this run appears to have completed the task successfully.
But from the training side, we need to think about many more questions:
A traditional text generation usually has only an input and a target output. An agent’s actions, however, change the environment, and the changed environment rewrites the subsequent inputs. The training object therefore expands from a single answer into a closed-loop trajectory:
\[\tau=(o_1,a_1,o_2,a_2,\ldots,o_T,a_T)\]where \(o_t\) is an environment observation and \(a_t\) is an internal decision or external action. The trajectory distribution is generated jointly by the policy and the environment:
\[\tau\sim p(\tau\mid \pi,E)\]Agent training seeks a policy:
\[\pi^* =\arg\max_\pi \mathbb E_{\tau\sim p(\tau\mid\pi,E)} [U(\tau)]\]This form reveals a key point: the agent’s policy simultaneously plays two roles — learner and data generator. Once the policy changes, the states it visits, the observations it obtains, and the mistakes it makes all change.
This is fundamentally different from ordinary supervised learning, which can be approximated as:
\[x\sim D,\qquad y\sim p(y\mid x)\]The data distribution \(D\) is externally given, and the model parameters do not change the next batch of inputs. An agent’s history distribution, by contrast, depends on the current policy:
\[h_t\sim d_E^\pi\]After a training update to \(\pi\), the \(d_E^\pi\) sampled in the next round changes with it. While learning how to act, the model continuously manufactures new training distributions.
From the classical RL perspective, this is precisely the policy-induced state-visitation distribution: the policy determines what data gets sampled, and after updating on that data, the new policy produces a new distribution. In such a setting, three classical RL questions stand out especially: how to obtain reliable feedback from a complex environment state, how to obtain training trajectories that cover the current policy’s behavior, and how to attribute a delayed outcome to specific decisions in a long trajectory. In practice, these are often summarized as Reward, State, and Credit:
Reward: what exactly should the policy maximize?
Reward turns the task result into an optimizable signal and determines the direction of learning.
State: does the training data cover the states the policy actually reaches?
State concerns which portion of the occupancy distribution the training data covers, and whether teacher-model demonstrations can transfer to the current policy’s actual operating distribution.
Credit: which decisions should a delayed outcome be attributed to?
Credit determines which actions in a long trajectory get reinforced and which need correcting.
Along these three perspectives, this piece discusses what information the various training methods — evolving from the arrival of large models into the agent era — each provide, and which part of the problem each one solves.
Training must first define what is good and what is not.
For ordinary question answering, the ground truth may be a reference answer. For an agent, the result usually shows up as a change in the state of the external environment, for example:
Therefore, a high-quality reward usually reads the environment state directly. An agent’s own description of its process is better used as an auxiliary signal.
You can think of the verifier as a measurement instrument. It compresses the full environment state into a training signal:
\[R(\tau)=V(s_T,\tau)\]where \(s_T\) is the final state. A good verifier must answer at least:
Coding is one of the earlier domains suited to directly verifiable rewards. Compilers, unit tests, static analysis, file diffs, and hidden tests can turn program state into machine-checkable feedback. When this feedback is used directly as the reward to update the policy, that is RLVR (Reinforcement Learning with Verifiable Rewards).
RLVR improves the repeatability and scalability of feedback, but “verifiable” only means the checker can stably execute a set of scoring rules; it does not mean those rules are equivalent to the real task objective.
SWE-RL is one example. It computes the training reward from the textual similarity between the predicted patch and the reference patch. A semantically correct but differently-written patch may thus score lower. This checker measures the textual similarity between the patch and the reference answer; it does not check whether the bug is fixed.
Unit tests are closer to program correctness, but still only check behavior that has already been encoded. When coverage is incomplete, hardcoding visible examples can also pass; when the evaluation channel lacks protection, the agent can also modify tests or exploit parser bugs. OpenAI’s audit of SWE-Bench Pro found problems in about thirty percent of tasks; BenchJack found 219 exploitable vulnerabilities across 10 agent benchmarks.
The problem in the first example is that the proxy metric is too far from the real objective; the problem in the second is insufficient test coverage. The specification problem therefore lands on the verifier’s objective definition and coverage. When evaluating a measurement, one must simultaneously check whether it can verify stably, what information it preserves, and what information it omits.
Any measurement compresses information:
\[M:\;(s_0,\tau,s_T)\longrightarrow z\]The full environment state, the action trajectory, and the final state are compressed into a binary label, a scalar score, or a preference ordering. Compression is a double-edged sword: the stronger the compression, the more direct and concise the signal and the easier the training — and the more that is omitted.
When choosing a measurement, look at at least five dimensions:
Program verifiers often have an advantage in directness, repeatability, and cost, yet may cover only a narrow objective. Human experts can judge broader quality dimensions, with weaker throughput and consistency. LLM judges are easy to scale, and also more prone to inheriting model bias and exploitable patterns. Different approaches suit being combined across different scenarios and different stages of training.
Typical forms include:
Search-R1 uses final exact match; ReTool uses final-answer equivalence; WebAgent-R1 checks whether a web task is completed; Agent-Diff checks the full state difference produced by enterprise API operations and zeroes out any undeclared side effect.
This kind of measurement suits tasks that satisfy three conditions:
the goal state is machine-readable
the consequences of actions can be isolated and recomputed
a valid solution can be decided programmatically
Its main risk is insufficient coverage. All-green tests only show that the tested behavior is correct; a correct final state can still hide a violating path behind it. If the agent modified tests, leaked hidden answers, or caused a transient side effect, a simple final-state checker may miss it entirely.
So program measurement should simultaneously check:
Tasks such as writing quality, patch maintainability, customer-service communication, and research-idea generation are hard to compress into a deterministic rule. But humans can compare two candidates well:
\[\tau_A \succ \tau_B\]Pairwise preference is usually easier than absolute scoring, because humans are good at comparison but find it hard to stably define a particular score.
Its advantage is broad coverage, absorbing judgments that are hard to formalize. Its costs include:
So human preference is better suited to soft quality dimensions such as clarity, maintainability, and communication style. Task correctness, permissions, and safety should still be handled by independent signals.
An LLM judge can scale an expert rubric to large-scale trajectory evaluation, and a learned RM can further distill preferences into a low-cost scalar.
This kind of method suits:
LLM-as-a-judge has long been controversial. This kind of method faces three kinds of risk.
The first is static bias: length, position, style, self-preference, and format all affect the score.
The second is distribution drift: the RM is trained on old-policy data, and the new trajectories produced after policy optimization may leave its trustworthy region.
The third is adversarial exploitation: One Token to Fool LLM-as-a-Judge found that a single colon or a fixed opening can raise the judge’s reward.
So an LLM judge is better used as a high-throughput filter. Periodic human spot-checks, hard-verifier calibration, and off-policy holdouts are all necessary supporting measures.
Common proxies include:
They are cheap to compute, but are usually just surface features correlated with the task result, and cannot directly indicate whether the task succeeded.
Format validity can be guaranteed directly by a grammar or tool schema; treating it as the main reward encourages the model to chase label correctness. Length and tool count are cost proxies, and optimizing them directly easily produces under-use or padding. Textual similarity suits quick filtering, but is still some distance from semantic correctness.
ToolRL’s length experiment shows this risk: directly rewarding response length dropped Qwen-1.5B on BFCL from 46.20% to 33.23%, and a dynamic length reward dropped it further to 28.51%. ReTool added no length reward, yet after RL its responses naturally shrank by about 40%. Once tools improve solving efficiency, cost improvements appear alongside task capability.
The most fitting role for a proxy is a gate, a diagnostic metric, or a secondary objective inside a successful trajectory.
Mature agent measurement is usually a composite structure:
Hard-constraint layer
permissions, safety, grader integrity — failure means the task fails
Task-outcome layer
hidden tests, state diff, goal state — defines whether the task is done
Soft-quality layer
human or LLM preference — compares the quality of successful solutions
Process-diagnosis layer
turn-level progress, cost, recovery, tool efficiency
The different layers of this framework solve different problems. They cannot be linearly weighted into a single reward, because that would let a high task score compensate for a safety violation, and let soft quality mask a task failure.
Training can use a hierarchical objective:
\[\text{satisfy Constraints first} \;\rightarrow\; \text{then maximize Task Success} \;\rightarrow\; \text{finally optimize Preference and Cost}\]This matches the real structure of tasks and training needs better than “tuning one weight per metric”.
Offline evaluation cares about whether the verifier judges accurately on a fixed data distribution. Once RL begins, the verifier’s score directly determines which behaviors get reinforced. As long as some behavior can stably game a high score, the optimization process will raise the probability of it appearing.
For example, a certain fixed opening makes the LLM judge wrongly give a high score. In a static test set, such samples are only 1%, and the verifier still looks 99% accurate. But once RL discovers this pattern, it will reinforce it repeatedly; the originally rare 1% may gradually become the policy’s main output. The average accuracy on the original distribution cannot represent reliability during training — the verifier must be re-evaluated on the continuously updated policy distribution.
So a training measurement needs extra checks:
This is also why verifiers need isolation, rotation, and red-team auditing. In agentic RL, the measurement system itself has become part of the environment’s safety boundary.
Looking at the whole architecture, one perhaps-effective strategy is: outcome provides ground truth; soft evaluation supplements quality that is hard to encode directly; process signals improve learning efficiency; hard constraints block unacceptable paths. Build the system by combining the strengths of each kind of measurement.
Reward answers “how to evaluate a trajectory”, but cannot answer “which trajectories enter training”. Even if the verifier is fully reliable, if the data only covers the teacher model’s successful paths, the error states the student creates itself still get no supervision.
State here refers to the entire history distribution the policy visits during interaction. Every action an agent takes changes the next input: if the student searches wrong once earlier or calls a wrong tool, the subsequent history may leave the teacher model’s trajectory. The places where the model needs to learn to recover will not appear in data generated only by the teacher model.
Therefore, who generates the data is part of the training mechanism. Trajectory SFT learns on the states the teacher model visits; on-policy methods let the training distribution follow the current policy.
The demonstrations for trajectory SFT can come from humans or models. The work discussed in this section mainly uses a stronger model to generate trajectories. The teacher model provides demonstrations, but is not assumed to be the optimal policy.
Trajectory SFT can be seen as behavior cloning applied to full agent trajectories. The teacher model first interacts with the environment, then decomposes each decision point in the trajectory into supervised samples:
\[\tau_T\sim p(\tau\mid\pi_T,E),\qquad \mathcal D_T=\{(h_t^T,a_t^T)\mid \tau_T\}\]where \(h_t^T\) is the history when the teacher model reaches step \(t\), and \(a_t^T\) is the action it gives. The student model fits these actions with standard cross-entropy:
\[\mathcal L_{\text{SFT}} =-\mathbb E_{(h_t^T,a_t^T)\sim\mathcal D_T} \log \pi_\theta(a_t^T\mid h_t^T)\]Compared with giving a single outcome only at the end of the trajectory, this supervision is dense: every action of the teacher model provides a token-level label. The task scenarios it suits are mainly:
AgentTuning, Agent-FLAN, and FireAct all demonstrate the value of trajectory distillation.
Dense labels can still be insufficient in coverage. The training history comes from the teacher model’s visitation distribution \(d_E^{\pi_T}\), while the deployment history comes from the student model’s visitation distribution \(d_E^{\pi_\theta}\). The difference between the two is the covariate shift in imitation learning.
If the student model searches the wrong file at some step, the subsequent context, hypotheses, and environment state all diverge from the teacher model’s trajectory. Teacher forcing teaches “how to move at the next step on a history generated by the teacher model”, but deployment further requires the student model to learn “how to recover after entering an abnormal state it created itself”.
Ordering methods by the source of states yields a continuous spectrum:
Teacher-model SFT
states produced by the teacher model
Policy rollout + verifier filtering
states produced by the sampling policy, outcome decides keep or rank
On-policy RL
states and actions both produced by the current policy, outcome provided by the environment
The core variable here is the occupancy distribution — which states the policy actually visits. CE, DPO, and policy gradient are just different update tools.
WebAgent-R1 provides a clear thread. Qwen2.5-3B rose from 6.1% to 20.0% via behavior cloning, then to 33.9% via RL. The version that did RL directly from the raw model degraded slightly: the action format was not yet mastered, and positive rewards almost never appeared.
This shows that BC serves as a long-standing exploration prior. It first moves the policy into a region where it “occasionally succeeds”, so that RL has a signal to keep optimizing.
Rollouts of the current policy produce both successful samples and real failure states. A trajectory that fails at the end is not suitable as a full positive demonstration, but the states it passed through and the effective actions within it can still be used for training. By the granularity of supervision used, failure trajectories can play a role at four levels:
SWE-Master uses failures to judge task difficulty; ETO uses failure trajectories as DPO negatives; Orchard extracts value-increasing segments from failure trajectories. They respectively exploit the difficulty, outcome, and process information in failure data.
The same failure trajectory can provide a negative signal at the outcome layer, keep effective segments at the process layer, and provide scarce error states at the state layer. A final failure only shows that the overall result is unsatisfactory; it does not show that every action in the trajectory was wrong. Further distinguishing the decisions that should be suppressed from those that should be kept requires solving the credit-assignment problem.
On-policy data makes training happen on the states the current policy visits, solving the training-distribution problem. But the verifier usually gives only one outcome at the end of the trajectory, which can provide only the overall result of the whole trajectory and lacks information about each step’s contribution. Training still needs to turn a trajectory-level result into action-level updates, distinguishing which decisions drove success, which were merely irrelevant steps, and which led to the later failure. This is credit assignment.
Suppose an 80-step trajectory finally succeeds. Giving all actions the same positive advantage reinforces the key decisions, the redundant searches, and the operations that once caused problems, all at once. A failure trajectory may also err only at the last step, and the large amount of correct behavior before it gets pushed down along with it.
Credit assignment needs to estimate:
\[Q(s_t,a_t) =\mathbb E[R(\tau)\mid s_t,a_t]\]A single trajectory can only tell us that some action co-occurred with success. To estimate an action’s causal contribution, the ideal is to try multiple actions in the same state and compare the subsequent results.
Forking from the same intermediate state:
state s_t
├── action a_1 → suffix τ_1 → reward 1
├── action a_2 → suffix τ_2 → reward 0
└── action a_3 → suffix τ_3 → reward 0.4
Such data is more valuable than three unrelated successful trajectories, because the state is fixed and the action differences are closer to a causal intervention.
In the language of causal inference, this kind of comparison is close to a counterfactual; in agent training and environment engineering, the more intuitive name is branched rollout or same-state branching: fix a shared prefix, change only the action at the branch point, then compare the suffixes.
RTMC organizes multiple rollouts of the same task into a tree by shared state, estimating step advantage at branch points, and beats GRPO by 3.2 points on SWE-bench Verified. C3 freezes the context, replaces the action, and replays a fixed continuation.
Their common premise is that the environment supports snapshot, fork, and replay: snapshot saves the full state at the branch point, fork copies multiple subsequent branches from the same state, and replay is used to reproduce and check execution results. If one can only rerun from the task’s starting point, the two trajectories may already differ before reaching the branch point due to random observations or tool return values, and the final reward will mix in the effect of these prefix differences. So the precision of credit assignment is jointly determined by algorithm quality and the environment’s ability to provide controlled, reproducible branches.
When the environment cannot branch heavily, one can train a function \(\Phi(s)\) that estimates the success potential of the current state, and use the change in state as a process signal:
\[r_t^{\text{process}} =\Phi(s_{t+1})-\Phi(s_t)\]This form has a key property: after summing along the trajectory, the intermediate terms cancel. Inserting redundant steps cannot increase the total reward out of thin air.
Rewarding Progress argues that process supervision measures “the change in the probability of a correct solution”, and reports an accuracy improvement of over 8% relative to outcome reward. TRACE computes turn reward from the log-ratio potential of a frozen reference against the gold answer, also exploiting the same telescoping structure.
Accumulating absolute \(V(s_t)\) or judge scores step by step produces another kind of incentive: as long as you stay in a high-value state, you can collect points repeatedly. In AgentPRM’s experiments, the validation PRM score kept rising while the real success rate dropped from 82% to 70%. This is a typical signal of over-optimized process reward.
Process signals and the final outcome solve problems at different granularities. Outcome provides trajectory-level ground truth but a sparse signal; process signals refine supervision to between adjacent states, but inherit the estimation error of \(\Phi\). So process signals are better used for advantage shaping, trajectory filtering, or auxiliary supervision, while the final outcome should remain the anchor of task success or failure:
The process signal answers “how much progress this step brought”; the final outcome judges “whether the task was ultimately completed”.
When combining the two, a high process score should not offset a final failure; and final success does not mean every action in the trajectory should be reinforced equally.
The potential difference in the previous section refines an episode-level outcome to the turn level, but it still depends on a learned \(\Phi\). If we also want to judge the contribution of a specific action, we need finer intermediate evaluation, more policy sampling, or the same-state branching introduced earlier. The resolution of credit and the cost of obtaining it thus form a direct trade-off.
By the resolution of attribution to actions, one can roughly order:
Episode outcome
→ Turn-level progress
→ Action-level advantage
→ Same-state branched comparison
The later it is, the more the signal can distinguish the contribution of adjacent decisions, but it usually also requires more forwards, more rollouts, or stronger environment capabilities. Same-state branching further requires fixing the starting state and executing multiple candidate actions, at a cost far higher than checking a single final result.
The training system must trade off among three goals: whether the signal is close to the real task result, whether it is dense enough, and whether the acquisition cost is acceptable. Common signals sit at different trade-off points:
Many new methods over the past year use extra computation to turn a sparse ground truth into a finer signal. When evaluating such a method, beyond final performance, one must also compare how much closer-to-real-result credit resolution is bought back per additional model call or environment rollout.
An agent’s training recipe is first of all an allocation of responsibility: which capabilities the model learns, and which boundaries the system guarantees. Handing deterministic, inviolable rules to RL wastes samples and cannot provide a hard guarantee; hard-coding decisions that need to adapt to the task and the current state into the system limits generalization. Supervision on the model side should shift with increasing uncertainty — from reusable prior knowledge and offline demonstrations toward on-policy experience — and concentrate expensive feedback on the branch points that most affect the result. The pipeline below unfolds along “system constraints — model prior — online optimization”.
Interface knowledge is usually stable and reusable across tasks. Tool schemas, action formats, common ACI patterns, and environment-feedback conventions can enter mid-training data through synthetic tool calls, full interaction trajectories, and error-recovery examples. The model thereby learns in advance to construct requests, read tool returns, and correct invalid calls, instead of mastering JSON syntax and tool protocols through trial and error only in the RL stage.
For example, a coding agent can learn in mid-training the format for calling file-read and patch tools; but whether it can write files outside the workspace, execute a deployment, or read secrets must be decided by the sandbox and permission system. The former is a model capability; the latter is a safety boundary that cannot be left to the model to observe on its own.
On top of existing interface capability, trajectory SFT further learns task-level workflows, state acquisition, and basic recovery. The goal is to move the policy into a region where it “occasionally completes the task”, providing a starting point for later exploration.
SFT’s metric should not be only the behavior-cloning score. In WebAgent-R1, long-CoT BC had a higher initial score yet a lower result after RL. An overly strong deterministic template compresses policy entropy. A good initialization must also preserve exploration space.
After finishing trajectory SFT, let the current policy enter the environment and collect the states it actually visits. The purpose of this step is to move the training distribution from the teacher model’s \(d_E^{\pi_T}\) to the current policy’s \(d_E^{\pi_\theta}\). Every policy update changes the subsequent state distribution, so the rollout data also needs to be continuously refreshed with training. Successful trajectories can enter rejection sampling, failure states can be handed to the teacher model for correction or used for recovery training, and high-uncertainty states are suited to being prioritized for branching.
The verifier should combine the initial state, the action trajectory, and the final state to check the goal condition, hidden side effects, permission violations, and evaluation integrity. This outcome decides which trajectories can serve as positive examples, and provides a ground-truth anchor for later RL and process signals.
A learned judge can supplement soft evaluations such as readability, efficiency, and solution quality, and is especially suited to ranking among candidate trajectories that have already completed the task. But a soft score cannot compensate for a task failure, nor replace hard metrics such as hidden tests, state diff, or environment invariants.
Teacher-model calls, branched rollout, and process audits all have relatively high cost. They should concentrate on:
This step turns training-signal design into active experiment design: the training system decides where to invest more cost to obtain more informative labels.
Only when the action prior already lets the policy occasionally succeed, the environment can continuously generate on-policy trajectories, and the verifier can stably judge results, does the RL loop have the conditions to run. Interface capability has been built by the earlier stages; RL focuses on the states the current policy actually visits, compares different decisions, reinforces high-return paths, and learns how to recover from errors it created itself. When positive rewards almost never appear, the policy lacks a usable exploration signal; when the verifier is unreliable, optimization instead prioritizes amplifying measurement holes.
In this loop, process signals are responsible for improving credit resolution and sample efficiency, the final outcome is responsible for anchoring the real task objective, and hard constraints continue to be enforced by the system. The three respectively address learning speed, optimization direction, and the safety boundary.
The full scheme can be written as:
Harness / ACI constraints
→ Trajectory SFT
→ Current-policy rollout
→ Verifier filtering
→ Additional feedback and branching at high-value states
→ Outcome-anchored agentic RL
→ Multi-dimensional independent evaluation
Whether a task suits agentic RL depends on whether the training loop can hold: whether the environment state can be read and reset, whether the consequences of actions can be sampled repeatedly, and whether the outcome can be verified within acceptable cost and latency. Different tasks lack different links, and the suitable training method changes accordingly. This section analyzes the loop bottleneck in different scenarios through a few representative task classes.
These tasks have digital state, executable actions, environment reset, and program verifiers, and most easily form a complete training loop.
Long-trajectory SFT can build a behavior prior for tool use and task workflows; hidden tests suit acting as a hard verifier, used to filter successful trajectories and provide the final reward. Continuously sampling the current policy can expose real failure states; these trajectories can serve as low-return samples and be used to construct recovery data for rollback, retry, and replanning. When environment throughput is sufficient, execution-grounded RL can directly optimize executable results; maintainability, resource cost, and code style suit being a secondary evaluation among correct candidates, handled separately from correctness.
The main risks are insufficient test coverage, grader tampering, harness overfitting, and environment startup cost.
Search’s short answers can be verified, but a long report also requires judging evidence support and source quality. A GUI simulator can check the final state, but a real website is hard to reset and contains irreversible actions such as payments, emails, and accounts.
Search and operation trajectory SFT can first teach query decomposition, page navigation, and basic tool use. For short answers, citations, and clear final states, one can use fact-checking or state-checking to provide hard feedback; for long reports, one still needs to evaluate evidence coverage, source quality, and whether the conclusion is supported. Continuously sampling the current policy in a resettable environment can collect recovery data after navigation failures, page changes, and tool errors. RL is better done in a search environment or a GUI simulator, while a real website is mainly used for permission-controlled evaluation and a small amount of data collection.
When a memory write happens, it is hard to immediately judge whether a piece of information is worth saving. It may help answer a query hundreds of turns later, or gradually go stale, conflict with new information, or create a privacy risk. Enterprise workflows have a similar delay: a ticket’s status can be updated immediately, but the business outcome may appear only weeks later, affected in the meantime by human handling and other systems.
The training system can only look back, after the future result appears, at which early writes or operations should receive credit. The time span, external interference, and multiple state modifications all increase attribution difficulty, so such tasks usually need a combination of immediate constraints, intermediate-state checks, and delayed outcomes.
Suitable signals include:
Theorem proving, numerical experiments, and simulated environments have strong verifiers, and suit rejection sampling, self-evolution, and RL.
Real rollouts for wet-lab work and robotics are expensive, slow, and possibly irreversible. Demonstrations and offline data carry the main body of training, and RL mostly happens in simulation and risk-limited real hardware.
Writing, aesthetics, strategy, and long-term interpersonal tasks lack a stable ground-truth anchor. A single LLM judge would compress preference into a set of exploitable proxy standards.
Such tasks suit high-quality SFT, individualized preference, user-edit feedback, and pluralistic reward. They are unlikely to see an RLVR-style jump like the one in math and code.
Across these scenarios, training efficiency mainly depends on whether the environment can produce reproducible, verifiable experience at low cost. Coding, terminal, SQL, closed search, and formal science can combine SFT, verifier filtering, and execution-grounded RL to continuously expand training data validated by real results. The higher the environment throughput and the more reliable the verifier, the easier this loop is to scale up.
Open-ended writing, aesthetics, social, and creative tasks lack a reproducible ground-truth measurement, and still rely on individualized preference, user feedback, and human judgment. The feedback cost is higher and the standards shift with users and context, so it is hard to scale up RL training the way verifiable domains do.
The basic unit of agent training can be understood as one repeatable experiment: the policy takes actions in the environment, the verifier reads the result, the optimization algorithm updates the policy based on the feedback, and the updated policy then generates the next round of experiments. Reward, State, and Credit form a continuous causal chain in this process. A tiny bias in the verifier changes which trajectories get high return; the policy then raises the probability of those trajectories, and the state-visitation distribution shifts along with it; once the new distribution enters the long tail, a sparse outcome makes it harder to explain each action’s contribution, and the credit error re-enters the next round of updates. Local errors thus amplify step by step along the loop.
This chain changes how training signals are evaluated. Accuracy on static data can only describe the verifier before the update; training truly depends on its reliability after the policy keeps changing. More rollouts and stronger optimization expand the policy’s search range, and both effective solutions and measurement holes get more chances to be explored. So while increasing training compute, one must simultaneously improve the environment’s observability, the verifier’s resistance to exploitation, and the attribution precision of feedback.
A mature training system continuously refreshes rollouts as the policy iterates, and incorporates newly appearing failure states into training and holdouts. The environment provides comparable experience through reset, replay, and same-state branching, and the verifier keeps the signal valid through isolated evaluation and adversarial auditing. What ultimately needs to be optimized is how much useful experience a unit of environment cost can produce. The algorithm decides how to use this experience; the quality of the loop decides the efficiency with which compute is converted into task capability. The long-term progress of agent training then depends on whether the training system can turn continuous interaction into experience that supports reliable updates.
从 Reward、State、Credit 理解 Agent 训练闭环
Agent 训练可以从 Reward、State 和 Credit 三个问题展开：如何用环境结果定义目标，如何让训练数据覆盖当前策略实际访问的状态，以及如何把延迟 Outcome 归因到具体动作。三者共同决定训练闭环能否产生可靠、可持续的学习信号。
设想一个 Coding Agent 接到任务：修复某个偶发的 Bug。
它搜索了 12 次代码，读了 8 个文件，提出 3 个假设，修改 2 处实现，运行 6 轮测试，最后让隐藏测试全部通过。
从部署侧看，这次运行似乎成功完成了该任务。
但从训练侧看，我们需要思考更多的问题：
一次传统的文本生成通常只有输入与目标输出。然而，Agent 的动作会改变环境，环境变化又会改写后续输入。训练对象因而由一个答案扩展成了一条闭环轨迹：
\[\tau=(o_1,a_1,o_2,a_2,\ldots,o_T,a_T)\]其中 \(o_t\) 是环境观察，\(a_t\) 是内部决策或外部行动。轨迹分布由策略与环境共同生成：
\[\tau\sim p(\tau\mid \pi,E)\]Agent 训练希望找到策略：
\[\pi^* =\arg\max_\pi \mathbb E_{\tau\sim p(\tau\mid\pi,E)} [U(\tau)]\]这个形式揭示了一个关键点：Agent 的策略同时承担学习与数据生成两个角色。一旦策略改变，它访问的状态、获得的观察、犯下的错误都会变化。
这和普通监督学习有根本差异。普通监督学习可以近似写成：
\[x\sim D,\qquad y\sim p(y\mid x)\]数据分布 \(D\) 由外部给定，模型参数不会改变下一批输入。Agent 的 History 分布则依赖当前策略：
\[h_t\sim d_E^\pi\]训练更新 \(\pi\) 后，下一轮采样的 \(d_E^\pi\) 也跟着改变。模型在学习如何行动的过程中，不断制造新的训练分布。
从经典 RL 视角看，这正是策略诱导的状态访问分布：策略决定会采到什么数据，基于这些数据进行更新后，新策略又会产生新的分布。在这样的场景中，经典 RL 的三个问题尤其突出：如何从复杂的环境状态中得到可靠反馈，如何获得覆盖当前策略行为的训练轨迹，以及如何把延迟结果归因到长轨迹中的具体决策。在具体实践中，人们常将它们概括为 Reward、State 与 Credit：
Reward：策略究竟应该最大化什么？
Reward 将任务结果转化为可优化的信号，决定学习方向。
State：训练数据是否覆盖策略实际到达的状态？
State 关注训练数据覆盖哪一段 Occupancy Distribution，以及教师模型示范能否迁移到当前策略的实际运行分布。
Credit：延迟结果应该归因给哪些决策？
Credit 决定一条长轨迹中的哪些 Action 得到强化，哪些 Action 需要修正。
全文将沿着这三个视角，讨论从大模型进入 Agent 时代至今，不断演进的各种训练方法分别提供了什么信息，又解决了哪一部分问题。
训练首先要定义什么是好的，什么是不好的。
对于普通问答，真值可能是一个标准答案。对于 Agent，结果通常表现为外部环境的状态发生变化，例如：
因此，高质量 Reward 通常直接读取环境状态。Agent 自身对过程的描述更适合作为辅助信号。
可以把 Verifier 看成一台测量仪器。它把完整环境状态压缩成一个训练信号：
\[R(\tau)=V(s_T,\tau)\]其中 \(s_T\) 是终局状态。一个好的 Verifier 至少要回答：
Coding 是较早适用可直接验证奖励的领域之一。编译器、单元测试、静态分析、文件 Diff 和隐藏测试能把程序状态转化为机器可检查的反馈。当这些反馈被直接用作 Reward 更新策略时，就是 RLVR（Reinforcement Learning with Verifiable Rewards）。
RLVR 提高了反馈的可重复性和扩展性，但“可验证”只表示 Checker 能稳定执行一套评分规则，不表示这套规则等同于真实任务目标。
SWE-RL 就是一个例子。它根据预测 Patch 与标准 Patch 的文本相似度计算训练 Reward。一份语义正确但写法不同的 Patch 可能因此得分较低。这个 Checker 衡量 Patch 与标准答案的文本相似度，没有检查 Bug 是否被修复。
单元测试离程序正确性更近，但仍只能检查已经编码的行为。覆盖不全时，Hardcode 可见样例也可能通过；评测通道缺少保护时，Agent 还可能修改测试或利用 Parser 漏洞。OpenAI 对 SWE-Bench Pro 的审计发现约三成任务存在问题；BenchJack 在 10 个 Agent Benchmark 中找到 219 个可利用漏洞。
前一个例子的问题是代理指标离真实目标太远，后一个例子的问题是测试覆盖不足。Specification Problem 因此落到了 Verifier 的目标定义与覆盖范围上。评价一种测量方式时，需要同时检查它能否稳定验证、保留了哪些信息，以及遗漏了哪些信息。
任何测量都在压缩信息：
\[M:\;(s_0,\tau,s_T)\longrightarrow z\]完整环境状态、行动轨迹和终局状态，被压缩成一个二值标签、标量分数或偏好排序。压缩是一把双刃剑，压缩越强，信号越直接简洁，训练越方便，遗漏也越多。
选择测量方式时，至少要看五个维度：
程序 Verifier 在 Directness、Repeatability 和 Cost 上常有优势，却可能只覆盖狭窄目标。人类专家能判断更宽的质量维度，吞吐与一致性较弱。LLM Judge 易于扩展，也更容易继承模型偏差和可利用模式。不同方案适合在不同场景和训练的不同阶段中结合使用。
典型形式包括：
Search-R1 用 Final Exact Match；ReTool 用最终答案等价；WebAgent-R1 检查网页任务是否完成；Agent-Diff 检查企业 API 操作产生的完整状态差异，并让未声明副作用直接清零。
这类测量适合满足三个条件的任务：
目标状态可以机器读取
行动后果可以隔离重算
有效解允许程序判定
它的主要风险是覆盖不足。测试全绿只能说明被测试的行为正确；终态正确也可能掩盖背后的违规路径。Agent 修改了测试、泄露了隐藏答案或造成短暂副作用，可能被简单的终态 Checker 完全忽视。
所以程序测量应同时检查：
写作质量、Patch 可维护性、客服沟通和研究创意生成等任务很难被压缩成一个确定规则。而人类可以很好地比较两个候选：
\[\tau_A \succ \tau_B\]Pairwise Preference 通常比绝对打分容易，因为人类擅长比较，但很难稳定定义出某个具体的分数。
它的优势是覆盖宽，能吸收难以形式化的判断。代价包括：
因此，人类偏好更适合处理软质量维度，例如清晰度、维护性和沟通风格。任务正确性、权限与安全仍应由独立信号负责。
LLM Judge 可以把专家 Rubric 扩展到成规模的轨迹评估中，Learned RM 则进一步地把偏好进一步蒸馏成低成本标量。
这类方法适合：
LLM as a Judge 一直以来争议较大。这一类方法面临三类风险。
第一类是静态偏差：长度、位置、文风、自偏好和格式会影响评分。
第二类是分布漂移：RM 在旧策略数据上训练，Policy 优化后产生的新轨迹可能离开它的可信区域。
第三类是对抗利用：One Token to Fool LLM-as-a-Judge 发现单个冒号和固定开头都能提高 Judge Reward。
因此，LLM Judge 更适合充当高吞吐的筛选器。定期人工抽检、Hard Verifier 校准和策略外 Holdout 都是必要的辅助措施。
常见 Proxy 包括：
它们计算便宜，但通常只是与任务结果相关的表面特征，不能直接说明任务是否成功。
格式合法可以通过 Grammar 或 Tool Schema 直接保证；把它当主要 Reward 会鼓励模型追求标签正确。长度和工具数属于成本代理，直接优化容易出现 Under-use 或 Padding。文本相似度适合快速筛选，但与语义正确性之间仍有距离。
ToolRL 的长度实验展示了这种风险：直接奖励回复长度让 Qwen-1.5B 在 BFCL 上从 46.20% 降到 33.23%，动态长度奖励进一步降到 28.51%。ReTool 没有加入长度 Reward，RL 后回复却自然缩短约 40%。工具提高解题效率后，成本改善会随任务能力一起出现。
Proxy 最合适的角色是 Gate、诊断指标或成功轨迹内部的次级目标。
成熟的 Agent 测量通常是组合结构：
硬约束层
权限、安全、Grader 完整性，不满足即失败
任务 Outcome 层
隐藏测试、State Diff、目标状态，定义任务是否完成
软质量层
人类或 LLM Preference，比较成功解的质量
过程诊断层
Turn-level Progress、成本、恢复、工具效率
这一框架的不同层次解决了不同的问题。不能把它们线性加权为单一 Reward，因为这会让高任务分补偿安全违规，也会让软质量掩盖任务失败。
训练时可以使用层级目标：
\[\text{先满足 Constraints} \;\rightarrow\; \text{再最大化 Task Success} \;\rightarrow\; \text{最后优化 Preference 与 Cost}\]这比“为每项指标调一个权重”更符合真实的任务结构和训练需求。
离线评测关注 Verifier 在固定数据分布上是否判断准确。进入 RL 后，Verifier 的分数会直接决定哪些行为得到强化。只要某种行为能够稳定骗取高分，优化过程就会提高它出现的概率。
例如，某个固定开头会让 LLM Judge 错误地给出高分。在静态测试集中，这类样本只占 1%，Verifier 看起来仍有 99% 的准确率。但 RL 一旦发现这个模式，就会反复强化它；原本罕见的 1% 可能逐渐成为 Policy 的主要输出。原始分布上的平均准确率无法代表训练时的可靠性，还需要在持续更新的 Policy 分布上重新评估 Verifier。
因此，训练用测量需要额外检查：
这也是为什么 Verifier 需要隔离、轮换和红队审计。Agentic RL 中，测量系统本身已经成为环境安全边界的一部分。
综合整个架构来看，一个也许行之有效的策略是：Outcome 提供真值；软评价补充难以直接编码的质量；过程信号提高学习效率；硬约束阻断不可接受的路径。结合每一种测量的优势构建系统。
Reward 回答了“怎样评价一条轨迹”，却无法回答“哪些轨迹会进入训练”。即使 Verifier 完全可靠，如果数据只覆盖教师模型的成功路径，学生自己造成的错误状态仍然得不到监督。
这里的 State 指策略在交互中访问到的整套 History Distribution。Agent 的每个动作都会改变下一步输入：学生只要在前面搜错一次、调用错一个工具，后续 History 就可能离开教师模型的轨迹。模型需要学习恢复的地方，不会出现在只由教师模型生成的数据中。
因此，数据由谁生成是训练机制的一部分。Trajectory SFT 在教师模型访问的状态上学习；On-policy 方法则让训练分布跟随当前策略。
Trajectory SFT 的示范可以来自人类或模型。本节讨论的工作主要使用更强的模型生成轨迹。教师模型负责提供示范，但不假设它是最优策略。
Trajectory SFT 可以看作应用在完整 Agent 轨迹上的 Behavior Cloning。教师模型先与环境交互，再把轨迹中的每个决策点拆成监督样本：
\[\tau_T\sim p(\tau\mid\pi_T,E),\qquad \mathcal D_T=\{(h_t^T,a_t^T)\mid \tau_T\}\]其中 \(h_t^T\) 是教师模型执行到第 \(t\) 步时的 History，\(a_t^T\) 是它给出的动作。学生模型使用标准交叉熵拟合这些动作：
\[\mathcal L_{\text{SFT}} =-\mathbb E_{(h_t^T,a_t^T)\sim\mathcal D_T} \log \pi_\theta(a_t^T\mid h_t^T)\]与只在轨迹结束时给一个 Outcome 相比，这种监督很密集：教师模型的每个动作都能提供 Token-level Label。它适合的任务场景主要是：
AgentTuning、Agent-FLAN 和 FireAct 都证明了轨迹蒸馏的价值。
密集标签仍可能覆盖不足。训练 History 来自教师模型的访问分布 \(d_E^{\pi_T}\)，部署 History 来自学生模型的访问分布 \(d_E^{\pi_\theta}\)。两者的差异就是模仿学习中的 Covariate Shift。
学生模型在某一步搜错一个文件，后面的 Context、假设和环境状态都会偏离教师模型的轨迹。Teacher Forcing 教的是“在教师模型生成的 History 上下一步怎么走”，而部署还要求学生模型学会“进入自己造成的异常状态后如何恢复”。
把方法按状态来源排列，会得到一条连续谱：
教师模型 SFT
状态由教师模型产生
策略 Rollout + Verifier 筛选
状态由采样策略产生，Outcome 决定保留或排序
On-policy RL
状态与动作都由当前策略产生，环境提供 Outcome
这里的核心变量是 Occupancy Distribution，也就是策略实际访问哪些状态。CE、DPO、Policy Gradient 只是不同的更新工具。
WebAgent-R1 提供了清晰的脉络。Qwen2.5-3B 从 6.1% 经 Behavior Cloning 提升到 20.0%，再经 RL 提升到 33.9%。直接从 Raw Model 做 RL 的版本略有退化：动作格式尚未掌握，正 Reward 几乎不出现。
这说明 BC 长期承担 Exploration Prior。它先把策略送进“偶尔能够成功”的区域，RL 才有信号继续优化。
当前策略的 Rollout 会同时产生成功样本和真实的失败状态。终局失败的轨迹不适合作为整段正向示范，但它经过的状态和其中的有效动作仍可用于训练。按照使用的监督粒度，失败轨迹可以在四个层面发挥作用：
SWE-Master 用失败判断任务难度；ETO 将失败轨迹作为 DPO 负例；Orchard 抽取失败轨迹中价值上升的片段。它们分别利用了失败数据中的难度、结果和过程信息。
同一条失败轨迹可以在 Outcome 层提供负信号，在 Process 层保留有效片段，并在 State 层提供稀缺的错误状态。终局失败只能说明整体结果不理想，不能说明轨迹中的每个动作都错误。进一步区分应当抑制和保留的决策，需要解决 Credit Assignment 问题。
On-policy 数据让训练发生在当前策略访问的状态上，解决的是训练分布问题。但 Verifier 通常只在轨迹结束时给出一个 Outcome，只能提供整条轨迹的总体结果，缺少各步决策的贡献信息。训练还需要把轨迹级结果转化为动作级更新，区分哪些决策推动了成功，哪些只是无关步骤，哪些导致了后续失败。这就是 Credit Assignment。
假设一条 80 步轨迹最终成功。给所有动作同一个正 Advantage，会同时强化关键决策、冗余搜索和曾经造成问题的操作。失败轨迹也可能只在最后一步犯错，前面的大量正确行为会被一并压低。
Credit Assignment 需要估计：
\[Q(s_t,a_t) =\mathbb E[R(\tau)\mid s_t,a_t]\]单条轨迹只能告诉我们某个动作与成功同时出现。要估计动作的因果贡献，理想情况是在相同状态尝试多个动作，再比较后续结果。
从同一个中间状态分叉：
state s_t
├── action a_1 → suffix τ_1 → reward 1
├── action a_2 → suffix τ_2 → reward 0
└── action a_3 → suffix τ_3 → reward 0.4
这样的数据比三条互不相关的成功轨迹更有价值，因为状态被固定，动作差异更接近因果干预。
在因果推断语境中，这类比较接近 Counterfactual；在 Agent 训练和环境工程中，更直观的叫法是 Branched Rollout 或同状态分支：固定共同前缀，只改变分叉点的 Action，再比较各个 Suffix。
RTMC 将同一任务的多条 Rollout 按共享状态组织成树，在分叉点估计 Step Advantage，SWE-bench Verified 比 GRPO 高 3.2 个百分点。C3 则冻结上下文，替换 Action 并重放固定 Continuation。
它们的共同前提是环境支持 Snapshot、Fork 和 Replay：Snapshot 保存分叉点的完整状态，Fork 从同一状态复制出多个后续分支，Replay 则用于复现并检查执行结果。如果只能从任务起点重新运行，两条轨迹在到达分叉点之前就可能因随机观察或工具返回值而不同，最终 Reward 会混入这些前缀差异的影响。因此，算法质量与环境提供受控、可复现分支的能力共同决定 Credit Assignment 的精度。
环境无法大量分叉时，可以训练一个函数 \(\Phi(s)\) 估计当前状态的成功潜力，再用状态变化作为过程信号：
\[r_t^{\text{process}} =\Phi(s_{t+1})-\Phi(s_t)\]这个形式有一个关键性质：沿轨迹求和后，中间项会抵消。插入冗余步骤无法凭空增加总 Reward。
Rewarding Progress 主张过程监督衡量“正确解概率的变化”，并报告相对 Outcome Reward 超过 8% 的准确率提升。TRACE 用 Frozen Reference 对 Gold Answer 的 Log-ratio Potential 计算 Turn Reward，也利用了相同的 Telescoping 结构。
绝对 \(V(s_t)\) 或 Judge 分逐步累加会产生另一种激励：只要停留在高价值状态，就能重复拿分。AgentPRM 的实验里，Validation PRM Score 持续上升，真实成功率却从 82% 降到 70%。这是过程 Reward 被过度优化的典型信号。
过程信号与终局 Outcome 解决的是不同粒度的问题。Outcome 提供轨迹级真值，但信号稀疏；过程信号把监督细化到相邻状态之间，却会继承 \(\Phi\) 的估计误差。因此，过程信号更适合用于 Advantage Shaping、轨迹筛选或辅助监督，终局 Outcome 仍应作为任务成败的锚点：
过程信号回答“这一步带来了多少进展”，终局 Outcome 判断“任务最终有没有完成”。
组合两者时，高过程分不应抵消终局失败；终局成功也不意味着轨迹中的每个动作都应得到同等强化。
上一节的 Potential Difference 把 Episode-level Outcome 细化到了 Turn-level，但它仍依赖学习得到的 \(\Phi\)。如果还要判断某个具体 Action 的贡献，就需要更细的中间评估、更多策略采样，或者前文介绍的同状态分支。Credit 的分辨率与获取成本由此形成一组直接的权衡。
按照对动作贡献的分辨率，可以大致排列为：
Episode Outcome
→ Turn-level Progress
→ Action-level Advantage
→ Same-state Branched Comparison
越靠后，信号越能区分相邻决策的贡献，但通常也需要更多 Forward、更多 Rollout 或更强的环境能力。同状态分支还要固定起始状态并执行多个候选 Action，成本远高于只检查一次终局结果。
训练系统需要在三个目标之间取舍：信号是否接近真实任务结果、是否足够稠密，以及获取成本是否可接受。常见信号位于不同的权衡点：
最近一年的许多新方法都在用额外计算将稀疏真值转化为更细的信号。评估这类方法时，除了最终性能，还需要比较每增加一次模型调用或环境 Rollout，换回了多少更接近真实结果的 Credit Resolution。
Agent 的训练配方首先是一项责任分配：哪些能力由模型学习，哪些边界由系统保证。把确定且不可违反的规则交给 RL 会浪费样本，也无法提供硬保证；将需要适应任务和当前状态的决策写死在系统中，则会限制泛化能力。模型侧的监督应随不确定性增加，从可复用的前置知识和离线示范转向 On-policy 经验，并把昂贵反馈集中在对结果影响较大的分叉点。下面的流程沿着“系统约束—模型先验—在线优化”展开。
接口知识通常稳定且可以跨任务复用。Tool Schema、动作格式、常见 ACI 模式和环境反馈约定，可以通过合成工具调用、完整交互轨迹和错误恢复示例进入 Mid-training 数据。模型由此提前学会构造请求、读取工具返回并修正无效调用，不必等到 RL 阶段再通过试错掌握 JSON 语法和工具协议。
例如，Coding Agent 可以在 Mid-training 中学会调用文件读取与补丁工具的格式；但它能否写入工作区外的文件、执行部署或读取密钥，必须由 Sandbox 和权限系统决定。前者属于模型能力，后者属于不可交给模型自行遵守的安全边界。
在已有接口能力上，Trajectory SFT 进一步学习任务级 Workflow、状态获取与基本恢复。目标是把策略送入“偶尔能够完成任务”的区域，为后续探索提供起点。
SFT 的指标不应只看 Behavior Cloning 分数。WebAgent-R1 中，Long-CoT BC 的初始分数更高，RL 后结果却更低。过强的确定性模板会压缩 Policy Entropy。好的初始化还要保留探索空间。
完成 Trajectory SFT 后，让当前 Policy 进入环境并收集它实际访问的状态。这一步的目的把训练分布从教师模型的 \(d_E^{\pi_T}\) 移到当前策略的 \(d_E^{\pi_\theta}\)。Policy 每次更新都会改变后续的状态分布，因此 Rollout 数据也需要随训练持续刷新。成功轨迹可以进入 Rejection Sampling，失败状态可以交给教师模型纠正或用作恢复训练，高不确定状态则适合优先分支。
Verifier 应结合初始状态、行动轨迹和终局状态，检查目标条件、隐藏副作用、权限违规与评测完整性。这个 Outcome 决定哪些轨迹可以作为正例，也为后续 RL 和过程信号提供真值锚点。
Learned Judge 可以补充可读性、效率和方案质量等软评价，尤其适合在已经完成任务的候选轨迹之间排序。但软分数不能补偿任务失败，也不能取代隐藏测试、State Diff 或环境 Invariant 等硬性指标。
教师模型调用、Branched Rollout 和 Process Audit 的成本都较高。它们应集中在：
这一步把训练信号设计转化为主动实验设计：由训练系统决定在哪里投入更多成本获取信息量更高的标签。
当 Action Prior 已经让策略能够偶尔成功，环境可以持续生成 On-policy 轨迹，Verifier 也能稳定判断结果时，RL 的闭环才具备运行条件。接口能力已由前面的阶段建立，RL 聚焦当前策略实际访问的状态，比较不同决策，强化高回报路径，并学习如何从自身造成的错误中恢复。正 Reward 几乎不出现时，策略缺少可用的探索信号；Verifier 不可靠时，优化又会优先放大测量漏洞。
在这个闭环中，过程信号负责提高 Credit Resolution 和样本效率，终局 Outcome 负责锚定真实任务目标，Hard Constraint 则继续由系统执行。三者分别解决学习速度、优化方向和安全边界。
完整方案可以写成：
Harness / ACI 约束
→ Trajectory SFT
→ 当前策略 Rollout
→ Verifier 筛选
→ 高价值状态追加反馈与分支
→ Outcome-anchored Agentic RL
→ 多维独立评测
一个任务是否适合 Agentic RL，取决于训练闭环能否成立：环境状态能否读取和重置，动作后果能否重复采样，Outcome 能否在可接受的成本与延迟内验证。不同任务缺失的环节不同，适合的训练方法也会随之变化。本节用几类代表性任务分析不同场景的闭环瓶颈。
这些任务具备数字化状态、可执行动作、环境 Reset 和程序 Verifier，最容易形成完整训练闭环。
长轨迹 SFT 可以建立工具使用和任务 Workflow 的行为先验；隐藏测试适合充当 Hard Verifier，用于筛选成功轨迹并提供终局 Reward。持续采样当前策略可以暴露真实失败状态；这些轨迹可作为低回报样本，并用于构造回滚、重试和重新规划的恢复数据。环境吞吐足够时，Execution-grounded RL 可以直接优化可执行结果；维护性、资源成本和代码风格适合作为正确候选之间的次级评价，与正确性分开处理。
主要风险是测试覆盖不足、篡改 Grader、Harness 过拟合和环境启动成本。
Search 的短答案可以验证，长报告还要判断证据支持和来源质量。GUI 模拟器可以检查最终状态，真实网站却难以 Reset，且包含支付、邮件、账户等不可逆动作。
搜索与操作轨迹 SFT 可以先教会查询拆解、页面导航和基本工具使用。对于短答案、引用和明确终态，可以使用事实检查或状态检查提供硬反馈；对于长报告，还需要评价证据覆盖、来源质量和结论是否得到支持。在可重置环境中持续采样当前策略，可以收集导航失败、页面变化和工具报错后的恢复数据。RL 更适合在搜索环境或 GUI 模拟器中进行，真实网站则主要用于带权限控制的评估和少量数据采集。
Memory 写入发生时，很难立即判断一条信息是否值得保存。它可能在数百 Turn 后帮助回答查询，也可能逐渐过期、与新信息冲突，或造成隐私风险。企业工作流也有类似延迟：工单状态可以立即更新，业务结果却可能数周后才出现，期间还会受到人工处理和其他系统的影响。
训练系统只能在未来结果出现后，回看哪些早期写入或操作应当获得 Credit。时间跨度、外部干扰和多次状态修改都会增加归因难度，因此这类任务通常需要组合即时约束、中间状态检查与延迟 Outcome。
适合的信号包括：
定理证明、数值实验和模拟环境拥有强 Verifier，适合 Rejection Sampling、自进化和 RL。
湿实验与 Robotics 的真实 Rollout 昂贵、缓慢且可能不可逆。示范和离线数据承担训练的主体，RL 主要发生在仿真与风险受限的实机环境中。
写作、审美、战略和长期人际任务缺少稳定真值锚点。单一 LLM Judge 会把偏好压成一套可被利用的代理标准。
这类任务更适合高质量 SFT、个体化 Preference、用户编辑反馈和 Pluralistic Reward。它们很难出现类似数学、代码领域的 RLVR 跃迁。
综合这些场景，训练效率主要取决于环境能否低成本地产生可重复、可验证的经验。Coding、Terminal、SQL、封闭搜索和形式科学可以结合 SFT、Verifier 筛选与 Execution-grounded RL，持续扩充由真实结果校验的训练数据。环境吞吐越高、Verifier 越可靠，这个闭环越容易扩大。
开放写作、审美、社交和创造类任务则缺少可重复的真值测量，仍需依赖个体化 Preference、用户反馈和人类判断。反馈成本更高、标准也会随用户和情境变化，因此很难按照可验证域的方式持续放大 RL 训练。
Agent 训练的基本单位可以理解为一次可重复实验：Policy 在环境中采取行动，Verifier 读取结果，优化算法根据反馈更新 Policy，更新后的 Policy 再生成下一轮实验。Reward、State 与 Credit 在这个过程中形成连续的因果链。Verifier 的微小偏差会改变哪些轨迹获得高回报，策略随后提高这些轨迹的概率，状态访问分布也跟着移动；新分布进入长尾区域后，稀疏 Outcome 更难解释每个动作的贡献，Credit 误差又会进入下一轮更新。局部误差由此沿闭环逐步放大。
这条链路改变了训练信号的评价方式。静态数据上的准确率只能描述更新前的 Verifier；训练真正依赖它在 Policy 持续变化后的可靠性。更多 Rollout 和更强优化会扩大策略的搜索范围，有效解法和测量漏洞都会获得更多探索机会。因此，增加训练计算的同时，需要同步提高环境的可观测性、Verifier 的抗利用能力和反馈的归因精度。
成熟的训练系统会随着 Policy 迭代持续刷新 Rollout，并把新出现的失败状态纳入训练与 Holdout。环境通过重置、重放和同状态分支提供可比较的经验，Verifier 通过隔离评测和对抗审计保持信号有效。最终需要优化的，是单位环境成本能够产生多少有益的经验。算法决定如何使用这些经验，闭环质量决定计算转化为任务能力的效率。Agent 训练的长期进展，则取决于训练系统能否把持续交互转化为支持可靠更新的经验。
There seem to be few places left for blogs to live on the Chinese internet. What remains is a restless sea: positions before views, display before expression, social traffic before sharing. As the internet grows more agitated, videos and image-text posts grow shorter. When we try to compress the time needed to transmit information to the extreme, thought has already lost its room to breathe. What remains is only dense emotion and hallucination.
I am not qualified to judge this environment. On the contrary, I am part of that density too. Over the past year, I found it increasingly hard to sit still and think. I got used to using noisy music in my headphones to cover up a noisy world, to sliding through short videos, to sinking into decaying time and venting emotion without purpose. I often felt powerless, tense, anxious, and irritable.
I decided to do something in the new year.
I realized that conversations with friends and teachers have always been when my mind is most alive. A strange thing often happens: only after I say a sentence does my thought arrive at the point. My own words suddenly make something clear to me. This also happens in writing. As this piece pours from my awareness onto the screen, I admit that I sometimes cannot keep up with it. I seem to have separated myself from “it”. What an interesting discovery. It feels as if, somewhere in my subconscious, I am guarding a generative intelligence that needs to be activated in an instruction-bearing environment. This calls to mind Libet’s classic experiment on free will: a decision appears in the brain before conscious awareness makes it. Our experience of decision and thought is an after-the-fact hallucination.
So I changed the title from “First Voice” to “The Illusion of the First Voice”. The first voice I write down is also the illusion of writing down the first voice.
Still, the voice was made. It carries language, and it is language itself: once the signifier forms a network, it takes on a life of its own and thought rushes outward. This is also why I respect large language models as much as I do. We sometimes care too much about the subject or the rules above symbols, and too little about symbols themselves. I do not intend to cast my insignificant vote in the Chomsky-Hinton debate, nor to choose a side between Wittgenstein and Lacan. I only want to talk about language: this small thing that fascinates me, this strange magic that may sometimes come before consciousness.
There is no doubt that, as a computer science student, I have barely entered the vast field of linguistics. But I will never forget the moment I encountered it. Not long after I entered Tsinghua, Professor Dun Deng wrote on the blackboard: “the morning star is the evening star”. In that instant, a flow state pushed open a heavy door. I love language, poetry, fiction, and all the great thoughts and brilliant lights of human civilization built on language. I also love language as language. I cannot fully describe the excitement I felt when I read that “metonymy is the diachronic sliding of concepts, while metaphor is the synchronic mapping of concepts”. I felt I was touching history, the present, and the future. I thought I possessed the whole world.
To fall in love with language today, amid the rapid rise of large language models, is fortunate. Overnight, language became a prominent discipline around the world. From California to Wudaokou, from computer science departments to humanities and social sciences, everyone has “language”, “computation”, and a few other words on their lips.
To fall in love with language today is also unfortunate. Countless people have already declared the death of computational linguistics. Fast-iterating models have planted flags of victory across language tasks. In the dust raised by their gallop, no one cares about the hills left conquered behind them.
When I found my way back to language, it felt as if everything had already ended. Yet I still felt unprecedented luck and happiness. I cannot see the distant mountains clearly. I cannot finish the books and papers. I only half understand long strings of models and formulas. I do not know where these roads lead, or whether all of this has enough meaning. But I do not want to drift with the current, and I do not want to deceive myself. I do not want to remain numb, dull, and wrapped in density. I want to pick language back up, write something, say something, speak with myself, speak with magic, and speak with that constructed reader: the unique you. I want to begin doing something, with enough motivation to learn and do research while I am still young.
I discovered that blogs were originally born on personal websites. I discovered that language appears when lived experience needs to be shared. I discovered that it is never too late to love something. I discovered that the last sunset of 2024 was unlike every day before it. I believe none of this is a hallucination.
To everyone who reads this far, I wish you courage, freedom, and fidelity to what you love.
Happy New Year.
]]>中文互联网上似乎没有几个适合博客生存的平台了。只有一片片狂欢的海——立场先于观点，展示重于表达，社交盖过分享。与躁动的互联网相适应的是越来越短的视频和图文，当我们努力把信息传递的时间压缩到极致时，思想早已失去了容身之地，虚空中只剩下稠密的情绪与幻觉。
我没有资格成为这一环境的审判者。相反，我也是稠密中的一员。在过去的一年里，我发现我越来越难以静下来思考问题，习惯于用嘈杂的耳机音乐遮蔽嘈杂的世界，在短视频的滑动中滑落，在腐烂的时间里无谓地发泄情绪。我时常感到无力、紧张、焦虑和烦躁。
我决意在新的一年里做点什么。
我发现同朋友和师长聊天时，一直是我思维最活跃的时候。时常出现这样一种情况：当我说出某句话之后，我的思维才抵达这个观点，我因我自己的言语而茅塞顿开。这一现象也发生在写作中，当这篇文章从我的意识里倾泻到屏幕上时，我承认我有时跟不上它。我似乎把“我”自己和“它”隔开了——多么有趣的发现！我仿佛在潜意识里守着一个生成式的智能，它需要在一个带指令的环境中被激活。这使我们想起利贝特那个经典的关于自由意志的实验，决定在大脑中先于人的意识做出。我们对于决定和思维的体验——是一场事后的幻觉。
于是我决定把题目从“先声”改为“先声的幻觉”。我写下的第一声，也是我写下第一声的幻觉。
声音总归发出了。它承载语言，它也是语言本身：能指在构建起网络的那一刻便拥有了生命，思想奔涌而出。这也是我那样尊重大语言模型的原因，我们有时过于看重符号之上的主体或规则，而忽视了符号本身。我无意在这篇文章里为乔姆斯基和辛顿的辩论投出我（无关紧要）的一票，也无意在维特根斯坦和拉康的分歧中选择我的倾向。我只想谈谈语言，谈谈这个令我着迷的小东西，谈谈这个也许有时先于意识的奇妙魔法。
毫无疑问地，作为一个计算机系学生，于语言学这一博大精深的学科我尚未入门。但我永远不会忘记我与它相遇的时刻，那时我刚进清华不久，邓盾老师在黑板上写下，“the morning star is the evening star”，一刹那的心流撞开了一扇沉重的大门。我喜欢语言，喜欢诗歌，喜欢小说，喜欢人类文明建构在语言上的一切伟大思想与璀璨光辉；我同样喜欢作为语言的语言，我无法言说当我读到“转喻是概念历时性的滑动，隐喻是概念共时性的映射”时的兴奋——我感到我在触碰历史、当下和未来，我以为我拥有了整个世界。
在大语言模型狂飙突进的今天喜欢上语言是幸运的。语言一夜之间成了全世界的显学。从加利福尼亚到五道口，从计算机学院到人文社科学院，每个人的嘴边都挂着“语言”“计算”和一些别的东西。
在今天喜欢上语言同样是不幸的。早有无数人为（计算）语言学宣告了死亡。快速迭代的大模型们早已在各类语言任务上插满了胜利的旗帜，马蹄扬起的飞尘里，没有人关心身后被自己攻克的山头。
当我兜兜转转回到语言身前时，好像一切都结束了，但我仍然感到前所未有的幸运与幸福。我看不清远山，看不完书和论文，对一大串模型和公式似懂非懂；我不知道这些道路通向何方，不知道这一切是否有足够的意义。但我不想随波逐流，不想自欺欺人。我不希望我继续麻木、木讷、包裹在稠密里。我希望重新拾起语言，写点什么，说点什么，与自己对话，与魔法对话，与那个被建构的读者——独一无二的你对话。我希望开始做点什么，带着充分的motivation去学习和科研，趁我还年轻。
我发现原来blog被发明时便在个人网页上。我发现原来语言出现于生命的经验与体验需要被分享之时。我发现原来热爱从来都不晚。我发现2024年的最后一道晚霞和过去每一天都不一样。我相信这一切都不是幻觉。
祝愿每一个读到这里的你，勇敢，自由，忠于热爱。
新年快乐！
]]>