Reinforcement Learning: Rewards, Value, and Sequential Decisions

In supervised learning, an example usually comes with a target. In reinforcement learning (RL), an agent selects an action, observes a consequence, and receives a reward signal. Its action can change which situations it encounters next. The objective must therefore account for a sequence, not merely whether the latest choice looks good.

This distinction matters for tool-using agents. Taking a quick shortcut may save one tool call but create a failure that costs several later steps. Conversely, calling a tool does not imply RL is being used: a scripted workflow or a prompted language model may act without updating a policy from rewards.

Open full-size image

Follow the loop: the agent acts, the environment changes, and state and reward return to inform the next action. Actions shape later experience. The state shown here may only be partially observable in practice; the learner seeks cumulative return across the sequence.

OpenAI, Spinning Up, Agent–environment interaction loop. · MIT

State, action, reward, and policy

A Markov decision process describes states, actions, transition probabilities, and rewards. A policy $\pi(a\mid s)$ determines how actions are selected. The state must contain enough information for the next-state and reward distribution to depend on it and the action rather than on an unrecorded history.

For a document-search agent, a state might include the question, evidence already retrieved, remaining budget, and unresolved claims. The latest user message alone may omit relevant history. A partially observed environment requires tracking a history or a belief about hidden state; calling the observation “state” does not make the Markov assumption true.

The discounted return from time $t$ is

$$G_t=\sum_{k=0}^{\infty}\gamma^k r_{t+k+1},\qquad 0\leq\gamma<1.$$

The discount gives less weight to distant rewards and helps make an infinite sum finite for bounded rewards. Finite episodes can also use an undiscounted objective when the task warrants it. The value $V^\pi(s)$ is expected return from a state under a policy; $Q^\pi(s,a)$ additionally fixes the first action. Neither is simply the next reward.

A short route with a delayed benefit

Consider an invented environment with two nonterminal states. At start, quit gives reward 1 and ends the episode. Alternatively, inspect gives reward 0 and moves to ready. At ready, finish gives reward 3 and ends the episode. With $\gamma=0.9$,

$$Q(\text{start},\text{quit})=1,\qquad Q(\text{start},\text{inspect})=0+0.9\times3=2.7.$$

Choosing only the largest immediate reward would quit, although inspecting has higher return. If inspection instead cost 2 reward units, its return would be $-2+2.7=0.7$, and quitting would become optimal under this objective. Rewards and costs define the problem; the algorithm does not decide what success ought to mean.

Q-learning propagates a discovered outcome

Tabular Q-learning updates a state–action estimate toward observed reward plus estimated best continuation:

$$Q(s,a)\leftarrow Q(s,a)+\alpha\left[r+\gamma\max_{a'}Q(s',a')-Q(s,a)\right].$$

For a terminal transition, continuation is zero. The following standard-library example uses a fixed schedule that visits every available action. It demonstrates credit propagation, not an exploration algorithm or a neural RL benchmark.

from math import isclose

q = {("start", "quit"): 0.0, ("start", "inspect"): 0.0,
     ("ready", "finish"): 0.0}
gamma, alpha = 0.9, 0.5
transitions = [
    ("start", "quit", 1.0, None),
    ("ready", "finish", 3.0, None),
    ("start", "inspect", 0.0, "ready"),
]
for _ in range(60):
    for state, action, reward, next_state in transitions:
        future = max((v for (s, a), v in q.items() if s == next_state), default=0.0)
        old = q[state, action]
        q[state, action] = old + alpha * (reward + gamma * future - old)
assert isclose(q["start", "inspect"], 2.7, abs_tol=1e-10)
assert q["start", "inspect"] > q["start", "quit"]
print({key: round(value, 3) for key, value in q.items()})

The first useful information appears when finish earns 3. Subsequent updates carry that value back to inspect. With a large state space, a neural network can approximate values, but the simple table’s behavior is not a blanket convergence guarantee for nonlinear function approximation.

Exploration changes the data you obtain

If an agent always quits, it never observes the reward after inspecting. Exploration tries alternatives so that the learned policy is not limited to its first guess. An epsilon-greedy rule sometimes samples an action instead of using the largest estimated value; more complex settings require more careful exploration strategies.

Exploration has a real cost when actions change external systems. A simulator or replay environment can support training without treating live users as arbitrary trial opportunities. Even in simulation, assess whether transition rules and rewards resemble the intended deployment. A policy can exploit a simulator bug as effectively as it exploits a valid shortcut.

Offline RL learns from already collected transitions. Coverage then becomes central: a log dominated by quit contains little evidence for inspect. A model assigning high value to unseen actions may be extrapolating, not discovering a good policy. Plain behavior cloning instead predicts recorded actions with supervised learning; it can reproduce a demonstrator without directly optimizing long-term return.

Rewards are an operational definition

Suppose a search assistant receives a point for every citation. It may generate many low-quality citations. Reward only short runtime and it may stop before checking evidence. These are concrete examples of a proxy diverging from the user’s goal. Define task completion, evidence quality, and costs explicitly, then inspect behavior rather than watching the reward curve alone.

Delayed rewards make credit assignment difficult: a final failure might come from the first search, a bad intermediate assumption, or the final response. Intermediate rewards can make learning easier but also change incentives. Retain a final task-success measure that is not merely the sum of convenient training signals.

Episodes and honest evaluation

Gymnasium’s environment interface separates terminated from truncated. A terminal task state has no future value; stopping an ongoing task only because an external time limit was reached may still require bootstrapping. Whether a horizon belongs to the task or to the data-collection wrapper determines the right target.

Evaluate policies on held-out initial conditions or tasks and report variation across runs, not only the best trajectory. Count environment interactions and failures as well as training compute. Compare against a simple fixed policy: if inspect always suffices in the toy environment, a learned policy offers no deployment advantage over writing that rule.

For language-model reward and preference training, continue with pretraining and post-training. For the execution loop that carries observations and actions regardless of how the policy was trained, see agent loop patterns.

My reading · Opened notes are remembered in this browser.