Credit Assignment Across Agents
Why a shared team reward gives every agent the same noisy signal, how counterfactual baselines and value factorisation isolate individual contributions, and the structural assumption each makes.
A cooperative team receives one reward for what it collectively achieved. Each agent must work out how its own action contributed, and with a shared scalar it cannot: an agent that acted badly is rewarded when its teammates succeed, and one that acted well is punished when they fail. The learning signal is dominated by other agents' behaviour, and the problem worsens as the team grows.
Counterfactual reasoning
The counterfactual approach asks what would have happened had this agent acted differently, holding everyone else fixed. COMA implements this with a centralised critic and a per-agent advantage computed against a baseline that marginalises over that agent's actions:
The baseline is the expected value if this agent had acted according to its policy while the others did what they did, so the advantage isolates the effect of the actual action. It requires a centralised critic that can evaluate counterfactual joint actions, which is exactly what centralised training provides.
Value factorisation
The alternative decomposes the team value into per-agent utilities, so each agent has its own learnable signal, with a constraint ensuring the decomposition is consistent with the joint objective.
VDN assumes the joint action-value is a sum of per-agent values. Simple, and it cannot represent tasks where an agent's contribution depends on what others did.
QMIX relaxes this to any monotonic combination, enforced by a mixing network with non-negative weights. Monotonicity is what preserves the crucial property that maximising each agent's utility individually maximises the joint value, which is what makes decentralised execution valid.
QTRAN and its successors relax further at the cost of more complex constraints, addressing tasks where monotonicity genuinely fails, such as those requiring coordinated simultaneous action.
When it breaks
Monotonicity is a real restriction. Any task where the best individual action depends on another agent's choice, coordination games where both must move together, violates it, and QMIX cannot represent the correct value function. The failure is a representational limit rather than a training problem.
Counterfactual baselines need an accurate critic. The advantage is a difference of critic estimates, so critic error enters twice and can dominate the signal it is meant to isolate, particularly early in training.
Lazy agents are an equilibrium. With a shared reward, an agent that does nothing while others succeed receives the team reward, so inaction can be locally optimal. Factorisation helps by giving each agent a distinct signal, and it does not remove the incentive where the factorisation is imperfect.
Scaling is the persistent problem. Both approaches involve a centralised component whose complexity grows with the number of agents, so methods validated on a handful of agents frequently do not transfer to dozens, and results should be read with the agent count prominent.
12 flashcards for this concept
Click a card to reveal the answer.