Formulation
ICU-Sepsis-v2 models a patient's stay in intensive care as a Markov decision process built from MIMIC-III records: 716 discrete health states and 25 treatments, each a combination of intravenous-fluid and vasopressor doses. The reward is 1 if the patient survives and 0 otherwise, with no discounting, so a policy's expected return is its survival rate.
Deep Q-learning
DQN learns , the expected return of giving treatment in state and acting well afterwards, with a neural network. Transitions go into a replay buffer , and the network regresses on targets computed by a slowly updated copy of itself, :
Each extension below changes one part of this loop.
Store transition
Prioritized replay
n-step
Double
Dueling
Distributional
Noisy layers
Extensions
- Double: the online network picks the next action and the target network scores it, , which curbs overestimation.
- Prioritized replay: transitions with large TD error are replayed more often, , with importance weights removing the bias.
- Dueling: separates how good a state is from how much the treatment matters.
- n-step: the target sums rewards before bootstrapping, , so the terminal reward travels back faster.
- Noisy layers: weights with learned replace ε-greedy with exploration that depends on the state.
- Distributional: the network predicts the whole return distribution on fixed atoms, with , trained against the projected Bellman target.
Rainbow combines all six.
Comparison
Starting from a two-layer DQN with 128 hidden units, the team added one extension at a time and then all six together as Rainbow. Each variant was tuned on a grid of about four settings and run for 10 seeds of 300,000 episodes.
Results
The dueling head reached the best return, at the cost of noisier learning; prioritized replay came second and converged about 100,000 episodes sooner. The combined methods failed to learn on grids of about four settings each, which says more about tuning budgets than about the methods.

