Preprint Explains Policy Necrosis and Redundant Coverage in Option-Critic

A new research preprint published on arXiv examines the mechanics behind the Option-Critic reinforcement learning architecture. The authors provide theoretical and experimental explanations for why model performance typically improves as more options are added.

The findings show that the termination rule learned by Option-Critic contributes very little and can sometimes hinder exploration. Furthermore, the sub-policy within an option barely explores the action space, causing states to lock onto the first seemingly viable action without updating. The researchers termed this phenomenon "policy necrosis" and found that three-fifths of states in a typical option suffer from it.

According to the study, adding extra options does not actually improve any single option; instead, it reduces the probability of all options failing simultaneously from 59% to 4%. When proper exploration is restored, a single option is capable of solving the task.