Abstract
<title>Abstract</title> <p> <bold>Background</bold> Knowledge distillation is expected to produce a policy that becomes even stronger when combined with Monte Carlo Tree Search (MCTS). We report a case in which the opposite occurred. <bold>Methods</bold> We distilled a 19x19 Go network (192 channels, 18 residual blocks, 11.98M parameters) from KataGo. All decisions were made from head-to-head games (272 games from the empty board, adjudicated by KataGo at 100 visits, reported with 95% Wilson confidence intervals). Every decision criterion was recorded in a pre-registration document before the corresponding experiment was run. <bold>Results</bold> MCTS at 400 simulations scored 0.317 against the same network playing policy argmax without search. The search machinery was intact: replacing only the leaf evaluation with KataGo raised the score to 0.983 (p = 1.9x10^-14). The defect was localised to the value head, which was well calibrated in absolute terms (mean residual -0.0091) but could not rank sibling positions (Spearman +0.005, against +0.146 for the policy head). Rebuilding the value head raised sibling ranking accuracy from 0.54 to 0.64 but did not make search useful (difference from policy argmax +0.000, n = 60). Two generations of self-play training in which the policy was held in place by a KL anchor produced no distinguishable improvement (generation 2 vs generation 1: 0.526, 95% CI [0.466, 0.584]). Changing a single component - training the policy by cross-entropy to the MCTS visit-count distribution - produced 0.824 [0.774, 0.864] against the preceding generation and 0.941 [0.907, 0.963] against the distillation baseline (z = +3.04, p = 0.0024), using less than half the self-play data. <bold>Conclusions</bold> The inference-time performance of an improvement operator and the teaching value of its output are distinct properties. MCTS selected a different move from the policy in 73% of positions without winning more games, yet its visit distribution was a substantially better training target than the policy itself. We note the same structural failure - a learned evaluator that remains calibrated while losing ranking fidelity, and an improvement operator that consequently runs backwards - in reinforcement learning for large language models. </p>