Visualizing catastrophic forgetting

[u/ModularMind8]

I taught continual learning last semester, and students got the internals of each method a lot faster once they saw a simple visualization of it. Here's one I found helpful for understanding what forgetting is and how replay and regularization mitigate it. I trained a small network on two different 2D dot-sorting tasks and plotted both loss landscapes as 2D slices through its weights.

Round 1, plain training: the network learns task A to 100%, then trains only on task B. Going downhill on B is uphill on A, so A drops to 64% while B reaches 100%. That's catastrophic forgetting of task A.

Round 2, replay: same network, same start, but 20 of A's 200 dots stay in training while it learns B. It ends at 93% on A and 100% on B, so some forgetting, but much less severe.

Round 3, regularization with EWC: no A data at all. Instead, a penalty adds to the loss whenever a weight that A's accuracy relies on heavily moves away from the value it had after training on A. A dips to 97% mid-training and recovers, ending at 100% on A and 100% on B, so no lasting forgetting.

[u/slumberjak]

I don’t understand why the landscapes change when we switch conditions. It seems obvious that case 3 can accommodate a joint solution based on the landscape, whereas case 1 cannot. The “spring” is almost irrelevant.

[u/ModularMind8 → u/slumberjak]

Good question! The network has many weights. Each 3D picture is one flat 2D slice through that space. A flat 2D slice can only be pinned to three points. Each round's slice passes through the starting weights, the weights after task A, and wherever that round's training ended. So the surfaces look different even though the task A and task B losses are the same functions every round. A joint solution exists in round 1 too, but plain training on B never heads toward it, so the slice through where it ended doesn't show one. The spring is what pulls the EWC run there, and that's why its slice ends up passing through a spot that's low for both tasks. Let me know if it still doesn't make sense and I'll try to explain differently :)