So I figured out what was causing the color problem. It turns out that when the ISR hits, it reads the VDP status register. Doing this cancels any command that may be in progress (in this case, set write address). So what happpens is that at the start of a screen write, instead of resetting the write pointer, writing continues from the last frame, overwritting the color tables and other stuff in VDP memory.

Disabling interrupts while VDP commands are being written seems to have fixed the problem.