I'm making this topic here to both present the RME staff a problem and provide future googlers a solution.
I've already done this search myself multiple times, and even found a video presenting the problem, but thought I should present everything I've found here in one place.

Basically, I like to keep my buffer at 96 or 128 when writing/recording. When I get to mixing/mastering or the plugins just get too heavy, I switch to 512. Previously, on my Focusrite Scarlett Solo 3rd Gen, when I would make the switch, any playing audio would stop for a second, then continue smoothly. I could jump back and forth between buffer sizes with no problems.
Now, on my Babyface, when I switch buffer sizes, either the ASIO audio, the Windows audio, or sometimes both, begin to become a crackly, slowed down mess, which is something I expect from an $80 interface, not my shiny new $1,000 one with the world's best drivers

After much experimentation and research, I found that switching my Windows audio device would stop the crackles. You know, when you click on the lil volume icon in the bottom right corner. So switch from phones to speakers, or one ADAT to another, or whatever. Really, the crackles get left behind, because if you switch back, say from the speakers to the phones, you resume the crackles.

So, the solution? Before you switch buffer size, change to a Windows output you aren't using, say, ADAT 7+8, change the buffer size, then switch back to whatever output device you actually wanted to use.

I have to ask, are the developers actually aware of this? I've found multiple posts about this over the years and they all peter out with a bunch of fingerpointing and zero conclusions. Yes I'm on the ultimate power plan, yes I turned selective USB suspending off, yes I have great stats on LatencyMon, yes it's plugged into the back of the pc in a USB3 slot. Is it true that I'm just supposed to pick a buffer size once and then never change it again forever, so really, it's my fault that it's glitching out? C'mon now!!