Make measuring with LatencyMon as suggested by MC. In properiert select method kernel latency timer and watch what the minimum and usual values are.
For an ideal system the min kernel latency timer is around <5 microseconds which should be reached by the system at least a few times during a 5 minute measuring interval. Usual values between 5 and 30 microseconds. Peaks should be low under 100 microseconds or something is blocking the CPU cores for too long, either:
- energy saving in the bios
- wrong energy settings in windows
- inefficient / bad drivers
- too many background tasks
Bad drivers you can identify also with LatencyMon. You can see it already during measuring with a higher number of ISR and DPC.

What other usb devices or hubs are connected to the systen besides mouse and keyboard? Disconnect all other and try all USB 2+3 ports whether one works better than the other.

What are your BIOS settings in terms of
- enervy saving ( c/p/t-states, c1n)
- Turbo
- speed step (EIST)
Disable Energy Saving and eventually also EIST and Turbo to get a fix cpu clock which is ideal for lowest system internal latencies to get a more agile reacting system.

Check whether Windows 10 is optimized for background tasks.

Use O&OWin10Shutup to reduce the amount of background tasks by disabling ie apps of the Start menu and telemetry data collecting  processes.

If you do nothing as user on your system then the system should be IDLE .. only very little background activity should happen. This needs to be reduced so that your CPU has full power and no lag fulfilling audio tasks which have near realtime requirements.

Therefore always disable automatic software updates.

Before making changes to your system make sure that you have a working backup and restore, best by using Macrium Reflect. It's fast and reliable and reduces the wear on your SSD because it changes only disk blocks during restore operation that need to be changed.

1. Yes, you talk about WDM and different drivers (USB/FW) and driver architecture. WDM is limited to buffer sizes of 256 samples, but somtimes that fails and one has to choose 128. 64 might work if the computer is capable for it.

2. Such are software errors which are not shown in the error display.

3. Yes, the FF blinking means complete loss of sync. That should be visible in the error display. Maybe only for a short moment as the automatic restart resets the error display.

Thanks, that pointed me in a good direction. In fact, the loss of sync is not visible in the error display, because the USB settings dialog closes automatically in the moment the error happens. Using this information, I was able to do long-term tests without being present, because I could see if the error had appeared from the USB settings dialog being still open or not.

I noticed that I could reproduce the error by wiggling the USB plug on the PC side slightly. That made me think that maybe my USB cable is not that good. In fact, the only two USB cables I had lying around were ~12 years old. With both cables the error was reproducable.

So I ordered a new high-quality USB cable. I'm really not into cable voodoo, but I did not want to make a compromise here, so I chose an Oehlbach USB plus for ridiculous 20 Euros (while this is still cheap, as I also found USB cables for 900 Euros #wtf?!).
And this was it: the error is gone. I have been running my PC for 48 hours now playing audio non-stop without any loss of sync. Still using buffer size of 64 samples, so I am happy again.

Thank you very much for your help!