Windows? You might see USB errors in the MADIface Series Settings dialog during DSD256 playback. That will most probably be your computer needing some optimizations. But first please answer ramses questions.

Although USB3 should be USB2 compatible it would be a good to test all ports.

You need to evaluate a test procedure which triggers the issue so that you are able to reproduce the issue reliably.
Once you have such a test procedure pls try this with all other USB ports whether the issue is the same or whether there are differences.

That's explained in the manual. Check what is connected internally to the ports. You might also want to run Latency Monitor from Resplendence to check the improvement will stay.

You need to find out with LatencyMon which driver is blocking cpu cores for so long.
You should read and try to understand the information given on the LatencyMon webpage.

https://www.resplendence.com/latencymon

"
LatencyMon checks if a system running Windows is suitable for processing real-time audio and other tasks. LatencyMon analyzes the possible causes of buffer underruns by measuring kernel timer latencies and reporting DPC and ISR excecution times as well as hard pagefaults. It will provide a comprehensible report and find the kernel modules and processes responsible for causing audio latencies which result in drop outs. It also provides the functionality of an ISR monitor, DPC monitor and a hard pagefault monitor.

LatencyMon will display the highest latencies of a kernel timer and report the highest execution times of ISR and DPC routines as well as hard pagefaults. In most cases it will also find the drivers and processes responsible for executing them. It will create a comprehensible report which also displays all sampled data in a detailed manner allowing you to perform in-depth analysis.

The audio latency problem

Windows is not a real-time operating system. All requests to the operating system are delivered on a best effort basis. There are no guarantees whatsoever that requests are delivered within a certain time frame, which are the characteristics of a real-time operating system. That is not a problem for most devices and tasks but this is bad news for audio applications (which are considered soft real-time) because they need to deliver data to the subsystem and the hardware in buffers several times per second. If one or more buffers miss their deadlines and are not delivered in time it has audible consequences which are recognized as dropouts, clicks and pops.

About DPCs and ISRs

The Windows thread disptacher (also known as scheduler) which is part of the kernel executes threads based on a priority scheme. Threads with higher priority will be given a longer execution time (also known as quantum or time slice) than threads with a lower priority. However the kernel also knows other types of units of execution known as interrupt service routines (ISRs). Devices connected to the system may interrupt on a connected CPU and cause their interrupt service routines to execute. An interrupt can occur on the same processor that an audio program is running on. Any thread that was running on the processor on which an interrupt occured will be temporarily halted regardless of its priority. The interrupt service routine (ISR) is executed and may schedule a DPC (Deferred Procedure Call) to offload an amount of work. The DPC will most likely run immediately on the same processor which means the audio application will halt until both the ISR and the DPC routines have finished execution. That is because ISRs and DPCs run at elevated IRQL which means they cannot become preempted by the thread dispatcher (scheduler). Therefore to guarantee responsiveness of the system, ISR and DPC routines should execute as fast as possible. Guidelines say that they should not spend more than 100 µs of execution time however this is often not reached due to hardware factors beyond the control of the driver developer. If execution time gets too high, the audio program may be unable to deliver audio buffers to the hardware in a timely manner.
"

I have uninstalled my heavy Trend Micro antivirus and am now back to using the windows defender app. The Latency monitor now says my computer is fine!
Next week I'll check the reading and recording capacities of my PC but it's looking good.

Thank you for your help again!

Chapter 31.12, USB Audio, explains exactly that.