EDIT1+2

Hi Bruce.

thanks for your detailed description, please also tell which Windows version you are using.

Warning: Before doing any change on Windows perform a system Backup by taking a disk image with Macrium Reflect, there is a free version available but I recommend to get this as backup solution and to install it later also to an USB stick so that you can boot from that stick for recovery. It copies all required drivers and add on drivers to the stick (USB3, Sonnet, Network, ...).

Along with your BIOS upgrades, did you perhaps  enable by this CPU microcode upgrades  against Spectre and Meltdown ? This can slow down the system for certain operations down to 50% of previous performance, for me ie when performing downmix in Cubase. In this case you should IMHO compensate this performance loss by reenabling Hyperthreading, see also next chapter. In Win7 you activate ucode upgrades by BIOS in Win10 the latest status was IIRR that Microsoft performs it, but changed for a while with 1804. There is a check tool available for this to be sure.

Does enabling of Hyperthreading make a difference? I would leave it enabled. For my system disabling of Hyperthreading kills Performance / reliability against audio drops pm high audio load (400 tracks test project in Cubase) although it's a E5-1650v3 which is very similar to your CPU.

What are your BIOS settings in regards to EIST, Turbo,  C-, P-, T-States, C1E ?
I recommend to keep EIST and Turbo enabled, C-states set to C0/C1 (or disable, depends on BIOS), disable P- and T-states, disable C1E.
With this settings my system runs with all cores constantly with 200 or 300 MHZ higher clock than Base clock.
See here how I am doing it on my Supermicro based socket  2011-3 system and what tools I use to control it.
http://www.tonstudio-forum.de/blog/inde … -X10SRi-F/

Use bitsum's parkcontrol tool to disable CPU core parking in the high performance energy profile and use this when you want to work with highest performance. Parking and waking up CPU cores cause also latency which also gamers notice as micro stutter on the screen/GPU.

In regards to DRAM. If your system runs otherwise stable I would not think that you have a memory related problem. Importal is to use modules of the same type. As a test you could take out 4 of them. Consult handbook which memory slot positions need to be used with only 4 modules built-in to ensure proper operation in quad-speed for socket 2011-3.

In regards to GPU, did eventually the driver change ? I had one time a bad driver which slowed down the whole system.

What PCIE slot do you use for the sonnet card which requires a x4 socket ? Is the socket that you use currently  eventually in conflict with other sockets or subsystems ? Does this slow down your GPU to work only at x8 instead of x16 speed? You can use cpu-z or gpu-z to find this out.

What happens if you connect the UFX II to different USB2 or USB3  port which comes definitively from Intel Chipset (not from the 3rd party ASMEDIA chip) ?

I also use and recommend this sonnet card, as it is very good and using message signalled interrupts (msi). It can't hurt though to fallback to Intel Chipset USB ports just for a test.

If you have Win 10 then use O&O1Oshutup to disable as many data grabber and background tasks as possible as there is too much data collection going on in the background, indeer,  cortana, weather apps and such crap.
You need to perform this also for the Administrator user if you reenabling it. Only some Oo10shutup settings are global, most seem to be on a per user setting.

On every system make sure that the indexer for Windows search / Cortana is disabled or at least fine tuned for only indexing meta data.

I would start with

0. Backup system disk by taking an image using Macrium Reflect (very reliable)

1. try ufx II on other usb2/3 usb  ports coming from Intel Chipset to exclude issues with sonnet card

2. Then BIOS related part to ensure energy saving i's disabled completely and Hyperthreading, EIST, Turbo are enabled.

3.  disable CPU core parking.

4. Check whether ucode upgrades slow down your system. On Win 7 you can deactivate them by flashing an older BIOS.

General remark.

1) Performance tuning you do in single steps, not more than one change then reamers ure.

2) The sum of all fine tuning steps together bring the best benefit. Keep suggested things even if they change not much at the moment as single step as long as it makes things not worse.

3) Upgrade system Disk to a SSD
which also can hold sample libraries to increase overall system Performance.
Get a second SSD for recording projects only.
By this you have split SATA load for system and applications to 2 different SSD's.
Samsung EVO's are very good.
SSD's became cheap these days.
You can use Macrium to restore your system Image to SSD for a quick migration.

In regards to the Sonnet. Fully supported by Win10, only Win 7 needs driver installation.
Which driver do you eventually use (if Win7).

Please check as with Windows 10 Microsoft delivers them as part of their upgrades. This was working so with 1709, 1804 broke it for a while, but it should be back now.

Full Turbo you will not get when disabling cpu core parking. But a little more for all cores.

Maybe misunderstanding here. If you place the card to the 2nd GPU socket it can happen that the speed of your primary GPU socket will be reduced from x16 to x8. Ensure to use a socket that does not impact the performance of your GPU.

It's a very good card and it solved some issues that I got after adding things like Bluetooth USB adapter and one USB 3.0 bridge to my system (I have C612 Chipset on a socket 2011-3 server board).

Now I can connect two UFX+ and ADI-2 Pro to the Sonnet card without any issue's and load a Cubase test project with 400 tracks and playback without audio loss with ASIO buffer size set to 32.

Also stable with Spectre Meltdown ucode upgrade but not when disabling Hyperthreading.

No I use 2 UFX+ and one ADI-2 Pro.
You can try an UFX+ of course.

In most cases if things fail like this then it's most likely your combination of mainboard / Chipset / BIOS. I also had an bogus Mainboard in the past where the successor model of the same board of all sudden performed much better, but didn't cure everything.

For some people Firewire gives better results. For me it was USB.

The best is to ask other people what works good or to buy a turnkey system where you pay that somebody else solved all issues.

I heard that Xeon based supermicro systems run very reliable. Built such a system and I can confirm, yes true. But even with this system I needed to get the Sonnet card as the USB ports of Chipset are not fully isolated and the connected equipment can influence each other.

Now where I isolated all three RME devices behind the Sonnet no issue's.  Neither on Win 7 nor on my Win 10 test system 1804.

Hi Timur,

Regarding the Intel USB port experiment, I have already undertaken testing the Intel ports with all flavors of XHCI handoff.  I tested the latest and a few earlier Intel-branded drivers, along with the Microsoft provided ones.  Additionally, I tried a few other PCIe USB 3.0 adapters that I had around the studio from previous machine builds.  These had varying chipsets.  I also tested against other USB "2.0 native" cards.  Finally, I purchased the Sonnet Allegro Pro 3.0.  All failed the same way, so I feel that I've exhaustively tested the USB subsystem(s).  My conclusion at this point (unless you see something else in all that which I missed) is that something else is interacting with ALL of these scenarios to destabilize my system.

I need to test with the standard VGA driver, I suppose.  That is difficult in my current predicament, because I am sleeping very little as is, and I can't effectively work to picture in Pro Tools in that configuration.  Unfortunately, the deadline for completion of this film is not adjustable to accommodate this issue, and over the few months I have been fighting this (ever since the Fireface 800 fried, at the beginning of production on the film's music), I have been struggling against the dual demands of keeping up with the work load and trying to solve this highly frustrating problem.

I am currently experimenting with limiting the AMD R 380 to the PCIe 2.0 standard.  I did some reading about the card's performance in systems which are actually limited to that standard, and the conclusions of experienced reviewers on that subject were that the performance of the card would not be greatly impacted in that scenario.

I would appreciate opinions on that.  My rationale for trying it is to determine if the PCIe 3.0 bandwidth in and of itself might be stressing a yet-unidentified weak link or undiscovered compromise in my system configuration, and whether limiting that throughput would lighten up and potentially walk the system back "from the edge."

Yesterday, at least until I became exhausted and needed sleep, the system seemed to stay up MUCH longer.  But, because I have been working through suggestions offered by Ramses and Matthias at a very quick pace, I am unsure whether I can confirm whether this did indeed help--or whether one of the other suggestions may have actually solved the problem.

Thoughts?

Today I am going to try to push the system hard until it either fails, or until I determine that something we've done has helped.  I will be very grateful if it is the latter.  And...if I get significant progress made on the film, it will start buying me some time.

I will report back to you all when/if new information manifests.

THANK YOU to Jeff, Matthias, Ramses and Timor.  I am grateful for your generosity, and for the time you have taken to offer suggestions.  I have been laughing about this experience, because I have configured and optimized many, many machines over the years and I don't think I've ever run into a problem quite as difficult as this one.  Usually, a few hours and some easy tweaks will do it, or I'll discover a piece of hardware failing or something stupid I did by accident.  This seems to be a genuine design issue with the Asus X99 board, which necessitates all sorts of implementation gymnastics to address.

Of course, when I built this machine over three years ago, the Haswell E and X99 chipset were bleeding edge, and the Fireface 800 did not present these issues.  My hope is that this may all fall under the category of deferred optimization.  Ironically, I didn't choose the AMD GPU specifically to avoid X99 conflicts.  I'd just become tired of a particular Nvidia board I had in a previous build (which didn't want to remember which monitor was primary), and was giving the AMD card a roll of the dice.  So, at least that choice seems to have been well-made, given the number of Nvidia-specific problems I've seen while researching my own issues.

Cheers, and please let me know any other thoughts that might come up.  One thing in particular that I did while working through the suggestions Ramses offered was to use the "O&O1Oshutup" utility to double-check my work on manually disabling various Windows 10 "junk processing."  I was glad to see that I had found a number of the offending processes already, but was surprised (and hopeful) to find some others that I had not specifically run across.  IT MAY BE...that one or more of these is actually responsible for the better performance I ***might, haha**** have seen late last night.

Wish me luck, and thanks again.  Any feedback on this report is appreciated.

Did you try to disable the mouse-pointer hardware acceleration yet (by enabling mouse-trails via Windows mouse options)? It's the mild version of trying to disable the AMD driver all together. As mentioned before I was only able to (re)produce this problem with the combination of (Asus) X99 + NVidia cards, but you never know. Even with full on dropouts there were no increased DPC latencies measurable.

In the end it very likely is a BIOS/UEFI issue present on Asus X99 boards and likely others as well. There is hardly any reason why something like accelerated mouse-pointer drawing should disrupt USB operation, but it does. Based on that experience alone I do not trust (Asus) X99 systems and would not be surprised if other strange combinations cause similar issues. So you may just have hit some similar BIOS bug.

Also this: Total Mix is putting a substantial load onto the USB subsystem, at least according to my experience.  I can break the system just by instantiating it.

I find it bizarre that Firewire was so much more solid on this same machine.  USB seems fragile in comparison.

Yes, I read that.  My results, when the Diagnostic is not throwing up errors, concur with these.  At 32 buffers, the system is able to handle simultaneous multitrack Audio, Video, and even some audio DSP.

Do you have any theories on the two data points that I found interesting?

First, that throttling the 16x AMD R-9 from PCIe 3.0 to 2.0 bandwidth seems to help make the system more stable.  That seems counterintuitive (that the CPU would take on more work if it can only pass half the data through a pipe that is otherwise solid).  I say that it is solid (at least in and of itself) because I've put that video card through exhaustive benchmarking and it has consistently overperformed its advertised spec when hit with multiple rounds of testing.  Why would artificially making it a less efficient subsystem help out here?

Second, that instantiating TotalMix is a pretty guaranteed way to bring it down and bring on the gapping and FiFo Errors quickly?

To be clear, these errors are in no way similar to typical buffer underrun errors.  Once this gapping occurs, other observable subsystems seem caught up in some kind of destabilized state of operation.  The GPU and video output frame rate becomes affected too.  In fact, once it has occurred, Pro Tools throws up a Video Engine "reported errors" message.  But it seems to be a symptom, not a root cause, since the FiFo Error can occur in an audio only session, without even having instantiated Avid Video Engine.

If I close Pro Tools after the error has occurred, and just go to YouTube and open a random video, the video "syncs" to the gapping heard in the audio.  If I leave the video playing and power cycle the UFX II at THAT point, the video will begin playing at correct frame rate.  Once I power cycle the UFX II back on at that point, the problem is "healed" and both audio and video play back normally.

Just to be clear that I am not complaining about USB itself.  Nevertheless, I am trying to make an expensive and well-built system function with the UFX II--an interface that was recommended to me as my best replacement path for my failed Fireface 800 by two different (and very competent and helpful) RME dealers.   Now I am in a hellish world of sleeplessness and embarrassment as my fellow filmmakers sit and wonder if I will hit my deadlines.

Any other suggestions as to what I might try?  I am curious if you have ever built systems with the UFX II versus the UFX+.  Theoretically, the UFX II does not need more bandwidth than USB II can offer.  BUT, what  I am experiencing is that simply adding whatever USB load that TotalMix adds to the equation TANKS my system dependably.

Hopefully everyone is clear that I am not trying to assess blame.  I am trying to not be in hell, and to come out of this film without losing any more money, so that I can recoup the losses I already incurred when the Fireface reached the end of its life (which was well lived).

Thank you for the reply Timur.

Yes, that was one of the first things I tried.  My methodology was working outward from any RME-specific settings into the system work.  However, I will try that again, because I have done quite a lot of further optimizing since my first post.

In my assessment, I would say that the grand total of all the optimizing has walked the system back maybe a step from the edge.  It seems to me that whatever constitutes the root cause is still present, and the only progress I'm really making is to keep inching the tipping point.

One of the clues, though, is that even though my track count and buffer performance is excellent at 32 samples (before the eventual FiFo Error manifests), what seems to hasten that tipping point is additional USB loading, either in the form of Total Mix or in the form of simultaneous record and playback.

One thing that really seems to trip it is ADR recording, where I'm looping 15-20 second segments of video/audio while Pro Tools is dropping each take into the same track (on different playlists).

If that offers anyone any clues, I'm all ears.  What I generally do is wake up and try anything that's suggested, and then get as much work done as possible until I fall asleep.  Rinse and repeat. 

Best regards,
B

I am in the exact same situation. I get Fifo some times in the USB Diagnosis, other times it's just numbers.
First time i connected the UFX II the sound would start stop start stop with about 1 second with sound and 1 second without.
Sometimes i get crazy buzzing sounds when audio is playing, but usually there are random dropouts.

Tried all my USB ports all of them produce the same issue.
I had a Babyface (original) before and had no issues. Updated drivers, updated firmware, no CC on etc.
I have a very similar setup as OP. Windows 10. Asus x99-e motherboard.
Everything has worked perfectly until i added the UFX II.

This seems like it's a common issue,
https://www.youtube.com/watch?v=8FIg_y8POFY
this guy had to buy a usb addon card for it not to do drop out, is there a specific USB prerequisite?

Ok, so i ordered an Allegro Pro USB 3.0 PCIe card... Still the same issue.