Your descriptions so far leave one thing unclear: did the FF800 work before you made the Nucleus Nuendo control work? In that case you could simply switch off the Nucleus control to see what happens. If not there is a basic problem with FireWire performance. What FW chip is on that Vantech card? You also might have to install the legacy FW drivers, like mentioned/described several times in this forum.

Do you have the Nucleus connected via USB as well?
Perhaps you can try with it, disconnected from USB.

Also, for the Legacy Firewire driver, which MC mentioned, just google-search: "Legacy Firewire Driver for Windows 10".
It's a Microsoft download.

Thanks for checking!

I think I might have figured out what causes the stuttering..

I assume that you use the Nucleus as a Monitor controller, as well, via its SPDIF. Are you, also, using SPDIF for sending the SSL Preamps into the FF800?
Based on that assumption, I checked the Nucleus' manual for where someone can configure its clocking capabilities(Master, Slave, Sample rate etc), but there was no mention of this in the manual(!!)

I, then, looked for similar issues with the Nucleus in other forums, but could not find a clear answer as to where such options exist in the Nucleus.

So, I 'd suggest that you try with the optical cables disconnected, using only the analog connections of the Nucleus!
Unless, you can find a way to succesfully clock the Nucleus either as Clock Slave or Master!

Ok!
Wrong assumption, then!

Some more ideas..

How many ports have you set up in the ipMidi app? Do you have 6, as SSL recommends? Can you try and set it to 8?(I think this requires a computer restart)

Also, did you try connecting the Nucleus to a different usb port? Preferably, one that comes directly from the motherboard's chipset.
Also, open Device manager -> Usb controllers -> Usb Hubs -> Right click on the one the Nucleus is connected to -> Properties -> Power Management -> UnCheck the "Allow windows to turn off this device" Option.

I 'd, also, recommend to call your PC tech and ask him to disable C-states in the BIOS and any power saving options.

You can, also, download LatencyMon from here:

https://resplendence.com/latencymon

and run it on an idle system with the FF800 and Nucleus connected, and post the results here, so we can all check!

And last, but not least, what is the CPU usage when the stuttering happens?(you can check that by having the task manager open)

Hello!

No worries that you couldn't upload the image!
The info that you provided is enough! We know, now, that it's the firewire driver that creates the stuttering.

I checked your MoBo's Manual, so I can offer some guidance!

Firstly, you need to enter BIOS by hitting the "Delete" button, when your computer boots.
When you enter BIOS, switch from "EZ" Mode to "Advanced"
Then, navigate to "CPU Features" Tab
There you will see some options(I will write the ones you need to change)

- Global C-state Control   ->  Switch that to "Disabled"
- Relaxed EDC throttling  -> Switch that to "Disabled"
- AMD Cool’n’Quiet  -> Switch that to "Disabled"

Save the changes and Exit BIOS

After performing these tweaks, repeat your tests and post an update whether the problem is eliminated, or it persists, still !

EDIT: Also, what happened with the ipMidi change? Did that help?

I dont have those options. I pull up BIOS by hitting delete while powering up > F7 to toggle EZ and Advanced > Settings... nothing that says C-State or any of those things. I do, however, see an "ErP" powering saving option (which has been disabled since day 1) but thats it.

Thanks. I see where it is in the manual. Kinda buried in there a little under subcategories.

I found CPU Features. I disabled AMD Cool N Quiet and Global C State Control (they were in auto) but I don't see Relaxed EDC Throttling.

I didn't notice an expert setting. Just the switch to advanced and back when I hit F7

No change. Still stutters so bad that its unusable. I wish I at least knew the cause. If I knew it was TotalMix that's causing it, then I could start looking at different brands of AD/DA converters. If I knew it was a conflict with using firewire, I could start looking at USB-C, AVB, or Dante systems.

I just remembered that I still have my Alesis MasterControl. As soon as I get a chance, I'm gonna check these out and reinstall MasterControl software so I can hook it up and see whether it works or gives me the same problem. If it works, the problem is specific to Fireface or TotalMix. If it gives me the same issue, its a firewire problem.

If I can't fix it in a satisfactory setting, I'll either have to get an Apollo 8 firewire (if the issue is RME specific) [which wouldn't be so bad because I could make up for much of the cost by selling my Twin Duo] or start looking at the converters and features on USB-C, Dante, and AVB interfaces. Hopefully it doesn't come to that.

Aah, this was to be expexted! Everyone here has great confidence in the RME Drivers!
They 're the best!

So, something else is hogging your system. Most propably a PCI lane needed for the FireWire gets used by something. I can't figure out what, though! (maybe it's the Usb Port you have the Nucleus connected to)

Here's a quote from the manual:

So, it's not RME specific, as you 've found out.
Did the 'Bandwidth' settings help at all?

According to the second paragraph of the quote above, we need to find out what occupies the PCI lane of the chipset!
So, what other cards do you have installed in your system?
How many hard drives?
Any NVME drives?
How many USB Devices in total?

Ok! Thanks for the info!

There is a block diagram in Page 21 of your MoBo's manual, that we can see what is connected through the Chipset and what is connected directly to the CPU.

My guess is that the hogging happens on the Chipset, where both the Firewire and the Ethernet controller are connected to, and that is why the stuttering happens when you turn the Nucleus or the FF 800 on.

So, let's begin with something easy:

Disable the Onboard Soundcard
This can be done in:
BIOS -> Settings -> Advanced -> Integrated Peripherals -> "HD Audio Controller" set this option to 'Disabled'
Save the settings, and exit BIOS

Perform a test to check if that improved the situation(even if it doesn't, this will release some Bandwidth from the CPU)

------  The next steps will require some familiarity with PC Internals! If you 're not comfortable opening PCs, please ask a friend who knows, or your PC Tech --------

Now, if you use your HDD only for storage(as you wrote), turn off your PC, disconnect from power, and disconnect the SATA cable from it.
Re-boot.
Perform a test again, and check if the stuttering persists.

If it stops, check in which SATA Port the HDD was connected and:
    if it was connected in SATA1 or 2, re - connect it in any of the SATA 3 to SATA 6 Ports
    if it was connected to any of the SATA3 to 6 Ports, re - connect it to either SATA1 or SATA2 Port
Perform a test again!

If this doen't stop the stuttering, you 'll need to check where the M2 drive is connected to

Is it on the first slot? (M2_1 directly from the CPU)
or the second one? (M2_2 from the Chipset)

I wouldn't advise to change slots for this one, since you have your Operating System installed there.
What you could do, is get another SATA Drive(SSD or HDD), install Windows on this one, then remove the M2 Drive and then perform another test!

These kind of drives occupy full PCI-e x4 lanes, so it might not be possible to have an optimal system with everything connected as they are, now(Firewire, Nucleus and M2 drive)

Or, finally, if you don't want to get in all this hastle, you can get a new USB or PCI-e Audio Interface(some recommendations are in my post #15)

I should have a little time Sunday during the day or Monday night (when I'm back to working on the renovations I decided to do now while the system is acting up).
That actually sounds like a reasonable solution. Thank you. If it doesn't work, I'm probably gonna have to suck it up and run only 6 inputs @96kHz with the Fireface800 to the Apollo Twin via ADAT for a bit while I choose a USB or Ethernet interface.

Just a little update. I haven't been able to do anything with the studio because the little renovation project turned into a bigger one so the entire studio is unplugged and shoved in the center of the control room area right now. Hopefully I can get everything straightened up and running this week and get back on track.

I've been looking at what to switch to that has USB2/USB-C/or Ethernet to PC ports, 2+ sets of ADAT I/O (to connect the RME FF800 and run it as a standalone converter). I'm undecided whether to shell out for an interface that has everything I'll want down the road (plenty of high quality analog inputs, analog in from a snake in case I go that route on a lunchbox,  Dante or AVB maybe to build a network, professional quality AD/DA conversion, etc) or just buy a Digiface USB to run my FW800 in and out of @96kHz via dual ADAT and maybe another standalone converter down the road if I need it. Before I do that though, it seems like I should sell the 800 and add the money I'd spend on the Digiface and get the version with USB instead though.
Idk. Any recommendations are welcome.

worst gear ever, worst costumer service, the most expensive ever! huge regret buying the Fireface uc. lotta issues, the most frustrating situation I've ever had its with RME BS! 1million thumps down!

Yeah, almost ten years for me!

Firstly with an HDSP 9632, and for the last year and a half, a DigiFace USB! Rock solid performance and not a single glitch!!
So, another 2million from me!!

@uriel
There's no need to be negative, man!
And please, don't hijack a thread that has nothing to do with your situation. If you had read a bit, you would have found out that this "problem" has nothing to do with the FF800 itself.
Instead of accusing people and RME, why don't you open a new thread explaining your situation?

I absolutely can't complain about the interface itself.  The quality is amazing. I just didn't realize I'd have a problem using my ethernet control surface (SSL Nucleus) with a firewire interface when I decided firwire was a great way to save money on a feature I didn't think would matter.

Back on topic:
I'm temporarily sucking it up and connecting the the FF800 to the Apollo Twin via ADAT for 6 total inputs at 96kHz tomorrow. I'm not planning on staying that way very long but I check with Sony every week or 2 and last night I finally got directed to a distributor I can order my C800G through, so I'm about to drain the budget and have to wait to stack some more away for the rest of my upgrade plans for maybe a month or 2.

What Im looking for is plenty of inputs (no less than 8 line), 2 ADAT in & 2 ADAT out (to add FF800 for an additional 8 I/O when working at 96kHz or less), 2 preamps that are worth putting up against my VT737sp, 1073, SSL SuperAnalogue, etc. and would be a plus but otherwise I just need the inputs to be separate from the pres, USB2 would be cheapest and easy in to integrate into any system but USB-C seems like a better option. Something that connects to the PC via an Ethernet Gigabit network router along with the Nucleus and internet cable would be best for future-proofing as my system grows, especially since I use Nuendo (which itself allows you to combine computer via a network or even SPDIF, allowing me to dedicate one computer/command station for MIDI composing, another for mixing, another for recording or mastering, etc) so that seems like a way to lean in the future (rather than selling PCs and building 1 new one everytime I want a PC upgrade, adding new PCs and turning them into unique stations would be preferable due to RAM/SSD/CPU/port limitations per mobo).
Oh, yeah, and I'm on Nuendo so PT-HD capabilities isn't necessary, unless the unit is what Im looking either way.

It gets complicated because, while the answer is obviously to just shell out a few grand and get a Dante enabled Red, Burl, etc unit; there's always another legendary compressor/preamp/EQ/channel strip that money could go to that'll never lose value, never be obsolete, etc (like digital gear does), and RME already gave me 8 amazing AD/DA converter ports via ADAT so I'm not gaining a whole lot.

I'm probably rambling trying to be thorough but I should clarify, I will not likely need to record a hell of a lot of inputs simultaneously any time soon. I prefer to record band members individually for more control in post when I win the debate anyway ("you want bleed? Here. I'll make a group channel in post" lol)
I dont need anymore simultaneously recording than the 16 from an 8 in interface plus my FF800 would provide pretty much ever.
What I need the inputs for is so that I can just leave everything ready to go. Have all my I/O setup in Nuendo and named, if I have more I/O than I have preamps for I can use them to route out to hardware for analog processing, but otherwise all I/O stay setup so that I can just plug mic/line cables into the preamp or HiZ input I wanna use and create a track with that in selected.
Oh, for outs I'll need 2 for sub, and 5 for monitors. coax SPDIF I/O would be nice to so that I can run out to a monitor controller for talkback and headphones and the would be SPDIF in would be incase I choose to System Link in Nuendo through it later on. (I think I could actually use the SPDIF ports on the FF800 as long as the other has ADAT out x2, I assume but never tried.