Your fader is down on the AN 1/2 output, this is why the signal is low.

The fader is probably pulled down because this is your main analog outs for monitoring. If so, you should pick a different unused I/O pair to utilize the loopback. For instance, route the playback and effects to SPDIF out, then activate loopback and record SPDIF in.

Regarding your last question.

To what product do you refer when talking about a "dedicated DSP" ? Do you mean the DSP on UCX or is this a more generic question in regards to UAD alike products vs native plugins ?

As I understood it, but I might be wrong, the DSP on totalmix is dedicated for everything that you do with totalmix (headphone mixes, etc etc). Totalmix sends the audio signals from its inputs unprocessed to your DAW. So I think there is no direct relationship between the DSP effects of Totalmix/UCX to whats going on on the DAW.

But there are products like UAD, which have DSPs on board (i.e. on a PCIe card), where the audio processing in your DAW is offloaded from the CPU. These Effects you can load on your DAW to shape the sound.

Some people still like their UAD effects but many people go for native effects as CPU power is increasing nowadays and then you are independend from the additional hardware which is also quite expensive.

The effect section in Totalmix is not there to compete with high quality native VST effects or UAD. They are useable, give you an add on value for the product and enable you to use effects without much latency directly for headphone mixes etc.

Thats good for standalone mode and also good, as its easier to add a reverb to a singer, than having to set this up in the DAW. If you implement Reverb by using DAW, then the signal needs to travel through the Firewire/USB, processing of the DAW, travel back through the Firewire/USB ... so its easier and straight forward to have it on the device.

But for audio processing (mixing / mastering) in a DAW you better use the VSTs that come either with your DAW or separately bought DAWs. If you want you can use UAD products to offload effect processing from the CPU. Its a matter of taste if you like the UADs more or native effects.

In regards to plugin vs native I would recommend asking in a pure musician forum like gearsluts. Such questions pop up regulary. From following other forums I have the perception, that buying native plugins in good quality does not result in any quality degradation and gives you more flexibility. You only need to choose the proper one which you like ;-) The Lexicon (PCM!) Reverbs are still very nice and have the sound of the 80's. Exponential Audio is also worth a look.

Tip: wait for discounts otherwise I regard these products as too expensive and it needs some time until you have the toolchain that fits your personal demand and taste.

If you are worried about performance you are maybe better off in investing into a good equipped audio workstation with maybe up to 2 CPUs (when you decide to take a server mainboard). Then you have invested the money into performance that supports you with every aspect of "working with the computer". Especially when thinking about downmixes with many tracks and VSTs.

In regards to RMEs built in FX. What you expect .. zillions of high quality FX ? At what price ? For what purpose ?
By design these FXs cant be used in a DAW as a VST. So most use cases are i.e. headphone mixes.
If you expect complex effect algorithms and a quality like i.e. Lexicon, well .. everything has its price ..
Would you spend the $$$ more for the FX on an RME ?

Its surely better RME keeps the good quaility in their key competence areas and gives Reverbs as add-on, so that the price of the units do not get much higher, as FX / VSTs is often also a matter of taste, so even if RME would choose FX with more "quality", people would still buy additional VSTs to get different "flavours" of Reverb, Delay, etc ...

So I think its maybe better to make the product not too expensive by pimping up the FXs ...

Of course .. if RME sees a possibility to make effects a little better without requiring "$$$ bucks more" .. wonderful .. but quality has usually its price ...

I hope my $0.02 on this topic helped you a bit to come to an own view/decision.