Thanks for the quick reply, waedi!  About the switch: no, it is correct (this would have affected the volume on the iMac too, but switching back to the iMac the same Babyface works fine..).
DIM is off too

Of course!  Here they are:

Will do! in a few hours!  Something I forgot to mention here: the panning behaves in a weird way: when I pan it towards left or right it just gets even more silent but plays equally from both speakers.  But on Audio & Midi settings "stereo" is selected, and things look normal there too..

To me that sounds like wrong wired or bad plugged cables. So my question: what exact cables do you use on the main out, what plug type is on the other end? What input do your speakers have?
And yes, to avoid additional noise or hiss on the main out, close all hardware input faders for the main mix.

So you switched back and forth between both your Macs with having always good results on the iMac and always bad results on the Mini? How can that be? I mean, in TotalMix all looks fine. And you know the app shows what is going within the Babyface. So normally I would assume a hardware fault which cannot be displayed in TotalMix because it happens after the DA conversion (or even outside the Babyface when bad cables or such would cause the issue).
I remember a reported similar issue where at the end a not proper plugged TRS cable was the reason, for this I asked about the cable connections before.
Maybe it would be good to double check the speaker settings in Audio-Midi-Setup. By the way, have you tried different sample rates?

Do you power the interface over USB only? Or over a separate power supply? I can’t really imagine that, but to sort things out I would check if there is an issue with insufficient power delivery…
Is the connection to the speakers balanced or not? The XLR outputs on the Babyface pro are not working servo symmetrical. From the manual:
When connecting unbalanced equipment, make sure pin 3 of the XLR output is not connected. A connection to ground might cause a decreased THD (higher distortion) and increased power consumption. So now I can imagine to have slight issues on the analog part of the unit as soon the power is close to insufficient… I’m really not a specialist, so this are only my random thoughts. Possibly the power delivery on the iMac is slightly higher. I know that some electronic (audio) devices behave really weird when there’s something wrong with the power delivery. For example I have a small effect insert mixing unit which works absolutely well with the right power adapter but does the weirdest things with a wrong one or even without any power supply (I once thought it could work like a passive unit - nope).

My system tells the exact same numbers (M1 Mac mini 2020, Ventura 13.5, RME USB driver 4.08(2), Babyface pro firmware v205), regardless to phantom power on / off, mics connected or not and even independent of using a powered hub in between or not...
Have you seen my updated post above?

???…
Another idea: you save your Totalmix workspace on the iMac and copy it to the Mini and load it there to have 100% the same settings.
And also if it doesn’t seem logical, I would double check all cables and connections or even switch them.

Unbelievable!
As soon as I unplugged the display monitor (the screen) the volume cranks up to normal!!
It was connected with a USB-C-to-DP cable, and it is a DELL U2415 screen.
Notice also that the M2 Pro version has 4 thunderbolt ports (all of them behave the same way in this aspect).

At first I thought that the cable might be too greedy (it has DP 1.4 8K output).  But as soon as I plug a traditional HDMI cable the same thing happens (volume drops down).

So, for the time being, in order to have normal volume with the BabyFace, I must have no display.

Also: if I connect BOTH monitors the volume doesn't drop further down.  As soon as any monitor is connected, via any cable (HDMI or USB-C-to-DP), the volume just drops down A LOT and that's it.

I am still not sure as what is causing this.  I don't think it's a power issue (external power should have fixed this, right?).

Maybe some audio subsystem of the Mac Mini gets confused since HDMI and DP can also carry audio signals?  Both my screen monitors appear as audio devices on the Audio Settings, but of course they are not selected, the only one that is selected to output any actual audio is the BabyFace Pro FS...

Ouch !
I didn't know this, but I know Apple has lots of special audio settings in the background.
Maybe it will save your ears from suddenly getting loud music from the monitor speakers ?
One thing I can tell is Bluetooth headphone.
It has also a mic for telephone use
The mac recognize this device as headset and is loading a special setting.
Only one channel output to both sides left and right and sample rate 32kHz.
If you are working on a music programm you have to select the internal mic instead of the external mic as audio input device.
Then it works fine, outdoor song mixing no problem. This happened with Reaper.

When you plug in the monitor watch the changes going on in the Audio-Midi-Setup, it will show you.

Anybody can think of a way to test if this is a problem with my specific unit (Mac Mini M2 Pro) or if all of them have the same problem?  I cannot get my hands on a different Mac Mini M2 Pro around here to test this.
And I'm afraid that Apple won't consider its computer faulty since it works fine with other audio interfaces.
Could someone from RME confirm if the drivers of RME BabyFace Pro FS have been tested with a Mac Mini M2 Pro?

Good catch!  Unfortunately it seems unrelated: I just tested plugging the BabyFace on the USB-A slot, and used the HDMI for my screen, so that all of the thunderbolt 4's were left intact.

I am back to thinking that the problem is power-related, because merely *touching* the HDMI plug to the screen's metal without actually plugging it (so that Mac OS doesn't know about this screen yet) has the same effect.  (Makes me think of grounding problems.  Mac Mini has 2 pins (Mac Studio has 3 (I was between these two and opted for buying this Mac Mini)).

I recorded a short video on youtube showing this:

https://youtube.com/shorts/TDgXYChbkbk

Note that: the meters on the BabyFace and the meters on TotalMix do not get affected by this, they show the same levels whether in "low volume" mode or in "normal volume" mode (I managed to see this without a screen plugged by taking a screenshot; it worked).
s'sh

Wondering if this happens with a headphone at the computer without the Babyface connected.
And how about using an external power supply on the Babyface ?
What about the speakers ? Are these active monitors ?
Can you plug them into another power socket ?

One pin cheaper, I guess!

I added my mixing console (Mackie 1202-VLZ3, with 3 pins ) in the middle and the problem went away.  Still not clear on the whys and hows.  But we can conclude that it's not an RME drivers problem.  Maybe plugging the BabyFace to my mains didn't help because the BabyFace also has only 2 "pins".  I can finally start using my new setup here, although this first blow made me think that maybe I should have gone with the Mac Studio instead.  It's something you don't really see advertised in a pros/cons comparison between the two.  Mac Studio: 3 pins / Mac Mini: 2 pins.

Oh, and no wonder it was working fine on the iMac M1.  Nothing to do with M1 vs M2 Pro: the iMac has 3 pins.

Thanks everyone for giving really helpful insights here! 

Thanks for your input, gmarinov!  There is one part I do not understand with your suggestion: would this be consistent with the fact that introducing an extra equipment (my mixer), and using the exact same cables to connect my RME to the mixer solves the problem?  (The cables are XLR-to-TRS.)

Indeed.. and in the absence of separate signal and power ground lines in our home studios, solutions to ground loops can veer off the gospel above. Plus, there's no way to ground-lift a display monitor, that I'm aware of, and plenty of ways to introduce noise into audio path from power and data (!) rails of adjacent equipment.