Hello Slavko.

Active monitors typically exhibit high loudness levels and, in most instances, lack a dedicated volume control. Occasionally, they may incorporate a DIP switch to reduce input sensitivity, thereby adjusting the final volume.

As a result, certain settings at the analogue output of the recording device and analogue input are necessary to prevent excessive loudness when the levels in TotalMix FX of "Main Out" are set to 0dB (= max digital level, perhaps "by accident").

Generally spoken: the ultimate loudness is contingent on the chosen output and input reference levels (on recording interface and active monitor.) the amplifier potency within the active monitor, and the speaker's efficiency.

The objective for your setup is to circumvent the need to set "Main Out" within TotalMix FX to levels such as -30 or -40 because too low levels in the digital domain might have an impact on sound quality. Additionally, if the level mismatches are too high, then you might get bad surprises, if you accidentally turn the volume to 0dB.

In your Yamaha monitors, you possess the capability to diminish the active monitor inputs by adjusting the level towards "minimum" (turning the knob leftward). Conversely, rotating it to the right would augment sensitivity, accommodating even weaker signals (such as consumer-level inputs) to attain appropriate levels.

Subsequently, I recommend configuring the output level on the BBF Pro to +4 dB. This is because an elevated reference level would correspondingly result in heightened output volume. In contrast, a lower reference level would yield reduced volume. Is this clear thus far?

These actions are all undertaken with the intention of averting any inadvertent auditory discomfort (or potential damage) arising from a 0dB setting in TotalMix FX.

Should the volume remain excessive even at a 0dB setting in TM FX, employing additional tools is necessary to rectify the "level mismatch" issue. This predicament is elaborated upon in a prominently featured post authored by MC within the RME forum. You can access the details here: https://forum.rme-audio.de/viewtopic.php?id=25399

For addressing this discrepancy, neutral-sounding attenuators are recommended. Furthermore, there are affordably priced, well-designed switchable attenuators available.
One such model that I suggest is the following: https://www.thomann.de/de/jts_ma_123.htm
You plug it best at the input of your active monitor.

On a related note, investing in pricier attenuators from Shure does not appear advantageous. This topic has already been deliberated within the forum, and RME (MC) confirmed the tonal neutrality of both options. Therefore, selecting the more costly variant doesn't yield any added benefits. Both products exhibit a solid build quality.

Still unclear?

OK, so you set on BBF Pro the analogue outputs (for "main out")  to a lower ref level of +4 dBu.
This makes it more silent.

But it might still be too loud when listening to music, if you move the output fader in TM FX towards 0dB.
If this is the case, the next step is to make the inputs of your active monitors less sensitive.

On your active monitor: I think you need to turn the knob to the left to make the inputs more insensitive so that the output volume decreases:

IF it is now still too loud if you move the output fader in TM FX to 0dB, then you still have a level mismatch and protect your ears and equipment by additionally using attenuators.

Otherwise, it might happen, that your volume settings change in TM FX of all sudden from -30 to 0 dB and this can become extremely loud in a case of bad luck, this we want to prevent.
We need to configure it in a way, that even 0 dB setting of the output fader in TM FX does not lead to volume levels that "nuke" your ears and/or equipment.