To which ADAT port are your digital monitors connected to ?
Do you route audio to this HW output ?

Please post a screenshot of
- driver settings and
- the TM FX window with the proper HW output selected in submix mode.

Many thanks.

Is it correct to assume that you connected your digital monitors to the first ADAT port "ADAT1" OUT ?
Which ports did you assign to Main Out ?

If you switch an ADAT port to SPDIF protocol, then usually the 1st two channels are being used for SPDIF
(as ADAT is 8ch at single speed and SPDIF is 2ch from single up to quad speed), so for
- the physical ADAT port ADAT1 OUT the TM FX channels -> ADAT 1/2 OUT
- the physical ADAT port ADAT2 OUT the TM FX channels-> ADAT 9/10 OUT
- the physical ADAT port ADAT3 OUT the TM FX channels-> ADAT 17/18 OUT
- the physical ADAT port ADAT4 OUT the TM FX channels-> ADAT 25/26 OUT

So if your monitors are connected to the port ADAT1 OUT, then in TM FX use HW Output ADAT 1/2 for the routing.
1. In the TotalMix FX Control room on the right side click "Assign" button.
2. Assign "Main Out" to port ADAT 1/2
3. Change "Main Out's" fader to -35 dB, so that it won't become too loud
4. Ensure that TM FX is set to "Submix Mode" (blue area on the right), next steps to be performed in submix mode
5. Select the "Main Out" channel in the control room, keep it selected for the next steps to create a submix for your monitors
Play Music, you should see a signal in the middle row under SW playback channels, maybe SW Playback ADAT 1/2.
6. Raise the fader of this channel slowly otherwise it might become too loud, you should hear sound.

Save this routing so snapshot 1 on the right side of TM FX
Save Snapshot, Worplace and Create a Workspace Quick Select entry to save your routing.

Similar steps to be done for phones ...
Click Assign Button and assign the analog outputs for phones to "Phones 1".
Create a routing for Phones1 in submix mode.
Select Channel "Phones1" in TM FX Control room and create a routing like shown above and save it to another snapshot and save the states to disk.

Next think to take care of ... that you find the proper volume level for your digital monitors and phones output.
These you should save to your two snapshots for using Monitors and Phones.
Update your saved Snapshot and Workspace file and Workspace Quick Select settings.

You need to make sure that these volume settings are being used whenever you boot your computer.
In TM FX Preferences you should configure: [x] always init DSP devices with TM FX settings

Its also very practical to configure there: "Snapshots Do not load "----""
By this the saved / proper volume level will be restored whenever your recall a snapshot for routing audio to monitors or phones.

If the other channels disturb you, then you can hide them under TM FX -> Options -> Channel Layout.