Your expectation is simply wrong that this can be operated like a normal mixer.
The good news, you only need to understand a few basic principles, then everything is easy and logic.

1. Basic setup is 1st of all to use the Assign button, to assign e.g. the Main Output, in your case AN 3/4 OUT.
By this you move this output to the TM FX control room, so that control room functions like DIM, etc are useable for your main monitors.

2. Check that you are in submix mode (right side blue area).

3. Each HW output has a dedicated submix of its own.
To route audio to a HW output you simply have to click to a HW output in submix mode,
e.g. to Main Out (AN 3/4 in cour case).
You will see that the bottom of each fader in the top and middle row will now display "AN3/4".
Now move the faders of
- top row, HW inputs and
- middle row, SW playbacks
to create the individual submix for your Main Monitors.

4. Default is that audio through HW inputs is unaltered for recording even if you use EQ for a submix.

Thats basically all ... and this is documented in the manual, the very good RME tutorial videos and I also wrote a primer for 1st setup.

Additionally there are some shortcuts for routing in TM FX to provide a basic routing to your Main Out, which sends every SW playback channel (audio from PC) to your Main Output. You find these functions under TM FX -> Options -> Reset Mix.

Here some resources for you:

RME tutorial videos, quick to the point, doesnt waste any time, pure useful information for main use cases
https://www.tonstudio-forum.de/blog/ind … al-Videos/

My blog article for a 1st time setup with additional useful information, in DE and EN:
https://www.tonstudio-forum.de/blog/ind … rnal-equi/

And the manuals are an orb .. you should browse through it to see whats in there and the go deeper piece by piece.
If you watched the RME Tutorials upfront it will be easier to understand as you have already some overview / knowledge.

If you want to have a routing which is exactly like DAW mode, then you can use this function in TM FX:
- full reset of mixer and
- straight playback
then every sw playback channel (from PC/application/DAW) will be routed to its corresponding HW output.
Based on this routing you can refine the submixes for every HW output as you like.

There are a few other things, like to save routings to snapshorts and how to backup your setup through  save workspace.
Workspace quick select is also nice to use, as you can restore your routings and window position with key combinations like ALT-0, ALT-1 .. This allows you to quickly restore up to 10 settings. In total there are 30 Quick workspace select slots.

But for a real backup of your TM FX routings use Save Workspace As, then you can store the settings to a file on disk, where you know where it is and to make it part of your computers backup. By this its also possible to copy over the routing to a different PC.