I think your feedback is caused by NI Machine, that routes a kind of software monitoring to the same outs, that you use to Loop Back. So you have to deactivate Machine's monitoring or route it to other outs. If using the latter, be aware of possible phasing problems in TotalmixFX ControlRoom, because two identical audio-souces play the same time, one from Machine with a small latency.

To use any outs of your RME device with OSX, open "Audio-Mdi-Setup" app, in left window select your device and activate it for System Sound Out, in right window click the below button to configurate speakers that opens a new window where you can assign every combination of your choice. You are free to use only stereo or different surround settings. All physical outs of your device are supporter. After setting up, all OSX audio (also all audio-apps without individual audio routing) can be controlled in the Playback row of TotalmixFX. There you can setup your individual Control Room mixing – very powerful