Yep - The RME stuff will PASS any AC3/DTS surround streams from other applications (sounds like white noise if not decoded and pegs the meter right around 0dBfs), but you will not be able to feed 6 independent channels from your RME into your AVR over SPDIF with any succsess. The RME is not a hardware AC3/DTS sourround encoder.
Please note for AC3/DTS Passthrough to work, the Software Send (middle row) and SPDIF/AES Output (bottom row) in TotalMix need to be set to unity gain (0.0), and I would also make sure none of the analog inputs are also feeding the SPDIF/AES output in TM (the noise floor of the analog inputs alone can "break" the encoded AC3/DTS stream when mixed in).
Your software player (I use WMP, Media Center, or VLC) and Windows will likely need to be configured for this to work properly. I use an older Multiface-I into a NAD T175HD over SPDIF Toslink - AC3/DTS 5.1 decoding works like a charm! :-)
MADIface-XT+ARC / 3x HDSP MADI / ADI648
2x SSL Alphalink MADI AX
2x Multiface / 2x Digiface /2x ADI8