hi,
at first make sure you're running TotalMix in SUBMIX mode (you can easily see that if you click any of the bottom line faders, others should be darker grey),
in this mode you can assign whole layout specificaly to the target fader on the bottom,
now, what you want to do is make one of the bottom faders Loopback, so choose some which you don't need and enable loopback,
if you clicked the fader, then whole TotalMix shows how faders are configured to that specific loopback I/O,
let's say you pick ADAT 1/2 for Ableton (software ASIO) input (=loopback), then you need to select (software ASIO) output in Traktor, which can be for ex. ADAT 3/4,
in TotalMix, you click ADAT 1/2 fader on the bottom row, enable loopback and in middle row of faders you drag up ADAT 3/4 (make sure everything else is down infinite),
bonus - in order to hear/monitor either Traktor output or Ableton input, you click on bottom line fader of your listening device (Main/Phones etc.) and in middle row you just drag up either ADAT 1/2 or ADAT 3/4 (dragging both should result in multiplied distorted volume though!)