Let my try to steer you through the configuration step by step.
And thanx for your offer, this advice comes for free.
If we ever meet I’ll come back to it
First try one single change:
• Set: Setup / Options / SPDIF/Remap Keys / Optical Out: “ADAT”
Do you get a signal now?
If not let’s run all through.
• Remove any possibly connected USB-cable.
USB can block changing some mode settings.
• Remove all digital cables from ADI-2, except the 2 optical ones.
• Check the optical cables, each In goes to Out.
Do yo see the red light on both?
• Reset ADI-2 to factory default to have a clean starting point:
Push and hold VOL button (NOT Volume Dial) and Encoder 1 while powering up, until you see a confirmation screen “Reset done”.
EQ Presets and stored user setups are not deleted by this.
• Set: Setup / Options / Device Mode/DSD / Basic Mode: “AD/DA”.
(Manual page 38).
• Set: Setup / Options / Device Mode/DSD / Dig. Out Source: “Default” .(should be there already) (Manual page 38).
• Set: Setup / Clock / Clock Source: “SPDIF”.
• Set: Setup / Options / SPDIF/Remap Keys / SPDIF In: “Optical”.
• Set: Setup / Options / SPDIF/Remap Keys / SRC: “OFF”.
• Set: Setup / Options / SPDIF/Remap Keys / Optical Out: “ADAT”.
• Set: I/O / Main Output 1/2 / Settings / Source: “SPDIF” (to monitor signals from Ableton on your speakers).
• Set: I/O / Phones Out 3/4 / Settings / Source: “Analog” (to check if “Analog” really gets a signal, plug your headphones into PH3/4).
TOTALMIX:
Reset to factory default, or route channels 1 and 2 straight through, for both ins and outs.
Hit “X” to see the matrix, the 0 dB level boxes should look like a straight diagonal, every input is routed to it’s same numbered output.
Here’s a collection of video tutorials about it, the 1st one covers this:
https://www.rme-audio.de/de_totalmix-fx.html
Something like the picture on the bottom.
Now:
you should see the 1/2 input meters of Totalmix show your signal,
and you should be able to record the signal from input 1/2 in Ableton,
and route it to ADI-2 for monitoring.
I’m not familiar with Ableton, so ... no further advice how to configure and route signal in Ableton.