> In the DAW and TotalMix the level coming through ADAT to the RME is so low, how can I change this?

Use a reference level with a higher sensitivity on the ADI-8.

> I noticed there isn't any controls for the ADI-8 in Totalmix itself.

TM FX Remote control options exist only for preamps with Auxcontrol (currently only Octamic XTC).
Remote control is possible for a few devices using MIDI remote or the newer MIDI remote AVB.
For the ADI-8 there is not remote control option only for ADI-8 QS.

> Reading the manual I saw something about the A/D reference control (on the input side),
> I pressed this a few times and it seemed to help but maybe not enough

Use the reference level with the highest sensitivity then, without overloading the analog input.
The level you see in TM FX.

> Plugging the DI directly to the RME doesn't have this issue however it looks like
> I still have to enable hi-z INST on the input and add some gain.

On the UFX you do not need the Radial DI box, there you can plug the guitar into any of the Mic inputs 9-12.
Plug in the normal TS plug of the guitar cable.
In TotalMix FX click to the wrench symbol and enable Hi-Z by clicking to "Inst" button.

> Sounds like the DI doesn't convert the signal to line level but only makes it balanced?
You also need to select the proper reference level on the UFX balanced inputs.
Did you enable PAD -15 dB on the Radial box by accident?
The Radial's box manual should tell you the output level.. select a proper reflevel on the inputs.
If you do not know anything about reflevels, choose the one where the signal is louder without overloading the inputs on UFX, same as with ADI-8.