No, the Dim setting is for the Dim feature, if you would use an ARC remote unit there is a DIM button.

The amp, is this a hifi amp or a guitar amp ?
How did you connect ?
In Totalmix on the output channel (the signal going to the amp) if you lower the fader is the hiss going away ?

The amp eventually has an optical input ? This would be great to test. Connect over optical cable only.

The outputs of your UFX II support three reference levels for studio: +19 dBu, +13 dBu, +4 dBu
See manual chapter 20.1 (analog outputs) and 39.1 Technical Specs - Analog.

Two Things to be performed to get consumer level (-10 dBV)

• Click in TM FX to the wrench symbol and select the reference level with the lowest volume: "+4 dBu".

• Additionally, you need to lower the volume by 12 dB (from 0 dB to -12 dB) at the output fader.
  Reason: the level difference between the +4 dBu studio level and the -10 dBV home level is Δ L = 11.78 dB (12 dB).

Formulas, see Sengpiel page: https://sengpielaudio.com/Rechner-db-volt.htm

Also look at this diagram:

Dim you can use later on demand to silence the signal by -20 dB, if needed (when talking or following a conversation).
The volume will be reduced by the amount set in the TM FX Settings (F3) (default -20 dB, adjustable from -65 to 0 dB).