You need to adjust your faders on the input and playback rows.
First, use Submix mode. It is much more intuitive that way.
Second, select (by clicking on) the hardware output your monitors or headphones are connected to. If I understand you correctly, you are trying to listen on the first headphone output. So in this case click on PH 9/10. Being in Submix mode, all the input and software playback faders now reflect what is being routed to your headphones.
Third, on the hardware input row make sure the fader on channel 9 is pulled all the way down. The meter will still show a signal as it reflects the incoming signal, not the post-fader signal.
Fourth, on the software playback row pull the fader on Analog 1 up until you hear the playback from Reaper.