From Wikipedia: Latency refers to a short period of delay (usually measured in milliseconds) between when an audio signal enters and when it emerges from a system. Potential contributors to latency in an audio system include analog-to-digital conversion, buffering, digital signal processing, transmission time, digital-to-analog conversion and the speed of sound in air.

Rather than give you specific numbers as a hard and fast solution, I'd suggest that you do a little experimentation with your system - you'll find that the trade-off is between too short a setting resulting in clicks and pops in your audio when playing back recorded material, and too large a setting resulting in an audible delay between the live sound of your voice and the sound of your voice coming back through the headphones from the system.

You'll probably discover that you'll need to use two settings, switching back and forth between them depending on what you're doing. For recording, select the shortest latency you can without experiencing clicks and pops in the headphones. When you are mixing, especially if you are using processor intensive plug-ins or performing other operations, select the maximum latency value.