From the way the OP is written it's impossible to tell if this question is actually about MIDI latency (as in the latency of the Babyface's MIDI I/O) or rather just about the latency experienced using software instruments which are controlled by MIDI, whether the MIDI is coming into the computer via the Babyface MIDI I/O or an external say a midi controller keyboard's MIDI-USB. I highly suspect this question is about the later, which is not an audio interface issue at all, nor a MIDI issue, or a Babyface issue.

mkok, my inkling is that the latency has nothing to do with any effects inline or the sample buffer inline, any of that, but just due to the software instruments themselves; I say this because even at the lowest sample buffers in Logic Pro X unless the separate low latency mode is engaged, I have the same problem as described in OP. I'd suggest looking for the equivalent in Ableton (and as mkok mentioned, lowering sample buffer as much as possible).