1st of all I would recommend to watch these 2 videos to get a better understanding of Totalmix FX:
https://www.youtube.com/watch?v=sCH3qVaLL-I
https://www.youtube.com/watch?v=7O5i3s29Id4

The choice for recording software (DAW) is independent from Totalmix.
Its more a choice based on workflow, budget, required features.

The usual way to work is to connect as much instruments as possible to your recording interface.
An instrument with stereo outputs requires 2 analog channels of your recording interface.
An instrument with mono outputs requires 1 analog channel of your recording interface.
It could be that the analog input allows to switch between "consumer" and "professional" to get the right input level, i.e. to avoid distortion or avoid having to turn the input poti to max.

The goal is to have each instrument recorded on a track of its own, so that you have later in the mixing phase every required possibility for each track/instrument/vocal: work on the audio material (de-essing), add VSTs to each individual track, place them in the stereo panorama and also to give the mix depth. Some instrument you want to hear more in the foreground, some in the background. For this you can use different tricks: a) volume, b) using different Reverbs which have different pre-delay settings.

If you do not have enough inputs, then you could think about expanding number of inputs, by adding another preamp or AD converter via i.e. ADAT if your interface allows.

Or you need to optimize your workflow in terms of plugging/unplugging devices on demand and record stuff in several stages, not in parallel.