Transferring 512 channels both ways on current operating systems is not a guaranteed task. Not to mention that many DAWs can't handle that amount of channels. The hardware resources in the FPGA to handle the data traffic exclude 768 channels as well.
TotalMix FX with 256 channels (rec or play) needs several tricks to work at all. The numbers of mixer knots / crosspoints and the possible combinations rises exponentially with channel count, and state changes (snapshot, workspace) need to be transferred in near real-time. There is a CPU and PCIe limit for that, 512 channels will clearly break it.
We have good reasons to limit our hardware the way it is.
Regarding MADI FX 390 channels: check the diagram again. Left is input, right is output, 256 channels go from TM FX to DAW and back, so this equals 512 channels (but for TM FX we refer to rec or play, not both summed).
Regards
Matthias Carstens
RME