Hi bigtree! Sounds like you are stepping up your sweet system yet again!
I believe proper BNC will allow less theoretical jitter than optical, but RME's Steadyclock should negate any appreciable difference between the two. I'm using SC MADI Optical in my setup.
I'm not intimately familiar with MADI Ring configurations (my MADI setup is strictly 1:1), but I would have to assume you can use the full 64 Channel MADI capability. This should allow 8 ADI8-QS's in a MADI Ring. However, I believe the 2-3 sample MADI Interface delay would be additive in a MADI chain configuration - which could add as much as 16-18 samples of latency in each direction once you sync up all units (more than the QS's AD/DA converters impart themselves!).
If you are looking at the basic PCIe HDSPe MADI card without TotalMix FX (TM-FX), I might suggest using a second (or third?) HDSPe AES card to allow the additional I/O you are looking for. You won't have any of the MADI Chain latency issues, and all of your QS AD/DA's will align perfectly as far as I'm aware. The complexity of a multi-card PCIe system is a point of contention, but should behave well on a good MoBo with a proven BIOS.
However, as a recent MADIface-XT owner (and HUGE Fan-boy!), I can say TM-FX opened up a lot of potential in my mixerless setup, and I would not want to revert back to a Non TM-FX based RME setup (like the standard HDSPe MADI card). In your case, the PCIe MADI-FX sounds like the way to go.
With a MADI-FX, you can have three ADI8-QS's feeding the MADI-FX via MADI - and they will be sample-accurate. I believe the ADI8-QS is a snazzy unit, and should then allow you to route the MADI I/O to additional QS units over AES or ADAT (each MADI QS unit feeding a slave QS unit over AES/DB25 - Allowing for 48 I/O, even @ 192K!). These additional units will have the 2-3 sample delay (latency), but if you manage your channel assignments carefully, you can be mindful about where to assign tracks that require relative phase-coherency (Common mics on a single source, Parallel processing chains, etc).
Best of luck in your pending studio expansion plans! Sounds sweet.
MADIface-XT+ARC / 3x HDSP MADI / ADI648
2x SSL Alphalink MADI AX
2x Multiface / 2x Digiface /2x ADI8