Generally speaking: do not interfere with gPTP priorities. It is not necessary to make a specific device gPTP grandmaster. If you unplug a grandmaster, another device becomes grandmaster. This does not interrupt streams (if devices have been connected for a couple of seconds). The reason is that gPTP syntonizes the clocks. It does not synchronize them. That means, the participating devices evaluate their internal clock drift w.r.t. the grandmaster to be able to provide accurate time (but are always using their own clock). Since the drift is pretty constant under normal conditions, the lack of a grandmaster can be compensated by simply continuing with the last known time and drift until a new grandmaster has been elected among the remaining devices.
Media clock is not directly derived from gPTP (in current AVB/MILAN). So you always have to clock one device to internal, and you do indeed need to stream CRF, AAF or AM824 to other devices on the network in order to synchronize them. Talker and listener both make use of the grandmaster's time, but neither one of them needs to be the grandmaster.
You can make the Digiface AVB media clock master and let the switch be the gPTP grandmaster and I don't see an issue. MOTU syncs to an incoming AM824, 12Mic and AVB Tool sync to CRF from Digiface AVB (make sure that either the first or last stream of the Digiface AVB is the CRF stream). This should work without issue, just changing sampling rates is a bit complicated at the moment because it requires that you disconnect the streams fully, then change the sampling rate of Digiface AVB and all listeners, and then re-connect them.
By the way, all devices use ATDECC (former AVDECC), not only the Digiface AVB.