So finally, after updating the babyface pro driver to 3.28A things have finally settled down. I'm on a M1pro MBP running Monterey. My conclusion is to stay away from Driverkit drivers from RME if you are on Monterey or lower and try it out if you are on Ventura...there were a couple of smaller updates within Monterey so I really don't know if RME fixed the problem or Apple... kudos to whoever finally did it...