it was two fold problem:
I had to set a static IP address for the google mini devices and restart the chromecast app. This fixed most of my flows
I was also using Google Services Cloud TTS to send some longer text to the speaker, I had to fix it according to this link: