Since you already have them:
You could consider keeping devices on your Hue Bridges.
You just have a lot of devices and Homey/Zigbee might not be able to handle them very well.
In THIS app, multiple bridges are supported.
They create have their own Zigbee network, but the devices can be added to Homey as well, so you can control them from one place.
Just make sure you have enough routers in each Zigbee network to keep a good coverage throughout the house and check Zigbee channels to keep them from interfering with each other (and your wifi, which could be the start of your problem as well).
There’s a lot of info in the Community here that handles interfering Zigbee/WiFi. Just search here for those terms.
This might be a good start as well:
