That’s because HUE is s bit quirky about it’s device settings.
There are some options on the restore cards you can test around with.
Like limit restored values to mode relevant and stuff like that.
Most of the special restore options are implemented especially for quirky things like hue.
The part with the brightness I am uncertain about. The problem with hue is that it turns the lights on when values are set. Not sure whether this is avoidable in your usecase.