No wifi is used in this setup(except for homey which does never show any problems with wifi).
Today I opened up the housing and cleaned it. It was pretty dirty inside (the wind pushes rain and mud up to the flap), hopefully this fixes things. It looks like the blocking pins worked easier after this.