I’m a developer of BleBox App, which is in the AppStore for quite a while. BleBoxes are small WiFi devices (switches, dimmers, gate openers, etc.). They’ve got their own http json API. Unfortunately there is no web hook capabilities, so to get current status I have to send requests to the devices. In most use cases you would need quite often updates, e.g. 1 per second. I’ve seen how Shelly App is made as those devices use similar method of communication and I’ve built mine similarly. In the beginning it seemed that everything worked fine. But the time passed and I’ve added much more devices to my setup. At the moment there are around 20 devices running at the same time, with polling interval of 1second. The problem which occurs is raising memory consumption. It’s very fast, from around 15MB up to around 80MB in just a day. Then the app freezes and you have to restart it to get it working again.
I use node-fetch for json requests, and I’ve tried a few approaches to pooling:
the same as in the shelly app - SetInterval -> memory rises the fastest
I’ve changed SetInterval into SetTimeout -> memory rises, but a bit slower
I’ve got back into SetInterval, but with usage of EventEmitter (as every Homey Devices implements this class). -> memory still rises, in the slowest way but still quite fast (attached is the screenshot from my insights).
The question is how to get rid of those memory rises, how to get garbage collector to clean up the mess. Or is there totally different way of making the pooling done in such a short intervals with so many devices?
What also is weird and you can see it on the screenshot is that the memory sometimes rises faster, and sometimes slower - I don’t see any reason it is not in the same way whole the time. The App was restarted yesterday 19:00 so you can see how fast did it consume memory until this moment.
Any help would be appreciated to make this app work better.
This has all the hallmarks of a memory leak. It might be in your code, in code that your app is using (like node-fetch) or in Homey’s core code. Try commenting out blocks of code from each of those parts to see if that stops the memory usage from growing.
But it didn’t help much
I’ll try commenting out other portions of code and I’ll look what is happening.
I thought that maybe any other Homey App developer already dealed with this problem so I would spare some time on investigations.
Yes, I’m trying to investigate it a bit further. It’s really weird, because after a while the memory has been freed in some part, but then it started to raise again. After 24h it was around 40MB…
BUT I restarted the App and then it started to behave totally different - it was stable at around 18MB for many hours (a bit up, a bit down -> memory allocated, memory freed). So I restarted it again to check what happens and then for a few hours it was stable (at around 22MB), then it started to raise, then it again became stable (at around 35MB)…
Totally weird. The same code behaves differently when restarted.
I also noticed that it behaves differently when installed on homey (homey app install) and differently when just run (homey app run) - in the latter the memory Is allocated and freed much faster (the “teeth” on the graph are much shorter).
Then I tried another scenario - I lowered number of devices using my app (I had 20, but I removed some from homey). And it helped - when the number of devices is up to around 10, the App behaves stable. So it looks like when there are more than around 10 intervals at the same moment, the app becomes unstable with the memory allocation. I will try also what happens when there are more devices (back to 20), but with lower interval (at the moment I used 1000ms, I’ll try more devices with e.g. 5000ms).
Those intervals are still not doing anything? Have you tried having 20 devices and no intervals at all? I don’t think that will show any issues, but who knows (perhaps there’s a limit to the number of devices Homey’s SDK can work with without getting unstable).
And have you tried the async loop I suggested? That would get rid of the intervals in favour of (shorter-lived) timeouts. I forgot to mention that await will only work in functions/methods that are marked async:
No, I haven’t tried your method yet, but I definately will do it. I just wanted to investigate this weird behavior further, because I became very curious why does it work like this, hoping to find an answer
I’ll let you know if that method worked for me. I also thought to try mix your method with EventEmitter (this.update() -> change to emit). I’ll check different scenarios to find optimal way of doing this.
Thanks for your support!
After investigations, changes, etc. it seems that the problem is with high amount of simultaneous short intervals with SetInterval. When I changed to your async/await method (mixed with EventEmitter) - the App became stable. I also replaced node-fetch with http-min, which gave me another memory optimizations (the App takes now around 2 times less memory than with node-fetch).
Thanks for all your help and suggestions. I’m happy to introduce v2 of my BleBox App.