-
-
Notifications
You must be signed in to change notification settings - Fork 32.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
High Memory Utilization Until 2023.3.0 #89885
Comments
Same here This seems to occur every few days, having to restart home assistant, and this last time, the whole container, as it was unresponsive. Seems a memory leak to me. Edit: I've duplicated both memory and disk size, but it had worked without it the last two years. |
In my case this could be (maybe, just a idea) a hacs plugin - but i dont know how to test mem consumption for a plugin ... This Would be then not a ha-issue |
I think that is not the case. I'm using the same plugins for a year and just now, with core-2023.3.0 and Python 3.11 support, it happened. |
Can you check if something is leaking new python objects? Call this service: Wait about 6 hours and check to see if new objects are still appearing in the log |
Can you please describe more detailed how to call this service? |
These do not grow but we do not need them in memory after startup. Noticed while investigating #89885
I have this leak also, please see here for more documentation including tests with HASSIO: henricm/ha-ferroamp#316 |
This is unlikely to get solved without the request data above |
I have installed and activated the profiler mentioned above, it will run for 12 hours and I will report back the result. |
I'd like to start the memory profiler, but documentation says, it's not supported on python 3.11 yet. |
The start_log_objects service does work on 3.11 |
The profiler is full of these lines: first lines: 2023-03-31 21:12:57.147 CRITICAL (SyncWorker_4) [homeassistant.components.profiler] Memory Growth: [('dict', 65538, 25)] Last lines: |
Looks like something is leaking anonymous dictionaries. You can use dump log objects to see what they are |
Sorry for a stupid question :-) What should I dump? |
I just did some comparison using python 3.11 and 3.10 using start_log_objects
Looks similar to me. |
It's good news that it's showing the leak. It's bad news that is not a named object because it's much harder to figure out where is coming from since it's an anonymous dict. In your case you want to dump 'dict' Unfortunately there are tens of thousands of them so it may not be able to dump them successfully The idea is to run a dump. Than run another dump and compare. Figure out which ones are new and than we might be able to figure out where they are coming from based on what's in them |
I figured that out when I tried to dump dict, and my HA came to a screeching halt when it was writing a huge amount of info into the log file. It wrote more than 150 MB in a couple of seconds. My second try resulted in a 98 MB log file, but comparing them was unfortunately a no go. |
I built a new service that will show the source / contents of the Once that merges that should hopefully give you a better shot at being able to get some useful data that doesn't need to be manually compared with 10000s of entries. |
@mickecamino If you install https://github.com/home-assistant/core/releases/tag/2023.4.0b5 via the beta channel and run the Call the |
I just got home, and I am running it now. Will post log in about an hour or so. |
One hour run:
|
I'm assuming you had 2023.3.x before. It would be quite ironic if the service I just built to track leaks can't find it because it was already fixed in 2023.4.x |
Ahhh, OK. |
I started Start logging growth of objects in memory and now it is showing Memory Growth on a couple of lines in the log. Memory Growth: [] ` |
Are you looking at the raw log or the UI? |
The latest was from the UI. I can zip the two log-files and upload it here if you want to look at them, EDIT: The log files are HUGE, more than 2GB |
My dropbox is [email protected] or gdrive is [email protected] |
I have shared the log files with you on Dropbox |
I've got the logs. Feel free to delete to free up space |
Whatever is creating these objects is causing the leak. Its some type of power sensor |
It looks like its coming from https://github.com/henricm/ha-ferroamp |
That is correct, I have an open issue there: henricm/ha-ferroamp#316 |
This isn't a bug in HA. Its a bug in the custom component. Its holding on to those |
All that data is coming from MQTT into ha-ferroamp. The original poster does not run ha-ferroamp, but is running MQTT. |
Its not the same problem (unless they are also using ha-ferroamp) |
OK, I have updated the issue at ha-ferroamp. |
I also have high memory utilization started from 2023.3, but no ha-ferroamp installed. New profile logger with source filled memory in few minutes. Some lines before crash:
|
That's unfortunate that you can't get it to run longer as its not enough data to know whats going on. It looks like something is leaking tuples but thats a bad guess without more data |
problem still exists, below some logs from another 10 minute run
|
Probably need it to run for another 10-15m to get it narrowed down some more |
ok, here is log from 1 hour run short after crash. Only way for longer profiler run is set "Maximum objects" to 1. How to read these logs? |
We disabled cleanup closed with python 3.11 and aiohttp a few days ago which should fix at least one leak #93013 (this only affects cpython 3.11.1, 3.11.2, 3.11.3) A better fix will come in cpython 3.11.4 via python/cpython#104485 If you haven't updated to 2023.5.3 yet, please do and if the leak persists you have another different leak and a new fresh log is needed. |
I have 2023.5.3 already, please check this log again. Leak for me is ~2000MB per day. |
New info about issue: 2023.6 limited memory leaks, but problem presists. Below log from ~1h profiler objects growth: @bdraco could you take a look? |
I've been using home-assistant core 2023.6.3 and I guess this memory issue was solved after Python 3.11.4 update on Fedora 38. It looks good for me now. Someone else have same thought? |
@jjk0giap You have a different leak. Something is leaking I'd start a new issue with a list of all your integrations |
Since most memory leaks are not the same its best to start a new issue for each leak instead of trying to do everything in a single issue as nothing will get solved as we will be chasing conflicting information and wasting everyone's time.
I'm going to close out this issue since there are at least 3 different leaks in here and the OP's leak was likely already fixed by a previous update. |
The problem
After upgrade to core-2023.3.0 I have seen a gradual memory utilization by Home Assistant without any visible explanation. I need to manual restart the application do free some memory to avoid problems.
core-2023.3.x sometimes increases up to 2 GiB memory utilization. Before update my setup don't use more than 0.5 GiB.
What version of Home Assistant Core has the issue?
core-2023.3.5
What was the last working version of Home Assistant Core?
core-2023.2.7
What type of installation are you running?
Home Assistant OS
Integration causing the issue
No response
Link to integration documentation on our website
No response
Diagnostics information
No response
Example YAML snippet
No response
Anything in the logs that might be useful for us?
No response
Additional information
No response
The text was updated successfully, but these errors were encountered: