-
Notifications
You must be signed in to change notification settings - Fork 17.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
x/tools/gopls: reduce memory usage #30309
Comments
I second this. Whilst using gopls, I notice a steady increase of memory until it OOMs my system (which has 30GiB spare). I've been using gopls on the go compiler itself, and on the x/tools repository. I regularly have to kill gopls to prevent it from using too much RAM. |
We are currently working on a number of improvements to gopls, particularly caching, which should solve a lot of these issues. As of right now, gopls calls |
Change https://golang.org/cl/164779 mentions this issue: |
Change https://golang.org/cl/165438 mentions this issue: |
This change adds an additional cache for type information, which here is just a *packages.Package for each package. The metadata cache maintains the import graph, which allows us to easily determine when a package X (and therefore any other package that imports X) should be invalidated. Additionally, rather than performing content changes as they happen, we queue up content changes and apply them the next time that any type information is requested. Updates golang/go#30309 Change-Id: Iaf569f641f84ce69b0c0d5bdabbaa85635eeb8bf Reviewed-on: https://go-review.googlesource.com/c/tools/+/165438 Run-TryBot: Rebecca Stambler <[email protected]> TryBot-Result: Gobot Gobot <[email protected]> Reviewed-by: Ian Cottrell <[email protected]>
Closing for now, we can reopen as needed. |
What is the size of the projects that you were working on? |
@stamblerre My go.mod has around 70 dependencies. |
@stamblerre I have about 10 dependencies Screenshot from 2019-05-11 21-12-32 go.mod file
go 1.12
go.som file
cloud.google.com/go v0.34.0 h1:eOI3/cP2VTU6uZLDYAoic+eyzzB9YyGmJ7eIjl8rOPg=
cloud.google.com/go v0.34.0/go.mod h1:aQUYkXzVsufM+DwF1aE+0xfcU+56JwCaLick0ClmMTw=
github.com/Akumzy/ipc v0.0.0-20190428150754-76128748c1e5 h1:tiD/RAuIZlj2Dy8f5NUhNdBrH5M0F0xPPVVtU6Scug8=
github.com/Akumzy/ipc v0.0.0-20190428150754-76128748c1e5/go.mod h1:KtE0oJxRkmbN0MyDlw/wULkL0ey21TF2SpDOfJ+sJUk=
github.com/asdine/storm v2.1.2+incompatible h1:dczuIkyqwY2LrtXPz8ixMrU/OFgZp71kbKTHGrXYt/Q=
github.com/asdine/storm v2.1.2+incompatible/go.mod h1:RarYDc9hq1UPLImuiXK3BIWPJLdIygvV3PsInK0FbVQ=
github.com/davecgh/go-spew v1.1.0 h1:ZDRjVQ15GmhC3fiQ8ni8+OwkZQO4DARzQgrnXU1Liz8=
github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/emersion/go-imap v1.0.0-beta.4 h1:QglkDofK1RhU471SqcHxzRlSuPsCL6YpFc+NR5O6H6Q=
github.com/emersion/go-imap v1.0.0-beta.4/go.mod h1:mOPegfAgLVXbhRm1bh2JTX08z2Y3HYmKYpbrKDeAzsQ=
github.com/emersion/go-imap-idle v0.0.0-20180114101550-2af93776db6b h1:q4qkNe/W10qFGD3RWd4meQTkD0+Zrz0L4ekMvlptg60=
github.com/emersion/go-imap-idle v0.0.0-20180114101550-2af93776db6b/go.mod h1:o14zPKCmEH5WC1vU5SdPoZGgNvQx7zzKSnxPQlobo78=
github.com/emersion/go-message v0.9.1/go.mod h1:m3cK90skCWxm5sIMs1sXxly4Tn9Plvcf6eayHZJ1NzM=
github.com/emersion/go-message v0.9.2 h1:rJmtGZO1Z71PJDQXbC31EwzlJCsA/8kya6GnebSGp6I=
github.com/emersion/go-message v0.9.2/go.mod h1:m3cK90skCWxm5sIMs1sXxly4Tn9Plvcf6eayHZJ1NzM=
github.com/emersion/go-sasl v0.0.0-20161116183048-7e096a0a6197 h1:rDJPbyliyym8ZL/Wt71kdolp6yaD4fLIQz638E6JEt0=
github.com/emersion/go-sasl v0.0.0-20161116183048-7e096a0a6197/go.mod h1:G/dpzLu16WtQpBfQ/z3LYiYJn3ZhKSGWn83fyoyQe/k=
github.com/emersion/go-textwrapper v0.0.0-20160606182133-d0e65e56babe h1:40SWqY0zE3qCi6ZrtTf5OUdNm5lDnGnjRSq9GgmeTrg=
github.com/emersion/go-textwrapper v0.0.0-20160606182133-d0e65e56babe/go.mod h1:aqO8z8wPrjkscevZJFVE1wXJrLpC5LtJG7fqLOsPb2U=
github.com/golang/protobuf v1.2.0/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U=
github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
github.com/stretchr/testify v1.3.0 h1:TivCn/peBQ7UY8ooIcPgZFpTNSz0Q2U6UrFlUfqbe0Q=
github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UVUgZn+9EI=
go.etcd.io/bbolt v1.3.2 h1:Z/90sZLPOeCy2PwprqkFa25PdkusRzaj9P8zm/KNyvk=
go.etcd.io/bbolt v1.3.2/go.mod h1:IbVyRI1SCnLcuJnV2u8VeU0CEYM7e686BmAb1XKL+uU=
golang.org/x/net v0.0.0-20180724234803-3673e40ba225/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
golang.org/x/net v0.0.0-20190108225652-1e06a53dbb7e h1:bRhVy7zSSasaqNksaRZiA5EEI+Ei4I1nO5Jh72wfHlg=
golang.org/x/net v0.0.0-20190108225652-1e06a53dbb7e/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
golang.org/x/oauth2 v0.0.0-20190402181905-9f3314589c9a h1:tImsplftrFpALCYumobsd0K86vlAs/eXGFms2txfJfA=
golang.org/x/oauth2 v0.0.0-20190402181905-9f3314589c9a/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw=
golang.org/x/sync v0.0.0-20181221193216-37e7f081c4d4/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/text v0.3.0 h1:g61tztE5qeGQ89tm6NTjjM9VPIm088od1l6aSorWRWg=
golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
google.golang.org/appengine v1.4.0/go.mod h1:xpcJRLb0r/rnEns0DIKYYv+WjYCduHsrkT7/EB5XEv4=
|
I had a few projects with modules open with ~70 dependencies between them. |
I have around 10 dependencies and memory goes up to 1GB right away |
I have experienced similar issues for what it is worth, ubuntu works flawlessly running with <1gb ram but initially running in windows quickly ate all available ram rendering gopls inoperative. Recent updates have improved this significantly, now gopls will only use ~5gb with ~40 dependencies for a single project. |
We have a few pending changes that will hopefully improve the memory consumption - I will update this issue once these changes are submitted. |
Change https://golang.org/cl/178719 mentions this issue: |
This change trims the function bodies from the ASTs of files belonging to dependency packages. In these cases, we do not necessarily need full file ASTs, so it's not necessary to store the function bodies in memory. This change will reduce memory usage. However, it will also slow down the case of a user opening a file in a dependency package, as we will have to re-typecheck the file to get the full AST. Hopefully, this increase in latency will not be significant, as we will only need to re-typecheck a single package (all the dependencies should be cached). Updates golang/go#30309 Change-Id: I7871ae44499c851d1097087bd9d3567bb27db691 Reviewed-on: https://go-review.googlesource.com/c/tools/+/178719 Run-TryBot: Rebecca Stambler <[email protected]> Reviewed-by: Ian Cottrell <[email protected]>
The change above was the first of these planned improvements. It would be really helpful to get feedback on it. Did memory consumption decrease for anyone on this thread? |
I switch to Windows from Linux (Ubuntu) recently and I haven't notice any lag since then |
|
Yep, that's what I have -- when I tried it last week it wasn't having any effect. |
Hopefully this is on the right track:
I think this was related to
I'm going to try setting |
|
Thank you. I didn't notice any issues with memory usage after removing |
Unfortunately, it looks like I'm still running into memory usage issues. I think this was related to imports again -- a typo in accessing a struct field seemed to cause a spike in CPU and memory usage. I had to manually kill Hope this is still helpful:
|
Yes, those numbers do seem problematic. Are you working on a public project that I can try reproducing on or is this private code? Also, do you experience the same issues at master? You can get master by running |
Unfortunately I'm working on private code. I'll try and see if I can get the behavior to trigger in a public codebase, though. I'll also set up master now -- thank you. |
It also happens to me ... 10G +... It is easy to reproduce, git clone https://github.com/asticode/go-astilectron-demo, With vscode, remote SSH to open this folder, open message.go and main.go files and scroll to browse the code. In one minute the memory will increase to 10G (insider version), while with release version of vscode, the memory up to 7.6G. |
@ajeecai: What version of |
Hi @stamblerre Is this the latest version? I don't know how to update standalone gopls and what is its latest version. |
@ajeecai: v0.2.2 is the latest released version, though we will be releasing an update sometime this month. If you'd like to try out the current master, you can run: |
hi @stamblerre , with this command, the version of gopls looks downgrading But as I edit the code in vscode, the memory usage looks extremely high, worse. Do you have any change to try with the project I code mentioned, I think it is easy to reproduce. |
@ajeecai: I am not able to reproduce with the example you've given, but it's possible there are other factors at play here. Can you share you enable debug logging and share your |
I have switched gopls back to 0.2.2, when the memory up to 9G, I have followed the instructions and captured the log information in attachments. Please take a look. Thanks |
Thank you for sharing this - something does appear to be going very wrong here, as the memory usage really is high, and you'll notice that there are many cancellation requests in the log, indicating that something is probably hanging. I'm afraid I won't be able to make much more progress in investigation without memory profiles, so if you'd be willing to collect some that would be very helpful. This can be done by running I would also highly recommend sticking with master (the |
This issue is almost a year old, and a great deal has changed in If you encounter a specific issue with your project, please open a new issue so that we can begin discussion separately.@ajeecai: If you are able to produce profiles, please do create a new issue to share them, and we will continue investigating. |
What version of Go are you using (
go version
)?Does this issue reproduce with the latest release?
Yes
What operating system and processor architecture are you using (
go env
)?go env
OutputWhat did you do?
Install gopls from latest x/tools/cmd/gopls, open daemon.go with vim-lsp.
And send textDocument/completion request from vim-lsp.
What did you expect to see?
In my personal experience, I thought gopls take about 500-800MB memory usage.
What did you see instead?
gopls take 3.5 GB memory usage.
The text was updated successfully, but these errors were encountered: