Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

DBT fails to close many connections #1021

Closed
beckjake opened this issue Sep 20, 2018 · 3 comments
Closed

DBT fails to close many connections #1021

beckjake opened this issue Sep 20, 2018 · 3 comments

Comments

@beckjake
Copy link
Contributor

beckjake commented Sep 20, 2018

Issue

During tests, with warning output turned on, I see many ResourceWarnings about failing to close sockets.

Issue description

When running make test or any other way of running integration tests, I get a ton of warnings like this:
ResourceWarning: unclosed <socket.socket fd=3564, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('10.0.0.32', 2183), raddr=('172.217.1.42', 443)>

Those warnings are generated when a socket object is garbage collected and has not yet been close()d, which means we're leaking sockets. It's possible this is test-only, but that seems unlikely - tests are just the only place where we don't suppress warnings. Since the warning happens during GC, the log part about file, line number, etc are totally useless.

It's possible that #962 would fix this, since we'd potentially be more careful with pooling issues, but I want this to be an explicit issue.

I thought there was also a related user-submitted issue about us holding open redshift or postgres connections indefinitely as well, but I cannot find it.

Results

I expect no warnings!

System information

This is running on dev/guion-bluford but I'm pretty sure dbt has always done this.

The operating system you're running on:
Inside a docker container, or on appveyor, it doesn't matter.

The python version you're using (probably the output of python --version):
Python 3.6

Steps to reproduce

Run the tests :)

@drewbanin
Copy link
Contributor

@beckjake do you think this is still an issue?

@beckjake
Copy link
Contributor Author

I actually haven't seen these in a while, though I don't think we did anything to deserve that!

@beckjake
Copy link
Contributor Author

Edit: I should note, we still do see these at DEBUG level when bigquery runs end, because google's libraries don't close their sockets, but I don't think that's going to get fixed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants