Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix(nest): Resource leak due to lack of closing HTTP response bodies #1297

Merged
merged 1 commit into from
Aug 7, 2024

Conversation

cthach
Copy link
Contributor

@cthach cthach commented Aug 6, 2024

Problem?

Looks like we're not closing the response bodies of HTTP responses and they must always be closed per docs.

// The http Client and Transport guarantee that Body is always
// non-nil, even on responses without a body or responses with
// a zero-length body. It is the caller's responsibility to
// close Body. The default HTTP client's Transport may not
// reuse HTTP/1.x "keep-alive" TCP connections if the Body is
// not read to completion and closed.

The leak compounds due to the ExtendStream method being called repeatedly to extend the life of the WebRTC stream (I think this happens every 5-10 minutes). Add in multiple concurrent Nest cam streams and you may exhaust system resources (e.g., file descriptors).

I only discovered this issue because my Nest camera streams would go down in Frigate after a few hours. The only way to bring them back up was to restart Frigate and go2rtc.

Solution

Defer the closure of HTTP response bodies after they are read or if a block returns.

How was this tested?

Deployed by building a custom version of Frigate using a Dockerfile and ensuring no regressions were introduced.

# Compile go2rtc
CGO_ENABLED=0 go build -ldflags "-s -w" -trimpath

# Override go2rtc in Frigate image
cat << EOF > Dockerfile
FROM ghcr.io/blakeblackshear/frigate:0.14.0-rc2

COPY go2rtc /usr/local/go2rtc/bin
EOF

# Build Frigate image
docker build -t ghcr.io/blakeblackshear/frigate:0.14.0-rc2-fix-nest .

# Deploy image and ensure Nest streams comes up

@cthach cthach changed the title fix(nest): Fix resource leak due to lack of closing HTTP response bodies fix(nest): Resource leak due to lack of closing HTTP response bodies Aug 6, 2024
@cthach cthach marked this pull request as ready for review August 6, 2024 23:12
@felipecrs
Copy link
Contributor

Btw you can just mount /config/go2rtc binary into frigate container and It Will use It.

1 similar comment
@felipecrs

This comment was marked as duplicate.

@cthach
Copy link
Contributor Author

cthach commented Aug 6, 2024

Btw you can just mount /config/go2rtc binary into frigate container and It Will use It.

I didn't know that, thanks! It'll make testing easier in the future.

I have some additional changes I'm planning to propose soon in separate PRs.

@AlexxIT AlexxIT merged commit afc8f4f into AlexxIT:master Aug 7, 2024
@AlexxIT
Copy link
Owner

AlexxIT commented Aug 7, 2024

Thanks!

@AlexxIT AlexxIT added this to the v1.9.5 milestone Aug 7, 2024
@cthach cthach deleted the fix-nest-extend-stream branch August 7, 2024 14:02
@AlexxIT
Copy link
Owner

AlexxIT commented Oct 28, 2024

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants