net/http: ResponseController to manipulate per-request timeouts (and other behaviors)

[neild]

This proposal seeks to address #16100 (no way of manipulating timeouts in Handler), and is inspired by #16100 (comment).

HTTP handler timeouts are specified on a per-Server basis: ReadTimeout, WriteTimeout. It would be very useful to permit these timeouts (and possibly other options) to be overridden on a per-handler basis. For example, a handler which serves a long-running streaming response may want to extend the WriteTimeout after each write, and will likely want a different WriteTimeout than a handler serving a short response.

A problem is that we have no good place at the moment to add functions that adjust these timeouts. We might add methods to the ResponseWriter implementation and access them via type assertions (as is done with the existing Flush and Hijack methods), but this proliferation of undiscoverable magic methods scales poorly and does not interact well with middleware which wraps the ResponseWriter type.

Proposal: Add a new concrete ResponseController type which provides additional per-request controls. (A ResponseWriter writes the response, a ResponseController provides additional controls.) This type will supersede the existing Flusher and Hijacker APIs.

// A ResponseController is used by an HTTP handler to control the response.
//
// A ResponseController may not be used after the Handler.ServeHTTP method has returned.
type ResponseController struct{}

// NewResponseController creates a ResponseController for a request.
//
// The Request must be the original value passed to the Handler.ServeHTTP method.
// The ResponseWriter must be the original value passed to the Handler.ServeHTTP method,
// or have an Unwrap() method returning the original ResponseWriter.
func NewResponseController(rw ResponseWriter, req *Request) *ResponseController

// Flush flushes buffered data to the client.
func (rc *ResponseController) Flush() error

// Hijack lets the caller take over the connection.
// See the Hijacker interface for details.
func (rc *ResponseController) Hijack() (net.Conn, *bufio.ReadWriter, error)

We additionally add the ability to set the read and write deadline on a per-request basis, via ResponseController. These functions take a deadline rather than a timeout, for consistency with net.Conn.

// SetReadDeadline sets the deadline for reading the entire request, including the body.
// Reads from the request body after the deadline has been exceeded will return an error.
// A zero value means no deadline.
//
// The read deadline may not be extended after it has been exceeded.
func (rc *ResponseController) SetReadDeadline(deadline time.Time) error

// SetWriteDeadline sets the deadline for writing the response.
// Writes to the response body after the deadline has been exceeded will not block,
// but may succeed if the data has been buffered.
// A zero value means no deadline.
//
// The write deadline may not be extended after it has been exceeded.
func (rc *ResponseController) SetWriteDeadline(deadline time.Time) error

The Handler returned by http.TimeoutHandler currently receives a ResponseWriter which does not implement the Flush or Hijack methods. This will not change under this proposal: The *ResponseController for a TimeoutHandler will return a not-implemented error from Flush, Hijack, SetWriteDeadline, and SetReadDeadline.

[gopherbot]

Change https://go.dev/cl/420174 mentions this issue: net/http: work in progress per-request timeouts

[mborsz]

Thanks for the proposal.

IIUC this will unblock removing timeoutHandler in kubernetes completely which has been a source of many hard to debug races in the past (kubernetes/kubernetes#105884).

[rsc]

This proposal has been added to the active column of the proposals project
and will now be reviewed at the weekly proposal review meetings.
— rsc for the proposal review group

[frioux]

If people want to understand a use case for this, at work we typically set a higher timeout duration for writes than reads (via http method.) The idea being that writes should have a higher likelihood of succeeding, and a faster timeout for reads forces engineers to work harder to make more efficient code. We haven’t implemented this in our go code because we couldn’t figure out how, the we do in other languages.

[rsc]

Does anyone object to adding this API?

[szuecs]

@neild @rsc Just to clarify if I understand this correctly. This proposal would make the timeouts only possible per separate http.Handler, right?
I am trying to understand if this will be usable for a http proxy application that wants to set these timeouts per route, such that developers with different applications can change for their route these timeouts. Right now a single route has not a separate http.Handler in our case. As far as I understand we would need to create a http.Handler per route to isolate the timeout handling.

Routes in this case are not static and can be created, changed, deleted every ~3 seconds. The proxy serves a variety of applications next to each other.
If we wrap the generic http.Handler with a ResponseController and set SetReadDeadline/SetWriteDeadline, then the deadline would change for the full proxy instance as far as I understand or we would need to create http.Handler per route or per group of different timeouts.

I hope my question is understandable. :)

[neild]

This proposal is for a mechanism to permit a handler to adjust the read and write deadlines for a request after the handler has been called.

Sample usage would be something like:

http.HandleFunc("/foo", func(w http.ResponseWriter, r *http.Request) {
  ctl := http.NewResponseController(w, r)
  ctl.SetWriteDeadline(time.Now().Add(1 * time.Minute)) // sets deadline for this request only
  fmt.Fprintln(w, "Hello, world.") // this write has a 1-minute timeout
})

[szuecs]

@neild thanks! That was also my understanding, thanks for the clarification and example.

[FZambia]

Hello, also need this for streaming connections, a couple more questions to clarify:

1. This will transparently work with both http/1.1 and http/2 connections right?
2. What about http/3 servers based on https://github.com/lucas-clemente/quic-go?

[neild]

This will transparently work with both http/1.1 and http/2 connections right?

Yes.

What about http/3 servers based on https://github.com/lucas-clemente/quic-go?

As proposed, this only works with ResponseWriters provided by net/http, or ones which wrap one.

My first thought was that the initial version of this proposal doesn't need to support third-party ResponseWriters, but on consideration I think that perhaps we should: We need some mechanism for golang.org/x/net/http2 to provide a ResponseController for its ResponseWriter. Every way of doing this I can think of will be usable by third-party packages as well. So we might as well document that mechanism and support it, since someone is going to rely on it anyway.

[neild]

Changes to the proposal:

• Drop the *Request parameter to NewResponseController. We don't need it, and I haven't been able to think of anything that would require it in the future.
• If a third-party ResponseWriter passed to NewResponseController implements a Flush, Hijack, etc. method, then the ResponseController will call them.

// NewResponseController creates a ResponseController for a request.
//
// The ResponseWriter should be the original value passed to the Handler.ServeHTTP method,
// or have an Unwrap() method returning the original ResponseWriter.
//
// If the ResponseWriter implements any of the following methods, the ResponseController will
// call them as appropriate:
//
//    Flush()
//    Flush() error // alternative Flush returning an error
//    Hijack() (net.Conn, *bufio.ReadWriter, error)
//    SetReadDeadline(deadline time.Time)
//    SetWriteDeadline(deadline time.Time)
func NewResponseController(rw ResponseWriter) *ResponseController

[szuecs]

@neild I think this helps also my use case :)

[gopherbot]

Change https://go.dev/cl/436890 mentions this issue: net/http: alternate work in progress per-request timeouts

[neild]

While working through a draft implementation of this, I came across a somewhat different approach to integrating non-net/http implementations (including x/net/http2) with ResponseController. A demonstration is in https://go.dev/cl/436890.

In this approach, ResponseController is an interface:

type ResponseController interface {
        Hijacker
        Flush() error
        SetReadDeadline(deadline time.Time) error
        SetWriteDeadline(deadline time.Time) error
        responseController() // can only be implemented within net/http
}

A ResponseWriter may have a ResponseController method returning the response controller. NewResponseController is a type assertion to look for a ResponseController method and call it. If there is no ResponseController method, NewResponseController returns a ResponseController that returns an "unimplemented" error for every method.

func NewResponseController(rw ResponseWriter) ResponseController {
        if rc, ok := rw.(interface{ ResponseController() ResponseController }); ok {
                return rc.ResponseController()
        }
        return unimplementedResponseController{}
}

We want to be able to add new methods to ResponseController in the future without breaking existing implementations. To ensure we can do this, the ResponseController interface contains an unexported method. Implementations outside net/http (including the one in x/net) embed the result of NewResponseController(nil), which ensures that they return an appropriate error from any methods they don't implement.

The advantage to this approach is less indirection: There's no need to pass each ResponseController method call through an intermediate function (the approach I was taking in https://go.dev/cl/420174). When we add a new method to ResponseController, we don't need to add a new corresponding magic method to ResponseWriter; we just add it to unimplementedResponseController and existing implementations automatically acquire reasonable default behavior.

A disadvantage is that the pattern of embedding a default implementation of an interface with unexported methods is not commonly used in the standard library. However, this only affects people writing ResponseController implementations (a very specialized and uncommon task); users of the ResponseController API don't need to worry about this at all.

[rsc]

@neild, it sounds like there are two possible APIs here. Which one do you suggest? And does anyone else have any opinions on which one we use?

[neild]

After discussion with @bradfitz, we think the original proposal with the update in #54136 (comment) is the right approach. Making ResponseController an interface is an interesting idea, but the benefits (if any) are minor and the mechanisms used to make it work are non-obvious.

[rsc]

Does anyone object to the API in #54136 (comment) ?

[neild]

This will be in 1.20.

[kostix]

Is it possible to somehow make deadlines be set also for individual Read and Write operations on the net.Conn underlying the request/response session?

At my $dayjob, we use long-lived HTTP connections (with chunked encoding) to transfer data ("soft-realtime"). These connections only end when either the client loses interest in receiving the data or the upstream shuts down, so a deadline for the full request has no sense—as basically there is no "end".
What we want, instead, is to be able to discover a data stream be stalled for too long: there are keepalives sent in the stream, so basically if Read is stuck for, like, ×2 times the keepalive period, the client should close the request and retry. Write has the same property: to be stuck in this call for too long means we should trash this connection and have its client to retry.

To implement such policy, we have a custom type which wraps the net.Conn and calls SetDeadline before each Write and Read (and resets it afterwards). This is certainly a rather hacky way to achieve what we're after.
As I can see we're not alone with such requirements.

So, can there be thought a way to implement such finer-grained controls?

[neild]

The http2.Transport has a WriteByteTimeout setting which sets the maximum amount of time to wait on any given write. I could see extending that to the HTTP/2 server and the HTTP/1 transport and server. We'd want to think through how that interacts with HTTP/2 health checks.

You could also useResponseController.SetReadDeadline and ResponseController.SetWriteDeadline to limit the time for reading/writing a single chunk on a long-lived connection, resetting the deadline after every read/write.

[kostix]

The http2.Transport has a WriteByteTimeout setting which sets the maximum amount of time to wait on any given write. I could see extending that to the HTTP/2 server and the HTTP/1 transport and server. We'd want to think through how that interacts with HTTP/2 health checks.

Sorry, I forgot to state that we're currently using HTTP 1.1.
So yes, I would love to see such support for both stacks, if possible ❤️

[drakkan]

Is it possible to somehow make deadlines be set also for individual Read and Write operations on the net.Conn underlying the request/response session?

At my $dayjob, we use long-lived HTTP connections (with chunked encoding) to transfer data ("soft-realtime"). These connections only end when either the client loses interest in receiving the data or the upstream shuts down, so a deadline for the full request has no sense—as basically there is no "end". What we want, instead, is to be able to discover a data stream be stalled for too long: there are keepalives sent in the stream, so basically if Read is stuck for, like, ×2 times the keepalive period, the client should close the request and retry. Write has the same property: to be stuck in this call for too long means we should trash this connection and have its client to retry.

To implement such policy, we have a custom type which wraps the net.Conn and calls SetDeadline before each Write and Read (and resets it afterwards). This is certainly a rather hacky way to achieve what we're after. As I can see we're not alone with such requirements.

the purpose of the above code is to extend the deadlines if we read/write a certain amount of bytes in a given time. This is required to protect long lived connections from Slowloris style attacks. I haven't tested the ResponseController yet but I think we can implement the same logic using it in combination with reader/writer wrappers. I'm not sure it's appropriate to simplify this use case directly into stdlib, but that would be great

So, can there be thought a way to implement such finer-grained controls?

[igor-kupczynski]

This proposal won't affect the idle timeout between the requests (http.Transport.IdleConnTimeout), right? I have a usecase, where some connections need a different, longer idle timeout.

Even if I use the ResponseController, say something like that:

# .. in a handler ..

if err := rc.SetReadDeadline(time.Now().Add(longDeadline)); err != nil {
	logger.Errorf("Failed to set the timeout ...", ..)
}

This is then overwritten by the idleTimeout() property [src]:

..
		serverHandler{c.server}.ServeHTTP(w, w.req)       # <-- ResponseController sets a deadline here
..
		if d := c.server.idleTimeout(); d != 0 {
			c.rwc.SetReadDeadline(time.Now().Add(d))  # <-- deadline gets overwritten here
		} else {
			c.rwc.SetReadDeadline(time.Time{})
		}
..

Do I get this right? Are there any workarounds / patterns to have a different per-connection idle timeout between the requests?

[neild]

This proposal won't affect the idle timeout between the requests (http.Transport.IdleConnTimeout), right?

Correct.

net/http doesn't really have any facilities for treating a connection differently based on a request received on it. (Aside from hijacking the conn entirely and removing it from the net/http package's control.)

[gopherbot]

Change https://go.dev/cl/495295 mentions this issue: net/http/httptest: add support for http.ResponseController to ResponseRecorder