std.http: add http server #15123

truemedian · 2023-03-30T15:52:49Z

extract http protocol into protocol.zig, as it is shared between client and server
coalesce Request and Response back into Client.zig, they don't contain any large chunks of code anymore
http.Server is implemented as basic as possible, a simple example below:

fn handler(res: *Server.Response) !void {
    while (true) {
        defer res.reset();

        try res.wait();
        res.headers.transfer_encoding = .{ .content_length = 14 };
        res.headers.connection = res.request.headers.connection;
        try res.do();
        _ = try res.write("Hello, World!\n");

        if (res.connection.closing) break;
    }
}

pub fn main() !void {
    var server = Server.init(std.heap.page_allocator, .{ .reuse_address = true });
    defer server.deinit();

    try server.listen(try net.Address.parseIp("127.0.0.1", 8080));

    while (true) {
        const res = try server.accept(.{ .dynamic = 8192 });

        const thread = try std.Thread.spawn(.{}, handler, .{res});
        thread.detach();
    }
}

Note: this server hasn't been tested very well, it works in the most simple of cases (like the above), but this is more a POC than a complete http server

tmm1 · 2023-03-30T18:38:38Z

cc #910

andrewrk · 2023-04-08T14:31:36Z

I believe when this PR was first opened there were some false positives in the CI. The failures now are true positives.

* extract http protocol into protocol.zig, as it is shared between client and server * coalesce Request and Response back into Client.zig, they don't contain any large chunks of code anymore * http.Server is implemented as basic as possible, a simple example below: ```zig fn handler(res: *Server.Response) !void { while (true) { defer res.reset(); try res.waitForCompleteHead(); res.headers.transfer_encoding = .{ .content_length = 14 }; res.headers.connection = res.request.headers.connection; try res.sendResponseHead(); _ = try res.write("Hello, World!\n"); if (res.connection.closing) break; } } pub fn main() !void { var server = Server.init(std.heap.page_allocator, .{ .reuse_address = true }); defer server.deinit(); try server.listen(try net.Address.parseIp("127.0.0.1", 8080)); while (true) { const res = try server.accept(.{ .dynamic = 8192 }); const thread = try std.Thread.spawn(.{}, handler, .{res}); thread.detach(); } } ```

…ssing through upgrades

…f read fix for 32bit arches curate error sets for api facing functions, expose raw errors in client.last_error fix bugged dependency loop, disable protocol tests (needs mocking) add separate mutex for bundle rescan

…ders end

truemedian · 2023-04-08T15:02:51Z

Sorry about that, missed the CI failure, was missing a single branch in findHeadersEnd that only broke with @Vector(32, u8), which zig build test-std wasn't hitting for me. I also just re-enabled the rest of the protocol tests.

andrewrk · 2023-04-09T14:45:09Z

Nice work ⚡

andrewrk · 2023-04-12T21:21:35Z

lib/std/http/Client.zig

+pub const ExtraError = union(enum) {
+    fn impliedErrorSet(comptime f: anytype) type {
+        const set = @typeInfo(@typeInfo(@TypeOf(f)).Fn.return_type.?).ErrorUnion.error_set;
+        if (@typeName(set)[0] != '@') @compileError(@typeName(f) ++ " doesn't have an implied error set any more.");
+        return set;
+    }
+
+    // There's apparently a dependency loop with using Client.DeflateDecompressor.
+    const FakeTransferError = proto.HeadersParser.ReadError || error{ReadFailed};
+    const FakeTransferReader = std.io.Reader(void, FakeTransferError, fakeRead);
+    fn fakeRead(ctx: void, buf: []u8) FakeTransferError!usize {
+        _ = .{ buf, ctx };
+        return 0;
+    }
+
+    const FakeDeflateDecompressor = std.compress.zlib.ZlibStream(FakeTransferReader);
+    const FakeGzipDecompressor = std.compress.gzip.Decompress(FakeTransferReader);
+    const FakeZstdDecompressor = std.compress.zstd.DecompressStream(FakeTransferReader, .{});
+
+    pub const TcpConnectError = std.net.TcpConnectToHostError;
+    pub const TlsError = std.crypto.tls.Client.InitError(net.Stream);
+    pub const WriteError = BufferedConnection.WriteError;
+    pub const ReadError = BufferedConnection.ReadError || error{HttpChunkInvalid};
+    pub const CaBundleError = impliedErrorSet(std.crypto.Certificate.Bundle.rescan);
+
+    pub const ZlibInitError = error{ BadHeader, InvalidCompression, InvalidWindowSize, Unsupported, EndOfStream, OutOfMemory } || Request.TransferReadError;
+    pub const GzipInitError = error{ BadHeader, InvalidCompression, OutOfMemory, WrongChecksum, EndOfStream, StreamTooLong } || Request.TransferReadError;
+    // pub const DecompressError = Client.DeflateDecompressor.Error || Client.GzipDecompressor.Error || Client.ZstdDecompressor.Error;
+    pub const DecompressError = FakeDeflateDecompressor.Error || FakeGzipDecompressor.Error || FakeZstdDecompressor.Error;
+
+    zlib_init: ZlibInitError, // error.CompressionInitializationFailed
+    gzip_init: GzipInitError, // error.CompressionInitializationFailed
+    connect: TcpConnectError, // error.ConnectionFailed
+    ca_bundle: CaBundleError, // error.CertificateAuthorityBundleFailed
+    tls: TlsError, // error.TlsInitializationFailed
+    write: WriteError, // error.WriteFailed
+    read: ReadError, // error.ReadFailed
+    decompress: DecompressError, // error.ReadFailed
+};


this is a bunch of meta garbage and should be deleted

The FakeDecompressors were required to work around a false positive dependency loop that made testing impossible, another part is working around the annoyance that is Certificate.Bundle.rescan not having a defined error set (and I could never get it defined such that it worked on CI, because it changes for every target), the rest of that is the full set of errors that come as part of the client.

Everything in the http client now returns a small set of curated errors and then dumps the actual error in the last_error field. Nearly all of the errors that the layers underneath http can return are likely useless to a user just trying to make a http request.

Moving the full set of errors into this union turns:

DoError: 134 errors ReadError: 38 errors ConnectError: 106 errors RequestError: 110 errors

into

DoError: 24 errors ( RequestError..., ReadError..., ShortHttpStatusLine, BadHttpVersion, HttpHeadersInvalid, HttpHeaderContinuationsUnsupported, HttpTransferEncodingUnsupported, HttpConnectionHeaderUnsupported, InvalidContentLength, CompressionNotSupported, Uri.ParseError..., TooManyHttpRedirects, HttpRedirectMissingLocation, CompressionInitializationFailed ) ReadError: 4 errors ( ReadFailed, HttpChunkInvalid, HttpHeadersExceededSizeLimit, OutOfMemory ) ConnectError: 3 errors ( ConnectionFailed, TlsInitializationFailed, OutOfMemory ) RequestError: 7 errors ( ...ConnectError, UnsupportedUrlScheme, UriMissingHost, CertificateAuthorityBundleFailed, WriteFailed )

DoError is longer than the rest because it's the function that handles the most.

nwtgck · 2023-04-14T14:02:00Z

Thank you so much for HTTP server implementation!

Are there res.waitForCompleteHead() and res.sendResponseHead()?
I fixed the example code in the first comment:

const std = @import("std");

pub fn main() !void {
    var server = std.http.Server.init(std.heap.page_allocator, .{ .reuse_address = true });
    defer server.deinit();

    try server.listen(try std.net.Address.parseIp("127.0.0.1", 8080));

    while (true) {
        const res = try server.accept(.{ .dynamic = 8192 });

        const thread = try std.Thread.spawn(.{}, handler, .{res});
        thread.detach();
    }
}

fn handler(res: *std.http.Server.Response) !void {
    defer res.reset();

    try res.wait(); // because res.waitForCompleteHead() not found
    std.debug.print("requested: {}\n", .{res.request});
    res.headers.transfer_encoding = .{ .content_length = 14 };
    res.headers.connection = res.request.headers.connection;
    try res.do(); // because res.sendResponseHead() not found
    _ = try res.write("Hello, World!\n");
}

$ curl localhost:8080
Hello, World!

M1 Mac
zig 0.11.0-dev.2582+25e3851fe

truemedian · 2023-04-14T14:57:01Z

Ah, sorry, yeah, that was a rather late addition to this PR. waitForCompleteHead was changed to wait and sendResponseHead to do to somewhat mirror the change in http.Client.

nwtgck · 2023-04-15T05:19:20Z

@truemedian Thank you so much for the HTTP server implementation and the first comment update! I'm trying the HTTP server now.

Ref: ziglang/zig#15123

andrewrk · 2023-04-27T21:44:22Z

lib/std/http/Server.zig

+            const left = buffer.len - out_index;
+
+            if (available > 0) {
+                const can_read = @truncate(u16, @min(available, left));


Why is there @truncate here? It looks like it should be @intCast.

The other @truncate in this file is suspicious as well.

This looks like remnants of some optimizations I was trying, will be changed @intCast

truemedian mentioned this pull request Mar 31, 2023

std.http.Client: api does not assert methods are called in the right order #15096

Open

truemedian force-pushed the http-server branch 4 times, most recently from 2f5c5fb to b1e0bf6 Compare April 5, 2023 20:50

andrewrk force-pushed the http-server branch from b1e0bf6 to a82558b Compare April 8, 2023 06:04

truemedian added 6 commits April 8, 2023 09:59

add buffering to connection instead of the http protocol, to allow pa…

aecbfa3

…ssing through upgrades

update package manager to use req.do(), fix chunked trailer reading

8250a92

std.http: add documentation

ef6d58e

std.http: reenable protocol read tests, add missing branch in findHea…

7f9a462

…ders end

truemedian force-pushed the http-server branch from 0726c0e to 7f9a462 Compare April 8, 2023 15:00

andrewrk merged commit 2ee3289 into ziglang:master Apr 9, 2023

andrewrk reviewed Apr 12, 2023

View reviewed changes

truemedian deleted the http-server branch April 13, 2023 03:15

kassane added a commit to kassane/Algorithms-Zig that referenced this pull request Apr 17, 2023

draft - http server

804a52f

Ref: ziglang/zig#15123

andrewrk reviewed Apr 27, 2023

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

std.http: add http server #15123

std.http: add http server #15123

truemedian commented Mar 30, 2023 •

edited

Loading

tmm1 commented Mar 30, 2023

andrewrk commented Apr 8, 2023

truemedian commented Apr 8, 2023 •

edited

Loading

andrewrk commented Apr 9, 2023

andrewrk Apr 12, 2023

truemedian Apr 12, 2023 •

edited

Loading

truemedian Apr 12, 2023

nwtgck commented Apr 14, 2023

truemedian commented Apr 14, 2023

nwtgck commented Apr 15, 2023

andrewrk Apr 27, 2023

truemedian Apr 27, 2023

std.http: add http server #15123

std.http: add http server #15123

Conversation

truemedian commented Mar 30, 2023 • edited Loading

tmm1 commented Mar 30, 2023

andrewrk commented Apr 8, 2023

truemedian commented Apr 8, 2023 • edited Loading

andrewrk commented Apr 9, 2023

andrewrk Apr 12, 2023

Choose a reason for hiding this comment

truemedian Apr 12, 2023 • edited Loading

Choose a reason for hiding this comment

truemedian Apr 12, 2023

Choose a reason for hiding this comment

nwtgck commented Apr 14, 2023

truemedian commented Apr 14, 2023

nwtgck commented Apr 15, 2023

andrewrk Apr 27, 2023

Choose a reason for hiding this comment

truemedian Apr 27, 2023

Choose a reason for hiding this comment

truemedian commented Mar 30, 2023 •

edited

Loading

truemedian commented Apr 8, 2023 •

edited

Loading

truemedian Apr 12, 2023 •

edited

Loading