Can io_uring_prep_sendto() make use of registered buffers?

[jmillan]

uname -r: 6.1.0-13-amd64
liburing: 2.4

I'm successfully using liburing to send out UDP and TCP packets using io_uring_prep_sendto and io_uring_prep_write respectively.

I'm trying to use registered buffers in order to avoid address mapping between user and kernel space.

Following the docs, I create the iovecs:

struct iovec iov[QueueDepth];

Then I initialize and register them:

for (size_t i{ 0 }; i < DepLibUring::QueueDepth; ++i)
{
  this->iov[i].iov_base = malloc(1500);
  this->iov[i].iov_len = 1500;
}

err = io_uring_register_buffers(std::addressof(this->ring), this->iov, DepLibUring::QueueDepth);

Then, I use it to prepare a UDP send request:

auto store = this->iov[idx];

std::memcpy(store.iov_base, data, len);
store.iov_len = len;

io_uring_prep_sendto(sqe, sockfd, store.iov_base, store.iov_len, 0, addr, addrlen);
sqe->buf_index = idx;
sqe->opcode |= IORING_OP_WRITE_FIXED;

Upon submitting the requests, I get a send() error "Invalid argument".

If I remove the lines:

sqe->buf_index = idx;
sqe->opcode |= IORING_OP_WRITE_FIXED;

Then everything works, but obviously the address mapping happen for each request.

I wonder if using registered buffers is supported for io_uring_prep_sendto() at all or whether I'm doing something wrong.

Any comment is much appreciated.

[isilence]

The short answer, it's not supported. It can be added to normal sends, but that wouldn't make any performance difference. Is there a use case for that?

Note, *_OP_SENDZC supports fixed buffers because it does make a difference for it.

sqe->buf_index = idx;
sqe->opcode |= IORING_OP_WRITE_FIXED;

IORING_OP_WRITE_FIXED is not a flag but an opcode, so it's not how it works. For example, _OP_SENDZC has a bit different way for enabling fixed buffers, you can find it in liburing examples.

[jmillan]

The short answer, it's not supported.

👍

but that wouldn't make any performance difference. Is there a use case for that?

I was just going to compare the performance difference. I can't foresee why it wouldn't make any performance difference. Could you please elaborate a bit so I can understand why it wouldn't make a different here and yes with *_OP_SENDZC?

I think next I'll try *_OP_SENDZC.

Thanks

[isilence]

It's because early in the submission the net stack will copy all data into kernel/system memory, and that's just a memcpy() like you'd do it in userspace. Registered buffers may help to save on MMU traversals, but it shouldn't show up. Especially since reg buffers will add some slight overhead here and there on top.

The zerocopy variant doesn't do a copy, and so internally it has to resolve virtual addresses to physical ones and do refcounting. That all is optimised out if you use registered buffers, and there is some more trickery inside.

[jmillan]

Thanks for the explanation @isilence, I'll try *_OP_SENDZC in my server next.