diff options
| author | Pavel Begunkov <asml.silence@gmail.com> | 2025-03-24 15:32:33 +0000 |
|---|---|---|
| committer | Jens Axboe <axboe@kernel.dk> | 2025-03-25 12:37:47 -0600 |
| commit | 3afcb3b2e3a4ead24e3ab476576e87877d55ee22 (patch) | |
| tree | 9da9556fac11f6b9831634511ec7825b891b43d7 /io_uring | |
| parent | 3f0cb8de56b9a5c052a9e43fa548856926059810 (diff) | |
io_uring: defer iowq cqe overflow via task_work
Don't handle CQE overflows in io_req_complete_post() and defer it to
flush_completions. It cuts some duplication, and I also want to limit
the number of places directly overflowing completions.
Signed-off-by: Pavel Begunkov <asml.silence@gmail.com>
Link: https://lore.kernel.org/r/9046410ac27e18f2baa6f7cdb363ec921cbc3b79.1742829388.git.asml.silence@gmail.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Diffstat (limited to 'io_uring')
| -rw-r--r-- | io_uring/io_uring.c | 11 |
1 files changed, 7 insertions, 4 deletions
diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index e6c462948273..1fcfe62cecd9 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -892,6 +892,7 @@ bool io_req_post_cqe(struct io_kiocb *req, s32 res, u32 cflags) static void io_req_complete_post(struct io_kiocb *req, unsigned issue_flags) { struct io_ring_ctx *ctx = req->ctx; + bool completed = true; /* * All execution paths but io-wq use the deferred completions by @@ -905,18 +906,20 @@ static void io_req_complete_post(struct io_kiocb *req, unsigned issue_flags) * the submitter task context, IOPOLL protects with uring_lock. */ if (ctx->lockless_cq || (req->flags & REQ_F_REISSUE)) { +defer_complete: req->io_task_work.func = io_req_task_complete; io_req_task_work_add(req); return; } io_cq_lock(ctx); - if (!(req->flags & REQ_F_CQE_SKIP)) { - if (!io_fill_cqe_req(ctx, req)) - io_req_cqe_overflow(req); - } + if (!(req->flags & REQ_F_CQE_SKIP)) + completed = io_fill_cqe_req(ctx, req); io_cq_unlock_post(ctx); + if (!completed) + goto defer_complete; + /* * We don't free the request here because we know it's called from * io-wq only, which holds a reference, so it cannot be the last put. |