linux.git/include/linux/blk-mq.h, branch v5.8-rc2

blk-mq: drain I/O when all CPUs in a hctx are offline

2020-05-29T16:23:25+00:00

Most of blk-mq drivers depend on managed IRQ's auto-affinity to setup
up queue mapping. Thomas mentioned the following point[1]:

"That was the constraint of managed interrupts from the very beginning:

 The driver/subsystem has to quiesce the interrupt line and the associated
 queue _before_ it gets shutdown in CPU unplug and not fiddle with it
 until it's restarted by the core when the CPU is plugged in again."

However, current blk-mq implementation doesn't quiesce hw queue before
the last CPU in the hctx is shutdown.  Even worse, CPUHP_BLK_MQ_DEAD is a
cpuhp state handled after the CPU is down, so there isn't any chance to
quiesce the hctx before shutting down the CPU.

Add new CPUHP_AP_BLK_MQ_ONLINE state to stop allocating from blk-mq hctxs
where the last CPU goes away, and wait for completion of in-flight
requests.  This guarantees that there is no inflight I/O before shutting
down the managed IRQ.

Add a BLK_MQ_F_STACKING and set it for dm-rq and loop, so we don't need
to wait for completion of in-flight requests from these drivers to avoid
a potential dead-lock. It is safe to do this for stacking drivers as those
do not use interrupts at all and their I/O completions are triggered by
underlying devices I/O completion.

[1] https://lore.kernel.org/linux-block/alpine.DEB.2.21.1904051331270.1802@nanos.tec.linutronix.de/

[hch: different retry mechanism, merged two patches, minor cleanups]

Signed-off-by: Ming Lei 
Signed-off-by: Christoph Hellwig 
Reviewed-by: Hannes Reinecke 
Reviewed-by: Daniel Wagner 
Signed-off-by: Jens Axboe

blk-mq: blk-mq: provide forced completion method

2020-05-29T16:21:59+00:00

Drivers may need to bypass error injection for error recovery. Rename
__blk_mq_complete_request() to blk_mq_force_complete_rq() and export
that function so drivers may skip potential fake timeouts after they've
reclaimed lost requests.

Signed-off-by: Keith Busch 
Reviewed-by: Daniel Wagner 
Signed-off-by: Jens Axboe

block: bypass ->make_request_fn for blk-mq drivers

2020-04-25T15:45:44+00:00

Call blk_mq_make_request when no ->make_request_fn is set.  This is
safe now that blk_alloc_queue always sets up the pointer for make_request
based drivers.  This avoids an indirect call in the blk-mq driver I/O
fast path, which is rather expensive due to spectre mitigations.

Signed-off-by: Christoph Hellwig 
Signed-off-by: Jens Axboe

blk-mq: Add blk_mq_delay_run_hw_queues() API call

2020-04-20T16:34:56+00:00

We have:
* blk_mq_run_hw_queue()
* blk_mq_delay_run_hw_queue()
* blk_mq_run_hw_queues()

...but not blk_mq_delay_run_hw_queues(), presumably because nobody
needed it before now.  Since we need it for a later patch in this
series, add it.

Signed-off-by: Douglas Anderson 
Reviewed-by: Ming Lei 
Signed-off-by: Jens Axboe

blk-mq: Replace zero-length array with flexible-array member

2020-04-18T20:44:54+00:00

The current codebase makes use of the zero-length array language
extension to the C90 standard, but the preferred mechanism to declare
variable-length types such as these ones is a flexible array member[1][2],
introduced in C99:

struct foo {
        int stuff;
        struct boo array[];
};

By making use of the mechanism above, we will get a compiler warning
in case the flexible array does not occur last in the structure, which
will help us prevent some kind of undefined behavior bugs from being
inadvertently introduced[3] to the codebase from now on.

Also, notice that, dynamic memory allocations won't be affected by
this change:

"Flexible array members have incomplete type, and so the sizeof operator
may not be applied. As a quirk of the original implementation of
zero-length arrays, sizeof evaluates to zero."[1]

This issue was found with the help of Coccinelle.

[1] https://gcc.gnu.org/onlinedocs/gcc/Zero-Length.html
[2] https://github.com/KSPP/linux/issues/21
[3] commit 76497732932f ("cxgb3/l2t: Fix undefined behaviour")

Signed-off-by: Gustavo A. R. Silva

block: add a blk_mq_init_queue_data helper

2020-03-27T16:23:43+00:00

This allows a driver to pass a queuedata member before ->init_hctx is
called.  null_blk currently open codes this logic, but I'd rather have
it in the core to ease future maintainance.

Reviewed-by: Johannes Thumshirn 
Signed-off-by: Christoph Hellwig 
Signed-off-by: Jens Axboe

blk-mq: Fix a comment in include/linux/blk-mq.h

2020-03-10T13:09:59+00:00

The 'hctx_list' member of struct blk_mq_hw_ctx is not a list head but
instead an entry in q->unused_hctx_list. Fix the comment above this
struct member.

Fixes: d386732bc142 ("blk-mq: fill header with kernel-doc")
Signed-off-by: Bart Van Assche 
Reviewed-by: Chaitanya Kulkarni 
Cc: André Almeida 
Signed-off-by: Jens Axboe

blk-mq: Delete blk_mq_has_free_tags() and blk_mq_can_queue()

2019-11-13T19:50:38+00:00

These functions are not referenced, so delete them.

Signed-off-by: John Garry 
Signed-off-by: Jens Axboe

blk-mq: Make blk_mq_run_hw_queue() return void

2019-11-01T14:42:41+00:00

Since commit 97889f9ac24f ("blk-mq: remove synchronize_rcu() from
blk_mq_del_queue_tag_set()"), the return value of blk_mq_run_hw_queue()
is never checked, so make it return void, which very marginally simplifies
the code.

Reviewed-by: Bob Liu 
Signed-off-by: John Garry 
Signed-off-by: Jens Axboe

blk-mq: fill header with kernel-doc

2019-10-25T20:28:10+00:00

Insert documentation for structs, enums and functions at header file.
Format existing and new comments at struct blk_mq_ops as
kernel-doc comments.

Reviewed-by: Bart Van Assche 
Signed-off-by: André Almeida 
Signed-off-by: Jens Axboe