linux-stable.git/fs/aio.c, branch linux-3.17.y

aio: fix uncorrent dirty pages accouting when truncating AIO ring buffer

2014-12-06T23:57:27+00:00

commit 835f252c6debd204fcd607c79975089b1ecd3472 upstream.

https://bugzilla.kernel.org/show_bug.cgi?id=86831

Markus reported that when shutting down mysqld (with AIO support,
on a ext3 formatted Harddrive) leads to a negative number of dirty pages
(underrun to the counter). The negative number results in a drastic reduction
of the write performance because the page cache is not used, because the kernel
thinks it is still 2 ^ 32 dirty pages open.

Add a warn trace in __dec_zone_state will catch this easily:

static inline void __dec_zone_state(struct zone *zone, enum
	zone_stat_item item)
{
     atomic_long_dec(&zone->vm_stat[item]);
+    WARN_ON_ONCE(item == NR_FILE_DIRTY &&
	atomic_long_read(&zone->vm_stat[item]) < 0);
     atomic_long_dec(&vm_stat[item]);
}

[   21.341632] ------------[ cut here ]------------
[   21.346294] WARNING: CPU: 0 PID: 309 at include/linux/vmstat.h:242
cancel_dirty_page+0x164/0x224()
[   21.355296] Modules linked in: wutbox_cp sata_mv
[   21.359968] CPU: 0 PID: 309 Comm: kworker/0:1 Not tainted 3.14.21-WuT #80
[   21.366793] Workqueue: events free_ioctx
[   21.370760] [] (unwind_backtrace) from []
(show_stack+0x20/0x24)
[   21.378562] [] (show_stack) from []
(dump_stack+0x24/0x28)
[   21.385840] [] (dump_stack) from []
(warn_slowpath_common+0x84/0x9c)
[   21.393976] [] (warn_slowpath_common) from []
(warn_slowpath_null+0x2c/0x34)
[   21.402800] [] (warn_slowpath_null) from []
(cancel_dirty_page+0x164/0x224)
[   21.411524] [] (cancel_dirty_page) from []
(truncate_inode_page+0x8c/0x158)
[   21.420272] [] (truncate_inode_page) from []
(truncate_inode_pages_range+0x11c/0x53c)
[   21.429890] [] (truncate_inode_pages_range) from
[] (truncate_pagecache+0x88/0xac)
[   21.439252] [] (truncate_pagecache) from []
(truncate_setsize+0x5c/0x74)
[   21.447731] [] (truncate_setsize) from []
(put_aio_ring_file.isra.14+0x34/0x90)
[   21.456826] [] (put_aio_ring_file.isra.14) from
[] (aio_free_ring+0x20/0xcc)
[   21.465660] [] (aio_free_ring) from []
(free_ioctx+0x24/0x44)
[   21.473190] [] (free_ioctx) from []
(process_one_work+0x134/0x47c)
[   21.481132] [] (process_one_work) from []
(worker_thread+0x130/0x414)
[   21.489350] [] (worker_thread) from []
(kthread+0xd4/0xec)
[   21.496621] [] (kthread) from []
(ret_from_fork+0x14/0x20)
[   21.503884] ---[ end trace 79c4bf42c038c9a1 ]---

The cause is that we set the aio ring file pages as *DIRTY* via SetPageDirty
(bypasses the VFS dirty pages increment) when init, and aio fs uses
*default_backing_dev_info* as the backing dev, which does not disable
the dirty pages accounting capability.
So truncating aio ring file will contribute to accounting dirty pages (VFS
dirty pages decrement), then error occurs.

The original goal is keeping these pages in memory (can not be reclaimed
or swapped) in life-time via marking it dirty. But thinking more, we have
already pinned pages via elevating the page's refcount, which can already
achieve the goal, so the SetPageDirty seems unnecessary.

In order to fix the issue, using the __set_page_dirty_no_writeback instead
of the nop .set_page_dirty, and dropped the SetPageDirty (don't manually
set the dirty flags, don't disable set_page_dirty(), rely on default behaviour).

With the above change, the dirty pages accounting can work well. But as we
known, aio fs is an anonymous one, which should never cause any real write-back,
we can ignore the dirty pages (write back) accounting by disabling the dirty
pages (write back) accounting capability. So we introduce an aio private
backing dev info (disabled the ACCT_DIRTY/WRITEBACK/ACCT_WB capabilities) to
replace the default one.

Reported-by: Markus Königshaus 
Signed-off-by: Gu Zheng 
Acked-by: Andrew Morton 
Signed-off-by: Benjamin LaHaise 
Signed-off-by: Greg Kroah-Hartman

aio: block exit_aio() until all context requests are completed

2014-09-04T20:54:47+00:00

It seems that exit_aio() also needs to wait for all iocbs to complete (like
io_destroy), but we missed the wait step in current implemention, so fix
it in the same way as we did in io_destroy.

Signed-off-by: Gu Zheng 
Signed-off-by: Benjamin LaHaise 
Cc: stable@vger.kernel.org

aio: add missing smp_rmb() in read_events_ring

2014-09-02T19:20:03+00:00

We ran into a case on ppc64 running mariadb where io_getevents would
return zeroed out I/O events.  After adding instrumentation, it became
clear that there was some missing synchronization between reading the
tail pointer and the events themselves.  This small patch fixes the
problem in testing.

Thanks to Zach for helping to look into this, and suggesting the fix.

Signed-off-by: Jeff Moyer 
Signed-off-by: Benjamin LaHaise 
Cc: stable@vger.kernel.org

aio: fix reqs_available handling

2014-08-24T22:47:27+00:00

As reported by Dan Aloni, commit f8567a3845ac ("aio: fix aio request
leak when events are reaped by userspace") introduces a regression when
user code attempts to perform io_submit() with more events than are
available in the ring buffer.  Reverting that commit would reintroduce a
regression when user space event reaping is used.

Fixing this bug is a bit more involved than the previous attempts to fix
this regression.  Since we do not have a single point at which we can
count events as being reaped by user space and io_getevents(), we have
to track event completion by looking at the number of events left in the
event ring.  So long as there are as many events in the ring buffer as
there have been completion events generate, we cannot call
put_reqs_available().  The code to check for this is now placed in
refill_reqs_available().

A test program from Dan and modified by me for verifying this bug is available
at http://www.kvack.org/~bcrl/20140824-aio_bug.c .

Reported-by: Dan Aloni 
Signed-off-by: Benjamin LaHaise 
Acked-by: Dan Aloni 
Cc: Kent Overstreet 
Cc: Mateusz Guzik 
Cc: Petr Matousek 
Cc: stable@vger.kernel.org      # v3.16 and anything that f8567a3845ac was backported to
Signed-off-by: Linus Torvalds

Merge git://git.kvack.org/~bcrl/aio-next

2014-08-16T14:56:27+00:00

Pull aio updates from Ben LaHaise.

* git://git.kvack.org/~bcrl/aio-next:
  aio: use iovec array rather than the single one
  aio: fix some comments
  aio: use the macro rather than the inline magic number
  aio: remove the needless registration of ring file's private_data
  aio: remove no longer needed preempt_disable()
  aio: kill the misleading rcu read locks in ioctx_add_table() and kill_ioctx()
  aio: change exit_aio() to load mm->ioctx_table once and avoid rcu_read_lock()

Merge branch 'for-3.17' of git://git.kernel.org/pub/scm/linux/kernel/git/tj/percpu

2014-08-04T17:09:27+00:00

Pull percpu updates from Tejun Heo:

 - Major reorganization of percpu header files which I think makes
   things a lot more readable and logical than before.

 - percpu-refcount is updated so that it requires explicit destruction
   and can be reinitialized if necessary.  This was pulled into the
   block tree to replace the custom percpu refcnting implemented in
   blk-mq.

 - In the process, percpu and percpu-refcount got cleaned up a bit

* 'for-3.17' of git://git.kernel.org/pub/scm/linux/kernel/git/tj/percpu: (21 commits)
  percpu-refcount: implement percpu_ref_reinit() and percpu_ref_is_zero()
  percpu-refcount: require percpu_ref to be exited explicitly
  percpu-refcount: use unsigned long for pcpu_count pointer
  percpu-refcount: add helpers for ->percpu_count accesses
  percpu-refcount: one bit is enough for REF_STATUS
  percpu-refcount, aio: use percpu_ref_cancel_init() in ioctx_alloc()
  workqueue: stronger test in process_one_work()
  workqueue: clear POOL_DISASSOCIATED in rebind_workers()
  percpu: Use ALIGN macro instead of hand coding alignment calculation
  percpu: invoke __verify_pcpu_ptr() from the generic part of accessors and operations
  percpu: preffity percpu header files
  percpu: use raw_cpu_*() to define __this_cpu_*()
  percpu: reorder macros in percpu header files
  percpu: move {raw|this}_cpu_*() definitions to include/linux/percpu-defs.h
  percpu: move generic {raw|this}_cpu_*_N() definitions to include/asm-generic/percpu.h
  percpu: only allow sized arch overrides for {raw|this}_cpu_*() ops
  percpu: reorganize include/linux/percpu-defs.h
  percpu: move accessors from include/linux/percpu.h to percpu-defs.h
  percpu: include/asm-generic/percpu.h should contain only arch-overridable parts
  percpu: introduce arch_raw_cpu_ptr()
  ...

aio: use iovec array rather than the single one

2014-07-24T14:59:40+00:00

Previously, we only offer a single iovec to handle all the read/write cases, so
the PREADV/PWRITEV request always need to alloc more iovec buffer when copying
user vectors.
If we use a tmp iovec array rather than the single one, some small PREADV/PWRITEV
workloads(vector size small than the tmp buffer) will not need to alloc more
iovec buffer when copying user vectors.

Reviewed-by: Jeff Moyer 
Signed-off-by: Gu Zheng 
Signed-off-by: Benjamin LaHaise

aio: fix some comments

2014-07-24T14:59:40+00:00

The function comments of aio_run_iocb and aio_read_events are out of date, so
fix them here.

Reviewed-by: Jeff Moyer 
Signed-off-by: Gu Zheng 
Signed-off-by: Benjamin LaHaise

aio: use the macro rather than the inline magic number

2014-07-24T14:59:40+00:00

Replace the inline magic number with the ready-made macro(AIO_RING_MAGIC),
just clean up.

Reviewed-by: Jeff Moyer 
Signed-off-by: Gu Zheng 
Signed-off-by: Benjamin LaHaise

aio: remove the needless registration of ring file's private_data

2014-07-24T14:59:40+00:00

Remove the registration of ring file's private_data, we do not use
it.

Reviewed-by: Jeff Moyer 
Signed-off-by: Gu Zheng 
Signed-off-by: Benjamin LaHaise