linux.git/mm/page-writeback.c, branch v2.6.20

Fix balance_dirty_page() calculations with CONFIG_HIGHMEM

2007-01-30T00:37:38+00:00

This makes balance_dirty_page() always base its calculations on the
amount of non-highmem memory in the machine, rather than try to base it
on total memory and then falling back on non-highmem memory if the
mapping it was writing wasn't highmem capable.

This not only fixes a situation where two different writers can have
wildly different notions about what is a "balanced" dirty state, but it
also means that people with highmem machines don't run into an OOM
situation when regular memory fills up with dirty pages.

We used to try to handle the latter case by scaling down the dirty_ratio
if the machine had a lot of highmem pages in page_writeback_init(), but
it wasn't aggressive enough for some situations, and since basing the
dirty ratio on highmem memory was broken in the first place, let's just
stop doing so.

(A variation of this theme fixed Justin Piszcz's OOM problem when
copying an 18GB file on a RAID setup).

Acked-by: Nick Piggin 
Cc: Justin Piszcz 
Cc: Andrew Morton 
Cc: Neil Brown 
Cc: Ingo Molnar 
Cc: Randy Dunlap 
Cc: Christoph Lameter 
Cc: Jens Axboe 
Cc: Peter Zijlstra 
Cc: Adrian Bunk 
Signed-off-by: Linus Torvalds

VM: Fix nasty and subtle race in shared mmap'ed page writeback

2006-12-29T18:00:58+00:00

The VM layer (on the face of it, fairly reasonably) expected that when
it does a ->writepage() call to the filesystem, it would write out the
full page at that point in time.  Especially since it had earlier marked
the whole page dirty with "set_page_dirty()".

But that isn't actually the case: ->writepage() does not actually write
a page, it writes the parts of the page that have been explicitly marked
dirty before, *and* that had not got written out for other reasons since
the last time we told it they were dirty.

That last caveat is the important one.

Which _most_ of the time ends up being the whole page (since we had
called "set_page_dirty()" on the page earlier), but if the filesystem
had done any dirty flushing of its own (for example, to honor some
internal write ordering guarantees), it might end up doing only a
partial page IO (or none at all) when ->writepage() is actually called.

That is the correct thing in general (since we actually often _want_
only the known-dirty parts of the page to be written out), but the
shared dirty page handling had implicitly forgotten about these details,
and had a number of cases where it was doing just the "->writepage()"
part, without telling the low-level filesystem that the whole page might
have been re-dirtied as part of being mapped writably into user space.

Since most of the time the FS did actually write out the full page, we
didn't notice this for a loong time, and this needed some really odd
patterns to trigger.  But it caused occasional corruption with rtorrent
and with the Debian "apt" database, because both use shared mmaps to
update the end result.

This fixes it. Finally. After way too much hair-pulling.

Acked-by: Nick Piggin 
Acked-by: Martin J. Bligh 
Acked-by: Martin Michlmayr 
Acked-by: Martin Johansson 
Acked-by: Ingo Molnar 
Acked-by: Andrei Popa 
Cc: High Dickins 
Cc: Andrew Morton ,
Cc: Peter Zijlstra 
Cc: Segher Boessenkool 
Cc: David Miller 
Cc: Arjan van de Ven 
Cc: Gordon Farquharson 
Cc: Guillaume Chazarain 
Cc: Theodore Tso 
Cc: Kenneth Cheng 
Cc: Tobias Diedrich 
Signed-off-by: Linus Torvalds

VM: Remove "clear_page_dirty()" and "test_clear_page_dirty()" functions

2006-12-21T17:19:57+00:00

They were horribly easy to mis-use because of their tempting naming, and
they also did way more than any users of them generally wanted them to
do.

A dirty page can become clean under two circumstances:

 (a) when we write it out.  We have "clear_page_dirty_for_io()" for
     this, and that function remains unchanged.

     In the "for IO" case it is not sufficient to just clear the dirty
     bit, you also have to mark the page as being under writeback etc.

 (b) when we actually remove a page due to it becoming inaccessible to
     users, notably because it was truncate()'d away or the file (or
     metadata) no longer exists, and we thus want to cancel any
     outstanding dirty state.

For the (b) case, we now introduce "cancel_dirty_page()", which only
touches the page state itself, and verifies that the page is not mapped
(since cancelling writes on a mapped page would be actively wrong as it
is still accessible to users).

Some filesystems need to be fixed up for this: CIFS, FUSE, JFS,
ReiserFS, XFS all use the old confusing functions, and will be fixed
separately in subsequent commits (with some of them just removing the
offending logic, and others using clear_page_dirty_for_io()).

This was confirmed by Martin Michlmayr to fix the apt database
corruption on ARM.

Cc: Martin Michlmayr 
Cc: Peter Zijlstra 
Cc: Hugh Dickins 
Cc: Nick Piggin 
Cc: Arjan van de Ven 
Cc: Andrei Popa 
Cc: Andrew Morton 
Cc: Dave Kleikamp 
Cc: Gordon Farquharson 
Cc: Martin Schwidefsky 
Cc: Trond Myklebust 
Signed-off-by: Linus Torvalds

[PATCH] io-accounting: write accounting

2006-12-10T17:55:41+00:00

Accounting writes is fairly simple: whenever a process flips a page from clean
to dirty, we accuse it of having caused a write to underlying storage of
PAGE_CACHE_SIZE bytes.

This may overestimate the amount of writing: the page-dirtying may cause only
one buffer_head's worth of writeout.  Fixing that is possible, but probably a
bit messy and isn't obviously important.

Cc: Jay Lan 
Cc: Shailabh Nagar 
Cc: Balbir Singh 
Cc: Chris Sturtivant 
Cc: Tony Ernst 
Cc: Guillaume Thouvenin 
Cc: David Wright 
Signed-off-by: Andrew Morton 
Signed-off-by: Linus Torvalds

[PATCH] clean up __set_page_dirty_nobuffers()

2006-12-10T17:55:41+00:00

Save a tabstop in __set_page_dirty_nobuffers() and __set_page_dirty_buffers()
and a few other places.  No functional changes.

Cc: Jay Lan 
Cc: Shailabh Nagar 
Cc: Balbir Singh 
Cc: Chris Sturtivant 
Cc: Tony Ernst 
Cc: Guillaume Thouvenin 
Cc: David Wright 
Signed-off-by: Andrew Morton 
Signed-off-by: Linus Torvalds

[PATCH] separate bdi congestion functions from queue congestion functions

2006-10-20T17:26:35+00:00

Separate out the concept of "queue congestion" from "backing-dev congestion".
Congestion is a backing-dev concept, not a queue concept.

The blk_* congestion functions are retained, as wrappers around the core
backing-dev congestion functions.

This proper layering is needed so that NFS can cleanly use the congestion
functions, and so that CONFIG_BLOCK=n actually links.

Cc: "Thomas Maier" 
Cc: "Jens Axboe" 
Cc: Trond Myklebust 
Cc: David Howells 
Cc: Peter Osterlund 
Signed-off-by: Andrew Morton 
Signed-off-by: Linus Torvalds

fix file specification in comments

2006-10-03T21:01:26+00:00

Many files include the filename at the beginning, serveral used a wrong one.

Signed-off-by: Uwe Zeisberger 
Signed-off-by: Adrian Bunk

[PATCH] BLOCK: Make it possible to disable the block layer [try #6]

2006-09-30T18:52:31+00:00

Make it possible to disable the block layer.  Not all embedded devices require
it, some can make do with just JFFS2, NFS, ramfs, etc - none of which require
the block layer to be present.

This patch does the following:

 (*) Introduces CONFIG_BLOCK to disable the block layer, buffering and blockdev
     support.

 (*) Adds dependencies on CONFIG_BLOCK to any configuration item that controls
     an item that uses the block layer.  This includes:

     (*) Block I/O tracing.

     (*) Disk partition code.

     (*) All filesystems that are block based, eg: Ext3, ReiserFS, ISOFS.

     (*) The SCSI layer.  As far as I can tell, even SCSI chardevs use the
     	 block layer to do scheduling.  Some drivers that use SCSI facilities -
     	 such as USB storage - end up disabled indirectly from this.

     (*) Various block-based device drivers, such as IDE and the old CDROM
     	 drivers.

     (*) MTD blockdev handling and FTL.

     (*) JFFS - which uses set_bdev_super(), something it could avoid doing by
     	 taking a leaf out of JFFS2's book.

 (*) Makes most of the contents of linux/blkdev.h, linux/buffer_head.h and
     linux/elevator.h contingent on CONFIG_BLOCK being set.  sector_div() is,
     however, still used in places, and so is still available.

 (*) Also made contingent are the contents of linux/mpage.h, linux/genhd.h and
     parts of linux/fs.h.

 (*) Makes a number of files in fs/ contingent on CONFIG_BLOCK.

 (*) Makes mm/bounce.c (bounce buffering) contingent on CONFIG_BLOCK.

 (*) set_page_dirty() doesn't call __set_page_dirty_buffers() if CONFIG_BLOCK
     is not enabled.

 (*) fs/no-block.c is created to hold out-of-line stubs and things that are
     required when CONFIG_BLOCK is not set:

     (*) Default blockdev file operations (to give error ENODEV on opening).

 (*) Makes some /proc changes:

     (*) /proc/devices does not list any blockdevs.

     (*) /proc/diskstats and /proc/partitions are contingent on CONFIG_BLOCK.

 (*) Makes some compat ioctl handling contingent on CONFIG_BLOCK.

 (*) If CONFIG_BLOCK is not defined, makes sys_quotactl() return -ENODEV if
     given command other than Q_SYNC or if a special device is specified.

 (*) In init/do_mounts.c, no reference is made to the blockdev routines if
     CONFIG_BLOCK is not defined.  This does not prohibit NFS roots or JFFS2.

 (*) The bdflush, ioprio_set and ioprio_get syscalls can now be absent (return
     error ENOSYS by way of cond_syscall if so).

 (*) The seclvl_bd_claim() and seclvl_bd_release() security calls do nothing if
     CONFIG_BLOCK is not set, since they can't then happen.

Signed-Off-By: David Howells 
Signed-off-by: Jens Axboe

[PATCH] BLOCK: Dissociate generic_writepages() from mpage stuff [try #6]

2006-09-30T18:52:26+00:00

Dissociate the generic_writepages() function from the mpage stuff, moving its
declaration to linux/mm.h and actually emitting a full implementation into
mm/page-writeback.c.

The implementation is a partial duplicate of mpage_writepages() with all BIO
references removed.

It is used by NFS to do writeback.

Signed-Off-By: David Howells 
Signed-off-by: Jens Axboe

[PATCH] BLOCK: Move functions out of buffer code [try #6]

2006-09-30T18:31:19+00:00

Move some functions out of the buffering code that aren't strictly buffering
specific.  This is a precursor to being able to disable the block layer.

 (*) Moved some stuff out of fs/buffer.c:

     (*) The file sync and general sync stuff moved to fs/sync.c.

     (*) The superblock sync stuff moved to fs/super.c.

     (*) do_invalidatepage() moved to mm/truncate.c.

     (*) try_to_release_page() moved to mm/filemap.c.

 (*) Moved some related declarations between header files:

     (*) declarations for do_invalidatepage() and try_to_release_page() moved
     	 to linux/mm.h.

     (*) __set_page_dirty_buffers() moved to linux/buffer_head.h.

Signed-Off-By: David Howells 
Signed-off-by: Jens Axboe