Most recent kernel where this bug did not occur: 2.6.22.11
Distribution: Debian Etch
Hardware Environment: Several x86 systems, Intel and AMD
Software Environment: Vanilla Debian Etch, self-compiled kernel from kernel.org
Problem Description: When writing sparse files with several applications onto nfs mounted filesystems, areas with NUL-bytes appear at locations where non-NUL-data should be. The problem occurs with multiple applications (depends on the file written, e.g. current firefox-bin triggers this error with both rsync and cpio - but at different places, maybe due to different write sizes) and only with 2.6.23 or 2.6.23.1 kernels - same setup with any 2.6.22.x kernel works fine, as do the same applications on non-Linux systems. Tested with different NFS servers, too (Linux, HP-UX, EMC) - all show the same symtoms. A local ext3 destination filesystem works fine.

Steps to reproduce:

# $dst is on a nfs-mounted, writable filesystem

wget ftp://mozilla.ussg.indiana.edu/pub/mozilla.org/firefox/releases/2.0.0.9/linux-i686/en-US/firefox-2.0.0.9.tar.gz
tar xzf firefox-2.0.0.9.tar.gz
cd firefox

# Using cpio:

echo firefox-bin | cpio -pdum --sparse $dst
md5sum firefox-bin $dst/firefox-bin
8cf961ebaaff03db222bc01d913ab7cc  firefox-bin
08d7baf9351c682309ac9181b1057716  .../firefox-bin

# The written file shows only NULs from from 0x9e7600 to 0x9e77ff where the source file has some non-NUL data

# Using rsync:

rsync -S firefox-bin $dst

8cf961ebaaff03db222bc01d913ab7cc  firefox-bin
cd0d218b489d32a7173faf4b0d901aa5  .../firefox-bin

# Here we find many blocks of bad NULs, the first from 0x000400 to 0x0007ff, from 0x004400 to 0x0047ff, from 0x007400 to 0x0077ff, from 0x00c400 to 0x00c7ff, from 0x010c00 to 0x010fff, from 0x019400 to 0x0197ff, and many more. 

# The problem can also be reproduced with rsync with the 2007-10-20T13:49:53 version of security.debian.org::debian-security/dists/etch/updates/main/binary-i386/Packages.gz