Hi, i'm trying to export a lvm volume via ata-over-ethernet daemon and noticed memory corruption / device out of range access. here's the testcase: 1). setup an empty linear lvm volume (from 4 devices /dev/sd[a-d]4). # lvdisplay --- Logical volume --- LV Name /dev/vg_storage/lv_storage VG Name vg_storage LV UUID T8z8Ko-wdP6-1962-01Oe-FBA1-N2I6-1bFp25 LV Write Access read/write LV Status available # open 0 LV Size 1.58 TiB Current LE 414800 Segments 4 Allocation inherit Read ahead sectors auto - currently set to 256 Block device 253:0 2). run AoE daemon. here's a config for http://code.google.com/p/ggaoed # cat /etc/ggaoed.conf [defaults] direct-io = true pid-file = /var/run/ggaoed.pid control-socket = /var/run/ggaoed.sock state-directory = /var/lib/ggaoed [lvm] interfaces = eth0 path = /dev/vg_storage/lv_storage shelf = 92 slot = 0 3). on first console run daemon in debug mode with command 'ggaoed -d' and on the second console run 'modprobe aoe'. 0:25:90:28:93:{62,63} are the MACs of eht0/1 interfaces. disk/lvm: Resetting AoE configuration space disk/lvm: Resetting AoE MAC Mask list disk/lvm: Resetting AoE Reserve list disk/lvm: Shelf 92, slot 0, path '/dev/vg_storage/lv_storage' (size 1620 GiB, sectors 3398041600) opened, using direct I/O net/lo: Set up 4096 KiB ring buffer (96 RX/96 TX packets) net/lo: Listener started (MTU: 16384) net/eth0: Set up 4096 KiB ring buffer (1344 RX/1344 TX packets) net/eth0: Listener started (MTU: 1500) disk/lvm: ff:ff:ff:ff:ff:ff/00000000: Completed, status 0 disk/lvm: ff:ff:ff:ff:ff:ff/00000000: Response sent net/eth1: Set up 4096 KiB ring buffer (1344 RX/1344 TX packets) net/eth1: Listener started (MTU: 1500) disk/lvm: 0:16:e6:80:81:94/08315fdf: Received ATA cmd IDENTIFY, LBA28 0, sect 1 [ ] disk/lvm: 0:16:e6:80:81:94/08315fdf: Completed, status 0 disk/lvm: 0:16:e6:80:81:94/08315fdf: Response sent disk/lvm: 0:16:e6:80:81:92/082dea51: Received ATA cmd IDENTIFY, LBA28 0, sect 1 [ ] disk/lvm: 0:16:e6:80:81:92/082dea51: Completed, status 0 disk/lvm: 0:16:e6:80:81:92/082dea51: Response sent disk/lvm: 6c:f0:49:b:53:88/019ce673: Received ATA cmd IDENTIFY, LBA28 0, sect 1 [ ] disk/lvm: 6c:f0:49:b:53:88/019ce673: Completed, status 0 disk/lvm: 6c:f0:49:b:53:88/019ce673: Response sent disk/lvm: 6c:f0:49:b:52:b5/0835f73b: Received ATA cmd IDENTIFY, LBA28 0, sect 1 [ ] disk/lvm: 6c:f0:49:b:52:b5/0835f73b: Completed, status 0 disk/lvm: 6c:f0:49:b:52:b5/0835f73b: Response sent disk/lvm: 6c:f0:49:b:5a:83/08313096: Received ATA cmd IDENTIFY, LBA28 0, sect 1 [ ] disk/lvm: 6c:f0:49:b:5a:83/08313096: Completed, status 0 disk/lvm: 6c:f0:49:b:5a:83/08313096: Response sent disk/lvm: 6c:f0:49:b:5a:83/00000000: Received CFG cmd READ disk/lvm: 6c:f0:49:b:5a:83/00000000: Completed, status 0 disk/lvm: 6c:f0:49:b:5a:83/00000000: Response sent disk/lvm: 6c:f0:49:b:5a:83/08323140: Received ATA cmd IDENTIFY, LBA28 0, sect 1 [ ] disk/lvm: 6c:f0:49:b:5a:83/08323140: Completed, status 0 disk/lvm: 6c:f0:49:b:5a:83/08323140: Response sent disk/lvm: 0:16:e6:80:81:92/00000000: Received CFG cmd READ disk/lvm: 0:16:e6:80:81:92/00000000: Completed, status 0 disk/lvm: 0:16:e6:80:81:92/00000000: Response sent disk/lvm: 0:16:e6:80:81:92/082eec80: Received ATA cmd IDENTIFY, LBA28 0, sect 1 [ ] disk/lvm: 0:16:e6:80:81:92/082eec80: Completed, status 0 disk/lvm: 0:16:e6:80:81:92/082eec80: Response sent disk/lvm: 0:25:90:28:93:63/00000000: Received CFG cmd READ disk/lvm: 0:25:90:28:93:63/00000000: Completed, status 0 disk/lvm: 0:25:90:28:93:63/00000000: Response sent disk/lvm: 0:25:90:28:93:63/0001c4e6: Received ATA cmd IDENTIFY, LBA28 0, sect 1 [ ] disk/lvm: 0:25:90:28:93:63/0001c4e6: Completed, status 0 disk/lvm: 0:25:90:28:93:63/0001c4e6: Response sent disk/lvm: 0:25:90:28:93:63/0002c4e6: Received ATA cmd READ_EXT, LBA48 0, sect 2 [ ] disk/lvm: 0:25:90:28:93:63/0003c4e6: Received ATA cmd READ_EXT, LBA48 2, sect 2 [ ] disk/lvm: 0:25:90:28:93:63/0004c4e6: Received ATA cmd READ_EXT, LBA48 4, sect 2 [ ] disk/lvm: 0:25:90:28:93:63/0005c4e6: Received ATA cmd READ_EXT, LBA48 6, sect 2 [ ] disk/lvm: 0:25:90:28:93:63/0002c4e6: Completed, status 0 disk/lvm: 0:25:90:28:93:63/0002c4e6: Response sent disk/lvm: 0:25:90:28:93:63/0003c4e6: Completed, status 0 disk/lvm: 0:25:90:28:93:63/0003c4e6: Response sent disk/lvm: 0:25:90:28:93:63/0004c4e6: Completed, status 0 disk/lvm: 0:25:90:28:93:63/0004c4e6: Response sent disk/lvm: 0:25:90:28:93:63/0005c4e6: Completed, status 0 disk/lvm: 0:25:90:28:93:63/0005c4e6: Response sent disk/lvm: 0:25:90:28:93:63/0006c4e6: Received ATA cmd READ_EXT, LBA48 8, sect 2 [ ] disk/lvm: 0:25:90:28:93:63/0007c4e6: Received ATA cmd READ_EXT, LBA48 10, sect 2 [ ] disk/lvm: 0:25:90:28:93:63/0008c4e6: Received ATA cmd READ_EXT, LBA48 12, sect 2 [ ] disk/lvm: 0:25:90:28:93:63/0009c4e6: Received ATA cmd READ_EXT, LBA48 14, sect 2 [ ] disk/lvm: 0:25:90:28:93:63/0006c4e6: Completed, status 0 disk/lvm: 0:25:90:28:93:63/0006c4e6: Response sent disk/lvm: 0:25:90:28:93:63/0007c4e6: Completed, status 0 disk/lvm: 0:25:90:28:93:63/0007c4e6: Response sent disk/lvm: 0:25:90:28:93:63/0008c4e6: Completed, status 0 disk/lvm: 0:25:90:28:93:63/0008c4e6: Response sent disk/lvm: 0:25:90:28:93:63/0009c4e6: Completed, status 0 disk/lvm: 0:25:90:28:93:63/0009c4e6: Response sent disk/lvm: 0:25:90:28:93:63/000ac4e7: Received ATA cmd READ_EXT, LBA48 24, sect 2 [ ] disk/lvm: 0:25:90:28:93:63/000bc4e7: Received ATA cmd READ_EXT, LBA48 26, sect 2 [ ] disk/lvm: 0:25:90:28:93:63/000cc4e7: Received ATA cmd READ_EXT, LBA48 28, sect 2 [ ] disk/lvm: 0:25:90:28:93:63/000dc4e7: Received ATA cmd READ_EXT, LBA48 30, sect 2 [ ] disk/lvm: 0:25:90:28:93:63/000ac4e7: Completed, status 0 disk/lvm: 0:25:90:28:93:63/000ac4e7: Response sent disk/lvm: 0:25:90:28:93:63/000bc4e7: Completed, status 0 disk/lvm: 0:25:90:28:93:63/000bc4e7: Response sent disk/lvm: 0:25:90:28:93:63/000cc4e7: Completed, status 0 disk/lvm: 0:25:90:28:93:63/000cc4e7: Response sent disk/lvm: 0:25:90:28:93:63/000dc4e7: Completed, status 0 disk/lvm: 0:25:90:28:93:63/000dc4e7: Response sent disk/lvm: 0:25:90:28:93:63/00000000: Received CFG cmd READ disk/lvm: 0:25:90:28:93:63/00000000: Completed, status 0 disk/lvm: 0:25:90:28:93:63/00000000: Response sent disk/lvm: 0:25:90:28:93:63/000ec4e7: Received ATA cmd IDENTIFY, LBA28 0, sect 1 [ ] disk/lvm: 0:25:90:28:93:63/000ec4e7: Completed, status 0 disk/lvm: 0:25:90:28:93:63/000ec4e7: Response sent disk/lvm: 0:25:90:28:93:63/000fc4e7: Received ATA cmd READ_EXT, LBA48 18446744072812625792, sect 2 [ ] disk/lvm: Attempt to access beyond end-of-device disk/lvm: 0:25:90:28:93:63/000fc4e7: Completed, status 0 disk/lvm: 0:25:90:28:93:63/000fc4e7: Response sent [93024.868914] aoe: AoE v47 initialised. [93024.869178] aoe: 002590289362 e92.0 v0100 has 3398041600 sectors [93024.869247] ============================================================================= [93024.869251] BUG blkdev_queue: Poison overwritten [93024.869252] ----------------------------------------------------------------------------- [93024.869254] [93024.869256] INFO: 0xffff8807ff2a9070-0xffff8807ff2a9077. First byte 0xb8 instead of 0x6b [93024.869266] INFO: Allocated in blk_alloc_queue_node+0x1f/0x1a0 age=90599 cpu=4 pid=1070 [93024.869271] INFO: Freed in blk_release_queue+0x6a/0x80 age=85026 cpu=13 pid=1859 [93024.869274] INFO: Slab 0xffffea001bfd14c0 objects=17 used=1 fp=0xffff8807ff2a8e60 flags=0x6000000000040c1 [93024.869277] INFO: Object 0xffff8807ff2a8e60 @offset=3680 fp=0xffff8807ff2a8730 [93024.869278] [93024.869280] Bytes b4 0xffff8807ff2a8e50: de ed a6 01 01 00 00 00 5a 5a 5a 5a 5a 5a 5a 5a ���.....ZZZZZZZZ [93024.869289] Object 0xffff8807ff2a8e60: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk [93024.869298] Object 0xffff8807ff2a8e70: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk [93024.869305] Object 0xffff8807ff2a8e80: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk [93024.869313] Object 0xffff8807ff2a8e90: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk [93024.869320] Object 0xffff8807ff2a8ea0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk [93024.869328] Object 0xffff8807ff2a8eb0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk [93024.869335] Object 0xffff8807ff2a8ec0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk [93024.869343] Object 0xffff8807ff2a8ed0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk [93024.869350] Object 0xffff8807ff2a8ee0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk [93024.869357] Object 0xffff8807ff2a8ef0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk [93024.869365] Object 0xffff8807ff2a8f00: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk [93024.869372] Object 0xffff8807ff2a8f10: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk [93024.869379] Object 0xffff8807ff2a8f20: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk [93024.869387] Object 0xffff8807ff2a8f30: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk [93024.869395] Object 0xffff8807ff2a8f40: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk [93024.869403] Object 0xffff8807ff2a8f50: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk [93024.869410] Object 0xffff8807ff2a8f60: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk [93024.869417] Object 0xffff8807ff2a8f70: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk [93024.869425] Object 0xffff8807ff2a8f80: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk [93024.869432] Object 0xffff8807ff2a8f90: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk [93024.869440] Object 0xffff8807ff2a8fa0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk [93024.869447] Object 0xffff8807ff2a8fb0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk [93024.869455] Object 0xffff8807ff2a8fc0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk [93024.869462] Object 0xffff8807ff2a8fd0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk [93024.869470] Object 0xffff8807ff2a8fe0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk [93024.869477] Object 0xffff8807ff2a8ff0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk [93024.869485] Object 0xffff8807ff2a9000: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk [93024.869492] Object 0xffff8807ff2a9010: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk [93024.869501] Object 0xffff8807ff2a9020: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk [93024.869509] Object 0xffff8807ff2a9030: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk [93024.869517] Object 0xffff8807ff2a9040: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk [93024.869526] Object 0xffff8807ff2a9050: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk [93024.869535] Object 0xffff8807ff2a9060: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk [93024.869543] Object 0xffff8807ff2a9070: b8 01 8a fe 03 88 ff ff 6b 6b 6b 6b 6b 6b 6b 6b �..�..��kkkkkkkk [93024.869550] Object 0xffff8807ff2a9080: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk [93024.869558] Object 0xffff8807ff2a9090: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk [93024.869568] Object 0xffff8807ff2a90a0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk [93024.869576] Object 0xffff8807ff2a90b0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk [93024.869590] Object 0xffff8807ff2a90c0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk [93024.869598] Object 0xffff8807ff2a90d0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk [93024.869605] Object 0xffff8807ff2a90e0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk [93024.869613] Object 0xffff8807ff2a90f0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk [93024.869623] Object 0xffff8807ff2a9100: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk [93024.869630] Object 0xffff8807ff2a9110: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk [93024.869638] Object 0xffff8807ff2a9120: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk [93024.869645] Object 0xffff8807ff2a9130: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk [93024.869652] Object 0xffff8807ff2a9140: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk [93024.869662] Object 0xffff8807ff2a9150: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk [93024.869670] Object 0xffff8807ff2a9160: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk [93024.869677] Object 0xffff8807ff2a9170: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk [93024.869685] Object 0xffff8807ff2a9180: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk [93024.869692] Object 0xffff8807ff2a9190: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk [93024.869699] Object 0xffff8807ff2a91a0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk [93024.869707] Object 0xffff8807ff2a91b0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk [93024.869714] Object 0xffff8807ff2a91c0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk [93024.869722] Object 0xffff8807ff2a91d0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk [93024.869730] Object 0xffff8807ff2a91e0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk [93024.869738] Object 0xffff8807ff2a91f0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk [93024.869745] Object 0xffff8807ff2a9200: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk [93024.869752] Object 0xffff8807ff2a9210: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk [93024.869765] Object 0xffff8807ff2a9220: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk [93024.869772] Object 0xffff8807ff2a9230: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk [93024.869780] Object 0xffff8807ff2a9240: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk [93024.869787] Object 0xffff8807ff2a9250: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk [93024.869794] Object 0xffff8807ff2a9260: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk [93024.869801] Object 0xffff8807ff2a9270: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk [93024.869810] Object 0xffff8807ff2a9280: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk [93024.869819] Object 0xffff8807ff2a9290: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk [93024.869826] Object 0xffff8807ff2a92a0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk [93024.869834] Object 0xffff8807ff2a92b0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk [93024.869843] Object 0xffff8807ff2a92c0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk [93024.869851] Object 0xffff8807ff2a92d0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk [93024.869860] Object 0xffff8807ff2a92e0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk [93024.869867] Object 0xffff8807ff2a92f0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk [93024.869876] Object 0xffff8807ff2a9300: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk [93024.869885] Object 0xffff8807ff2a9310: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk [93024.869893] Object 0xffff8807ff2a9320: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk [93024.869900] Object 0xffff8807ff2a9330: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk [93024.869910] Object 0xffff8807ff2a9340: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk [93024.869918] Object 0xffff8807ff2a9350: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk [93024.869926] Object 0xffff8807ff2a9360: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk [93024.869934] Object 0xffff8807ff2a9370: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk [93024.869941] Object 0xffff8807ff2a9380: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk [93024.869948] Object 0xffff8807ff2a9390: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk [93024.869958] Object 0xffff8807ff2a93a0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk [93024.869965] Object 0xffff8807ff2a93b0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk [93024.869975] Object 0xffff8807ff2a93c0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk [93024.869982] Object 0xffff8807ff2a93d0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk [93024.869990] Object 0xffff8807ff2a93e0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk [93024.869997] Object 0xffff8807ff2a93f0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk [93024.870004] Object 0xffff8807ff2a9400: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk [93024.870012] Object 0xffff8807ff2a9410: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk [93024.870020] Object 0xffff8807ff2a9420: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk [93024.870028] Object 0xffff8807ff2a9430: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk [93024.870036] Object 0xffff8807ff2a9440: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk [93024.870043] Object 0xffff8807ff2a9450: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk [93024.870050] Object 0xffff8807ff2a9460: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk [93024.870058] Object 0xffff8807ff2a9470: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk [93024.870065] Object 0xffff8807ff2a9480: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk [93024.870072] Object 0xffff8807ff2a9490: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk [93024.870080] Object 0xffff8807ff2a94a0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk [93024.870087] Object 0xffff8807ff2a94b0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk [93024.870094] Object 0xffff8807ff2a94c0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk [93024.870105] Object 0xffff8807ff2a94d0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk [93024.870112] Object 0xffff8807ff2a94e0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk [93024.870120] Object 0xffff8807ff2a94f0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk [93024.870127] Object 0xffff8807ff2a9500: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk [93024.870134] Object 0xffff8807ff2a9510: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk [93024.870142] Object 0xffff8807ff2a9520: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk [93024.870149] Object 0xffff8807ff2a9530: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk [93024.870156] Object 0xffff8807ff2a9540: 6b 6b 6b 6b 6b 6b 6b a5 kkkkkkk� [93024.870163] Redzone 0xffff8807ff2a9548: bb bb bb bb bb bb bb bb �������� [93024.870170] Padding 0xffff8807ff2a9588: 5a 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZ [93024.870183] Pid: 1070, comm: kworker/4:1 Not tainted 2.6.37-rc8 #4 [93024.870186] Call Trace: [93024.870195] [<ffffffff8111ea1e>] print_trailer+0xfe/0x160 [93024.870198] [<ffffffff8111f074>] check_bytes_and_report+0xf4/0x130 [93024.870202] [<ffffffff8111f2da>] check_object+0x22a/0x270 [93024.870206] [<ffffffff811fd0df>] ? blk_alloc_queue_node+0x1f/0x1a0 [93024.870209] [<ffffffff811fd0df>] ? blk_alloc_queue_node+0x1f/0x1a0 [93024.870213] [<ffffffff81120380>] alloc_debug_processing+0x110/0x1f0 [93024.870217] [<ffffffff811211c9>] __slab_alloc+0x3a9/0x410 [93024.870222] [<ffffffff810de600>] ? mempool_alloc_slab+0x10/0x20 [93024.870225] [<ffffffff8112166f>] ? kmem_cache_alloc_node_notrace+0xbf/0xe0 [93024.870229] [<ffffffff810de7fe>] ? mempool_create_node+0x7e/0x1a0 [93024.870232] [<ffffffff811216fc>] kmem_cache_alloc_node+0x6c/0x140 [93024.870236] [<ffffffff810de87b>] ? mempool_create_node+0xfb/0x1a0 [93024.870239] [<ffffffff811fd0df>] ? blk_alloc_queue_node+0x1f/0x1a0 [93024.870246] [<ffffffffa050df20>] ? aoecmd_sleepwork+0x0/0xf0 [aoe] [93024.870250] [<ffffffff811fd0df>] blk_alloc_queue_node+0x1f/0x1a0 [93024.870254] [<ffffffffa050df20>] ? aoecmd_sleepwork+0x0/0xf0 [aoe] [93024.870257] [<ffffffff811fd26e>] blk_alloc_queue+0xe/0x10 [93024.870261] [<ffffffffa050c6fd>] aoeblk_gdalloc+0x6d/0x1f0 [aoe] [93024.870265] [<ffffffffa050df20>] ? aoecmd_sleepwork+0x0/0xf0 [aoe] [93024.870269] [<ffffffffa050dff8>] aoecmd_sleepwork+0xd8/0xf0 [aoe] [93024.870272] [<ffffffffa050df20>] ? aoecmd_sleepwork+0x0/0xf0 [aoe] [93024.870277] [<ffffffff81062048>] process_one_work+0x118/0x400 [93024.870281] [<ffffffff81063b73>] worker_thread+0x163/0x330 [93024.870286] [<ffffffff81035f39>] ? __wake_up_common+0x59/0x90 [93024.870290] [<ffffffff81063a10>] ? worker_thread+0x0/0x330 [93024.870295] [<ffffffff81068276>] kthread+0x96/0xa0 [93024.870299] [<ffffffff81003b54>] kernel_thread_helper+0x4/0x10 [93024.870303] [<ffffffff810681e0>] ? kthread+0x0/0xa0 [93024.870306] [<ffffffff81003b50>] ? kernel_thread_helper+0x0/0x10 [93024.870309] FIX blkdev_queue: Restoring 0xffff8807ff2a9070-0xffff8807ff2a9077=0x6b
btw, afaics the aoe driver in kernel tree looks unmaintained while the one from the http://support.coraid.com/support/linux/ is in active development.
Has any action taken place to resolve this issue? (Is it still a problem in 2.6.38.y / 2.6.39-rc*?)
Ed, are you aware of this issue?
Created attachment 62762 [details] slowpath warning triggered on 2.6.39.1 by hdparm -t /dev/etherd/eX.Y
Created attachment 62772 [details] task-block triggered after hdparm benchmarking.
update: i don't see the 'poison overwritten' on the 2.6.39.1 with ~1.6TB exported volume but the 'attempt to access beyond end-of-device' is present. # ggaoed -d disk/testdev: Shelf 92, slot 0, path '/dev/vg_test/lv_test' (size 1620 GiB, sectors 3398041600) opened R/O, using direct I/O net/lo: Set up 4096 KiB ring buffer (96 RX/96 TX packets) net/lo: Listener started (MTU: 16384) disk/testdev: ff:ff:ff:ff:ff:ff/00000000: Completed, status 0 disk/testdev: ff:ff:ff:ff:ff:ff/00000000: Response sent net/eth0: Set up 4096 KiB ring buffer (1344 RX/1344 TX packets) net/eth0: Listener started (MTU: 1500) disk/testdev: ff:ff:ff:ff:ff:ff/00000000: Completed, status 0 disk/testdev: ff:ff:ff:ff:ff:ff/00000000: Response sent net/eth1: Does not match the configured pattern list, ignoring disk/testdev: 0:25:90:1d:b:f7/01737f73: Received ATA cmd IDENTIFY, LBA28 0, sect 1 [ ] disk/testdev: 0:25:90:1d:b:f7/01737f73: Completed, status 0 disk/testdev: 0:25:90:1d:b:f7/01737f73: Response sent disk/testdev: 0:16:e6:80:81:92/016b7af9: Received ATA cmd IDENTIFY, LBA28 0, sect 1 [ ] disk/testdev: 0:16:e6:80:81:92/016b7af9: Completed, status 0 disk/testdev: 0:16:e6:80:81:92/016b7af9: Response sent disk/testdev: 0:25:90:1d:b:f6/01747f73: Received ATA cmd IDENTIFY, LBA28 0, sect 1 [ ] disk/testdev: 0:25:90:1d:b:f6/01747f73: Completed, status 0 disk/testdev: 0:25:90:1d:b:f6/01747f73: Response sent disk/testdev: 6c:f0:49:b:53:88/0a078679: Received ATA cmd IDENTIFY, LBA28 0, sect 1 [ ] disk/testdev: 6c:f0:49:b:53:88/0a078679: Completed, status 0 disk/testdev: 6c:f0:49:b:53:88/0a078679: Response sent disk/testdev: 6c:f0:49:b:52:b5/016a8f39: Received ATA cmd IDENTIFY, LBA28 0, sect 1 [ ] disk/testdev: 6c:f0:49:b:52:b5/016a8f39: Completed, status 0 disk/testdev: 6c:f0:49:b:52:b5/016a8f39: Response sent disk/testdev: 6c:f0:49:b:5a:65/016acbca: Received ATA cmd IDENTIFY, LBA28 0, sect 1 [ ] disk/testdev: 6c:f0:49:b:5a:65/016acbca: Completed, status 0 disk/testdev: 6c:f0:49:b:5a:65/016acbca: Response sent disk/testdev: 6c:f0:49:b:5a:83/0166b554: Received ATA cmd IDENTIFY, LBA28 0, sect 1 [ ] disk/testdev: 6c:f0:49:b:5a:83/0166b554: Completed, status 0 disk/testdev: 6c:f0:49:b:5a:83/0166b554: Response sent disk/testdev: 6c:f0:49:b:53:88/00000000: Received CFG cmd READ disk/testdev: 6c:f0:49:b:53:88/00000000: Completed, status 0 disk/testdev: 6c:f0:49:b:53:88/00000000: Response sent disk/testdev: 6c:f0:49:b:53:88/000195c0: Received ATA cmd IDENTIFY, LBA28 0, sect 1 [ ] disk/testdev: 6c:f0:49:b:53:88/000195c0: Completed, status 0 disk/testdev: 6c:f0:49:b:53:88/000195c0: Response sent disk/testdev: 6c:f0:49:b:53:88/000295c1: Received ATA cmd READ_EXT, LBA48 0, sect 2 [ ] disk/testdev: 6c:f0:49:b:53:88/000395c1: Received ATA cmd READ_EXT, LBA48 2, sect 2 [ ] disk/testdev: 6c:f0:49:b:53:88/000495c1: Received ATA cmd READ_EXT, LBA48 4, sect 2 [ ] disk/testdev: 6c:f0:49:b:53:88/000595c1: Received ATA cmd READ_EXT, LBA48 6, sect 2 [ ] disk/testdev: 6c:f0:49:b:53:88/000295c1: Completed, status 0 disk/testdev: 6c:f0:49:b:53:88/000295c1: Response sent disk/testdev: 6c:f0:49:b:53:88/000395c1: Completed, status 0 disk/testdev: 6c:f0:49:b:53:88/000395c1: Response sent disk/testdev: 6c:f0:49:b:53:88/000495c1: Completed, status 0 disk/testdev: 6c:f0:49:b:53:88/000495c1: Response sent disk/testdev: 6c:f0:49:b:53:88/000595c1: Completed, status 0 disk/testdev: 6c:f0:49:b:53:88/000595c1: Response sent disk/testdev: 6c:f0:49:b:53:88/000695c5: Received ATA cmd READ_EXT, LBA48 8, sect 2 [ ] disk/testdev: 6c:f0:49:b:53:88/000795c5: Received ATA cmd READ_EXT, LBA48 10, sect 2 [ ] disk/testdev: 6c:f0:49:b:53:88/000895c5: Received ATA cmd READ_EXT, LBA48 12, sect 2 [ ] disk/testdev: 6c:f0:49:b:53:88/000995c5: Received ATA cmd READ_EXT, LBA48 14, sect 2 [ ] disk/testdev: 6c:f0:49:b:53:88/000695c5: Completed, status 0 disk/testdev: 6c:f0:49:b:53:88/000695c5: Response sent disk/testdev: 6c:f0:49:b:53:88/000795c5: Completed, status 0 disk/testdev: 6c:f0:49:b:53:88/000795c5: Response sent disk/testdev: 6c:f0:49:b:53:88/000895c5: Completed, status 0 disk/testdev: 6c:f0:49:b:53:88/000895c5: Response sent disk/testdev: 6c:f0:49:b:53:88/000995c5: Completed, status 0 disk/testdev: 6c:f0:49:b:53:88/000995c5: Response sent disk/testdev: 6c:f0:49:b:53:88/000a95c5: Received ATA cmd READ_EXT, LBA48 24, sect 2 [ ] disk/testdev: 6c:f0:49:b:53:88/000b95c5: Received ATA cmd READ_EXT, LBA48 26, sect 2 [ ] disk/testdev: 6c:f0:49:b:53:88/000c95c5: Received ATA cmd READ_EXT, LBA48 28, sect 2 [ ] disk/testdev: 6c:f0:49:b:53:88/000d95c5: Received ATA cmd READ_EXT, LBA48 30, sect 2 [ ] disk/testdev: 6c:f0:49:b:53:88/000a95c5: Completed, status 0 disk/testdev: 6c:f0:49:b:53:88/000a95c5: Response sent disk/testdev: 6c:f0:49:b:53:88/000b95c5: Completed, status 0 disk/testdev: 6c:f0:49:b:53:88/000b95c5: Response sent disk/testdev: 6c:f0:49:b:53:88/000c95c5: Completed, status 0 disk/testdev: 6c:f0:49:b:53:88/000c95c5: Response sent disk/testdev: 6c:f0:49:b:53:88/000d95c5: Completed, status 0 disk/testdev: 6c:f0:49:b:53:88/000d95c5: Response sent disk/testdev: 6c:f0:49:b:53:88/00000000: Received CFG cmd READ disk/testdev: 6c:f0:49:b:53:88/00000000: Completed, status 0 disk/testdev: 6c:f0:49:b:53:88/00000000: Response sent disk/testdev: 6c:f0:49:b:53:88/000e95c6: Received ATA cmd IDENTIFY, LBA28 0, sect 1 [ ] disk/testdev: 6c:f0:49:b:53:88/000e95c6: Completed, status 0 disk/testdev: 6c:f0:49:b:53:88/000e95c6: Response sent disk/testdev: 6c:f0:49:b:53:88/000f95c6: Received ATA cmd READ_EXT, LBA48 18446744072812625792, sect 2 [ ] disk/testdev: Attempt to access beyond end-of-device (...)
I'm glad 2.6.39.1 doesn't show poison overwritten. If you use vblade instead of qqaoed, can you still get the out-of-bounds access to occur?