Bug 26012

Summary:	BUG blkdev_queue: Poison overwritten / ATA cmd READ_EXT, LBA48 18446744072812625792, sect 2 [ ] disk/lvm: Attempt to access beyond end-of-device.
Product:	IO/Storage	Reporter:	Pawel Sikora (pluto)
Component:	Block Layer	Assignee:	Jens Axboe (axboe)
Status:	CLOSED OBSOLETE
Severity:	high	CC:	alan, ecashin, florian, neilb
Priority:	P1
Hardware:	All
OS:	Linux
Kernel Version:	2.6.37-rc8	Subsystem:
Regression:	No	Bisected commit-id:
Bug Depends on:
Bug Blocks:	26102
Attachments:	slowpath warning triggered on 2.6.39.1 by hdparm -t /dev/etherd/eX.Y task-block triggered after hdparm benchmarking.

Description Pawel Sikora 2011-01-01 15:54:45 UTC

Hi,

i'm trying to export a lvm volume via ata-over-ethernet daemon
and noticed memory corruption / device out of range access.

here's the testcase:

1). setup an empty linear lvm volume (from 4 devices /dev/sd[a-d]4).

# lvdisplay

  --- Logical volume ---
  LV Name                /dev/vg_storage/lv_storage
  VG Name                vg_storage
  LV UUID                T8z8Ko-wdP6-1962-01Oe-FBA1-N2I6-1bFp25
  LV Write Access        read/write
  LV Status              available
  # open                 0
  LV Size                1.58 TiB
  Current LE             414800
  Segments               4
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           253:0

2). run AoE daemon. here's a config for http://code.google.com/p/ggaoed

# cat /etc/ggaoed.conf

[defaults]
direct-io = true
pid-file = /var/run/ggaoed.pid
control-socket = /var/run/ggaoed.sock
state-directory = /var/lib/ggaoed

[lvm]
interfaces = eth0
path = /dev/vg_storage/lv_storage
shelf = 92
slot = 0

3). on first console run daemon in debug mode with command 'ggaoed -d'
and on the second console run 'modprobe aoe'. 0:25:90:28:93:{62,63} are
the MACs of eht0/1 interfaces.

disk/lvm: Resetting AoE configuration space
disk/lvm: Resetting AoE MAC Mask list
disk/lvm: Resetting AoE Reserve list
disk/lvm: Shelf 92, slot 0, path '/dev/vg_storage/lv_storage' (size 1620 GiB, sectors 3398041600) opened, using direct I/O
net/lo: Set up 4096 KiB ring buffer (96 RX/96 TX packets)
net/lo: Listener started (MTU: 16384)
net/eth0: Set up 4096 KiB ring buffer (1344 RX/1344 TX packets)
net/eth0: Listener started (MTU: 1500)
disk/lvm: ff:ff:ff:ff:ff:ff/00000000: Completed, status 0
disk/lvm: ff:ff:ff:ff:ff:ff/00000000: Response sent
net/eth1: Set up 4096 KiB ring buffer (1344 RX/1344 TX packets)
net/eth1: Listener started (MTU: 1500)
disk/lvm: 0:16:e6:80:81:94/08315fdf: Received ATA cmd IDENTIFY, LBA28 0, sect 1 [   ]
disk/lvm: 0:16:e6:80:81:94/08315fdf: Completed, status 0
disk/lvm: 0:16:e6:80:81:94/08315fdf: Response sent
disk/lvm: 0:16:e6:80:81:92/082dea51: Received ATA cmd IDENTIFY, LBA28 0, sect 1 [   ]
disk/lvm: 0:16:e6:80:81:92/082dea51: Completed, status 0
disk/lvm: 0:16:e6:80:81:92/082dea51: Response sent
disk/lvm: 6c:f0:49:b:53:88/019ce673: Received ATA cmd IDENTIFY, LBA28 0, sect 1 [   ]
disk/lvm: 6c:f0:49:b:53:88/019ce673: Completed, status 0
disk/lvm: 6c:f0:49:b:53:88/019ce673: Response sent
disk/lvm: 6c:f0:49:b:52:b5/0835f73b: Received ATA cmd IDENTIFY, LBA28 0, sect 1 [   ]
disk/lvm: 6c:f0:49:b:52:b5/0835f73b: Completed, status 0
disk/lvm: 6c:f0:49:b:52:b5/0835f73b: Response sent
disk/lvm: 6c:f0:49:b:5a:83/08313096: Received ATA cmd IDENTIFY, LBA28 0, sect 1 [   ]
disk/lvm: 6c:f0:49:b:5a:83/08313096: Completed, status 0
disk/lvm: 6c:f0:49:b:5a:83/08313096: Response sent
disk/lvm: 6c:f0:49:b:5a:83/00000000: Received CFG cmd READ
disk/lvm: 6c:f0:49:b:5a:83/00000000: Completed, status 0
disk/lvm: 6c:f0:49:b:5a:83/00000000: Response sent
disk/lvm: 6c:f0:49:b:5a:83/08323140: Received ATA cmd IDENTIFY, LBA28 0, sect 1 [   ]
disk/lvm: 6c:f0:49:b:5a:83/08323140: Completed, status 0
disk/lvm: 6c:f0:49:b:5a:83/08323140: Response sent
disk/lvm: 0:16:e6:80:81:92/00000000: Received CFG cmd READ
disk/lvm: 0:16:e6:80:81:92/00000000: Completed, status 0
disk/lvm: 0:16:e6:80:81:92/00000000: Response sent
disk/lvm: 0:16:e6:80:81:92/082eec80: Received ATA cmd IDENTIFY, LBA28 0, sect 1 [   ]
disk/lvm: 0:16:e6:80:81:92/082eec80: Completed, status 0
disk/lvm: 0:16:e6:80:81:92/082eec80: Response sent
disk/lvm: 0:25:90:28:93:63/00000000: Received CFG cmd READ
disk/lvm: 0:25:90:28:93:63/00000000: Completed, status 0
disk/lvm: 0:25:90:28:93:63/00000000: Response sent
disk/lvm: 0:25:90:28:93:63/0001c4e6: Received ATA cmd IDENTIFY, LBA28 0, sect 1 [   ]
disk/lvm: 0:25:90:28:93:63/0001c4e6: Completed, status 0
disk/lvm: 0:25:90:28:93:63/0001c4e6: Response sent
disk/lvm: 0:25:90:28:93:63/0002c4e6: Received ATA cmd READ_EXT, LBA48 0, sect 2 [   ]
disk/lvm: 0:25:90:28:93:63/0003c4e6: Received ATA cmd READ_EXT, LBA48 2, sect 2 [   ]
disk/lvm: 0:25:90:28:93:63/0004c4e6: Received ATA cmd READ_EXT, LBA48 4, sect 2 [   ]
disk/lvm: 0:25:90:28:93:63/0005c4e6: Received ATA cmd READ_EXT, LBA48 6, sect 2 [   ]
disk/lvm: 0:25:90:28:93:63/0002c4e6: Completed, status 0
disk/lvm: 0:25:90:28:93:63/0002c4e6: Response sent
disk/lvm: 0:25:90:28:93:63/0003c4e6: Completed, status 0
disk/lvm: 0:25:90:28:93:63/0003c4e6: Response sent
disk/lvm: 0:25:90:28:93:63/0004c4e6: Completed, status 0
disk/lvm: 0:25:90:28:93:63/0004c4e6: Response sent
disk/lvm: 0:25:90:28:93:63/0005c4e6: Completed, status 0
disk/lvm: 0:25:90:28:93:63/0005c4e6: Response sent
disk/lvm: 0:25:90:28:93:63/0006c4e6: Received ATA cmd READ_EXT, LBA48 8, sect 2 [   ]
disk/lvm: 0:25:90:28:93:63/0007c4e6: Received ATA cmd READ_EXT, LBA48 10, sect 2 [   ]
disk/lvm: 0:25:90:28:93:63/0008c4e6: Received ATA cmd READ_EXT, LBA48 12, sect 2 [   ]
disk/lvm: 0:25:90:28:93:63/0009c4e6: Received ATA cmd READ_EXT, LBA48 14, sect 2 [   ]
disk/lvm: 0:25:90:28:93:63/0006c4e6: Completed, status 0
disk/lvm: 0:25:90:28:93:63/0006c4e6: Response sent
disk/lvm: 0:25:90:28:93:63/0007c4e6: Completed, status 0
disk/lvm: 0:25:90:28:93:63/0007c4e6: Response sent
disk/lvm: 0:25:90:28:93:63/0008c4e6: Completed, status 0
disk/lvm: 0:25:90:28:93:63/0008c4e6: Response sent
disk/lvm: 0:25:90:28:93:63/0009c4e6: Completed, status 0
disk/lvm: 0:25:90:28:93:63/0009c4e6: Response sent
disk/lvm: 0:25:90:28:93:63/000ac4e7: Received ATA cmd READ_EXT, LBA48 24, sect 2 [   ]
disk/lvm: 0:25:90:28:93:63/000bc4e7: Received ATA cmd READ_EXT, LBA48 26, sect 2 [   ]
disk/lvm: 0:25:90:28:93:63/000cc4e7: Received ATA cmd READ_EXT, LBA48 28, sect 2 [   ]
disk/lvm: 0:25:90:28:93:63/000dc4e7: Received ATA cmd READ_EXT, LBA48 30, sect 2 [   ]
disk/lvm: 0:25:90:28:93:63/000ac4e7: Completed, status 0
disk/lvm: 0:25:90:28:93:63/000ac4e7: Response sent
disk/lvm: 0:25:90:28:93:63/000bc4e7: Completed, status 0
disk/lvm: 0:25:90:28:93:63/000bc4e7: Response sent
disk/lvm: 0:25:90:28:93:63/000cc4e7: Completed, status 0
disk/lvm: 0:25:90:28:93:63/000cc4e7: Response sent
disk/lvm: 0:25:90:28:93:63/000dc4e7: Completed, status 0
disk/lvm: 0:25:90:28:93:63/000dc4e7: Response sent
disk/lvm: 0:25:90:28:93:63/00000000: Received CFG cmd READ
disk/lvm: 0:25:90:28:93:63/00000000: Completed, status 0
disk/lvm: 0:25:90:28:93:63/00000000: Response sent
disk/lvm: 0:25:90:28:93:63/000ec4e7: Received ATA cmd IDENTIFY, LBA28 0, sect 1 [   ]
disk/lvm: 0:25:90:28:93:63/000ec4e7: Completed, status 0
disk/lvm: 0:25:90:28:93:63/000ec4e7: Response sent
disk/lvm: 0:25:90:28:93:63/000fc4e7: Received ATA cmd READ_EXT, LBA48 18446744072812625792, sect 2 [   ]
disk/lvm: Attempt to access beyond end-of-device
disk/lvm: 0:25:90:28:93:63/000fc4e7: Completed, status 0
disk/lvm: 0:25:90:28:93:63/000fc4e7: Response sent




[93024.868914] aoe: AoE v47 initialised.
[93024.869178] aoe: 002590289362 e92.0 v0100 has 3398041600 sectors
[93024.869247] =============================================================================
[93024.869251] BUG blkdev_queue: Poison overwritten
[93024.869252] -----------------------------------------------------------------------------
[93024.869254] 
[93024.869256] INFO: 0xffff8807ff2a9070-0xffff8807ff2a9077. First byte 0xb8 instead of 0x6b
[93024.869266] INFO: Allocated in blk_alloc_queue_node+0x1f/0x1a0 age=90599 cpu=4 pid=1070
[93024.869271] INFO: Freed in blk_release_queue+0x6a/0x80 age=85026 cpu=13 pid=1859
[93024.869274] INFO: Slab 0xffffea001bfd14c0 objects=17 used=1 fp=0xffff8807ff2a8e60 flags=0x6000000000040c1
[93024.869277] INFO: Object 0xffff8807ff2a8e60 @offset=3680 fp=0xffff8807ff2a8730
[93024.869278] 
[93024.869280] Bytes b4 0xffff8807ff2a8e50:  de ed a6 01 01 00 00 00 5a 5a 5a 5a 5a 5a 5a 5a ���.....ZZZZZZZZ
[93024.869289]   Object 0xffff8807ff2a8e60:  6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[93024.869298]   Object 0xffff8807ff2a8e70:  6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[93024.869305]   Object 0xffff8807ff2a8e80:  6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[93024.869313]   Object 0xffff8807ff2a8e90:  6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[93024.869320]   Object 0xffff8807ff2a8ea0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[93024.869328]   Object 0xffff8807ff2a8eb0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[93024.869335]   Object 0xffff8807ff2a8ec0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[93024.869343]   Object 0xffff8807ff2a8ed0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[93024.869350]   Object 0xffff8807ff2a8ee0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[93024.869357]   Object 0xffff8807ff2a8ef0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[93024.869365]   Object 0xffff8807ff2a8f00:  6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[93024.869372]   Object 0xffff8807ff2a8f10:  6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[93024.869379]   Object 0xffff8807ff2a8f20:  6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[93024.869387]   Object 0xffff8807ff2a8f30:  6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[93024.869395]   Object 0xffff8807ff2a8f40:  6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[93024.869403]   Object 0xffff8807ff2a8f50:  6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[93024.869410]   Object 0xffff8807ff2a8f60:  6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[93024.869417]   Object 0xffff8807ff2a8f70:  6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[93024.869425]   Object 0xffff8807ff2a8f80:  6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[93024.869432]   Object 0xffff8807ff2a8f90:  6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[93024.869440]   Object 0xffff8807ff2a8fa0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[93024.869447]   Object 0xffff8807ff2a8fb0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[93024.869455]   Object 0xffff8807ff2a8fc0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[93024.869462]   Object 0xffff8807ff2a8fd0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[93024.869470]   Object 0xffff8807ff2a8fe0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[93024.869477]   Object 0xffff8807ff2a8ff0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[93024.869485]   Object 0xffff8807ff2a9000:  6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[93024.869492]   Object 0xffff8807ff2a9010:  6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[93024.869501]   Object 0xffff8807ff2a9020:  6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[93024.869509]   Object 0xffff8807ff2a9030:  6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[93024.869517]   Object 0xffff8807ff2a9040:  6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[93024.869526]   Object 0xffff8807ff2a9050:  6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[93024.869535]   Object 0xffff8807ff2a9060:  6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[93024.869543]   Object 0xffff8807ff2a9070:  b8 01 8a fe 03 88 ff ff 6b 6b 6b 6b 6b 6b 6b 6b �..�..��kkkkkkkk
[93024.869550]   Object 0xffff8807ff2a9080:  6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[93024.869558]   Object 0xffff8807ff2a9090:  6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[93024.869568]   Object 0xffff8807ff2a90a0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[93024.869576]   Object 0xffff8807ff2a90b0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[93024.869590]   Object 0xffff8807ff2a90c0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[93024.869598]   Object 0xffff8807ff2a90d0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[93024.869605]   Object 0xffff8807ff2a90e0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[93024.869613]   Object 0xffff8807ff2a90f0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[93024.869623]   Object 0xffff8807ff2a9100:  6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[93024.869630]   Object 0xffff8807ff2a9110:  6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[93024.869638]   Object 0xffff8807ff2a9120:  6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[93024.869645]   Object 0xffff8807ff2a9130:  6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[93024.869652]   Object 0xffff8807ff2a9140:  6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[93024.869662]   Object 0xffff8807ff2a9150:  6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[93024.869670]   Object 0xffff8807ff2a9160:  6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[93024.869677]   Object 0xffff8807ff2a9170:  6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[93024.869685]   Object 0xffff8807ff2a9180:  6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[93024.869692]   Object 0xffff8807ff2a9190:  6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[93024.869699]   Object 0xffff8807ff2a91a0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[93024.869707]   Object 0xffff8807ff2a91b0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[93024.869714]   Object 0xffff8807ff2a91c0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[93024.869722]   Object 0xffff8807ff2a91d0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[93024.869730]   Object 0xffff8807ff2a91e0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[93024.869738]   Object 0xffff8807ff2a91f0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[93024.869745]   Object 0xffff8807ff2a9200:  6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[93024.869752]   Object 0xffff8807ff2a9210:  6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[93024.869765]   Object 0xffff8807ff2a9220:  6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[93024.869772]   Object 0xffff8807ff2a9230:  6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[93024.869780]   Object 0xffff8807ff2a9240:  6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[93024.869787]   Object 0xffff8807ff2a9250:  6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[93024.869794]   Object 0xffff8807ff2a9260:  6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[93024.869801]   Object 0xffff8807ff2a9270:  6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[93024.869810]   Object 0xffff8807ff2a9280:  6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[93024.869819]   Object 0xffff8807ff2a9290:  6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[93024.869826]   Object 0xffff8807ff2a92a0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[93024.869834]   Object 0xffff8807ff2a92b0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[93024.869843]   Object 0xffff8807ff2a92c0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[93024.869851]   Object 0xffff8807ff2a92d0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[93024.869860]   Object 0xffff8807ff2a92e0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[93024.869867]   Object 0xffff8807ff2a92f0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[93024.869876]   Object 0xffff8807ff2a9300:  6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[93024.869885]   Object 0xffff8807ff2a9310:  6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[93024.869893]   Object 0xffff8807ff2a9320:  6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[93024.869900]   Object 0xffff8807ff2a9330:  6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[93024.869910]   Object 0xffff8807ff2a9340:  6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[93024.869918]   Object 0xffff8807ff2a9350:  6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[93024.869926]   Object 0xffff8807ff2a9360:  6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[93024.869934]   Object 0xffff8807ff2a9370:  6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[93024.869941]   Object 0xffff8807ff2a9380:  6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[93024.869948]   Object 0xffff8807ff2a9390:  6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[93024.869958]   Object 0xffff8807ff2a93a0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[93024.869965]   Object 0xffff8807ff2a93b0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[93024.869975]   Object 0xffff8807ff2a93c0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[93024.869982]   Object 0xffff8807ff2a93d0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[93024.869990]   Object 0xffff8807ff2a93e0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[93024.869997]   Object 0xffff8807ff2a93f0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[93024.870004]   Object 0xffff8807ff2a9400:  6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[93024.870012]   Object 0xffff8807ff2a9410:  6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[93024.870020]   Object 0xffff8807ff2a9420:  6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[93024.870028]   Object 0xffff8807ff2a9430:  6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[93024.870036]   Object 0xffff8807ff2a9440:  6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[93024.870043]   Object 0xffff8807ff2a9450:  6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[93024.870050]   Object 0xffff8807ff2a9460:  6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[93024.870058]   Object 0xffff8807ff2a9470:  6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[93024.870065]   Object 0xffff8807ff2a9480:  6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[93024.870072]   Object 0xffff8807ff2a9490:  6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[93024.870080]   Object 0xffff8807ff2a94a0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[93024.870087]   Object 0xffff8807ff2a94b0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[93024.870094]   Object 0xffff8807ff2a94c0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[93024.870105]   Object 0xffff8807ff2a94d0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[93024.870112]   Object 0xffff8807ff2a94e0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[93024.870120]   Object 0xffff8807ff2a94f0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[93024.870127]   Object 0xffff8807ff2a9500:  6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[93024.870134]   Object 0xffff8807ff2a9510:  6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[93024.870142]   Object 0xffff8807ff2a9520:  6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[93024.870149]   Object 0xffff8807ff2a9530:  6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[93024.870156]   Object 0xffff8807ff2a9540:  6b 6b 6b 6b 6b 6b 6b a5                         kkkkkkk�        
[93024.870163]  Redzone 0xffff8807ff2a9548:  bb bb bb bb bb bb bb bb                         ��������        
[93024.870170]  Padding 0xffff8807ff2a9588:  5a 5a 5a 5a 5a 5a 5a 5a                         ZZZZZZZZ        
[93024.870183] Pid: 1070, comm: kworker/4:1 Not tainted 2.6.37-rc8 #4
[93024.870186] Call Trace:
[93024.870195]  [<ffffffff8111ea1e>] print_trailer+0xfe/0x160
[93024.870198]  [<ffffffff8111f074>] check_bytes_and_report+0xf4/0x130
[93024.870202]  [<ffffffff8111f2da>] check_object+0x22a/0x270
[93024.870206]  [<ffffffff811fd0df>] ? blk_alloc_queue_node+0x1f/0x1a0
[93024.870209]  [<ffffffff811fd0df>] ? blk_alloc_queue_node+0x1f/0x1a0
[93024.870213]  [<ffffffff81120380>] alloc_debug_processing+0x110/0x1f0
[93024.870217]  [<ffffffff811211c9>] __slab_alloc+0x3a9/0x410
[93024.870222]  [<ffffffff810de600>] ? mempool_alloc_slab+0x10/0x20
[93024.870225]  [<ffffffff8112166f>] ? kmem_cache_alloc_node_notrace+0xbf/0xe0
[93024.870229]  [<ffffffff810de7fe>] ? mempool_create_node+0x7e/0x1a0
[93024.870232]  [<ffffffff811216fc>] kmem_cache_alloc_node+0x6c/0x140
[93024.870236]  [<ffffffff810de87b>] ? mempool_create_node+0xfb/0x1a0
[93024.870239]  [<ffffffff811fd0df>] ? blk_alloc_queue_node+0x1f/0x1a0
[93024.870246]  [<ffffffffa050df20>] ? aoecmd_sleepwork+0x0/0xf0 [aoe]
[93024.870250]  [<ffffffff811fd0df>] blk_alloc_queue_node+0x1f/0x1a0
[93024.870254]  [<ffffffffa050df20>] ? aoecmd_sleepwork+0x0/0xf0 [aoe]
[93024.870257]  [<ffffffff811fd26e>] blk_alloc_queue+0xe/0x10
[93024.870261]  [<ffffffffa050c6fd>] aoeblk_gdalloc+0x6d/0x1f0 [aoe]
[93024.870265]  [<ffffffffa050df20>] ? aoecmd_sleepwork+0x0/0xf0 [aoe]
[93024.870269]  [<ffffffffa050dff8>] aoecmd_sleepwork+0xd8/0xf0 [aoe]
[93024.870272]  [<ffffffffa050df20>] ? aoecmd_sleepwork+0x0/0xf0 [aoe]
[93024.870277]  [<ffffffff81062048>] process_one_work+0x118/0x400
[93024.870281]  [<ffffffff81063b73>] worker_thread+0x163/0x330
[93024.870286]  [<ffffffff81035f39>] ? __wake_up_common+0x59/0x90
[93024.870290]  [<ffffffff81063a10>] ? worker_thread+0x0/0x330
[93024.870295]  [<ffffffff81068276>] kthread+0x96/0xa0
[93024.870299]  [<ffffffff81003b54>] kernel_thread_helper+0x4/0x10
[93024.870303]  [<ffffffff810681e0>] ? kthread+0x0/0xa0
[93024.870306]  [<ffffffff81003b50>] ? kernel_thread_helper+0x0/0x10
[93024.870309] FIX blkdev_queue: Restoring 0xffff8807ff2a9070-0xffff8807ff2a9077=0x6b

Comment 1 Pawel Sikora 2011-01-11 15:34:01 UTC

btw, afaics the aoe driver in kernel tree looks unmaintained while the one
from the http://support.coraid.com/support/linux/ is in active development.

Comment 2 Florian Mickler 2011-03-29 21:01:04 UTC

Has any action taken place to resolve this issue? (Is it still a problem in 2.6.38.y / 2.6.39-rc*?)

Comment 3 Florian Mickler 2011-04-17 16:41:16 UTC

Ed, are you aware of this issue?

Comment 4 Pawel Sikora 2011-06-19 06:28:26 UTC

Created attachment 62762 [details]
slowpath warning triggered on 2.6.39.1 by hdparm -t /dev/etherd/eX.Y

Comment 5 Pawel Sikora 2011-06-19 06:29:38 UTC

Created attachment 62772 [details]
task-block triggered after hdparm benchmarking.

Comment 6 Pawel Sikora 2011-06-20 18:23:16 UTC

update:

i don't see the 'poison overwritten' on the 2.6.39.1 with ~1.6TB exported
volume but the 'attempt to access beyond end-of-device' is present.

# ggaoed -d
disk/testdev: Shelf 92, slot 0, path '/dev/vg_test/lv_test' (size 1620 GiB, sectors 3398041600) opened R/O, using direct I/O
net/lo: Set up 4096 KiB ring buffer (96 RX/96 TX packets)
net/lo: Listener started (MTU: 16384)
disk/testdev: ff:ff:ff:ff:ff:ff/00000000: Completed, status 0
disk/testdev: ff:ff:ff:ff:ff:ff/00000000: Response sent
net/eth0: Set up 4096 KiB ring buffer (1344 RX/1344 TX packets)
net/eth0: Listener started (MTU: 1500)
disk/testdev: ff:ff:ff:ff:ff:ff/00000000: Completed, status 0
disk/testdev: ff:ff:ff:ff:ff:ff/00000000: Response sent
net/eth1: Does not match the configured pattern list, ignoring
disk/testdev: 0:25:90:1d:b:f7/01737f73: Received ATA cmd IDENTIFY, LBA28 0, sect 1 [   ]
disk/testdev: 0:25:90:1d:b:f7/01737f73: Completed, status 0
disk/testdev: 0:25:90:1d:b:f7/01737f73: Response sent
disk/testdev: 0:16:e6:80:81:92/016b7af9: Received ATA cmd IDENTIFY, LBA28 0, sect 1 [   ]
disk/testdev: 0:16:e6:80:81:92/016b7af9: Completed, status 0
disk/testdev: 0:16:e6:80:81:92/016b7af9: Response sent
disk/testdev: 0:25:90:1d:b:f6/01747f73: Received ATA cmd IDENTIFY, LBA28 0, sect 1 [   ]
disk/testdev: 0:25:90:1d:b:f6/01747f73: Completed, status 0
disk/testdev: 0:25:90:1d:b:f6/01747f73: Response sent
disk/testdev: 6c:f0:49:b:53:88/0a078679: Received ATA cmd IDENTIFY, LBA28 0, sect 1 [   ]
disk/testdev: 6c:f0:49:b:53:88/0a078679: Completed, status 0
disk/testdev: 6c:f0:49:b:53:88/0a078679: Response sent
disk/testdev: 6c:f0:49:b:52:b5/016a8f39: Received ATA cmd IDENTIFY, LBA28 0, sect 1 [   ]
disk/testdev: 6c:f0:49:b:52:b5/016a8f39: Completed, status 0
disk/testdev: 6c:f0:49:b:52:b5/016a8f39: Response sent
disk/testdev: 6c:f0:49:b:5a:65/016acbca: Received ATA cmd IDENTIFY, LBA28 0, sect 1 [   ]
disk/testdev: 6c:f0:49:b:5a:65/016acbca: Completed, status 0
disk/testdev: 6c:f0:49:b:5a:65/016acbca: Response sent
disk/testdev: 6c:f0:49:b:5a:83/0166b554: Received ATA cmd IDENTIFY, LBA28 0, sect 1 [   ]
disk/testdev: 6c:f0:49:b:5a:83/0166b554: Completed, status 0
disk/testdev: 6c:f0:49:b:5a:83/0166b554: Response sent
disk/testdev: 6c:f0:49:b:53:88/00000000: Received CFG cmd READ
disk/testdev: 6c:f0:49:b:53:88/00000000: Completed, status 0
disk/testdev: 6c:f0:49:b:53:88/00000000: Response sent
disk/testdev: 6c:f0:49:b:53:88/000195c0: Received ATA cmd IDENTIFY, LBA28 0, sect 1 [   ]
disk/testdev: 6c:f0:49:b:53:88/000195c0: Completed, status 0
disk/testdev: 6c:f0:49:b:53:88/000195c0: Response sent
disk/testdev: 6c:f0:49:b:53:88/000295c1: Received ATA cmd READ_EXT, LBA48 0, sect 2 [   ]
disk/testdev: 6c:f0:49:b:53:88/000395c1: Received ATA cmd READ_EXT, LBA48 2, sect 2 [   ]
disk/testdev: 6c:f0:49:b:53:88/000495c1: Received ATA cmd READ_EXT, LBA48 4, sect 2 [   ]
disk/testdev: 6c:f0:49:b:53:88/000595c1: Received ATA cmd READ_EXT, LBA48 6, sect 2 [   ]
disk/testdev: 6c:f0:49:b:53:88/000295c1: Completed, status 0
disk/testdev: 6c:f0:49:b:53:88/000295c1: Response sent
disk/testdev: 6c:f0:49:b:53:88/000395c1: Completed, status 0
disk/testdev: 6c:f0:49:b:53:88/000395c1: Response sent
disk/testdev: 6c:f0:49:b:53:88/000495c1: Completed, status 0
disk/testdev: 6c:f0:49:b:53:88/000495c1: Response sent
disk/testdev: 6c:f0:49:b:53:88/000595c1: Completed, status 0
disk/testdev: 6c:f0:49:b:53:88/000595c1: Response sent
disk/testdev: 6c:f0:49:b:53:88/000695c5: Received ATA cmd READ_EXT, LBA48 8, sect 2 [   ]
disk/testdev: 6c:f0:49:b:53:88/000795c5: Received ATA cmd READ_EXT, LBA48 10, sect 2 [   ]
disk/testdev: 6c:f0:49:b:53:88/000895c5: Received ATA cmd READ_EXT, LBA48 12, sect 2 [   ]
disk/testdev: 6c:f0:49:b:53:88/000995c5: Received ATA cmd READ_EXT, LBA48 14, sect 2 [   ]
disk/testdev: 6c:f0:49:b:53:88/000695c5: Completed, status 0
disk/testdev: 6c:f0:49:b:53:88/000695c5: Response sent
disk/testdev: 6c:f0:49:b:53:88/000795c5: Completed, status 0
disk/testdev: 6c:f0:49:b:53:88/000795c5: Response sent
disk/testdev: 6c:f0:49:b:53:88/000895c5: Completed, status 0
disk/testdev: 6c:f0:49:b:53:88/000895c5: Response sent
disk/testdev: 6c:f0:49:b:53:88/000995c5: Completed, status 0
disk/testdev: 6c:f0:49:b:53:88/000995c5: Response sent
disk/testdev: 6c:f0:49:b:53:88/000a95c5: Received ATA cmd READ_EXT, LBA48 24, sect 2 [   ]
disk/testdev: 6c:f0:49:b:53:88/000b95c5: Received ATA cmd READ_EXT, LBA48 26, sect 2 [   ]
disk/testdev: 6c:f0:49:b:53:88/000c95c5: Received ATA cmd READ_EXT, LBA48 28, sect 2 [   ]
disk/testdev: 6c:f0:49:b:53:88/000d95c5: Received ATA cmd READ_EXT, LBA48 30, sect 2 [   ]
disk/testdev: 6c:f0:49:b:53:88/000a95c5: Completed, status 0
disk/testdev: 6c:f0:49:b:53:88/000a95c5: Response sent
disk/testdev: 6c:f0:49:b:53:88/000b95c5: Completed, status 0
disk/testdev: 6c:f0:49:b:53:88/000b95c5: Response sent
disk/testdev: 6c:f0:49:b:53:88/000c95c5: Completed, status 0
disk/testdev: 6c:f0:49:b:53:88/000c95c5: Response sent
disk/testdev: 6c:f0:49:b:53:88/000d95c5: Completed, status 0
disk/testdev: 6c:f0:49:b:53:88/000d95c5: Response sent
disk/testdev: 6c:f0:49:b:53:88/00000000: Received CFG cmd READ
disk/testdev: 6c:f0:49:b:53:88/00000000: Completed, status 0
disk/testdev: 6c:f0:49:b:53:88/00000000: Response sent
disk/testdev: 6c:f0:49:b:53:88/000e95c6: Received ATA cmd IDENTIFY, LBA28 0, sect 1 [   ]
disk/testdev: 6c:f0:49:b:53:88/000e95c6: Completed, status 0
disk/testdev: 6c:f0:49:b:53:88/000e95c6: Response sent
disk/testdev: 6c:f0:49:b:53:88/000f95c6: Received ATA cmd READ_EXT, LBA48 18446744072812625792, sect 2 [   ]
disk/testdev: Attempt to access beyond end-of-device
(...)

Comment 7 Ed Cashin 2011-06-27 16:50:51 UTC

I'm glad 2.6.39.1 doesn't show poison overwritten.  If you use vblade instead of qqaoed, can you still get the out-of-bounds access to occur?