Created attachment 112261 [details] Diagnostics with a 3.11.0 kernel After an unexpected power failure on a Gentoo Hardened 3.11.3 system, one of my btrfs partitions causes a General Protection Fault when doing df -h (and probably other interesting things.). I've attached the disk to and done some diagnostics on a Kubuntu system with both a Ubuntu 3.11.3 and Ubuntu 3.11.6 kernel. Transcripts of those sessions are attached.
Created attachment 112271 [details] Diagnostics with a 3.11.6 kernel
Unfortunately, I followed *all* of the suggestions at http://www.mail-archive.com/linux-btrfs@vger.kernel.org/msg26816.html and have run btrfsck --repair on the partition. Fortunately, I have the output of btrfs-image -c9 -t4 . It's a ~160 MB file that doesn't compress well, and I'm not sure who would be interested in it.
Sorry, I misspoke. The first Ubuntu kernel that I used was a 3.11.0 kernel. Also, I have no idea why so many modules were force-loaded with that kernel.
This is remotely close to bug 60834, where it crashes in memcpy after reading extents. It's most probably caused by corrupted data read back from disk and based on previous debugging the data were corrupted in memory before they were written to disk so checksums will not catch that.
Let me know where to pull down the image, you can email me the location at jbacik at fusionio dot com.
(In reply to David Sterba from comment #4) > This is remotely close to bug 60834, where it crashes in memcpy after > reading extents. Okay, it was a different issue and the proposed patch should fix that. This memcpy is in a different place: read_extent_buffer(): 4884 while (len > 0) { 4885 page = extent_buffer_page(eb, i); 4886 4887 cur = min(len, (PAGE_CACHE_SIZE - offset)); 4888 kaddr = page_address(page); 4889 memcpy(dst, kaddr + offset, cur); 4890 4891 dst += cur; 4892 len -= cur; 4893 offset = 0; 4894 i++; 4895 } 4896 }
For further reference, from comment 0: [ 4554.137008] CPU: 2 PID: 5066 Comm: df Tainted: GF 3.11.0-12-generic #19-Ubuntu [ 4554.137148] Hardware name: To be filled by O.E.M. To be filled by O.E.M./M5A99X EVO, BIOS 1708 04/10/2013 [ 4554.137310] task: ffff880423b2c650 ti: ffff8802fa8bc000 task.ti: ffff8802fa8bc000 [ 4554.137435] RIP: 0010:[<ffffffff8136929d>] [<ffffffff8136929d>] memcpy+0xd/0x110 [ 4554.137571] RSP: 0018:ffff8802fa8bd968 EFLAGS: 00010206 [ 4554.137662] RAX: ffff88035149d7bc RBX: 0000000000000035 RCX: 0000000000000006 [ 4554.137804] RDX: 0000000000000005 RSI: 0005080000000000 RDI: ffff88035149d7bc [ 4554.137917] RBP: ffff8802fa8bd9a0 R08: 0000000000001000 R09: ffff88035149d000 [ 4554.138030] R10: 0000000000000000 R11: 0000000000000000 R12: ffff88033e3b1610 [ 4554.138142] R13: 0000160000000000 R14: ffff88035149d7f1 R15: 0000000000000035 [ 4554.138257] FS: 00007fd98edeb7c0(0000) GS:ffff88043fc80000(0063) knlGS:00000000f75926c0 [ 4554.138384] CS: 0010 DS: 002b ES: 002b CR0: 000000008005003b [ 4554.138476] CR2: 00000000f820185c CR3: 00000003331b7000 CR4: 00000000000007e0 [ 4554.138588] Stack: [ 4554.138621] ffffffffa033fd4c 0000000000001000 ffff8803af466800 0000000000000000 [ 4554.138747] ffff8802f30fb090 ffff88033e36e4e0 ffff880335f56130 ffff8802fa8bda60 [ 4554.138871] ffffffffa03243b6 0000000000000000 00000000000007ca 00000000ffffffff [ 4554.139020] Call Trace: [ 4554.139112] [<ffffffffa033fd4c>] ? read_extent_buffer+0xbc/0x110 [btrfs] [ 4554.139256] [<ffffffffa03243b6>] btrfs_get_extent+0x906/0x9b0 [btrfs] [ 4554.139396] [<ffffffffa033d04e>] __extent_read_full_page+0x2ee/0x710 [btrfs] [ 4554.139516] [<ffffffff8119c79a>] ? __mem_cgroup_commit_charge+0x9a/0x380 [ 4554.139657] [<ffffffffa0323ab0>] ? btrfs_real_readdir+0x5c0/0x5c0 [btrfs] [ 4554.139773] [<ffffffff8115b873>] ? __inc_zone_page_state+0x33/0x40 [ 4554.139905] [<ffffffffa0323ab0>] ? btrfs_real_readdir+0x5c0/0x5c0 [btrfs] [ 4554.140050] [<ffffffffa033e465>] extent_readpages+0x1a5/0x210 [btrfs] [ 4554.140188] [<ffffffff81183969>] ? alloc_pages_current+0xa9/0x160 [ 4554.140319] [<ffffffffa0321a5f>] btrfs_readpages+0x1f/0x30 [btrfs] [ 4554.140424] [<ffffffff8114a888>] __do_page_cache_readahead+0x1a8/0x240 [ 4554.140534] [<ffffffff8114ad96>] ondemand_readahead+0x126/0x250 [ 4554.140634] [<ffffffff8114aef3>] page_cache_sync_readahead+0x33/0x50 [ 4554.140739] [<ffffffff811406a5>] generic_file_aio_read+0x4b5/0x700 [ 4554.140844] [<ffffffff811a64e0>] do_sync_read+0x80/0xb0 [ 4554.140933] [<ffffffff811a6b3e>] vfs_read+0x9e/0x170 [ 4554.141017] [<ffffffff811a7669>] SyS_read+0x49/0xa0 [ 4554.141101] [<ffffffff816f712c>] cstar_dispatch+0x7/0x21
*** Bug 64511 has been marked as a duplicate of this bug. ***
Reported on 3.12: [ 2276.914634] general protection fault: 0000 [#1] PREEMPT SMP¤ [ 2276.914704] Modules linked in: x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm arc4 crct10dif_pclmul crct10dif_common iwldvm crc32_pclmul ghash_clmulni_intel mac80211 cryptd uvcvideo i915 intel_agp iTCO_wdt intel_gtt videobuf2_vmalloc rtsx_pci_ms joydev iTCO_vendor_support videobuf2_memops videobuf2_core drm_kms_helper videodev drm media atl1c thinkpad_acpi i2c_algo_bit iwlwifi microcode evdev wmi memstick nvram snd_hda_codec_hdmi snd_hda_codec_conexant cfg80211 rfkill snd_hda_intel snd_hda_codec snd_hwdep thermal snd_pcm snd_page_alloc snd_timer i2c_i801 snd soundcore i2c_core ac video battery shpchp button psmouse lpc_ich mei_me processor mei serio_raw btrfs libcrc32c xor raid6_pq sd_mod rtsx_pci_sdmmc mmc_core crc32c_intel ahci ehci_pci libahci ehci_hcd libata scsi_mod rtsx_pci usbcore [ 2276.915469] usb_common [last unloaded: pcspkr] [ 2276.915503] CPU: 2 PID: 1073 Comm: hg Not tainted 3.12.0-1-ARCH #1 [ 2276.915554] Hardware name: LENOVO 30456UG/30456UG, BIOS 8QET56WW (1.17 ) 10/23/2012 [ 2276.915616] task: ffff8800aaeb70f0 ti: ffff88012c2c8000 task.ti: ffff88012c2c8000 [ 2276.915674] RIP: 0010:[<ffffffff8129adf4>] [<ffffffff8129adf4>] memcpy+0x54/0x110 [ 2276.915743] RSP: 0018:ffff88012c2c98f0 EFLAGS: 00210206 [ 2276.915786] RAX: ffff88008f425d05 RBX: 0000000000000154 RCX: ffff880000000000 [ 2276.915842] RDX: 0000000000000114 RSI: 0005080000000154 RDI: ffff88008f425e59 [ 2276.915897] RBP: ffff88012c2c9928 R08: 0000000000010001 R09: ffffffffffdf0000 [ 2276.915957] R10: 00000000006a0700 R11: 007d030000000000 R12: ffff8800ab691750 [ 2276.916012] R13: 0000160000000000 R14: ffff88008f425e59 R15: 0000000000000154 [ 2276.916069] FS: 0000000000000000(0000) GS:ffff88013fa80000(0063) knlGS:00000000f73ff700 [ 2276.916132] CS: 0010 DS: 002b ES: 002b CR0: 0000000080050033 [ 2276.916177] CR2: 00000000f77d5000 CR3: 000000012e948000 CR4: 00000000000407e0 [ 2276.916233] Stack: [ 2276.916252] ffffffffa021adbc 0000000000001000 ffff880138280800 0000000000000000 [ 2276.916321] ffff8800928bc6c0 ffff88008f425000 ffff8800979c4098 ffff88012c2c99f0 [ 2276.916389] ffffffffa01ff0e1 0000000000000000 0000000000001000 00000000ffffffff [ 2276.916457] Call Trace: [ 2276.916503] [<ffffffffa021adbc>] ? read_extent_buffer+0xbc/0x110 [btrfs] [ 2276.916573] [<ffffffffa01ff0e1>] btrfs_get_extent+0x911/0x9b0 [btrfs] [ 2276.916640] [<ffffffffa0216ce6>] __do_readpage+0x376/0x7d0 [btrfs] [ 2276.916703] [<ffffffffa01fe7d0>] ? __btrfs_submit_bio_start_direct_io+0x30/0x30 [btrfs] [ 2276.916769] [<ffffffff811989ee>] ? __mem_cgroup_commit_charge+0xce/0x3f0 [ 2276.916836] [<ffffffffa02174a2>] __extent_readpages.constprop.44+0x282/0x2b0 [btrfs] [ 2276.916909] [<ffffffffa01fe7d0>] ? __btrfs_submit_bio_start_direct_io+0x30/0x30 [btrfs] [ 2276.916986] [<ffffffffa02194a6>] extent_readpages+0x1b6/0x1c0 [btrfs] [ 2276.917050] [<ffffffffa01fe7d0>] ? __btrfs_submit_bio_start_direct_io+0x30/0x30 [btrfs] [ 2276.917116] [<ffffffff8117dd49>] ? alloc_pages_current+0xa9/0x160 [ 2276.917177] [<ffffffffa01fd5bf>] btrfs_readpages+0x1f/0x30 [btrfs] [ 2276.917229] [<ffffffff81141506>] __do_page_cache_readahead+0x1b6/0x260 [ 2276.917283] [<ffffffff811419ea>] ondemand_readahead+0x14a/0x2c0 [ 2276.917332] [<ffffffff81141b93>] page_cache_sync_readahead+0x33/0x50 [ 2276.917386] [<ffffffff811365d5>] generic_file_aio_read+0x4b5/0x710 [ 2276.917439] [<ffffffff811a333a>] do_sync_read+0x5a/0x90 [ 2276.917485] [<ffffffff811a38fa>] vfs_read+0x9a/0x170 [ 2276.917528] [<ffffffff811a4449>] SyS_read+0x49/0xa0 [ 2276.917572] [<ffffffff814fc8ac>] sysenter_dispatch+0x7/0x21 [ 2276.917617] Code: 4c 89 07 4c 89 4f 08 4c 89 57 10 4c 89 5f 18 48 8d 7f 20 73 d4 83 c2 20 eb 44 48 01 d6 48 01 d7 48 83 ea 20 0f 1f 00 48 83 ea 20 <4c> 8b 46 f8 4c 8b 4e f0 4c 8b 56 e8 4c 8b 5e e0 48 8d 76 e0 4c¤ [ 2276.917938] RIP [<ffffffff8129adf4>] memcpy+0x54/0x110 [ 2276.917985] RSP <ffff88012c2c98f0> [ 2276.928497] ---[ end trace e3d1618455379d30 ]---
*** This bug has been marked as a duplicate of bug 68411 ***