running btrfs on kernel 3.10.3, i got the following two backtraces, first an info about a hung task and a kernel bug. -- [ 479.949428] INFO: task btrfs-transacti:1761 blocked for more than 120 seconds. [ 479.949430] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 479.949431] btrfs-transacti D 0000000000000001 0 1761 2 0x00000000 [ 479.949433] ffff88041d195d58 0000000000000002 ffff88041d195fd8 0000000000004000 [ 479.949435] ffff88041d195fd8 ffffffff81a10440 ffff880419fec540 0000000000000000 [ 479.949436] 0000000000000010 ffff88041d195cf8 ffffffff8111a8db 0000805000000287 [ 479.949437] Call Trace: [ 479.949442] [<ffffffff8111a8db>] ? cache_alloc_refill+0x1ab/0x2e0 [ 479.949456] [<ffffffffa0201fe1>] ? btrfs_free_path+0x21/0x30 [btrfs] [ 479.949458] [<ffffffff815927f4>] schedule+0x24/0x70 [ 479.949460] [<ffffffff81590871>] schedule_timeout+0x181/0x1c0 [ 479.949466] [<ffffffffa022811a>] btrfs_commit_transaction+0x22a/0xac0 [btrfs] [ 479.949469] [<ffffffff81059960>] ? abort_exclusive_wait+0xb0/0xb0 [ 479.949475] [<ffffffffa0229043>] ? start_transaction+0x93/0x460 [btrfs] [ 479.949481] [<ffffffffa022024d>] transaction_kthread+0x19d/0x220 [btrfs] [ 479.949486] [<ffffffffa02200b0>] ? btrfs_alloc_root+0x30/0x30 [btrfs] [ 479.949488] [<ffffffff81058e0b>] kthread+0xbb/0xc0 [ 479.949489] [<ffffffff81058d50>] ? kthread_freezable_should_stop+0x60/0x60 [ 479.949491] [<ffffffff8159976c>] ret_from_fork+0x7c/0xb0 [ 479.949493] [<ffffffff81058d50>] ? kthread_freezable_should_stop+0x60/0x60 [ 1009.450789] warning: `VirtualBox' uses 32-bit capabilities (legacy support in use) [ 5847.442443] ------------[ cut here ]------------ [ 5847.442446] kernel BUG at fs/btrfs/ctree.c:3000! [ 5847.442447] invalid opcode: 0000 [#1] PREEMPT SMP [ 5847.442449] Modules linked in: ecryptfs sha256_generic encrypted_keys sha1_generic trusted tpm tpm_bios pci_stub vboxpci(O) vboxnetadp(O) vboxnetflt(O) vboxdrv(O) parport_pc ppdev bluetooth nfsd auth_rpcgss binfmt_misc oid_registry exportfs nfs_acl nfs lockd sunrpc fscache dm_crypt wacom lp parport arc4 rt2800usb rt2x00usb rt2800lib rt2x00lib mac80211 cfg80211 crc_ccitt snd_hda_codec_hdmi nvidia(PO) snd_hda_codec_realtek coretemp snd_usb_audio kvm_intel snd_usbmidi_lib kvm snd_seq_dummy snd_hda_intel snd_hda_codec snd_seq_oss snd_seq_midi snd_seq_midi_event snd_hdsp snd_hwdep mxm_wmi snd_pcm_oss snd_mixer_oss snd_rawmidi microcode snd_seq snd_pcm snd_seq_device snd_page_alloc snd_timer serio_raw snd soundcore lpc_ich btrfs xor zlib_deflate raid6_pq libcrc32c hid_generic usbhid hid ghash_clmulni_intel firewire_ohci firewire_core crc_itu_t r8169 aesni_intel i915 aes_x86_64 glue_helper lrw gf128mul ablk_helper video i2c_algo_bit drm_kms_helper cryptd drm xhci_hcd ahci libahci [ 5847.442479] CPU: 0 PID: 6645 Comm: btrfs-endio-wri Tainted: P O 3.10.3 #4 [ 5847.442480] Hardware name: Gigabyte Technology Co., Ltd. Z68X-UD3H-B3/Z68X-UD3H-B3, BIOS F3 05/13/2011 [ 5847.442481] task: ffff88031d2040c0 ti: ffff88036afb8000 task.ti: ffff88036afb8000 [ 5847.442482] RIP: 0010:[<ffffffffa0205299>] [<ffffffffa0205299>] btrfs_set_item_key_safe+0x149/0x150 [btrfs] [ 5847.442490] RSP: 0018:ffff88036afb9b10 EFLAGS: 00010286 [ 5847.442491] RAX: 00000000ffffffff RBX: 000000000000000e RCX: 0000000b6f8ff000 [ 5847.442492] RDX: 00000000ffffffff RSI: ffff88036afb9c0e RDI: ffff88036afb9aef [ 5847.442492] RBP: ffff88036afb9b58 R08: 0000000000000000 R09: 0000000000000000 [ 5847.442493] R10: 0000000b6f909000 R11: 0000000000000000 R12: ffff88036afb9b1f [ 5847.442494] R13: ffff88038f9480a0 R14: ffff88036afb9c0e R15: ffff88032c9a4560 [ 5847.442495] FS: 0000000000000000(0000) GS:ffff88042fa00000(0000) knlGS:0000000000000000 [ 5847.442496] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5847.442497] CR2: fffff98060fc2000 CR3: 000000033c618000 CR4: 00000000000427f0 [ 5847.442497] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 5847.442498] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 5847.442499] Stack: [ 5847.442500] ffff88041aba7000 8200000000000ce5 6c00000000004f0e 0000000b6f8fa000 [ 5847.442501] ffff88032c9a4560 0000000000006000 0000000000000000 ffff88038f9480a0 [ 5847.442502] 0000000000000ce5 ffff88036afb9c58 ffffffffa023b9e1 00000000004f0e82 [ 5847.442504] Call Trace: [ 5847.442512] [<ffffffffa023b9e1>] __btrfs_drop_extents+0x591/0xb30 [btrfs] [ 5847.442518] [<ffffffffa023ca66>] btrfs_drop_extents+0x66/0xa0 [btrfs] [ 5847.442524] [<ffffffffa022e092>] insert_reserved_file_extent.constprop.66+0x72/0x2b0 [btrfs] [ 5847.442530] [<ffffffffa0229043>] ? start_transaction+0x93/0x460 [btrfs] [ 5847.442536] [<ffffffffa0235200>] btrfs_finish_ordered_io+0x530/0x990 [btrfs] [ 5847.442538] [<ffffffff81046048>] ? try_to_del_timer_sync+0x48/0x60 [ 5847.442540] [<ffffffff810460aa>] ? del_timer_sync+0x4a/0x60 [ 5847.442542] [<ffffffff810457b0>] ? usleep_range+0x40/0x40 [ 5847.442547] [<ffffffffa0235670>] finish_ordered_fn+0x10/0x20 [btrfs] [ 5847.442553] [<ffffffffa025417f>] worker_loop+0x14f/0x540 [btrfs] [ 5847.442555] [<ffffffff81065b2d>] ? default_wake_function+0xd/0x10 [ 5847.442561] [<ffffffffa0254030>] ? btrfs_queue_worker+0x310/0x310 [btrfs] [ 5847.442563] [<ffffffff81058e0b>] kthread+0xbb/0xc0 [ 5847.442565] [<ffffffff81058d50>] ? kthread_freezable_should_stop+0x60/0x60 [ 5847.442566] [<ffffffff8159976c>] ret_from_fork+0x7c/0xb0 [ 5847.442568] [<ffffffff81058d50>] ? kthread_freezable_should_stop+0x60/0x60 [ 5847.442569] Code: 00 00 4c 89 e6 48 98 4c 89 ff 48 8d 04 80 48 8d 54 80 65 e8 9a 29 04 00 4c 89 f6 4c 89 e7 e8 8f f2 ff ff 85 c0 0f 8f 5c ff ff ff <0f> 0b 0f 0b 0f 1f 00 48 b8 00 00 00 00 00 16 00 00 55 48 ba b7 [ 5847.442584] RIP [<ffffffffa0205299>] btrfs_set_item_key_safe+0x149/0x150 [btrfs] [ 5847.442588] RSP <ffff88036afb9b10> [ 5847.442589] ---[ end trace fbc9487dfa341622 ]--- --
btrfsck /dev/sdb1 checking extents checking fs roots root 5 inode 3080015 errors 100 root 5 inode 3080016 errors 100 root 5 inode 3080017 errors 100 root 5 inode 3080018 errors 100 root 5 inode 3080019 errors 100 root 5 inode 3080020 errors 100 root 5 inode 3080021 errors 100 root 5 inode 5181058 errors 80 found 2285968412672 bytes used err is 1 total csum bytes: 2227494076 total tree bytes: 5014478848 total fs tree bytes: 1873141760 btree space waste bytes: 1055234213 file data blocks allocated: 11017304006656 referenced 2265867509760 Btrfs v0.20-rc1
Ok could you create an image of this file system and upload it somewhere so I can try and reproduce? You can do this with btrfs-image -c9 -t4 /dev/whatever somefile.img and upload the image somewhere so I can download it. You can email me (jbacik at fusionio dot com) with the link if you don't feel comfortable posting it in bugzilla. btrfs-image only captures metadata, but sometimes your file names are sensitive too, so if that is the case then use -s which will generate random garbage for your filenames so I can't see your file names either.
... 2.5TB partition ... but i'd happily run a patch with more debugging info
Created attachment 107096 [details] debug patch Yeah that's definitely too big. Here is a debug patch, it will print the key it's trying to add and then dump the leaf contents and then panic, so make sure to capture all of that output. I've put this patch on top of btrfs-next, if you prefer a different kernel version let me know and I can rebase the patch onto that kernel.
Created attachment 107097 [details] debug patch Sorry I uploaded the wrong version of the patch, use this one instead.
can try a patch in the next few days ... but could you send me one that applies on top of 3.10(.3)? i cannot run 3.11, because the nvidia driver currently does not compile
Created attachment 107137 [details] debug patch for 3.10.3 Here's one based on 3.10.3.
just for the record: i haven't been able to reproduce it, although i have been running this patch for some time on top of 3.10.5.
That's ok, just keep chugging along and hopefully we'll catch it.
Josef, I think it makes sense to merge the debugging patch, the crash is random and was hit by various people in the past.
I'm hitting what appears to be the same BUG_ON() during BTRFS_IOC_CLONE_RANGE on a vanilla 3.14.0-rc1 kernel: [ 120.067100] kernel BUG at /home/abuild/rpmbuild/BUILD/kernel-vanilla-3.14.rc1/linux-3.14-rc1/fs/btrfs/file.c:844! [ 120.067102] invalid opcode: 0000 [#1] SMP [ 120.067104] Modules linked in: bnep bluetooth rfkill 6lowpan_iphc xt_tcpudp xt_pkttype xt_LOG xt_limit af_packet ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 ip6table_raw ipt_REJECT iptable_raw xt_CT iptable_filter ip6table_mangle nf_conntrack_netbios_ns nf_conntrack_broadcast nf_conntrack_ipv4 nf_defrag_ipv4 ip_tables xt_conntrack nf_conntrack ip6table_filter ip6_tables x_tables crct10dif_pclmul crc32_pclmul ghash_clmulni_intel joydev aesni_intel hid_generic ablk_helper cryptd lrw gf128mul glue_helper aes_x86_64 sr_mod cdrom hv_utils ata_generic hid_hyperv floppy i2c_piix4 ata_piix serio_raw pcspkr button sg dm_mod autofs4 btrfs raid6_pq xor libcrc32c hv_netvsc hv_storvsc hyperv_keyboard hyperv_fb crc32c_intel processor thermal_sys hv_vmbus scsi_dh_rdac scsi_dh_hp_sw scsi_dh_emc scsi_dh_alua scsi_dh [ 120.067132] CPU: 0 PID: 4599 Comm: cloner Not tainted 3.14.0-rc1-1-vanilla #1 [ 120.067133] Hardware name: Microsoft Corporation Virtual Machine/Virtual Machine, BIOS 090006 05/23/2012 [ 120.067135] task: ffff88000435c190 ti: ffff8800072c8000 task.ti: ffff8800072c8000 [ 120.067136] RIP: 0010:[<ffffffffa00eff7b>] [<ffffffffa00eff7b>] __btrfs_drop_extents+0xc0b/0xc70 [btrfs] [ 120.067154] RSP: 0018:ffff8800072c99e8 EFLAGS: 00010246 [ 120.067155] RAX: 0000000000000000 RBX: ffff880006bc65b0 RCX: 0000000000000000 [ 120.067156] RDX: 00000000ffffffff RSI: 0000000000000129 RDI: ffff880006bc65b0 [ 120.067157] RBP: ffff8800072c9b08 R08: 0000000000001000 R09: ffff8800072c99a0 [ 120.067158] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000 [ 120.067159] R13: 0000000000000014 R14: 0000000000000e0d R15: ffff88002eca07c0 [ 120.067161] FS: 00007f2ab6d8c700(0000) GS:ffff88002c400000(0000) knlGS:0000000000000000 [ 120.067162] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 120.067163] CR2: 00007f2ab62a6760 CR3: 00000000073ff000 CR4: 00000000000006f0 [ 120.067166] Stack: [ 120.067167] ffff8800072c9ad8 ffffffffa00c45f0 ffff88002ffdc6c0 ffffe8ff00000000 [ 120.067169] ffff8800072c9a28 ffffffff8116c1ea ffff8800072c9a48 0000000000000000 [ 120.067171] 000000000101c190 0000000000000000 ffff88002edac380 ffff880006bcbd98 [ 120.067172] Call Trace: [ 120.067183] [<ffffffffa00c45f0>] ? reserve_metadata_bytes+0x70/0x890 [btrfs] [ 120.067186] [<ffffffff8116c1ea>] ? __insert_vmap_area+0x7a/0xd0 [ 120.067189] [<ffffffff811882c6>] ? kmem_cache_alloc+0x176/0x190 [ 120.067203] [<ffffffffa00f0c9f>] btrfs_drop_extents+0x7f/0xb0 [btrfs] [ 120.067216] [<ffffffffa010d652>] btrfs_clone+0x582/0xa60 [btrfs] [ 120.067228] [<ffffffffa010dfd8>] btrfs_ioctl_clone+0x4a8/0x4e0 [btrfs] [ 120.067241] [<ffffffffa0110899>] btrfs_ioctl+0x9e9/0x29c0 [btrfs] [ 120.067244] [<ffffffff811b07ad>] ? path_openat+0xbd/0x650 [ 120.067246] [<ffffffff811b16de>] ? do_filp_open+0x3e/0xa0 [ 120.067249] [<ffffffff811b3d79>] do_vfs_ioctl+0x439/0x500 [ 120.067251] [<ffffffff811ac321>] ? final_putname+0x21/0x50 [ 120.067253] [<ffffffff811ac516>] ? putname+0x26/0x40 [ 120.067255] [<ffffffff811b3ed1>] SyS_ioctl+0x91/0xb0 [ 120.067259] [<ffffffff815d4fbd>] system_call_fastpath+0x1a/0x1f [ 120.067260] Code: 41 89 c0 b9 9a 03 00 00 48 c7 c2 d0 3a 15 a0 41 89 c4 e8 09 fb fb ff c7 85 4c ff ff ff 01 00 00 00 e9 16 fc ff ff 45 85 e4 75 3e <0f> 0b 0f 1f 00 4c 89 ff e8 68 3d fc ff 4c 89 6d 88 c7 85 24 ff [ 120.067276] RIP [<ffffffffa00eff7b>] __btrfs_drop_extents+0xc0b/0xc70 [btrfs] [ 120.067287] RSP <ffff8800072c99e8> [ 120.067289] ---[ end trace 0f26fec1c7b14c44 ]--- 692 int __btrfs_drop_extents(struct btrfs_trans_handle *trans, ... 839 /* 840 * | ---- range to drop ----- | 841 * | -------- extent -------- | 842 */ 843 if (start <= key.offset && end < extent_end) { 844 BUG_ON(extent_type == BTRFS_FILE_EXTENT_INLINE); 845 846 memcpy(&new_key, &key, sizeof(new_key)); I've written up a simple xfstests based reproducer. I'll attach it and send it to the ML.
Created attachment 124641 [details] xfstests reproducer
I am has this bug been fixed or is it still open if it is please let me know what the traces for this bug are and what the issue(s) are assumed to be off your traces. In addition let me known if there are any other major issues with brtfs that need to be fixed as I am using this filesystem and feel it's great in terms of features but needs to be way more stable before I start using it for major production use.
I think this bug can be closed. Filipe addressed the inline extent clone bug via 8039d87d9e473aeb740d4fdbd59b9d2f89b2ced9 .