Bug 203317

Just hit this on the newly released kernel 5.2:

```
Jul 08 14:49:04 i87k kernel: i2c i2c-3: NAK from device addr 0x50 msg #0
Jul 08 14:49:22 i87k kernel: WARNING: CPU: 11 PID: 1021 at fs/ext4/inode.c:3925 ext4_set_page_dirty+0x39/0x50
Jul 08 14:49:22 i87k kernel: Modules linked in: msr xt_TCPMSS iptable_mangle iptable_security iptable_nat nf_nat iptable_raw nf_log_ipv4 nf_log_common xt_owner xt_LOG xt_connlimit nf_conncount xt_conntrack nf_conntrack nf_defrag_ipv4 xt_hashlimit xt_multiport xt_addrtype snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic intel_rapl x86_pkg_temp_thermal intel_powerclamp coretemp i915 crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel intel_cstate snd_hda_intel i2c_algo_bit snd_hda_codec drm_kms_helper snd_hwdep snd_hda_core syscopyarea sysfillrect snd_pcm sysimgblt fb_sys_fops snd_timer intel_uncore iTCO_wdt drm snd iTCO_vendor_support pcspkr intel_rapl_perf e1000e soundcore i2c_i801 drm_panel_orientation_quirks xhci_pci xhci_hcd
Jul 08 14:49:22 i87k kernel: CPU: 11 PID: 1021 Comm: Xorg Kdump: loaded Tainted: G     U            5.2.0-g0ecfebd2b524 #16
Jul 08 14:49:22 i87k kernel: Hardware name: System manufacturer System Product Name/PRIME Z370-A, BIOS 1002 07/02/2018
Jul 08 14:49:22 i87k kernel: RIP: 0010:ext4_set_page_dirty+0x39/0x50
Jul 08 14:49:22 i87k kernel: Code: 48 8b 00 a8 01 75 16 48 8b 57 08 48 8d 42 ff 83 e2 01 48 0f 44 c7 48 8b 00 a8 08 74 0d 48 8b 07 f6 c4 20 74 0f e9 87 eb f8 ff <0f> 0b 48 8b 07 f6 c4 20 75 f1 0f 0b e9 76 eb f8 ff 66 0f 1f 44 00
Jul 08 14:49:22 i87k kernel: RSP: 0018:ffffb04a8622bc88 EFLAGS: 00010246
Jul 08 14:49:22 i87k kernel: RAX: 002fffc000002036 RBX: ffff99c48020e3c0 RCX: 0000000000000000
Jul 08 14:49:22 i87k kernel: RDX: 0000000000000000 RSI: 000000076c000000 RDI: ffffdd669db10840
Jul 08 14:49:22 i87k kernel: RBP: ffffdd669db10840 R08: 0000000000000000 R09: 0000000000000000
Jul 08 14:49:22 i87k kernel: R10: 0000000000000000 R11: 0000000000000000 R12: 000000000076c421
Jul 08 14:49:22 i87k kernel: R13: ffff99c4477923c0 R14: ffff99c4896cbbd0 R15: 0000000000000000
Jul 08 14:49:22 i87k kernel: FS:  00007bd2ce71edc0(0000) GS:ffff99c4edac0000(0000) knlGS:0000000000000000
Jul 08 14:49:22 i87k kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Jul 08 14:49:22 i87k kernel: CR2: 000070a6f8001920 CR3: 00000008253e2006 CR4: 00000000003606e0
Jul 08 14:49:22 i87k kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Jul 08 14:49:22 i87k kernel: DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Jul 08 14:49:22 i87k kernel: Call Trace:
Jul 08 14:49:22 i87k kernel:  i915_gem_userptr_put_pages+0x135/0x180 [i915]
Jul 08 14:49:22 i87k kernel:  __i915_gem_object_put_pages+0x56/0x90 [i915]
Jul 08 14:49:22 i87k kernel:  userptr_mn_invalidate_range_start+0x17f/0x210 [i915]
Jul 08 14:49:22 i87k kernel:  __mmu_notifier_invalidate_range_start+0x52/0x90
Jul 08 14:49:22 i87k kernel:  unmap_vmas+0xb3/0xd0
Jul 08 14:49:22 i87k kernel:  unmap_region+0xa3/0x100
Jul 08 14:49:22 i87k kernel:  ? ep_poll+0x29d/0x450
Jul 08 14:49:22 i87k kernel:  __do_munmap+0x1eb/0x450
Jul 08 14:49:22 i87k kernel:  __vm_munmap+0x6a/0xc0
Jul 08 14:49:22 i87k kernel:  __x64_sys_munmap+0x12/0x20
Jul 08 14:49:22 i87k kernel:  do_syscall_64+0x50/0x170
Jul 08 14:49:22 i87k kernel:  entry_SYSCALL_64_after_hwframe+0x44/0xa9
Jul 08 14:49:22 i87k kernel: RIP: 0033:0x7bd2cfd7230b
Jul 08 14:49:22 i87k kernel: Code: ff ff 0f 1f 44 00 00 f7 d8 48 8b 15 7f 8b 0c 00 64 89 02 48 c7 c0 ff ff ff ff e9 76 ff ff ff f3 0f 1e fa b8 0b 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 55 8b 0c 00 f7 d8 64 89 01 48
Jul 08 14:49:22 i87k kernel: RSP: 002b:00007ffe5dcb0e88 EFLAGS: 00000202 ORIG_RAX: 000000000000000b
Jul 08 14:49:22 i87k kernel: RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007bd2cfd7230b
Jul 08 14:49:22 i87k kernel: RDX: 0000000000000000 RSI: 00000000003d8600 RDI: 00007bd2c9c83000
Jul 08 14:49:22 i87k kernel: RBP: 000063b039dfb230 R08: 000063b0394fc014 R09: 0000000000000007
Jul 08 14:49:22 i87k kernel: R10: 000063b039d110a0 R11: 0000000000000202 R12: 0000000001600013
Jul 08 14:49:22 i87k kernel: R13: 000063b039d11138 R14: 0000000000000012 R15: 000063b037c857c0
Jul 08 14:49:22 i87k kernel: ---[ end trace 41ece918a42b1617 ]---
Jul 08 14:49:39 i87k kernel: snd_hda_intel 0000:00:1f.3: PME# enabled
```

`fs/ext4/inode.c`:

```
  static int ext4_set_page_dirty(struct page *page)
  {
    WARN_ON_ONCE(!PageLocked(page) && !PageDirty(page)); // this is line 3925
    WARN_ON_ONCE(!page_has_buffers(page));
    return __set_page_dirty_buffers(page);
  }
```

I'm using ext4 only on zram, like (/etc/fstab):
/dev/zram1                   /tmp        ext4    defaults,auto,rw,exec,nosuid,strictatime,nodev,discard,delalloc,block_validity,user_xattr,nojournal_checksum,barrier 0 0
/dev/zram2               /var/tmp        ext4    defaults,auto,rw,exec,nosuid,strictatime,nodev,discard,delalloc,block_validity,user_xattr,nojournal_checksum,barrier 0 0

```
$ mount|grep ext4
/dev/zram1 on /tmp type ext4 (rw,nosuid,nodev,discard)
/dev/zram2 on /var/tmp type ext4 (rw,nosuid,nodev,discard)
```

my root filesystem is btrfs/zstd:5

Thanks for report! We know about this problem but the solution is a work in progress and probably will take quite a while to sort that out properly.

Got similar here:
[ 3651.932517] WARNING: CPU: 3 PID: 3625 at fs/ext4/inode.c:3925 ext4_set_page_dirty+0x3e/0x50 [ext4]
[ 3651.932522] Modules linked in: fuse ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter nct6775 msr hwmon_vid sunrpc intel_rapl x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm mousedev joydev crct10dif_pclmul crc32_pclmul ghash_clmulni_intel snd_hda_codec_hdmi aesni_intel snd_hda_codec_realtek snd_hda_codec_generic ledtrig_audio aes_x86_64 crypto_simd snd_hda_intel iTCO_wdt iTCO_vendor_support cryptd snd_hda_codec glue_helper snd_hda_core i2c_i801 snd_hwdep snd_pcm snd_timer snd e1000e mxm_wmi soundcore intel_pch_thermal mei_me mei input_leds intel_cstate led_class evdev intel_uncore mac_hid pcc_cpufreq intel_rapl_perf wmi loop sg ip_tables x_tables ext4 crc32c_generic crc16 mbcache jbd2 sd_mod serio_raw atkbd libps2 ahci libahci libata crc32c_intel scsi_mod i8042 serio hid_logitech ff_memless i915 intel_gtt i2c_algo_bit cec rc_core drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops drm agpgart
[ 3651.932597] CPU: 3 PID: 3625 Comm: kworker/u8:3 Not tainted 5.2.1-1-mainline #1
[ 3651.932599] Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./Z170M Extreme4, BIOS P7.20 12/13/2016
[ 3651.932610] Workqueue: writeback wb_workfn (flush-8:0)
[ 3651.932662] RIP: 0010:ext4_set_page_dirty+0x3e/0x50 [ext4]
[ 3651.932669] Code: 48 8b 00 a8 01 75 16 48 8b 57 08 48 8d 42 ff 83 e2 01 48 0f 44 c7 48 8b 00 a8 08 74 0d 48 8b 07 f6 c4 20 74 0f e9 72 75 8d f1 <0f> 0b 48 8b 07 f6 c4 20 75 f1 0f 0b e9 61 75 8d f1 90 0f 1f 44 00
[ 3651.932672] RSP: 0018:ffffb250c86ab700 EFLAGS: 00010246
[ 3651.932677] RAX: 02ffff0000002036 RBX: ffff9ee52c494880 RCX: 0000000000000000
[ 3651.932679] RDX: 0000000000000000 RSI: 0000000206000000 RDI: ffffe7a6481cdc40
[ 3651.932684] RBP: ffffe7a6481cdc40 R08: 0000000206000000 R09: 0000000000000000
[ 3651.932686] R10: ffff9ee55baacb98 R11: 0000000000000000 R12: 0000000000207371
[ 3651.932688] R13: ffff9ee52c7556c0 R14: ffff9ee54e66a790 R15: 0000000000000000
[ 3651.932692] FS:  0000000000000000(0000) GS:ffff9ee56eb80000(0000) knlGS:0000000000000000
[ 3651.932695] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 3651.932697] CR2: 0000556097c527c0 CR3: 0000000159a0a003 CR4: 00000000003606e0
[ 3651.932699] Call Trace:
[ 3651.932888]  i915_gem_userptr_put_pages+0x13f/0x1c0 [i915]
[ 3651.933030]  __i915_gem_object_put_pages+0x5e/0xa0 [i915]
[ 3651.933187]  userptr_mn_invalidate_range_start+0x209/0x260 [i915]
[ 3651.933212]  __mmu_notifier_invalidate_range_start+0x57/0xa0
[ 3651.933229]  page_mkclean_one+0x1b6/0x1d0
[ 3651.933243]  rmap_walk_file+0xf4/0x260
[ 3651.933253]  page_mkclean+0xa4/0xc0
[ 3651.933261]  ? page_referenced_one+0x170/0x170
[ 3651.933269]  ? pmdp_collapse_flush+0x10/0x10
[ 3651.933276]  clear_page_dirty_for_io+0x9c/0x240
[ 3651.933353]  mpage_submit_page+0x1f/0x70 [ext4]
[ 3651.933410]  mpage_process_page_bufs+0xe7/0xf0 [ext4]
[ 3651.933458]  mpage_prepare_extent_to_map+0x1c4/0x280 [ext4]
[ 3651.933513]  ext4_writepages+0x4f7/0xef0 [ext4]
[ 3651.933527]  ? update_blocked_averages+0x894/0xa90
[ 3651.933538]  ? cpumask_next_and+0x19/0x20
[ 3651.933547]  ? do_writepages+0x43/0xd0
[ 3651.933599]  ? ext4_mark_inode_dirty+0x1e0/0x1e0 [ext4]
[ 3651.933606]  do_writepages+0x43/0xd0
[ 3651.933645]  __writeback_single_inode+0x3d/0x3d0
[ 3651.933653]  writeback_sb_inodes+0x1f0/0x430
[ 3651.933666]  __writeback_inodes_wb+0x4c/0xc0
[ 3651.933674]  wb_writeback+0x28f/0x340
[ 3651.933688]  ? get_nr_inodes+0x32/0x50
[ 3651.933695]  wb_workfn+0x3b6/0x480
[ 3651.933714]  process_one_work+0x1d1/0x3e0
[ 3651.933730]  worker_thread+0x4a/0x3d0
[ 3651.933752]  kthread+0xfb/0x130
[ 3651.933761]  ? process_one_work+0x3e0/0x3e0
[ 3651.933774]  ? kthread_park+0x90/0x90
[ 3651.933786]  ret_from_fork+0x35/0x40
[ 3651.933799] ---[ end trace 7a0001b6900a9f55 ]---

Kernel version: 5.2.1

The commit that attempts to fix this(and thus links to this issue) is causing a system freeze (where sysctl+s,u,b still works though) when attempting to cause a memory pressure via script `memfreeze`(ran as variant 1) which I used in order to test [le9g.patch](https://www.phoronix.com/forums/forum/phoronix/general-discussion/1118164-yes-linux-does-bad-in-low-ram-memory-pressure-situations-on-the-desktop?p=1119440#post1119440) that prevents disk thrashing, but this system freeze happens without that patch also.

```
commit aa56a292ce623734ddd30f52d73f527d1f3529b5
    drm/i915/userptr: Acquire the page lock around set_page_dirty()
```

The freeze seems to be happening right before OOM-killer is about to kill something, apparently.

I double checked that v5.3-rc4 with this commit reverted does indeed get rid of the system freeze.

`memfreeze` script is this:

```

#!/bin/bash

#run this multiple times consecutively, but not at once.

#this should freeze i87k for about 1-2 seconds(actually 5 seconds - try running top of watch -n1 -d cat /proc/meminfo) while running out of memory temporarily just before OOM triggers and kills one of the threads:
#^ old comments

#Ensure watchers are running:
ensure_this_runs_already() {
  local cmdtorun="$*"
  if ! pgrep --full "^${cmdtorun}$" >/dev/null; then
    #shellcheck disable=SC2086
    xfce4-terminal -x $cmdtorun  #XXX: unquoted! else it will fail to launch!
  fi
}
ensure_this_runs_already "watch -n.1 -d cat /proc/meminfo"
#ensure_this_runs_already watch -n.1 -d cat /proc/meminfo #yes this works too! I even tested it.
ensure_this_runs_already "top -d 0.5"
ensure_this_runs_already "sudo iotop -d 5 -o -b"
#exit 1


#XXX tested on i87k host
systemctl --user stop psd #otherwise have to manually rename 2+1 profile dirs from *-backup to * after the system crashes(and it does with 5.3.0-rc4-gd45331b00ddb kernel, not with 5.2.0 (git) though, or 5.2.4 (stable) )
if test "${0##*/}" == "memfreeze2"; then
  variant=2
elif test "${0##*/}" == "memfreeze3"; then
  variant=3
else
  variant=1
fi
echo "!! Using memfreeze variant '$variant'"

if test "$1" != "keepswap"; then
  echo "! swapoff first"
  sudo swapoff -a
  threads=2
  timeout=10s
else
  threads=6
  timeout=30s
fi
sync #because possibly crash! in fact guaranteed crash in 5.3.0-rc4-gd45331b00ddb even without any le*.patch-es

if test "$variant" == "1"; then
  #this is a remnant from when I've tested this on QubesOS
  alloc="$(awk '/MemAvailable/{printf "%d\n", $2 + 4000;}' < /proc/meminfo)"
  echo "Will alloc: $alloc kB for each of the $threads concurrent threads."
  echo "MemTotal before: $(awk '/MemTotal/{printf "%d kB\n", $2;}' < /proc/meminfo)"
  time stress --vm-bytes "${alloc}k" --vm-keep -m $threads --timeout $timeout
  echo "exit code: $?"
  awk '/MemTotal/{printf "MemTotal afterwards: %d kB\n", $2;}' < /proc/meminfo
elif test "$variant" == "2"; then
  time stress -m 220 --vm-bytes 10000000000 --timeout 20
elif test "$variant" == "3"; then
  #XXX say bye bye to Xorg, courtesy of kernel's OOM-killer FIXME: somehow disable oom-killer or what?!
  time stress -m 3000 --vm-bytes 10M --timeout 50
else
  echo "!! memfreeze variant not implemented"
fi
```

Just to be clear, this is a regression that commit aa56a292ce623734ddd30f52d73f527d1f3529b5 introduces, which is manifested by a deadlock-like system freeze which is different than the one that le9g.patch is addressing. (all of this is mentioned in prev. comment)

I don't know what the fix is, but temporarily reverting that commit gets rid of the regression.

Please let me know whether I should stick around if you want me to test any new patch about this or not. (because otherwise I won't even check my email/notifications for this account, anymore)

Cheers.

Freeze still present on 5.3.0-rc8 kernel, but only on Intel/i915, not on AMD/Radeon.

I've gotten this freeze during compilation(s).

I've attempted to get a kdump by triggering the freeze via `memfreeze`(script) then sysrq+c to crash kernel, however, I'm unable to (for some reason) read the kdump, but here's what it says anyway: https://gist.github.com/howaboutsynergy/d9818dcf462c071251a236787d8fbea6#file-gistfile1-txt-L1

Note that you filed this bug against the ext4 file system (because of the ext4 warning).   As a result, regressions about the i915 driver are not going to be seen by the i915 driver developers (and ext4 developers will generally ignroe them).

It looks like you are trying to stress test Linux kernels under very high memory pressure, and this is no doubt going to cause multiple issues to pop out.   Please don't conflate them into a single bug, and please note that not all kernel subsystems (not even a majority) prefer to use bugzilla for people to report bugs. 

If you can't find a component in the bugzilla (and perhaps even if you can), you are better off sending a note to the mailing list entry (L:) in the MAINTAINERS file.

But, I 've already tested that reverting that commit gets rid of the freeze. Even if the cause is somewhere else, it appears that the regression is introduced by the very commit I mentioned: aa56a292ce623734ddd30f52d73f527d1f3529b5 (and that commit mentions this issue - which is why I'm posting here)

I mean, the only thing I could try to do is to somehow find a way to get a stacktrace while the system is frozen, but for some reason even manually causing kernel to crash(sysrq+c) fails to allow me to read the dump via 'crash' afterwards ...

Also, I've encountered the freeze during chromium compilation (possibly because it uses at least 15+GiB of 32G of RAM due to jumbo). Even though it's easier for me to trigger the freeze via the 'memfreeze' script. So I expect users will hit this soon. (assuming they use some kind of ext4, I use it on the /tmp and /var/tmp zram)

Not sure how else to say it. I used bisect to find the commit(a while ago), tested with and without it. 

Ahh forgot to mention that what I said in Comment 6 (that freeze/race) happens when the commit is NOT reverted.

But this isn't the same freeze that happens during the memory pressure which simply does a ton of disk reading. I know because I'm already patching against it by keeping Active(file) above 300MiB. (see Comment 4)

I'm happy to try other things to help with this. Please let me know what I could do.

(In reply to Theodore Tso from comment #7)
> you are better off sending a note to the mailing list entry (L:) in the MAINTAINERS file.

ok I did that, thanks! :)

(In reply to howaboutsynergy from comment #9)
> (In reply to Theodore Tso from comment #7)
> > you are better off sending a note to the mailing list entry (L:) in the
> MAINTAINERS file.
> 
> ok I did that, thanks! :)

Where?

If you think this is about i915, and want i915 developers to look, please file bugs at [1], or send email to intel-gfx@lists.freedesktop.org where the likely first response is to ask you to file bugs at [1].

Generally the only action the i915 developers do at bugzilla.kernel.org is to close, and ask bugs to be filed at [1]. You get the picture. ;)


[1] https://bugs.freedesktop.org/enter_bug.cgi?product=DRI&component=DRM/Intel

(In reply to Jani Nikula from comment #10)
> Where?

To: intel-gfx@lists.freedesktop.org

> If you think this is about i915, and want i915 developers to look, please file bugs at [1], or send email to intel-gfx@lists.freedesktop.org where the likely first response is to ask you to file bugs at [1].

The only reason I think it's about i915 is because of:
1. "i915" in the stacktrace in Comment 1
2. it only happens on my Intel(i915) computer, not on Amd/Radeon laptop(I simply cannot reproduce the freeze on the AMD one)

> Generally the only action the i915 developers do at bugzilla.kernel.org is to close, and ask bugs to be filed at [1]. You get the picture. ;)
> [1] https://bugs.freedesktop.org/enter_bug.cgi?product=DRI&component=DRM/Intel


Good info, thanks! I'll file a bug there then. Sorry for the noise.

> Good info, thanks! I'll file a bug there then. Sorry for the noise.

filed: https://bugs.freedesktop.org/show_bug.cgi?id=111601

(In reply to howaboutsynergy from comment #11)
> (In reply to Jani Nikula from comment #10)
> > Where?
> 
> To: intel-gfx@lists.freedesktop.org

Thanks, found it in the moderation queue.

(In reply to howaboutsynergy from comment #12)
> filed: https://bugs.freedesktop.org/show_bug.cgi?id=111601

Thanks.