Bug 217165
| Summary: | NFS cache stops working if accessed from process with PPID of 1 | ||
|---|---|---|---|
| Product: | File System | Reporter: | artlav |
| Component: | NFS | Assignee: | Trond Myklebust (trondmy) |
| Status: | NEW --- | ||
| Severity: | normal | CC: | artlav, chengen.du, regressions |
| Priority: | P1 | ||
| Hardware: | All | ||
| OS: | Linux | ||
| Kernel Version: | 6.2 | Subsystem: | |
| Regression: | No | Bisected commit-id: | |
| Attachments: | Bisect log and reproduction scripts | ||
I am pleased to announce that we have successfully resolved the issue that was affecting the NFS client's access cache.
The commit 029085b8949f5d269ae2bbd14915407dd0c7f902 has been made to complete the new mechanism,
which clears the NFS access cache as soon as the cache timestamp becomes older than the user's login time.
The primary objective of this mechanism is to ensure that the NFS client's access cache does not become stale due to any changes made to the user's group membership on the server after the user has already logged in on the client.
Our investigation revealed that the issue was caused by the timing of initializing the access cache timestamp.
Whenever a user logs in, a new entry is created to replace the original entry in the RB-tree only if the login time is more recent than the cache's timestamp.
However, the timestamp is only set if the entry is not found in the RB-tree, which can lead to the timestamp being undefined when the entry already exists.
If the timestamp is zero, it can result in cache invalidation and a considerable surge in ACCESS operations.
To resolve this issue, the commit 21fd9e8700de86d1169f6336e97d7a74916ed04a was made.
Author: Chengen Du <chengen.du@canonical.com>
Date: Wed Mar 8 16:03:27 2023 +0800
NFS: Correct timing for assigning access cache timestamp
Please give it a try.
Commit 21fd9e8700de86d1169f6336e97d7a74916ed04a appears to have fixed this. (In reply to Chengen Du from comment #1) > To resolve this issue, the commit 21fd9e8700de86d1169f6336e97d7a74916ed04a > was made. Thx for this. That commit has no "CC: <stable@..." tag, hence it not ensured that the stable tree will pick it up (see https://docs.kernel.org/process/stable-kernel-rules.html). If such a backport looks safe to you, could you ask Greg to pick it up for 6.2.y? (In reply to The Linux kernel's regression tracker (Thorsten Leemhuis) from comment #3) > (In reply to Chengen Du from comment #1) > > To resolve this issue, the commit 21fd9e8700de86d1169f6336e97d7a74916ed04a > > was made. > > Thx for this. That commit has no "CC: <stable@..." tag, hence it not ensured > that the stable tree will pick it up (see > https://docs.kernel.org/process/stable-kernel-rules.html). If such a > backport looks safe to you, could you ask Greg to pick it up for 6.2.y? I apologize for my lack of familiarity with the procedure. Trond kindly requested Anna to pick up the commit for 6.2-rcX from the mail, as can be seen in this link (https://lore.kernel.org/lkml/CAPza5qe=e-CkG+j4NYzF24cv5ZDFbheOFfAi+LQ_OW0pZmfc9g@mail.gmail.com/T/). May I ask if it means that the backport process is currently underway? I received an email stating that the patch 'NFS: Correct timing for assigning access cache timestamp' has been added to the 6.2-stable tree. |
Created attachment 303907 [details] Bisect log and reproduction scripts In 6.2 kernel versions an NFS mount would often start behaving as if it was mounted with noac after various triggers commonly encountered in desktop environment. A reliable way i found to reproduce it is to touch a mounted NFS share from a process that has PPID of 1, i.e. disowned and terminal closed (practical examples: XFCE icons and keyboard shortcuts, mc extension handlers). I've bisected it to commit 029085b8949f5d269ae2bbd14915407dd0c7f902 being the first bad one. Author: Chengen Du <chengen.du canonical com> Date: Fri Dec 30 11:04:32 2022 +0800 NFS: Judge the file access cache's timestamp in rcu path To reproduce, see scripts in the attached file (also contains the bisect log). Assuming the NFS is mounted at /mnt/nas, mount it, time how long it takes to ls -la a directory with a few dozen or more files, run test.sh, check the ls time again. If the issue is triggered, the time would be significantly larger and more or less the same as you would get if the share was mounted with noac option. Usually it takes one test.sh run to trigger it, if it's not triggered after 3-5 then it's good. Once triggered the issue persists until the share is unmounted and remounted. Dropping caches (echo 2 > /proc/sys/vm/drop_caches) seems to reset it too. It does not appear to be dependent on the server, NFS version or common mount options. Explicitly specifying "acdirmin=3600,acdirmax=3600" does not help. Running as root does not help. This issue makes NFS annoying to use on fast connections and borderline unusable on slow/high latency connections. As of 6.3-rc1 it is not fixed.