Bug 81031

Summary: inotify does not send events for a binary that's running
Product: File System Reporter: Eric Radman (ericshane)
Component: VFSAssignee: fs_vfs
Status: NEW ---    
Severity: normal CC: michael2012zhao, pranjas, szg00000
Priority: P1    
Hardware: All   
OS: Linux   
Kernel Version: 3.13.0 Subsystem:
Regression: No Bisected commit-id:

Description Eric Radman 2014-07-24 15:33:56 UTC
If a binary is modified or removed while the program is running IN_ATTRIB is sent by inotify, but all other events block until the program terminates

./inotify-touch
wd=1 mask = 32 (IN_OPEN)           # ./a.out &
wd=1 mask = 4 (IN_ATTRIB)          # rm a.out
wd=1 mask = 16 (IN_CLOSE_NOWRITE)  # pkill a.out
wd=1 mask = 1024 (IN_DELETE_SELF)

This is my simple test program

#include <stdio.h>
#include <unistd.h>
int main(int argc, char *argv[]) {
    printf("started, sleeping for 5 minutes\n");
    sleep(300);
    printf("exiting\n");
    return 0;
}

I'm watching for events using

https://sourceforge.net/p/libkqueue/tickets/_discuss/thread/ab1bb661/b50f/attachment/inotify-touch.c

with a small modification

- #define TEST_FILE "test-file"
+ #define TEST_FILE "a.out"
Comment 1 Eric Radman 2014-07-24 18:52:19 UTC
Also tried and old kernel, 2.6.32 from CentOS 6.5. It has the same behavior

I should add a note about why this matters: I'm the author of entr(1) (http://entrproject.org), and this causes a deadlock in some cases.

I can conceive of a manual workaround by recording and comparing inode numbers, but it would be much easier if inotify(7) events never blocked on a running process. (this problem doesn't exist using kqueue(2) on BSD and MacOS)
Comment 2 PKS 2014-09-10 07:50:34 UTC
I don't think that it is blocking any event. It's just that it's not giving you the one you think it should and this is not related to a process running.

Try the above with tail -f $TEST_FILE. 

It would produce the same result, both IN_CLOSE_NOWRITE and IN_DELETE_SELF show up after you kill tail. 

I believe this has to do with file and dentry refcount. I'm not sure a patch is required since you do get an event when it's deleted but still I'll try to see if it's really needed.
Comment 3 Eric Radman 2014-09-10 16:15:58 UTC
`tail -f` wouldn't provide anything meaningful since it's a binary. Here's another way of illustrating this problem:

In the first test I start `inotifywait` and then remove the file with `rm a.out`

$ inotifywait -m a.out 
Setting up watches.
Watches established.
a.out ATTRIB 
a.out DELETE_SELF 

Now I'll run `./a.out`, start `inotifywait` and then remove the file with `rm a.out`

$ inotifywait -m a.out 
Setting up watches.
Watches established.
a.out ATTRIB 

This shows that DELETE_SELF will only be sent if a.out terminates
Comment 4 PKS 2014-09-15 10:08:58 UTC
Yes you are correct tail on a.out is meaningless but you can try with any text file, the idea is to have an open descriptor while you are trying to remove the file and waiting for DELETE_SELF event to be fired.

I'll try to explain output of your both runs,

Run 1) 
removing a.out or say any file not being accessed currently, on removal will fire DELETE_SELF which you do get.

Run 2)
This time you have an open file descriptor on a.out so when you delete this file, vfs isn't firing the DELETE_SELF it's not holding it either you'll get exactly one DELETE_SELF when that inode is destroyed but ATTRIB is sent out as file's meta data has changed.

The application can trap this ATTRIB event instead. I understand you'll have to check what changed but lstat won't give you anything so your application/script should not be in trouble. If your Run 2 didn't generated any event then it would've been a problem but that's not the case.

Can you try your script/application to check for ATTRIB event instead? It doesn't say which attribute that I understand but it should not be that hard to code in application.
Comment 5 Eric Radman 2014-09-15 22:09:29 UTC
Thanks for the followup. The technique you describe does work; the patch in my application looks like this:

diff -r 9ab56bfde2f1 missing/kqueue_inotify.c
--- a/missing/kqueue_inotify.c  Tue Sep 09 02:25:39 2014 -0400
+++ b/missing/kqueue_inotify.c  Mon Sep 15 17:55:34 2014 -0400
@@ -15,0 +16,1 @@
+#include <sys/stat.h>
+#include <sys/types.h>
@@ -82,0 +85,1 @@
+       struct stat sb;
@@ -95,1 +98,1 @@
-                                   IN_CLOSE_WRITE|IN_DELETE_SELF|IN_MODIFY|IN_MOVE_SELF);
+                                   IN_CLOSE_WRITE|IN_DELETE_SELF|IN_MODIFY|IN_MOVE_SELF|IN_ATTRIB);
@@ -135,0 +139,4 @@
+                       if (iev->mask & IN_ATTRIB) {
+                               if ((fstat(iev->wd, &sb) == -1) && errno == ENOENT)
+                                       fflags |= NOTE_DELETE;
+                       }

To me it seems better if DELETE_SELF was issued immediately, but this workaround is not too intrusive.
Comment 6 Eric Radman 2018-04-20 13:15:52 UTC
On the Linux 3.x I was using `fstat(fd, &sb) == 0` when IN_ATTRIB was received as a means of detecting a file was removed.

In the Linux 4.x series kernel fstat(2) does not fail for a binary that has been unlinked while running.

Here is an example trace of the stat structure for a file on Linux 4.4

stat st_dev: 0x00000801
stat st_ino: 0x00ac0253
stat st_mode: 0x000081ed
stat st_nlink: 0x00000001
stat st_uid: 0x000003ea
stat st_gid: 0x000003eb
stat st_rdev: 0000000000
stat st_size: 0x00007ab0
stat st_atime: 0x5ad9027b
stat st_ctime: 0x5ad90275
stat st_mtime: 0x5ad90275

And a again after the file is unlinked

stat st_dev: 0x0000000e
stat st_ino: 0x0000000c
stat st_mode: 0x00002190
stat st_nlink: 0x00000001
stat st_uid: 0x000003ea
stat st_gid: 0x00000005
stat st_rdev: 0x00008809
stat st_size: 0000000000
stat st_atime: 0x5ad902b0
stat st_ctime: 0x5ad8fcad
stat st_mtime: 0x5ad902b0

To me this almost appears to be uninitialized data. (This may not be wrong; my understanding is that POSIX guarantees that you will get an error if you try to write or read from a closed file, but not much else.)

Nonetheless I am now comparing inode numbers as a substitute for IN_DELETE_SELF.

This works but it would still be much nicer for notifications to work on Linux. BSD and MacOS issue NOTE_DELETE immediately.