Created attachment 294621 [details] all-mentionned-logs-and-reproducers-source-code.zip Overview: git clone and other programs that creates a lot of files seems to fail to create folders or files. It's seems only related to cases when there is an interrupted (and restarted) syscall for the calling programm. It happens for for on 2 distinct machines : Host : Microsoft Windows [Version 10.0.19041.329] (seems to happens also on 20H2 update) Virtualbox : Version 6.1.12 r139181 (Qt5.6.2) (seems to happen in 6.0.x too) Guest : Fresh installed Debian 10.7, *NO* VBoxAdditions.iso installed (guest step by step install into guest-debian-install.txt) Steps to see it in my primary use case (git clone) : # Login as root into the VM echo http://deb.debian.org/debian buster-backports main > /etc/apt/sources.list.d/buster-backports.list apt update apt install -t buster-backports linux-image-amd64 apt install git reboot # Login as root again uname -a # Linux debian 5.9.0-0.bpo.2-amd64 #1 SMP Debian 5.9.6-1~bpo10+1 (2020-11-19) x86_64 GNU/Linux mount -t vboxsf /shared /mnt # /mnt should mount without error messages and have 777 permissions # Do once : cd /dev/shm; git clone --bare https://github.com/Ylianst/MeshCentral # this .git is 1 Gio. master is 705 files. This project is a very nice one with pandemic situation. # Do a few times : cd /mnt; git clone /dev/shm/MeshCentral.git 2>&1 | tee -a /dev/shm/git-clone.log Checking out files: 96% (677/705) error: unable to create file views/agentinvite.handlebars: File exists Checking out files: 96% (682/705) [...] fatal: unable to checkout working tree warning: Clone succeeded, but checkout failed. You can inspect what was checked out with 'git status' and retry the checkout with 'git checkout -f HEAD' It's seems racy, it's happens not at any time but 1 or 2 times out of 4 tries for me. It happens with a 1 vCPU and with a 6 vCPU VM without noticable changes. It's "never" on the same file or folder. # Other "tries" : git status 2>&1 | tee /mnt/git-status.log git checkout -f HEAD 2>&1 | tee /mnt/git-checkout.log git checkout -f HEAD 2>&1 | tee /mnt/git-checkout2.log # Not attached, exact same output than git-checkout.log # Can't delete from guest (linux) nor from host (windows) the problematic file even after "umount /mnt" in guest. # I was thinking that explorer.exe could interfere (thumbnails or so), I replay the whole thing with any exeplorer.ee running. It happens to fail in the very same ways. # I enabled on my windows, cmd.exe as administrator : openfiles /local on # I rebooted # I reproduced the failing git clone. It was common.js that is the affected file this time. # In cmd as administrator : openfiles /query /fo table /v > \shared\openfiles-after-3rd-try.log 2>&1 It's show that common.js is open by VirtualboxVM.exe : ID: 4868 Accessed By: username PID: 1352 Process Name: VirtualBoxVM.exe Open File (Path\executable): C:\shared\MeshCentral\common.js I tried umount /mnt on guest, and run again openfiles : it is still opened. I still can't remove the file, even from host, without killing VirtualBoxVM.exe. It seems to break the savestate poweroff functionnality in VirtualBox (VM will be avorted). I wrote dumb reproducers, they start to reproduce the bug easily if there is interrupted syscalls (on open or mkdir at least) in the guest calling program. apt install build-essential cd /mnt gcc -Wall -o r1 reproducer1-mkdir-rmdir.c gcc -Wall -o r2 reproducer2-creat-unlink.c ./r1 2>&1 | tee -a /dev/shm/r1.log Ctrl-Z fg Ctrl-Z fg Ctrl-Z fg Ctrl-Z fg # BANG ! cp /dev/shm/r1.log /mnt r1.log was manually anotated and is attached. Same thing with ./r2 r2.log is not anotated, it goes bad as well, i've done 4 ctrl-Z and fg to get it. I can't have this reproduced with a Debian 10.7 with 4.19.160-2 (2020-11-28) x86_64 stock kernel and VBoxGuestAdditions.iso' modules installed. syscall speed is very similar in the two cases. # In the original VM : timeout 5 strace -c ./r1 > vboxsf-kernel5-r1-syscall-rate-no-int-syscall.log # I've got 15153 syscalls # In a modified copy of the VM to have vboxsf from VBoxGuestAdditions.iso : timeout 5 strace -c ./r1 > vboxsf-kernel4-r1-syscall-rate-no-int-syscall.log # I've got 17277 syscalls Extra : if you while(1) and not while(errcount < 7) you can full DoS VirtualBox. I didn't found any solution to get it alive again without killing VirtualBoxVM.exe. Trying a vm force poweroff from Virtualbox GUI even fails. My guess is : it's related to interrupted system calls, at least for open(...,O_CREAT,...) and mkdir(). If a program on the guest is waiting for one of those syscall to complete and the syscall is interrupted (by a signal or may be in the git use case, because of a multi-threading thing), on the host side, they don't cancel nor rollback. On the guest side an errno=4 (Interrupted system call) is returned which is normal but after that condition any further try to recover is going to fail because of a stale file descriptor on the host in the virtualbox process. Best regards, Ludovic Pouzenc
As already discussed by email, there are really 2 independent issues here: 1. There was a race condition where a wait_on could be interrupted even though it was an in-kernel vbox ipc call. This was causing the errors above and this has been fixed upstream now: https://git.kernel.org/pub/scm/linux/kernel/git/gregkh/char-misc.git/commit/?h=char-misc-next&id=c35901b39ddc20077f4ae7b9f7bf344487f62212 2. When running "git clone" on the guest inside a vbox-shared-folder and the host is a Linux machine then it will fail with -EPERM when trying to create a tmpfile. I've posted a patch-series for the vboxsf fs driver which fixes 2. here is a copy of the cover-letter which explain the problem in more details: Opening a new file is done in 2 steps on regular filesystems: 1. Call the create inode-op on the parent-dir to create an inode to hold the meta-data related to the file. 2. Call the open file-op to get a handle for the file. vboxsf however does not really use disk-backed inodes because it is based on passing through file-related system-calls through to the hypervisor. So both steps translate to an open(2) call being passed through to the hypervisor. With the handle returned by the first call immediately being closed again. Making 2 open calls for a single open(..., O_CREATE, ...) calls has 2 problems: a) It is not really efficient. b) It actually breaks some apps. An example of b) is doing a git clone inside a vboxsf mount. When git clone tries to create a tempfile to store the pak files which is downloading the following happens: 1. vboxsf_dir_mkfile() gets called with a mode of 0444 and succeeds. 2. vboxsf_file_open() gets called with file->f_flags containing O_RDWR. When the host is a Linux machine this fails because doing a open(..., O_RDWR) on a file which exists and has mode 0444 results in an -EPERM error. This series fixes this by adding support for the atomic_open directory-inode op. The patch series fixing 2. is awaiting review.