Created attachment 110491 [details]
Kernel.org format filed infos

Sorry, but I have not idea which part of kernel this bug is related,

so I chose ext4 (becaise in call trace I saw ext4_es_shrink+0xc8/0x140)

I logged several of this almost every day.

Kernel is running on a VMware Esxi 4.1 last patched Host on a HP ML350 G5 server.

Thanks. This looks like a problem in the shrinker for ext4 extent cache. Apparently we burn too much time (the longest stall is over 2 seconds) searching and freeing extents without giving CPU a break. Ideally we should make the shrinker more efficient (since I don't believe it was called with so many entries to free) but if that won't work out easily, we should at least sprinkle some cond_resched() into the shrinker code as a band aid. I'll see what we can do.

Update to kernel 3.11.4-generic, problem still persist, I added another attach (bug-20131010084533.txt)

Created attachment 110571 [details]
Infos about bug evidence.

Created attachment 112191 [details]
Additional evidence, kernel 3.11.6

Jan - its on a vmware virtual platform so it could just be that vmware ran off with the CPU. Can you duplicate this on a native platform, and if you disable the NMI watchdog does the kernel hang or merely stall for a moment in this situation ?

Hello,

what can I do on my side?

I've done the following modifications:

1) Installed latest mainline kernel 3.12

2) Replaced SCSI drivers from vmw_pvsci to LSI Logic SAS (mptsas)

3) reduced number of CPUs from 8 to 4

4) enabled on ESXi (4.1) setting for this machine "CPU Hot add" feature

The issue is still affecting ths machine... other 2 almost equals machine
running on this host (same kernel) but with XFS on root are not affected and very
stable.

I reattached another info file (bug-20131118084007.txt) which contains a lot of infos.

Thanks.

Created attachment 115021 [details]
Additional evidence, kernel 3.12

Hi Loris,

Sorry for my late reply.  I haven't had a chance to take a closer look at this problem until now.

(In reply to Loris Luise from comment #8)
> Hello,
> 
> what can I do on my side?

I am not sure whether you use 'delalloc' on your system.  I don't see these details from the log that you pasted.  So now only I could suggest is that you can try to turn off the delalloc.  The step is as below:

 % sudo mount -t ext4 -o remount,nodelalloc ${DEV} ${MNT}

After this, it won't track the delayed entry on extent status tree.  So it will avoid es shrinker to take too much time to scan delayed entries that couldn't be reclaimed.

Then I will try to reproduce this problem on my sand box and think about how to improve es shrinker.

Thanks,
- Zheng

> 
> I've done the following modifications:
> 
> 1) Installed latest mainline kernel 3.12
> 
> 2) Replaced SCSI drivers from vmw_pvsci to LSI Logic SAS (mptsas)
> 
> 3) reduced number of CPUs from 8 to 4
> 
> 4) enabled on ESXi (4.1) setting for this machine "CPU Hot add" feature
> 
> The issue is still affecting ths machine... other 2 almost equals machine
> running on this host (same kernel) but with XFS on root are not affected and
> very
> stable.
> 
> I reattached another info file (bug-20131118084007.txt) which contains a lot
> of infos.
> 
> Thanks.

Hello,

I had cloned root partition to a XFS one and curently I have no more problems.

Previously I used a mount line in fstab like this

/dev/mapper/h30server2-root /               ext4    errors=remount-ro,noatime,nodiratime,data=writeback 0       1

and later (after your suggestion)

/dev/mapper/h30server2-root /               ext4    errors=remount-ro,noatime,nodiratime,nodelalloc,data=writeback 0       1


but I did not test it with this new option for a sufficient amount of time to be considered valid and come to a conclusion.

(I had to stabilize the system so I was forced to switch to XFS).

I collected more buginfos If they can be usefult I can post them here.

Thanks a lot.

Hi Louis,

Can you talk a bit about your workload?   What sort of applications are you running on the file system which is having the problems?   A database?  Squid cache, something else?   Is it a lot of random writes?    What sort of files does it create?  Does it create and delete a lot of files?   Etc.

It looks like your system has 8GB of memory --- is that right?   How much free memory does your machine normally has?   Also, if you still have your ext4 file system image, could you run "e2freefrag -v /dev/mapper/XXX"?

Many thanks!!