Bug 99471

Summary: System locks with kswapd0 and kworker taking full IO and mem
Product: Memory Management Reporter: Gonzalo Aguilar (gaguilar)
Component: Page AllocatorAssignee: Andrew Morton (akpm)
Status: NEW ---    
Severity: blocking CC: 32170745, howaboutsynergy, larsnostdal, meisterplanlos, serianox, sgh, spam, viktorpal, Wilhelm.Buchmueller
Priority: P1    
Hardware: x86-64   
OS: Linux   
Kernel Version: 4.0.4-303.fc22.x86_64 Subsystem:
Regression: No Bisected commit-id:
Attachments: This is situation while blocked
Kernel log with tasks an memory
1 week historic load

Description Gonzalo Aguilar 2015-06-05 17:07:36 UTC
Hi, 

I found a problem with kernel 4.0.3 and later where the system got locked due to full usage because kswapd0 and kworkers taking full cpu (cores) and IO. I see 99% sys on top when I can issue the command. 

It takes about 10 min after reboot to happen. What I have to do is to open firefox, and start eclipse instance, after a while working it start paging and locks. I tried to schedule higher the kswapd process and this way I can get a little info while full. 

Also I tried to disable completely the swap. But the problem is the same and kswapd is in the list of more cpu usage even with no swap. 

I have SSD disk and crypted filesystem but with more and more pressure over IO it got locked anyway.


It's also worth mentioning that I have only 2Gb RAM but I set eclipse to use 512Mb. But anyway it seems to get more than 512Mb. I see it to take 46% of the memory machine. 


Linux azul-laptop.level2crm.com 4.0.4-303.fc22.x86_64 #1 SMP Thu May 28 12:37:06 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux


cat /proc/cpuinfo 
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 15
model name	: Intel(R) Core(TM)2 Duo CPU     T7500  @ 2.20GHz
stepping	: 10
microcode	: 0x95
cpu MHz		: 800.000
cache size	: 4096 KB
physical id	: 0
siblings	: 2
core id		: 0
cpu cores	: 2
apicid		: 0
initial apicid	: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 10
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx lm constant_tsc arch_perfmon pebs bts rep_good nopl aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm lahf_lm ida dtherm tpr_shadow vnmi flexpriority
bugs		:
bogomips	: 4388.95
clflush size	: 64
cache_alignment	: 64
address sizes	: 36 bits physical, 48 bits virtual
power management:

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 15
model name	: Intel(R) Core(TM)2 Duo CPU     T7500  @ 2.20GHz
stepping	: 10
microcode	: 0x95
cpu MHz		: 800.000
cache size	: 4096 KB
physical id	: 0
siblings	: 2
core id		: 1
cpu cores	: 2
apicid		: 1
initial apicid	: 1
fpu		: yes
fpu_exception	: yes
cpuid level	: 10
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx lm constant_tsc arch_perfmon pebs bts rep_good nopl aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm lahf_lm ida dtherm tpr_shadow vnmi flexpriority
bugs		:
bogomips	: 4388.95
clflush size	: 64
cache_alignment	: 64
address sizes	: 36 bits physical, 48 bits virtual
power management:



cat /proc/filesystems 
nodev	sysfs
nodev	rootfs
nodev	ramfs
nodev	bdev
nodev	proc
nodev	cgroup
nodev	cpuset
nodev	tmpfs
nodev	devtmpfs
nodev	debugfs
nodev	securityfs
nodev	sockfs
nodev	pipefs
nodev	configfs
nodev	devpts
	ext3
	ext2
	ext4
nodev	hugetlbfs
nodev	autofs
nodev	pstore
nodev	mqueue
nodev	selinuxfs
nodev	rpc_pipefs
nodev	nfsd
nodev	binfmt_misc
	fuseblk
nodev	fuse
nodev	fusectl

cat /proc/meminfo 
MemTotal:        2040100 kB
MemFree:          186592 kB
MemAvailable:     517364 kB
Buffers:           24904 kB
Cached:           592468 kB
SwapCached:          172 kB
Active:          1117004 kB
Inactive:         559312 kB
Active(anon):     886452 kB
Inactive(anon):   370620 kB
Active(file):     230552 kB
Inactive(file):   188692 kB
Unevictable:          32 kB
Mlocked:              32 kB
SwapTotal:       2097148 kB
SwapFree:        2096328 kB
Dirty:                56 kB
Writeback:             0 kB
AnonPages:       1058804 kB
Mapped:           314120 kB
Shmem:            198128 kB
Slab:              93376 kB
SReclaimable:      48300 kB
SUnreclaim:        45076 kB
KernelStack:        8208 kB
PageTables:        40968 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:     3117196 kB
Committed_AS:    4735668 kB
VmallocTotal:   34359738367 kB
VmallocUsed:      545656 kB
VmallocChunk:   34359186648 kB
HardwareCorrupted:     0 kB
AnonHugePages:         0 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
DirectMap4k:      107328 kB
DirectMap2M:     1980416 kB


mount
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime,seclabel)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
devtmpfs on /dev type devtmpfs (rw,nosuid,seclabel,size=1009056k,nr_inodes=252264,mode=755)
securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev,seclabel)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,seclabel,gid=5,mode=620,ptmxmode=000)
tmpfs on /run type tmpfs (rw,nosuid,nodev,seclabel,mode=755)
tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,seclabel,mode=755)
cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd)
pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime,seclabel)
cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpu,cpuacct)
cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls,net_prio)
cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory)
cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset)
cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer)
cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio)
cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec,relatime,hugetlb)
cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event)
cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices)
configfs on /sys/kernel/config type configfs (rw,relatime)
/dev/mapper/fedora-root on / type ext4 (rw,relatime,seclabel,data=ordered)
selinuxfs on /sys/fs/selinux type selinuxfs (rw,relatime)
systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=30,pgrp=1,timeout=300,minproto=5,maxproto=5,direct)
mqueue on /dev/mqueue type mqueue (rw,relatime,seclabel)
debugfs on /sys/kernel/debug type debugfs (rw,relatime,seclabel)
hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,seclabel)
tmpfs on /tmp type tmpfs (rw,seclabel)
nfsd on /proc/fs/nfsd type nfsd (rw,relatime)
/dev/sda1 on /boot type ext4 (rw,relatime,seclabel,data=ordered)
/dev/mapper/fedora-home on /home type ext4 (rw,relatime,seclabel,data=ordered)
sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime)
binfmt_misc on /proc/sys/fs/binfmt_misc type binfmt_misc (rw,relatime)
tmpfs on /run/user/42 type tmpfs (rw,nosuid,nodev,relatime,seclabel,size=204012k,mode=700,uid=42,gid=42)
tmpfs on /run/user/1000 type tmpfs (rw,nosuid,nodev,relatime,seclabel,size=204012k,mode=700,uid=1000,gid=1000)
gvfsd-fuse on /run/user/1000/gvfs type fuse.gvfsd-fuse (rw,nosuid,nodev,relatime,user_id=1000,group_id=1000)


And the log:




[ 1645.007155] CPU0: Core temperature above threshold, cpu clock throttled (total events = 1)
[ 1645.007803] CPU0: Core temperature/speed normal
[ 1800.701028] mce: [Hardware Error]: Machine check events logged
[ 2162.123944] Purging GPU memory, 368640 bytes freed, 615292928 bytes still pinned.
[ 2175.996060] Purging GPU memory, 499712 bytes freed, 615251968 bytes still pinned.
[ 2175.998841] bash invoked oom-killer: gfp_mask=0x20858, order=0, oom_score_adj=0
[ 2175.998844] bash cpuset=/ mems_allowed=0
[ 2175.998851] CPU: 0 PID: 3838 Comm: bash Not tainted 4.0.3-300.fc22.x86_64 #1
[ 2175.998854] Hardware name: Sony Corporation VGN-SZ61WN_C/VAIO                            , BIOS R0122S5 04/04/2008
[ 2175.998856]  0000000000000000 00000000970ce22f ffff8800037975d8 ffffffff81782254
[ 2175.998860]  0000000000000000 ffff8800794b13e0 ffff880003797668 ffffffff81781748
[ 2175.998863]  ffffffff03797648 fffeefff00000000 0000000000000001 0000000024ac0003
[ 2175.998866] Call Trace:
[ 2175.998874]  [<ffffffff81782254>] dump_stack+0x45/0x57
[ 2175.998878]  [<ffffffff81781748>] dump_header+0x89/0x20f
[ 2175.998882]  [<ffffffff811a3616>] oom_kill_process+0x1e6/0x3c0
[ 2175.998886]  [<ffffffff811a3b26>] __out_of_memory+0x336/0x560
[ 2175.998889]  [<ffffffff811a3f3b>] out_of_memory+0x5b/0x80
[ 2175.998893]  [<ffffffff811a9f4d>] __alloc_pages_nodemask+0x7dd/0xa10
[ 2175.998898]  [<ffffffff811f2df1>] alloc_pages_current+0x91/0x110
[ 2175.998901]  [<ffffffff8119fb5f>] __page_cache_alloc+0xaf/0xd0
[ 2175.998904]  [<ffffffff8119fcf4>] pagecache_get_page+0x84/0x1e0
[ 2175.998908]  [<ffffffff81253f6f>] __getblk_slow+0xcf/0x2d0
[ 2175.998911]  [<ffffffff812541d1>] __getblk_gfp+0x61/0x70
[ 2175.998914]  [<ffffffff812552b6>] __breadahead+0x26/0x70
[ 2175.998919]  [<ffffffff812a990f>] __ext4_get_inode_loc+0x3df/0x3f0
[ 2175.998922]  [<ffffffff812ac996>] ext4_iget+0x96/0xab0
[ 2175.998925]  [<ffffffff812ad3e5>] ext4_iget_normal+0x35/0x40
[ 2175.998928]  [<ffffffff812b744a>] ext4_lookup+0xca/0x160
[ 2175.998932]  [<ffffffff8122706d>] lookup_real+0x1d/0x70
[ 2175.998935]  [<ffffffff81228602>] __lookup_hash+0x42/0x60
[ 2175.998938]  [<ffffffff81228663>] lookup_slow+0x43/0xc0
[ 2175.998941]  [<ffffffff8122d64a>] path_lookupat+0x87a/0xc90
[ 2175.998945]  [<ffffffff811fe695>] ? kmem_cache_alloc+0x1a5/0x230
[ 2175.998948]  [<ffffffff8122e56f>] ? getname_flags+0x4f/0x1c0
[ 2175.998951]  [<ffffffff8122da8a>] filename_lookup+0x2a/0xd0
[ 2175.998954]  [<ffffffff8122f883>] user_path_at_empty+0x63/0xd0
[ 2175.998958]  [<ffffffff812228b6>] ? cp_new_stat+0x156/0x190
[ 2175.998962]  [<ffffffff8122f901>] user_path_at+0x11/0x20
[ 2175.998965]  [<ffffffff8122239a>] vfs_fstatat+0x6a/0xd0
[ 2175.998968]  [<ffffffff8122291e>] SYSC_newstat+0x2e/0x60
[ 2175.998973]  [<ffffffff810141bb>] ? math_state_restore+0xdb/0x1f0
[ 2175.998976]  [<ffffffff81014303>] ? do_device_not_available+0x33/0x70
[ 2175.998980]  [<ffffffff81222aae>] SyS_newstat+0xe/0x10
[ 2175.998984]  [<ffffffff81788909>] system_call_fastpath+0x12/0x17
[ 2175.998986] Mem-Info:
[ 2175.998988] Node 0 DMA per-cpu:
[ 2175.998991] CPU    0: hi:    0, btch:   1 usd:   0
[ 2175.998993] CPU    1: hi:    0, btch:   1 usd:   0
[ 2175.998994] Node 0 DMA32 per-cpu:
[ 2175.998997] CPU    0: hi:  186, btch:  31 usd: 161
[ 2175.998999] CPU    1: hi:  186, btch:  31 usd:  60
[ 2175.999016] active_anon:305425 inactive_anon:141206 isolated_anon:0
                active_file:5109 inactive_file:4666 isolated_file:0
                unevictable:4 dirty:2 writeback:0 unstable:0
                free:13218 slab_reclaimable:6552 slab_unreclaimable:11310
                mapped:21203 shmem:155079 pagetables:10921 bounce:0
                free_cma:0
[ 2175.999021] Node 0 DMA free:8236kB min:348kB low:432kB high:520kB active_anon:2836kB inactive_anon:3408kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:288kB shmem:3696kB slab_reclaimable:196kB slab_unreclaimable:548kB kernel_stack:176kB pagetables:292kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:40256 all_unreclaimable? yes
[ 2175.999028] lowmem_reserve[]: 0 1974 1974 1974
[ 2175.999032] Node 0 DMA32 free:44636kB min:44704kB low:55880kB high:67056kB active_anon:1218864kB inactive_anon:561416kB active_file:20436kB inactive_file:18664kB unevictable:16kB isolated(anon):128kB isolated(file):0kB present:2071360kB managed:2024200kB mlocked:16kB dirty:8kB writeback:0kB mapped:84524kB shmem:616620kB slab_reclaimable:26012kB slab_unreclaimable:44692kB kernel_stack:8160kB pagetables:43392kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:10928244 all_unreclaimable? yes
[ 2175.999038] lowmem_reserve[]: 0 0 0 0
[ 2175.999042] Node 0 DMA: 15*4kB (UEM) 98*8kB (UEM) 88*16kB (UM) 35*32kB (UM) 8*64kB (EM) 2*128kB (EM) 0*256kB 0*512kB 0*1024kB 2*2048kB (UR) 0*4096kB = 8236kB
[ 2175.999056] Node 0 DMA32: 187*4kB (M) 3242*8kB (UM) 314*16kB (EM) 72*32kB (UEMR) 24*64kB (UEMR) 9*128kB (UEMR) 5*256kB (UEMR) 3*512kB (MR) 1*1024kB (R) 0*2048kB 1*4096kB (R) = 44636kB
[ 2175.999072] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
[ 2175.999074] 169619 total pagecache pages
[ 2175.999076] 4752 pages in swap cache
[ 2175.999078] Swap cache stats: add 468915, delete 464163, find 76521/98873
[ 2175.999080] Free swap  = 1615656kB
[ 2175.999082] Total swap = 2097148kB
[ 2175.999083] 521838 pages RAM
[ 2175.999084] 0 pages HighMem/MovableOnly
[ 2175.999086] 11811 pages reserved
[ 2175.999087] 0 pages hwpoisoned
[ 2175.999088] [ pid ]   uid  tgid total_vm      rss nr_ptes nr_pmds swapents oom_score_adj name
[ 2175.999098] [  726]     0   726    13504      495      30       2       66             0 systemd-journal
[ 2175.999101] [  733]     0   733    32319        0      27       2      379             0 lvmetad
[ 2175.999105] [  757]     0   757    11715       72      24       2      604         -1000 systemd-udevd
[ 2175.999110] [  915]     0   915    12266       24      24       2       86         -1000 auditd
[ 2175.999113] [  925]     0   925    20056        0       9       2       42             0 audispd
[ 2175.999116] [  927]     0   927     4196        0      13       2       52             0 alsactl
[ 2175.999120] [  929]     0   929    84733       42      67       3      265             0 ModemManager
[ 2175.999123] [  931]     0   931     5496       13      16       3       31             0 sedispatch
[ 2175.999126] [  934]     0   934    97539       14      45       3      265             0 accounts-daemon
[ 2175.999129] [  935]    81   935    16042      581      30       2      103          -900 dbus-daemon
[ 2175.999132] [  937]     0   937     1877        8       8       3       20             0 gpm
[ 2175.999136] [  941]     0   941   155788       69     158       3      169             0 rsyslogd
[ 2175.999139] [  944]   172   944    41159        4      15       2       46             0 rtkit-daemon
[ 2175.999142] [  946]     0   946     6060       36      15       2       45             0 systemd-logind
[ 2175.999145] [  956]     0   956     9927       70      23       2       69             0 bluetoothd
[ 2175.999149] [  958]     0   958    81543      233      80       3     4159             0 firewalld
[ 2175.999152] [  960]    70   960     6993       35      19       2       34             0 avahi-daemon
[ 2175.999155] [  967]     0   967    89760      140      54       3      339             0 abrtd
[ 2175.999158] [  973]   989   973    29027      149      26       2       70             0 chronyd
[ 2175.999161] [  977]     0   977     1626       26       9       3       10             0 mcelog
[ 2175.999164] [  987]    70   987     6993       10      17       2       48             0 avahi-daemon
[ 2175.999167] [  990]   995   990   133190      294      56       2     1112             0 polkitd
[ 2175.999170] [  996]     0   996    50831       10      38       3      113             0 gssproxy
[ 2175.999174] [ 1017]     0  1017    52777       37      54       2      381             0 abrt-watch-log
[ 2175.999177] [ 1022]     0  1022   148115      279     171       2      396             0 abrt-dump-journ
[ 2175.999180] [ 1076]     0  1076   144815     1115      95       3      344             0 NetworkManager
[ 2175.999189] [ 1111]     0  1111    98875       35      44       2      250             0 gdm
[ 2175.999191] [ 1113]     0  1113     5950        0      16       2       48             0 atd
[ 2175.999195] [ 1117]     0  1117    31444      250      17       2      129             0 crond
[ 2175.999197] [ 1150]     0  1150    12838       58      28       2      148             0 wpa_supplicant
[ 2175.999200] [ 1249]     0  1249    89750       35      60       2      397             0 gdm-session-wor
[ 2175.999203] [ 1297]    42  1297    11249       49      24       2      124             0 systemd
[ 2175.999206] [ 1301]    42  1301    21028      101      41       2      635             0 (sd-pam)
[ 2175.999209] [ 1303]    42  1303    77356        0      36       2      152             0 gdm-wayland-ses
[ 2175.999212] [ 1305]    42  1305    15666      119      30       2      109             0 dbus-daemon
[ 2175.999215] [ 1316]    42  1316   143299       72     107       4      370             0 gnome-session
[ 2175.999217] [ 1345]    42  1345   365420     7438     295       5     7734             0 gnome-shell
[ 2175.999220] [ 1429]    42  1429    80168       13      40       3      190             0 gvfsd
[ 2175.999223] [ 1434]     0  1434    93614       88      61       2      308             0 upowerd
[ 2175.999226] [ 1467]    42  1467    60422       39     106       3     3705             0 Xwayland
[ 2175.999229] [ 1472]    42  1472    65544        0      30       3      149             0 at-spi-bus-laun
[ 2175.999232] [ 1475]    42  1475    15582       32      32       2      118             0 dbus-daemon
[ 2175.999235] [ 1479]    42  1479    31394        5      29       3      166             0 at-spi2-registr
[ 2175.999238] [ 1484]    42  1484   131371       60      79       2      644             0 pulseaudio
[ 2175.999241] [ 1527]     0  1527    30125       92      59       3     3073             0 dhclient
[ 2175.999244] [ 1559]    42  1559   120523       69      53       4      663             0 ibus-daemon
[ 2175.999247] [ 1567]    42  1567   101163       32      47       4      236             0 ibus-dconf
[ 2175.999250] [ 1569]    42  1569   118208        5     113       3      583             0 ibus-x11
[ 2175.999253] [ 1581]     0  1581   172028     1809     125       3     7320             0 packagekitd
[ 2175.999256] [ 1584]    42  1584   243187      497     191       4     1278             0 gnome-settings-
[ 2175.999259] [ 1591]    42  1591   103159      189      52       3      168             0 gvfs-udisks2-vo
[ 2175.999262] [ 1593]     0  1593    92902      194      45       3      178             0 udisksd
[ 2175.999265] [ 1602]    42  1602    81011        0      43       3      204             0 gvfs-mtp-volume
[ 2175.999268] [ 1606]    42  1606    83284        0      47       3      251             0 gvfs-gphoto2-vo
[ 2175.999271] [ 1610]    42  1610   108988        0      62       4      353             0 gvfs-afc-volume
[ 2175.999274] [ 1615]    42  1615    78346        0      39       4      188             0 gvfs-goa-volume
[ 2175.999276] [ 1618]    42  1618   167142       74     159       3     1145             0 goa-daemon
[ 2175.999279] [ 1624]    42  1624    80562      233      58       3      180             0 mission-control
[ 2175.999282] [ 1667]    42  1667    82712       35      46       3      217             0 ibus-engine-sim
[ 2175.999285] [ 1704]   994  1704   101773       35      52       2      462             0 colord
[ 2175.999288] [ 1743]     0  1743    93406       35      68       2      543             0 gdm-session-wor
[ 2175.999291] [ 1754]  1000  1754    11249       48      25       2      124             0 systemd
[ 2175.999294] [ 1759]  1000  1759    21028      103      41       2      634             0 (sd-pam)
[ 2175.999297] [ 1770]  1000  1770    97068      179      39       4       71             0 gnome-keyring-d
[ 2175.999300] [ 1832]  1000  1832    77875        0      36       3      154             0 gdm-x-session
[ 2175.999303] [ 1834]  1000  1834   124780    25443     206       4     1250             0 Xorg
[ 2175.999305] [ 1854]     0  1854     1042        0       7       3       19             0 xf86-video-inte
[ 2175.999308] [ 1876]  1000  1876    15836      312      30       2       88             0 dbus-daemon
[ 2175.999311] [ 1882]  1000  1882   165442      202     115       3      362             0 gnome-session
[ 2175.999314] [ 1939]  1000  1939    65545       35      30       3      147             0 at-spi-bus-laun
[ 2175.999316] [ 1942]  1000  1942    15614       74      29       2       81             0 dbus-daemon
[ 2175.999319] [ 1946]  1000  1946    31391       74      30       3       83             0 at-spi2-registr
[ 2175.999322] [ 1954]  1000  1954    80168       69      42       3      128             0 gvfsd
[ 2175.999325] [ 1958]  1000  1958    76342        0      48       4      241             0 gvfsd-fuse
[ 2175.999328] [ 1979]  1000  1979   277334      827     223       4     1301             0 gnome-settings-
[ 2175.999331] [ 1993]  1000  1993   139762      380      97       3      409             0 pulseaudio
[ 2175.999334] [ 2013]  1000  2013   428298    15574     338       5    11979             0 gnome-shell
[ 2175.999336] [ 2019]     0  2019    53356       19      57       2      356             0 cupsd
[ 2175.999339] [ 2025]  1000  2025   121322        0     112       3      508             0 gsd-printer
[ 2175.999342] [ 2053]  1000  2053   193266       83     200       4     1726             0 gnome-shell-cal
[ 2175.999345] [ 2057]  1000  2057   297794      551     256       4     2583             0 evolution-sourc
[ 2175.999347] [ 2059]  1000  2059   120473       24      52       4      609             0 ibus-daemon
[ 2175.999350] [ 2064]  1000  2064   101166       28      48       3      240             0 ibus-dconf
[ 2175.999353] [ 2066]  1000  2066   114827        0     108       3      508             0 ibus-x11
[ 2175.999356] [ 2077]  1000  2077   365249     3225     204       5     3766             0 goa-daemon
[ 2175.999358] [ 2085]  1000  2085    80597      280      58       3      181             0 mission-control
[ 2175.999361] [ 2089]  1000  2089    44705      118      24       3       57             0 dconf-service
[ 2175.999364] [ 2107]  1000  2107   103159      211      51       3      151             0 gvfs-udisks2-vo
[ 2175.999367] [ 2116]  1000  2116    81011       90      44       3      116             0 gvfs-mtp-volume
[ 2175.999370] [ 2123]  1000  2123    62598       83      39       4      120             0 gvfsd-metadata
[ 2175.999373] [ 2126]  1000  2126    83284       84      46       3      170             0 gvfs-gphoto2-vo
[ 2175.999375] [ 2137]  1000  2137   108988       53      62       4      294             0 gvfs-afc-volume
[ 2175.999378] [ 2152]  1000  2152    82216      104      46       3      143             0 gvfs-goa-volume
[ 2175.999381] [ 2186]  1000  2186   278081       51     246       4     9956             0 evolution-calen
[ 2175.999384] [ 2223]  1000  2223   152367      474     143       2      918             0 abrt-applet
[ 2175.999386] [ 2224]  1000  2224   162874      139      90       4      719             0 tracker-miner-f
[ 2175.999389] [ 2228]  1000  2228   139654        9      77       4      728             0 tracker-miner-a
[ 2175.999392] [ 2241]  1000  2241   155076       86      73       3     1745             0 tracker-store
[ 2175.999395] [ 2258]  1000  2258    77437       48     103       3      529             0 seapplet
[ 2175.999397] [ 2268]  1000  2268   121156        3      74       3      655             0 tracker-miner-u
[ 2175.999400] [ 2271]  1000  2271   266581      205     227       4     2189             0 evolution-alarm
[ 2175.999403] [ 2272]  1000  2272   208207      334     174       4    12450             0 gnome-software
[ 2175.999405] [ 2273]  1000  2273   193787      393     155       3     1135             0 tracker-extract
[ 2175.999408] [ 2288]  1000  2288   425093      264     244       5     4834             0 evolution-calen
[ 2175.999411] [ 2302]  1000  2302    36878       64      27       3       85             0 gconfd-2
[ 2175.999413] [ 2321]  1000  2321   256759       73     209       4     1471             0 evolution-calen
[ 2175.999416] [ 2334]  1000  2334    82712       35      45       3      219             0 ibus-engine-sim
[ 2175.999419] [ 2346]  1000  2346   248895      115     226       4     1520             0 evolution-addre
[ 2175.999422] [ 2415]  1000  2415   279936      118     214       4     1507             0 evolution-addre
[ 2175.999424] [ 2445]  1000  2445   289535      104     215       4     1458             0 evolution-calen
[ 2175.999427] [ 2500]  1000  2500   228855      134     228       4     1490             0 evolution-calen
[ 2175.999430] [ 2552]  1000  2552   151469      349     143       4     1639             0 gnome-terminal-
[ 2175.999432] [ 2606]  1000  2606    59302        0      50       3      277             0 telepathy-logge
[ 2175.999435] [ 2607]  1000  2607     5349        0      16       2       82             0 obexd
[ 2175.999438] [ 2648]  1000  2648     2116       25       9       2       29             0 gnome-pty-helpe
[ 2175.999440] [ 2649]  1000  2649    29531       37      12       2      302             0 bash
[ 2175.999444] [ 2962]  1000  2962   289192    53455     380       4       48             0 firefox
[ 2175.999446] [ 3073]  1000  3073   340123     7522     249       5        0             0 evolution-addre
[ 2175.999449] [ 3189]     0  3189    58396      280      65       2        0             0 sudo
[ 2175.999452] [ 3197]     0  3197    29532      342      13       2        0             0 bash
[ 2175.999454] [ 3523]  1000  3523    28963       72      13       3        0             0 eclipse
[ 2175.999457] [ 3540]  1000  3540   323738     7509     253       4        3             0 evolution-addre
[ 2175.999460] [ 3566]  1000  3566   854261   175914     547       7       11             0 java
[ 2175.999462] [ 3736]    42  3736    44645      139      24       3        0             0 dconf-service
[ 2175.999465] [ 3832]  1000  3832   101453     1166     117       3        0             0 zenity
[ 2175.999468] [ 3837]     0  3837    31966      477      18       2      107             0 crond
[ 2175.999470] [ 3838]     0  3838    28712      658      11       2        0             0 bash
[ 2175.999473] Out of memory: Kill process 3566 (java) score 170 or sacrifice child
[ 2175.999477] Killed process 3566 (java) total-vm:3417044kB, anon-rss:703656kB, file-rss:0kB
[ 2176.000641] bash invoked oom-killer: gfp_mask=0x20858, order=0, oom_score_adj=0
[ 2176.000644] bash cpuset=/ mems_allowed=0
[ 2176.000651] CPU: 0 PID: 3838 Comm: bash Not tainted 4.0.3-300.fc22.x86_64 #1
[ 2176.000653] Hardware name: Sony Corporation VGN-SZ61WN_C/VAIO                            , BIOS R0122S5 04/04/2008
[ 2176.000655]  0000000000000000 00000000970ce22f ffff8800037975d8 ffffffff81782254
[ 2176.000658]  0000000000000000 ffff8800794b13e0 ffff880003797668 ffffffff81781748
[ 2176.000661]  ffffffff03797648 fffeefff00000000 0000000000000001 0000000024ac0003
[ 2176.000664] Call Trace:
[ 2176.000671]  [<ffffffff81782254>] dump_stack+0x45/0x57
[ 2176.000674]  [<ffffffff81781748>] dump_header+0x89/0x20f
[ 2176.000678]  [<ffffffff811a3616>] oom_kill_process+0x1e6/0x3c0
[ 2176.000681]  [<ffffffff811a3b26>] __out_of_memory+0x336/0x560
[ 2176.000686]  [<ffffffff8101265b>] ? __switch_to+0x19b/0x5e0
[ 2176.000689]  [<ffffffff811a3f3b>] out_of_memory+0x5b/0x80
[ 2176.000692]  [<ffffffff811a9f4d>] __alloc_pages_nodemask+0x7dd/0xa10
[ 2176.000697]  [<ffffffff811f2df1>] alloc_pages_current+0x91/0x110
[ 2176.000700]  [<ffffffff8119fb5f>] __page_cache_alloc+0xaf/0xd0
[ 2176.000703]  [<ffffffff8119fcf4>] pagecache_get_page+0x84/0x1e0
[ 2176.000706]  [<ffffffff81253f6f>] __getblk_slow+0xcf/0x2d0
[ 2176.000710]  [<ffffffff812541d1>] __getblk_gfp+0x61/0x70
[ 2176.000713]  [<ffffffff812552b6>] __breadahead+0x26/0x70
[ 2176.000717]  [<ffffffff812a990f>] __ext4_get_inode_loc+0x3df/0x3f0
[ 2176.000720]  [<ffffffff812ac996>] ext4_iget+0x96/0xab0
[ 2176.000723]  [<ffffffff812ad3e5>] ext4_iget_normal+0x35/0x40
[ 2176.000727]  [<ffffffff812b744a>] ext4_lookup+0xca/0x160
[ 2176.000730]  [<ffffffff8122706d>] lookup_real+0x1d/0x70
[ 2176.000733]  [<ffffffff81228602>] __lookup_hash+0x42/0x60
[ 2176.000736]  [<ffffffff81228663>] lookup_slow+0x43/0xc0
[ 2176.000739]  [<ffffffff8122d64a>] path_lookupat+0x87a/0xc90
[ 2176.000743]  [<ffffffff811fe695>] ? kmem_cache_alloc+0x1a5/0x230
[ 2176.000746]  [<ffffffff8122e56f>] ? getname_flags+0x4f/0x1c0
[ 2176.000749]  [<ffffffff8122da8a>] filename_lookup+0x2a/0xd0
[ 2176.000752]  [<ffffffff8122f883>] user_path_at_empty+0x63/0xd0
[ 2176.000756]  [<ffffffff812228b6>] ? cp_new_stat+0x156/0x190
[ 2176.000759]  [<ffffffff8122f901>] user_path_at+0x11/0x20
[ 2176.000762]  [<ffffffff8122239a>] vfs_fstatat+0x6a/0xd0
[ 2176.000765]  [<ffffffff8122291e>] SYSC_newstat+0x2e/0x60
[ 2176.000768]  [<ffffffff810141bb>] ? math_state_restore+0xdb/0x1f0
[ 2176.000772]  [<ffffffff81014303>] ? do_device_not_available+0x33/0x70
[ 2176.000775]  [<ffffffff81222aae>] SyS_newstat+0xe/0x10
[ 2176.000779]  [<ffffffff81788909>] system_call_fastpath+0x12/0x17
[ 2176.000781] Mem-Info:
[ 2176.000782] Node 0 DMA per-cpu:
[ 2176.000785] CPU    0: hi:    0, btch:   1 usd:   0
[ 2176.000787] CPU    1: hi:    0, btch:   1 usd:   0
[ 2176.000788] Node 0 DMA32 per-cpu:
[ 2176.000791] CPU    0: hi:  186, btch:  31 usd: 161
[ 2176.000793] CPU    1: hi:  186, btch:  31 usd:  74
[ 2176.000798] active_anon:305425 inactive_anon:141206 isolated_anon:0
                active_file:5109 inactive_file:4666 isolated_file:0
                unevictable:4 dirty:2 writeback:0 unstable:0
                free:13187 slab_reclaimable:6552 slab_unreclaimable:11310
                mapped:21203 shmem:155079 pagetables:10921 bounce:0
                free_cma:0
[ 2176.000803] Node 0 DMA free:8236kB min:348kB low:432kB high:520kB active_anon:2836kB inactive_anon:3408kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:288kB shmem:3696kB slab_reclaimable:196kB slab_unreclaimable:548kB kernel_stack:176kB pagetables:292kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:40256 all_unreclaimable? yes
[ 2176.000809] lowmem_reserve[]: 0 1974 1974 1974
[ 2176.000813] Node 0 DMA32 free:44512kB min:44704kB low:55880kB high:67056kB active_anon:1218864kB inactive_anon:561416kB active_file:20436kB inactive_file:18664kB unevictable:16kB isolated(anon):0kB isolated(file):0kB present:2071360kB managed:2024200kB mlocked:16kB dirty:8kB writeback:0kB mapped:84524kB shmem:616620kB slab_reclaimable:26012kB slab_unreclaimable:44692kB kernel_stack:8160kB pagetables:43392kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:10929960 all_unreclaimable? yes
[ 2176.000820] lowmem_reserve[]: 0 0 0 0
[ 2176.000823] Node 0 DMA: 15*4kB (UEM) 98*8kB (UEM) 88*16kB (UM) 35*32kB (UM) 8*64kB (EM) 2*128kB (EM) 0*256kB 0*512kB 0*1024kB 2*2048kB (UR) 0*4096kB = 8236kB
[ 2176.000837] Node 0 DMA32: 188*4kB (UM) 3236*8kB (M) 315*16kB (UEM) 69*32kB (UEMR) 24*64kB (UEMR) 9*128kB (UEMR) 5*256kB (UEMR) 3*512kB (MR) 1*1024kB (R) 0*2048kB 1*4096kB (R) = 44512kB
[ 2176.000853] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
[ 2176.000855] 169619 total pagecache pages
[ 2176.000857] 4752 pages in swap cache
[ 2176.000859] Swap cache stats: add 468915, delete 464163, find 76521/98873
[ 2176.000861] Free swap  = 1615656kB
[ 2176.000862] Total swap = 2097148kB
[ 2176.000864] 521838 pages RAM
[ 2176.000865] 0 pages HighMem/MovableOnly
[ 2176.000866] 11811 pages reserved
[ 2176.000868] 0 pages hwpoisoned
[ 2176.000869] [ pid ]   uid  tgid total_vm      rss nr_ptes nr_pmds swapents oom_score_adj name
[ 2176.000875] [  726]     0   726    13504      495      30       2       66             0 systemd-journal
[ 2176.000878] [  733]     0   733    32319        0      27       2      379             0 lvmetad
[ 2176.000882] [  757]     0   757    11715       72      24       2      604         -1000 systemd-udevd
[ 2176.000886] [  915]     0   915    12266       24      24       2       86         -1000 auditd
[ 2176.000889] [  925]     0   925    20056        0       9       2       42             0 audispd
[ 2176.000892] [  927]     0   927     4196        0      13       2       52             0 alsactl
[ 2176.000894] [  929]     0   929    84733       42      67       3      265             0 ModemManager
[ 2176.000897] [  931]     0   931     5496       13      16       3       31             0 sedispatch
[ 2176.000900] [  934]     0   934    97539       14      45       3      265             0 accounts-daemon
[ 2176.000903] [  935]    81   935    16042      581      30       2      103          -900 dbus-daemon
[ 2176.000906] [  937]     0   937     1877        8       8       3       20             0 gpm
[ 2176.000909] [  941]     0   941   155788       69     158       3      169             0 rsyslogd
[ 2176.000912] [  944]   172   944    41159        4      15       2       46             0 rtkit-daemon
[ 2176.000915] [  946]     0   946     6060       36      15       2       45             0 systemd-logind
[ 2176.000918] [  956]     0   956     9927       70      23       2       69             0 bluetoothd
[ 2176.000922] [  958]     0   958    81543      233      80       3     4159             0 firewalld
[ 2176.000925] [  960]    70   960     6993       35      19       2       34             0 avahi-daemon
[ 2176.000927] [  967]     0   967    89760      140      54       3      339             0 abrtd
[ 2176.000930] [  973]   989   973    29027      149      26       2       70             0 chronyd
[ 2176.000933] [  977]     0   977     1626       26       9       3       10             0 mcelog
[ 2176.000936] [  987]    70   987     6993       10      17       2       48             0 avahi-daemon
[ 2176.000938] [  990]   995   990   133190      294      56       2     1112             0 polkitd
[ 2176.000942] [  996]     0   996    50831       10      38       3      113             0 gssproxy
[ 2176.000945] [ 1017]     0  1017    52777       37      54       2      381             0 abrt-watch-log
[ 2176.000948] [ 1022]     0  1022   148115      279     171       2      396             0 abrt-dump-journ
[ 2176.000951] [ 1076]     0  1076   144815     1115      95       3      344             0 NetworkManager
[ 2176.000953] [ 1111]     0  1111    98875       35      44       2      250             0 gdm
[ 2176.000956] [ 1113]     0  1113     5950        0      16       2       48             0 atd
[ 2176.000959] [ 1117]     0  1117    31444      250      17       2      129             0 crond
[ 2176.000962] [ 1150]     0  1150    12838       58      28       2      148             0 wpa_supplicant
[ 2176.000965] [ 1249]     0  1249    89750       35      60       2      397             0 gdm-session-wor
[ 2176.000968] [ 1297]    42  1297    11249       49      24       2      124             0 systemd
[ 2176.000971] [ 1301]    42  1301    21028      101      41       2      635             0 (sd-pam)
[ 2176.000974] [ 1303]    42  1303    77356        0      36       2      152             0 gdm-wayland-ses
[ 2176.000977] [ 1305]    42  1305    15666      119      30       2      109             0 dbus-daemon
[ 2176.000980] [ 1316]    42  1316   143299       72     107       4      370             0 gnome-session
[ 2176.000983] [ 1345]    42  1345   365420     7438     295       5     7734             0 gnome-shell
[ 2176.000986] [ 1429]    42  1429    80168       13      40       3      190             0 gvfsd
[ 2176.000989] [ 1434]     0  1434    93614       88      61       2      308             0 upowerd
[ 2176.000992] [ 1467]    42  1467    60422       39     106       3     3705             0 Xwayland
[ 2176.000995] [ 1472]    42  1472    65544        0      30       3      149             0 at-spi-bus-laun
[ 2176.000998] [ 1475]    42  1475    15582       32      32       2      118             0 dbus-daemon
[ 2176.001001] [ 1479]    42  1479    31394        5      29       3      166             0 at-spi2-registr
[ 2176.001022] [ 1484]    42  1484   131371       60      79       2      644             0 pulseaudio
[ 2176.001025] [ 1527]     0  1527    30125       92      59       3     3073             0 dhclient
[ 2176.001028] [ 1559]    42  1559   120523       69      53       4      663             0 ibus-daemon
[ 2176.001031] [ 1567]    42  1567   101163       32      47       4      236             0 ibus-dconf
[ 2176.001034] [ 1569]    42  1569   118208        5     113       3      583             0 ibus-x11
[ 2176.001037] [ 1581]     0  1581   172028     1809     125       3     7320             0 packagekitd
[ 2176.001040] [ 1584]    42  1584   243187      497     191       4     1278             0 gnome-settings-
[ 2176.001042] [ 1591]    42  1591   103159      189      52       3      168             0 gvfs-udisks2-vo
[ 2176.001046] [ 1593]     0  1593    92902      194      45       3      178             0 udisksd
[ 2176.001048] [ 1602]    42  1602    81011        0      43       3      204             0 gvfs-mtp-volume
[ 2176.001052] [ 1606]    42  1606    83284        0      47       3      251             0 gvfs-gphoto2-vo
[ 2176.001054] [ 1610]    42  1610   108988        0      62       4      353             0 gvfs-afc-volume
[ 2176.001057] [ 1615]    42  1615    78346        0      39       4      188             0 gvfs-goa-volume
[ 2176.001060] [ 1618]    42  1618   167142       74     159       3     1145             0 goa-daemon
[ 2176.001064] [ 1624]    42  1624    80562      233      58       3      180             0 mission-control
[ 2176.001066] [ 1667]    42  1667    82712       35      46       3      217             0 ibus-engine-sim
[ 2176.001070] [ 1704]   994  1704   101773       35      52       2      462             0 colord
[ 2176.001073] [ 1743]     0  1743    93406       35      68       2      543             0 gdm-session-wor
[ 2176.001076] [ 1754]  1000  1754    11249       48      25       2      124             0 systemd
[ 2176.001079] [ 1759]  1000  1759    21028      103      41       2      634             0 (sd-pam)
[ 2176.001083] [ 1770]  1000  1770    97068      179      39       4       71             0 gnome-keyring-d
[ 2176.001085] [ 1832]  1000  1832    77875        0      36       3      154             0 gdm-x-session
[ 2176.001088] [ 1834]  1000  1834   124780    25443     206       4     1250             0 Xorg
[ 2176.001091] [ 1854]     0  1854     1042        0       7       3       19             0 xf86-video-inte
[ 2176.001094] [ 1876]  1000  1876    15836      312      30       2       88             0 dbus-daemon
[ 2176.001097] [ 1882]  1000  1882   165442      202     115       3      362             0 gnome-session
[ 2176.001100] [ 1939]  1000  1939    65545       35      30       3      147             0 at-spi-bus-laun
[ 2176.001103] [ 1942]  1000  1942    15614       74      29       2       81             0 dbus-daemon
[ 2176.001106] [ 1946]  1000  1946    31391       74      30       3       83             0 at-spi2-registr
[ 2176.001109] [ 1954]  1000  1954    80168       69      42       3      128             0 gvfsd
[ 2176.001111] [ 1958]  1000  1958    76342        0      48       4      241             0 gvfsd-fuse
[ 2176.001114] [ 1979]  1000  1979   277334      827     223       4     1301             0 gnome-settings-
[ 2176.001118] [ 1993]  1000  1993   139762      380      97       3      409             0 pulseaudio
[ 2176.001121] [ 2013]  1000  2013   428298    15574     338       5    11979             0 gnome-shell
[ 2176.001124] [ 2019]     0  2019    53356       19      57       2      356             0 cupsd
[ 2176.001127] [ 2025]  1000  2025   121322        0     112       3      508             0 gsd-printer
[ 2176.001130] [ 2053]  1000  2053   193266       83     200       4     1726             0 gnome-shell-cal
[ 2176.001133] [ 2057]  1000  2057   297794      551     256       4     2583             0 evolution-sourc
[ 2176.001136] [ 2059]  1000  2059   120473       24      52       4      609             0 ibus-daemon
[ 2176.001139] [ 2064]  1000  2064   101166       28      48       3      240             0 ibus-dconf
[ 2176.001142] [ 2066]  1000  2066   114827        0     108       3      508             0 ibus-x11
[ 2176.001145] [ 2077]  1000  2077   365249     3225     204       5     3766             0 goa-daemon
[ 2176.001148] [ 2085]  1000  2085    80597      280      58       3      181             0 mission-control
[ 2176.001151] [ 2089]  1000  2089    44705      118      24       3       57             0 dconf-service
[ 2176.001154] [ 2107]  1000  2107   103159      211      51       3      151             0 gvfs-udisks2-vo
[ 2176.001157] [ 2116]  1000  2116    81011       90      44       3      116             0 gvfs-mtp-volume
[ 2176.001160] [ 2123]  1000  2123    62598       83      39       4      120             0 gvfsd-metadata
[ 2176.001162] [ 2126]  1000  2126    83284       84      46       3      170             0 gvfs-gphoto2-vo
[ 2176.001165] [ 2137]  1000  2137   108988       53      62       4      294             0 gvfs-afc-volume
[ 2176.001168] [ 2152]  1000  2152    82216      104      46       3      143             0 gvfs-goa-volume
[ 2176.001172] [ 2186]  1000  2186   278081       51     246       4     9956             0 evolution-calen
[ 2176.001175] [ 2223]  1000  2223   152367      474     143       2      918             0 abrt-applet
[ 2176.001177] [ 2224]  1000  2224   162874      139      90       4      719             0 tracker-miner-f
[ 2176.001180] [ 2228]  1000  2228   139654        9      77       4      728             0 tracker-miner-a
[ 2176.001183] [ 2241]  1000  2241   155076       86      73       3     1745             0 tracker-store
[ 2176.001186] [ 2258]  1000  2258    77437       48     103       3      529             0 seapplet
[ 2176.001189] [ 2268]  1000  2268   121156        3      74       3      655             0 tracker-miner-u
[ 2176.001192] [ 2271]  1000  2271   266581      205     227       4     2189             0 evolution-alarm
[ 2176.001195] [ 2272]  1000  2272   208207      334     174       4    12450             0 gnome-software
[ 2176.001199] [ 2273]  1000  2273   193787      393     155       3     1135             0 tracker-extract
[ 2176.001202] [ 2288]  1000  2288   425093      264     244       5     4834             0 evolution-calen
[ 2176.001205] [ 2302]  1000  2302    36878       64      27       3       85             0 gconfd-2
[ 2176.001208] [ 2321]  1000  2321   256759       73     209       4     1471             0 evolution-calen
[ 2176.001211] [ 2334]  1000  2334    82712       35      45       3      219             0 ibus-engine-sim
[ 2176.001214] [ 2346]  1000  2346   248895      115     226       4     1520             0 evolution-addre
[ 2176.001216] [ 2415]  1000  2415   279936      118     214       4     1507             0 evolution-addre
[ 2176.001219] [ 2445]  1000  2445   289535      104     215       4     1458             0 evolution-calen
[ 2176.001222] [ 2500]  1000  2500   228855      134     228       4     1490             0 evolution-calen
[ 2176.001225] [ 2552]  1000  2552   151469      349     143       4     1639             0 gnome-terminal-
[ 2176.001228] [ 2606]  1000  2606    59302        0      50       3      277             0 telepathy-logge
[ 2176.001231] [ 2607]  1000  2607     5349        0      16       2       82             0 obexd
[ 2176.001234] [ 2648]  1000  2648     2116       25       9       2       29             0 gnome-pty-helpe
[ 2176.001237] [ 2649]  1000  2649    29531       37      12       2      302             0 bash
[ 2176.001240] [ 2962]  1000  2962   289192    53455     380       4       48             0 firefox
[ 2176.001243] [ 3073]  1000  3073   340123     7522     249       5        0             0 evolution-addre
[ 2176.001246] [ 3189]     0  3189    58396      280      65       2        0             0 sudo
[ 2176.001249] [ 3197]     0  3197    29532      342      13       2        0             0 bash
[ 2176.001252] [ 3523]  1000  3523    28963       72      13       3        0             0 eclipse
[ 2176.001255] [ 3540]  1000  3540   323738     7509     253       4        3             0 evolution-addre
[ 2176.001258] [ 3736]    42  3736    44645      139      24       3        0             0 dconf-service
[ 2176.001261] [ 3832]  1000  3832   101453     1166     117       3        0             0 zenity
[ 2176.001263] [ 3837]     0  3837    31966      477      18       2      107             0 crond
[ 2176.001266] [ 3838]     0  3838    28712      658      11       2        0             0 bash
[ 2176.001269] Out of memory: Kill process 2962 (firefox) score 52 or sacrifice child
[ 2176.001274] Killed process 2962 (firefox) total-vm:1156768kB, anon-rss:198532kB, file-rss:15288kB
[root@azul-laptop gaguilar]#
Comment 1 Gonzalo Aguilar 2015-07-14 08:33:30 UTC
It's very strange that linux completly hangs when on memmory pressure. That seems to happen with java 8 but I don't understand why. 

I set swappines to 1 and then 0. Nothing changes. 

I removed swap. Nothing changes. 

The problem with this bug is that the system takes 100% of system usage. Blocks IO by using hd to full. Even without swap!!! How can this be? Kernel is saying ok, I'm without memory, so I will use hard disk (without looking for memory) intesively until I decide to kill the program is taking too much RAM. And selects Java. Why?

The normal behaviour should be: I have no ram. What's best candidate to be killed. Java. Kill. Finish. 

Normally I have to shutdown the computer because is not responsive. And this cannot be. 

I have 2G ram that's not much, but it used to work with Ubuntu 14.10 kernels. 

I attach an screenshot of the computer when completely locked. Without swap.

Situation is awful:

4 kworkers. Taking 75% of the system. 
1 kswap worker. Taking the rest and the IO wait. More than the 24% it shows.
Comment 2 Gonzalo Aguilar 2015-07-14 08:34:06 UTC
Current kernel:

Linux azul-laptop.level2crm.com 4.0.5-300.fc22.x86_64 #1 SMP Mon Jun 8 16:15:26 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
Comment 3 Gonzalo Aguilar 2015-07-14 08:36:44 UTC
Created attachment 182541 [details]
This is situation while blocked

The system is so unresponsive that I have to shudown hard to recover it. Normally kernel should kill offending program before going unresponsive.
Comment 4 Gonzalo Aguilar 2015-07-14 11:28:54 UTC
Created attachment 182571 [details]
Kernel log with tasks an memory

I added everything I thought it was relevant
Comment 5 Gonzalo Aguilar 2015-07-14 11:30:03 UTC
I changed to scheduler since I don't think is related to the filesystem but the scheduler and memory management.
Comment 6 Søren Holm 2015-09-01 12:32:10 UTC
I have the same problem, although it has nothing to do with the amount of memory used. I'm willing to do every possible attempt to track this down. I have tried upgrading the kenrel to 4.1.3, but the same thing happens.

https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1462919

<quoote>
In 15.04 even when having plenty of RAM and swap kswpd uses lots of CPU and might bring the machine down to a crawl making is unable to access it over the network.

I say "might", because it's only a very clear feeling that I have. I do not know it for sure.

Anyway the fact is that I have two server that once every 2 week ends up totally unaccessible from the network. They can be pinged though. Logging in locally takes a long time. Rebooting them locally takes a long time. They have absolutely no noticeable disk IO when in this stalled state.
</quote>
Comment 7 Søren Holm 2015-09-05 14:01:41 UTC
I have cpuload plotted on this site.
http://sgh.dk/~sgh/graphs.html

Notice the system time just climbing after every reboot.
Comment 8 Søren Holm 2015-09-07 21:47:11 UTC
It happens with 4.2.0 too.
Comment 9 Andrew Morton 2015-09-10 21:04:21 UTC
(switched to email.  Please respond via emailed reply-to-all, not via the
bugzilla web interface).

On Tue, 01 Sep 2015 12:32:10 +0000 bugzilla-daemon@bugzilla.kernel.org wrote:

> https://bugzilla.kernel.org/show_bug.cgi?id=99471

Guys, could you take a look please?

The machine went oom when there's heaps of unused swap and most memory
is being used on active_anon and inactive_anon.  We should have just
swapped that stuff out and kept going.
Comment 10 Raymond Jennings 2015-09-12 06:18:25 UTC
On 09/10/15 14:04, Andrew Morton wrote:
> (switched to email.  Please respond via emailed reply-to-all, not via the
> bugzilla web interface).
>
> On Tue, 01 Sep 2015 12:32:10 +0000 bugzilla-daemon@bugzilla.kernel.org wrote:
>
>> https://bugzilla.kernel.org/show_bug.cgi?id=99471
> Guys, could you take a look please?
>
> The machine went oom when there's heaps of unused swap and most memory
> is being used on active_anon and inactive_anon.  We should have just
> swapped that stuff out and kept going.

Isn't there already logic in the kernel that disables OOM if there's 
swap space available?

I saw it once, what happened to it?
>
>
> --
> To unsubscribe, send a message with 'unsubscribe linux-mm' in
> the body to majordomo@kvack.org.  For more info on Linux MM,
> see: http://www.linux-mm.org/ .
> Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
Comment 11 Søren Holm 2015-09-14 06:28:59 UTC
Created attachment 187511 [details]
1 week historic load

It is clear to see the system load "singularity".

The system has 8 physical cores and 12 GB of memory. The sigularity happended on a sunday no activity on the system.
Comment 12 Johannes Weiner 2015-09-15 09:21:46 UTC
On Thu, Sep 10, 2015 at 02:04:18PM -0700, Andrew Morton wrote:
> (switched to email.  Please respond via emailed reply-to-all, not via the
> bugzilla web interface).
> 
> On Tue, 01 Sep 2015 12:32:10 +0000 bugzilla-daemon@bugzilla.kernel.org wrote:
> 
> > https://bugzilla.kernel.org/show_bug.cgi?id=99471
> 
> Guys, could you take a look please?
> 
> The machine went oom when there's heaps of unused swap and most memory
> is being used on active_anon and inactive_anon.  We should have just
> swapped that stuff out and kept going.

I think we need to re-evaluate the way we balance file and anon scan
pressure. It's not just the "not swapping" aspect that bugs me, it's
also the fact that the machine has been thrashing page cache at full
load for *minutes* before signalling the OOM.

SSDs can flush and reload pages quick enough that on memory pressure
there are always reclaimable cache pages and the scanner never goes
after anonymous memory. If anonymous memory does not leave enough room
for page cache to hold the libraries and executables, userspace goes
into a state where it's mostly waiting for cache to become uptodate.

It's a very frustrating problem because it's hard to even detect.

One idea I had to address the LRU balance problem in the past was to
always reclaim the pages in the following order: inactive file, active
file, anon*. As one set becomes empty, go after the next one. If the
workingset code detects cache thrashing, it depends on the refault
distances what to do: if they are smaller than the active file size,
deactivate; if they are bigger than that, but smaller than active file
+ anon, we need to start swapping to alleviate the cache thrashing.

Now, if the refault distances are bigger than active file + anon, no
amount of deactivating and swapping are going to stop the thrashing
and we have to think about triggering OOM. But OOM is drastic and the
refaults might happen at a very slow pace (or, with sparse files, not
require any IO at all) and the system might be completely fine. So in
addition this would require a measure of overall time spent on
thrashing IO, comparable to what Tejun proposed in "[RFD] memory
pressure and sizing problem", where we say if thrashing IO takes up X
percent of all execution time spent, we trigger the OOM killer--not to
free memory, but to reduce the tasks that contribute to the thrashing
and let the remaining tasks make progress, similar to the swap token
or a BSD style memory scheduler.

* we can ignore the difference between inactive and active anon here
  as anon is not aged the same way as the file LRU is aged
Comment 13 Søren Holm 2015-09-20 10:37:39 UTC
FYI. It's still a problem in 4.3-rc1
Comment 14 Viktor Pal 2015-10-14 18:34:31 UTC
I experience the same on a desktop computer (Ubuntu 15.04) maybe once a week.
When this happens the whole machine gets unresponsive immediately and only restarting helps.
It only turns out from the atop logs what happened after the machine was restarted. Atop is configured to run every 30 secs.
It is visible from the atop logs that kswapd0 is eating up all the CPU and IO resources.
Strange thing is that this happens despite there is no swap enabled.

Kernel version:
Linux 3.19.0-30-generic #34-Ubuntu SMP Fri Oct 2 22:08:41 UTC 2015 x86_64 GNU/Linux

If you need any additional info please let me know.
I would really appreciate if this problem could be fixed as it makes the OS very unreliable.

If there is any way (a workaround) to avoid this from happening please share it.
Comment 15 Søren Holm 2015-10-29 09:43:50 UTC
For what it's worth it's not as such a problem in ubuntu 15.10. I do not know if the patched the hell out of that kernel but it's in any case not destructive anymore.

There are however a strange thing about my cpu load graph.

http://sgh.dk/~sgh/system-cpu-ripples.png

As you can see the ripples that where there in period with high cpu load is still there when it's low. The interesting thing is that the ripples are now smaller.
Comment 16 Lars Rune Nøstdal 2015-11-01 01:16:03 UTC
(In reply to Søren Holm from comment #15)
> For what it's worth it's not as such a problem in ubuntu 15.10.

For me this is definitely a problem in 15.10 (4.2.0-17-generic #21-Ubuntu SMP x86_64). The moment the system has some memory pressure (but not much needed; even at 2 GB free of my 4 GB in total) kswapd0 will start using 100% CPU sometimes locking the system down permanently and I have to pull the plug or remove the laptop battery.

..pretty darn lame.
Comment 17 S. Kalteis 2016-01-06 06:50:18 UTC
Still present in 4.3.0 (Linux thinkpad 4.3.0 #2 SMP PREEMPT Mon Nov 9 00:11:27 CET 2015 x86_64 GNU/Linux) and 4.4.0-rc4.
Comment 18 serianox 2016-02-14 06:17:48 UTC
still present
Comment 19 Andrew Morton 2016-02-16 22:42:04 UTC
On Mon, 5 Oct 2015 22:03:46 +0200 Michal Hocko <mhocko@kernel.org> wrote:

> On Tue 15-09-15 10:39:19, Johannes Weiner wrote:
> > On Thu, Sep 10, 2015 at 02:04:18PM -0700, Andrew Morton wrote:
> > > (switched to email.  Please respond via emailed reply-to-all, not via the
> > > bugzilla web interface).
> > > 
> > > On Tue, 01 Sep 2015 12:32:10 +0000 bugzilla-daemon@bugzilla.kernel.org
> wrote:
> > > 
> > > > https://bugzilla.kernel.org/show_bug.cgi?id=99471
> > > 
> > > Guys, could you take a look please?

So this isn't fixed and a number of new reporters (cc'ed) are chiming
in (let's please keep this going via email, not via the bugzilla UI!).

We have various theories but I don't think we've nailed it down yet.

Are any of the reporters able to come up with a set of instructions
which will permit the developers to reproduce this bug locally?

Can we think up a way of adding some form of debug/instrumentation to
the kernel which will permit us to diagnose and fix this?  It could be
something which a tester manually adds or it could be something
permanent, perhaps controlled via a procfs knob.
Comment 20 serianox 2016-02-17 01:46:00 UTC
Hi, I'm just a regular user but i'd like to point out some relevant info on the topic.
fix proposed that partialy solved the problem
http://lkml.iu.edu//hypermail/linux/kernel/1601.2/03564.html
also:
https://bugzilla.kernel.org/show_bug.cgi?id=65201
in this github issue you can see a small program made to easily reproduce this issue
https://github.com/GalliumOS/galliumos-distro/issues/52#issuecomment-174261443
Comment 21 Wilhelm Buchmüller 2017-01-06 04:57:34 UTC
Happens with no swap, no mount, only reading from SSD:
 -> https://bugzilla.kernel.org/show_bug.cgi?id=65201#c50
Comment 22 Adam Wang 2017-08-09 09:16:39 UTC
I have set "swapoff -a" to disable the swap function when this issue is happen.