Bug 121241 - AML stack leak - Acpi-State leaks consume all of memory
Summary: AML stack leak - Acpi-State leaks consume all of memory
Status: RESOLVED MOVED
Alias: None
Product: ACPI
Classification: Unclassified
Component: ACPICA-Core (show other bugs)
Hardware: All Linux
: P1 blocking
Assignee: Robert Moore
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2016-06-30 15:50 UTC by Matthew L. Martin
Modified: 2017-03-23 06:15 UTC (History)
4 users (show)

See Also:
Kernel Version: 4.1.8
Subsystem:
Regression: No
Bisected commit-id:


Attachments
requested acpidump file. (4.07 MB, text/plain)
2016-07-21 18:14 UTC, Matthew L. Martin
Details
fio script (1.51 KB, application/octet-stream)
2016-07-21 18:27 UTC, Matthew L. Martin
Details
system dmesg output (161.20 KB, application/x-gzip)
2016-07-22 13:17 UTC, Matthew L. Martin
Details
acpi state debug patch (7.26 KB, patch)
2016-07-22 23:19 UTC, David Box
Details | Diff
debug patch for 4.1 kernel (7.47 KB, patch)
2016-07-25 18:55 UTC, David Box
Details | Diff
grep ACPI /var/log/messages output (752.83 KB, text/plain)
2016-07-26 15:14 UTC, Matthew L. Martin
Details
Print ACPI Methods associated with Acpi-State. (8.18 KB, patch)
2016-07-27 04:02 UTC, David Box
Details | Diff

Description Matthew L. Martin 2016-06-30 15:50:27 UTC

    
Comment 1 Matthew L. Martin 2016-06-30 15:55:42 UTC
Running an fio (fio-2.1.10) job against a software RAID-10 device built from multipathed SSDs causes all memory to be consumed by Acpi-State objects. This is the report from slabtop:

 Active / Total Objects (% used)    : -1033565817 / -1033409379 (100.0%)
 Active / Total Slabs (% used)      : 65344141 / 65345079 (100.0%)
 Active / Total Caches (% used)     : 69 / 122 (56.6%)
 Active / Total Size (% used)       : 256421282.18K / 256452679.19K (100.0%)
 Minimum / Average / Maximum Object : 0.02K / 0.08K / 4096.00K

  OBJS ACTIVE  USE OBJ SIZE  SLABS OBJ/SLAB CACHE SIZE NAME                   
3260693700 3260690441   1%    0.08K 65213874	   50 260855496K Acpi-State
193936 163748  84%    0.03K   1564	124      6256K kmalloc-32

This is the script running the fio jobs:

DEV='/dev/md6'=~ [yY](es)* ]]
DRIVESIZEBYTES=$(($(cat /sys/block/`basename  $DEV`/size) *512))
DRIVESIZEGiB=$(($DRIVESIZEBYTES/1024/1024/1024))e drive fill x 2"
FIO="sudo /opt/clearsky/bin/fio"
echo Drive Size=$DRIVESIZEGiB GiB
        $FIO --readwrite=rw --bs=4k --invalidate=1 --bsrange=4k-512k\
trap "exit 1;" SIGHUP SIGINT SIGTERM --direct=1 --size="$DRIVESIZEGiB"G\
        --group_reporting --ioengine=libaio --numjobs=1\
devs=` ls /dev/md[345] `-minimal $names >> '/tmp/seq_precon_log.txt';
names=""Pass # $n, complete drive fill"
        done
for dev in $devs; do
  names=`echo $names "--name=${dev} --iodepth=128 --filename=${dev}"`;
doneit
fi
echo Conditioning $devs
read -p "Are you sure you want to continue? <y/N> " prompt                                                                                                   
~                                                                                                                                                            
if [[ $prompt =~ [yY](es)* ]]                                                                                                                                
then                                                                                                                                                         
  echo "Starting 512KB sequential write, complete drive fill x 2"                                                                                            
  date                                                                                                                                                       
  n=0; while (( n++ < 2000 )); do                                                                                                                            
        $FIO --readwrite=rw --bs=4k --invalidate=1 --bsrange=4k-512k\                                                                                        
        --rwmixread=33 --end_fsync=0 --direct=1 --size="$DRIVESIZEGiB"G\                                                                                     
        --group_reporting --ioengine=libaio --numjobs=1\                                                                                                     
        --randrepeat=0 --minimal $names >> '/tmp/seq_precon_log.txt';                                                                                        
  echo "Pass # $n, complete drive fill"                                                                                                                      
        done
  date
else
  exit
fi
Comment 2 Matthew L. Martin 2016-06-30 19:59:37 UTC
If --ioengine=libaio is removed the issue goes away. This is the slabtop from the run without ioengine:

Active / Total Objects (% used)    : 1172329 / 1264471 (92.7%)
 Active / Total Slabs (% used)      : 147877 / 148087 (99.9%)
 Active / Total Caches (% used)     : 72 / 113 (63.7%)
 Active / Total Size (% used)       : 1770060.91K / 1782121.35K (99.3%)
 Minimum / Average / Maximum Object : 0.02K / 1.41K / 4096.00K

  OBJS ACTIVE  USE OBJ SIZE  SLABS OBJ/SLAB CACHE SIZE NAME                   
274833 269544  98%    0.10K   7047	 39     28188K buffer_head
203980 168866  82%    0.03K   1645	124      6580K kmalloc-32
121248 112230  92%    0.25K   7578	 16     30312K kmalloc-256
112728 111322  98%    0.07K   2013	 56      8052K Acpi-ParseExt
 97544  97521  99%   16.00K  97544        1   1560704K kmalloc-16384
 78435  77473  98%    0.19K   3735	 21     14940K dentry
 59829  58472  97%    0.12K   1813	 33      7252K kernfs_node_cache
 52794  43897  83%    0.06K    838	 63      3352K kmalloc-64
 30250  28222  93%    0.08K    605	 50      2420K Acpi-State
 26922  26894  99%    0.53K   3846        7     15384K inode_cache
 22785  22366  98%    0.12K    735	 31      2940K kmalloc-96
 22572  16167  71%    0.04K    228       99       912K Acpi-Namespace
 17556  16389  93%    0.19K    836       21      3344K kmalloc-192
 15344  15193  99%    0.50K   1918        8      7672K kmalloc-512

Adding --ioengine=libaio in again shows Acpi-State slab growing without bound.

I was also unable to reproduce this with a similar vdbench script:

*workload:  Single run, 3 raw disks, 0% random, 33% read of 4k blocks at unlimited rate

*HD:	Host Definition
*SD:		Storage Definition
*WD:	Workload Definition
*RD:	Run Definition
*
hd=default,user=root,shell=ssh,jvms=8

sd=sd1,lun=/dev/md3,openflags=o_direct,hitarea=0,range=(0,100),threads=8
sd=sd2,lun=/dev/md4,openflags=o_direct,hitarea=0,range=(0,100),threads=8
sd=sd3,lun=/dev/md5,openflags=o_direct,hitarea=0,range=(0,100),threads=8
wd=wd1,sd=(sd1,sd2,sd3),xfersize=(4096,100),rdpct=33,seekpct=0
rd=run1,wd=wd1,iorate=max,elapsed=1800,warmup=600
Comment 3 Matthew L. Martin 2016-07-05 14:01:43 UTC
Please remove the "'=~ [yY](es)* ]] from the first line of the script in the original comment. It is a cut and paste error.
Comment 4 Matthew L. Martin 2016-07-05 17:08:57 UTC
I upgraded fio:

# ./fio --version
fio-2.12

using ioengine=libaio, the results are the same:

 Active / Total Objects (% used)    : 170404994 / 170852831 (99.7%)
 Active / Total Slabs (% used)      : 3574679 / 3574916 (100.0%)
 Active / Total Caches (% used)     : 70 / 122 (57.4%)
 Active / Total Size (% used)       : 15112939.94K / 15169514.47K (99.6%)
 Minimum / Average / Maximum Object : 0.02K / 0.09K / 4096.00K

  OBJS ACTIVE  USE OBJ SIZE  SLABS OBJ/SLAB CACHE SIZE NAME                   
168280800 168280800  23%    0.08K 3365616	50  13462464K Acpi-State
645918 444330  68%    0.10K  16562	 39     66248K buffer_head
324508 200318  61%    0.03K   2617	124     10468K kmalloc-32
323296 268628  83%    0.25K  20206	 16     80824K kmalloc-256
254331 242560  95%    0.19K  12111	 21     48444K dentry
165528 158701  95%    0.04K   1672	 99      6688K Acpi-Namespace
117306 111044  94%    0.06K   1862	 63      7448K kmalloc-64
115008 115008 100%    0.31K   9584       12     38336K bio-3
112840 111079  98%    0.07K   2015	 56	 8060K Acpi-ParseExt
Comment 5 Matthew L. Martin 2016-07-06 15:17:03 UTC
I eliminated both the MD RAID-10 and the device mapper multi-path to run the test on the raw devices (/dev/sd?) with the same defect:

 Active / Total Caches (% used)     : 72 / 112 (64.3%)
 Active / Total Size (% used)       : 149415419.76K / 149439781.30K (100.0%)
 Minimum / Average / Maximum Object : 0.02K / 0.08K / 4096.00K

  OBJS ACTIVE  USE OBJ SIZE  SLABS OBJ/SLAB CACHE SIZE NAME                   
1890042650 1890042295   0%    0.08K 37800853       50 151203412K Acpi-State
193869 187373  96%    0.10K   4971       39     19884K buffer_head
257052 167477  65%    0.03K   2073      124      8292K kmalloc-32
152192 115624  75%    0.25K   9512	 16     38048K kmalloc-256
112896 111877  99%    0.07K   2016       56      8064K Acpi-ParseExt
 97565  97489  99%   16.00K  97565	  1   1561040K kmalloc-16384
 85932  81944  95%    0.19K   4092	 21     16368K dentry
 57288  53843  93%    0.12K   1736	 33      6944K kernfs_node_cache
 48636  42996  88%    0.06K    772	 63      3088K kmalloc-64
 40572  40403  99%    0.53K   5796	  7     23184K inode_cache
 22847  21776  95%    0.12K    737       31      2948K kmalloc-96
 15225  13729  90%    0.19K    725	 21      2900K kmalloc-192
 15939  13178  82%    0.04K    161       99       644K Acpi-Namespace
 13008  12336  94%    0.50K   1626	  8	 6504K kmalloc-5
Comment 6 Matthew L. Martin 2016-07-06 16:14:45 UTC
Correction to comment 5: MD RAID-10 was eliminated. Multi-path was not. The defect does not appear to be present on the raw devices. The following is the slabtop output after twice as much I/O as shown in comment 5:

 Active / Total Objects (% used)    : 1293649 / 1470729 (88.0%)
 Active / Total Slabs (% used)      : 148968 / 149123 (99.9%)
 Active / Total Caches (% used)     : 67 / 110 (60.9%)
 Active / Total Size (% used)       : 1756033.81K / 1785582.46K (98.3%)
 Minimum / Average / Maximum Object : 0.02K / 1.21K / 4096.00K

  OBJS ACTIVE  USE OBJ SIZE  SLABS OBJ/SLAB CACHE SIZE NAME                   
270777 266887  98%    0.10K   6943	 39     27772K buffer_head
235200 234330  99%    0.08K   4704	 50     18816K Acpi-State
232252 164813  70%    0.03K   1873	124      7492K kmalloc-32
112952 110903  98%    0.07K   2017	 56      8068K Acpi-ParseExt
163200 109900  67%    0.25K  10200       16     40800K kmalloc-256
 97555  97518  99%   16.00K  97555	  1   1560880K kmalloc-16384
 69174  61546  88%    0.19K   3294	 21     13176K dentry
 55143  53400  96%    0.12K   1671	 33	 6684K kernfs_node_cache
 48888  40602  83%    0.06K    776	 63      3104K kmalloc-64
Comment 7 Matthew L. Martin 2016-07-07 21:06:00 UTC
I have demonstrated that the defect is only on the write path.
Comment 8 Robert Moore 2016-07-15 20:46:04 UTC
Please post the acpidump for this machine, thanks.
Comment 9 Matthew L. Martin 2016-07-21 18:14:40 UTC
Created attachment 224461 [details]
requested acpidump file.

As requested.
Comment 10 David Box 2016-07-21 18:22:12 UTC
Thanks. Please also attach your fio script as a file. Even after handling the copy/paste error I cannot run the script and don't want to assume any corrections.
Comment 11 Matthew L. Martin 2016-07-21 18:27:21 UTC
Created attachment 224471 [details]
fio script

As requested.
Comment 12 Robert Moore 2016-07-21 21:13:34 UTC
dmesg would also help.
thanks.
Comment 13 Matthew L. Martin 2016-07-22 13:17:53 UTC
Created attachment 225271 [details]
system dmesg output

As requested
Comment 14 Matthew L. Martin 2016-07-22 13:24:11 UTC
I booted linux 4.6.0 and the fault is not present.
Comment 15 Matthew L. Martin 2016-07-22 14:17:07 UTC
I downloaded and built 4.4.15, the fault did not reproduce.
Comment 16 Robert Moore 2016-07-22 17:42:24 UTC
It would be helpful to us to know the exact platform where this was seen.
thanks.
Comment 17 David Box 2016-07-22 23:19:02 UTC
Created attachment 225531 [details]
acpi state debug patch

This patch will print a number that corresponds to the Acpi-State descriptor being created. Types are enumerated in drivers/acpi/acpica/acobject.h. Look for the prefix ACPI_DESC_TYPE.
Comment 18 David Box 2016-07-22 23:23:23 UTC
Hi Matthew. If you can move to a newer kernel that would be best. If you can't or would like to pursue the root cause if this problem please try the above patch on the problem kernel to see which descriptor is behind the memory problem. Should print to dmesg. You may want to clear dmesg (save it first if desired) before running your script.
Comment 19 Matthew L. Martin 2016-07-25 14:53:07 UTC
I get the following compilation errors when building linux 4.1.8 with your patch:

In file included from include/acpi/acpi.h:61:0,
                 from drivers/acpi/acpica/utstate.c:44:
drivers/acpi/acpica/utstate.c: In function ‘acpi_ut_create_generic_state’:
include/acpi/acoutput.h:213:41: error: too few arguments to function ‘acpi_info’
 #define ACPI_INFO(plist)                acpi_info plist
                                         ^
drivers/acpi/acpica/utstate.c:133:2: note: in expansion of macro ‘ACPI_INFO’
  ACPI_INFO(("Acpi Generic State Created, Type: 0x%x\n",
  ^
In file included from include/acpi/acpi.h:64:0,
                 from drivers/acpi/acpica/utstate.c:44:
include/acpi/acpixf.h:868:5: note: declared here
     acpi_info(const char *module_name,
     ^
include/acpi/acpixf.h:320:2: note: in definition of macro ‘ACPI_MSG_DEPENDENT_RETURN_VOID’
  prototype;
  ^
scripts/Makefile.build:258: recipe for target 'drivers/acpi/acpica/utstate.o' failed
make[3]: *** [drivers/acpi/acpica/utstate.o] Error 1
make[3]: *** Waiting for unfinished jobs....
scripts/Makefile.build:403: recipe for target 'drivers/acpi/acpica' failed
make[2]: *** [drivers/acpi/acpica] Error 2
scripts/Makefile.build:403: recipe for target 'drivers/acpi' failed
make[1]: *** [drivers/acpi] Error 2
Makefile:947: recipe for target 'drivers' failed
make: *** [drivers] Error 2


We are not in a position to move forward to a new kernel at this time. We have not yet targeted our next kernel. We may want to wait until the next new long term support kernel is available.
Comment 20 David Box 2016-07-25 18:55:54 UTC
Created attachment 226361 [details]
debug patch for 4.1 kernel

My apologies. It applies to the current kernel. Simple fix. Compiled tested on 4.1.8.
Comment 21 Matthew L. Martin 2016-07-25 19:29:18 UTC
Here is the BOM for the systems in question:

Description	Quantity
OEM Dell Storage MD1420, 24 HDs 2.5", 2Us (210-ADBU)	1
OEM SHIP,MD1420,DAO (340-AKRE)	1
No Raid, no Controller (405-AAFC)	1
Brand/No-Bezel, 2U, OEM Storage (325-BBIQ)	1
Enclosure Management Module, dual (403-BBGM)	1
Hard Drive Filler 2.5in, single blank (400-AEPR)	10
200GB Solid State Drive SAS Write Intensive MLC 12Gbps 2.5in Hot-plug Drive, PX04SH (400-AMEY)	14
Rack rail, 2Us, Static (770-BBJE)	1
12Gb HD-Mini to HD-Mini SAS cable, 0.5m (470-ABDN)	4
Solutions Program Support (927-3179)	1
	
	
	
Description	Quantity
OEM PowerEdge R630 (210-ADCY)	1
PowerEdge R630 Motherboard MLK (329-BCZI)	1
OEM PowerEdge R630 Shipping-8 Drive Chassis (340-AKRZ)	1
Intel X710 Quad Port 10Gb DA/SFP+ Ethernet, Network Daughter Card (555-BCKP)	1
SAS 12Gbps HBA External Controller, Low Profile (405-AAFB)	1
iDRAC8 Enterprise, integrated Dell Remote Access Controller, Enterprise (385-BBHO)	1
Chassis with up to 8, 2.5" Hard Drives, 3 PCIe Slots (321-BBKL)	1
Brand/No-Bezel, OEM PowerEdge R630, 8 Drive Chassis (325-BBJC)	1
Performance BIOS Settings (384-BBBL)	1
RAID 1 for H330/H730/H730P (2 HDDs or SSDs) (780-BBJK)	1
PERC H330 Integrated RAID Controller (405-AAEF)	1
Intel Xeon E5-2680 v4 2.4GHz,35M Cache,9.60GT/s QPI,Turbo,HT,14C/28T (120W) Max Mem 2400MHz (338-BJDO)	1
Intel Xeon E5-2680 v4 2.4GHz,35M Cache,9.60GT/s QPI,Turbo,HT,14C/28T (120W) Max Mem 2400MHz (338-BJEE)	1
32GB RDIMM, 2400MT/s, Dual Rank, x4 Data Width (370-ACNS)	8
2400MT/s RDIMMs (370-ACPH)	1
Performance Optimized (370-AAIP)	1
200GB Solid State Drive SATA Mix Use MLC 6Gpbs 2.5in Hot-plug Drive, S3610 (400-AIFV)	2
Description	Quantity
OEM Dell Storage MD1420, 24 HDs 2.5", 2Us (210-ADBU)	1
Dell Hardware Limited Warranty Plus On Site Service Initial 15 Months (977-8949)	1
Dell Hardware Limited Warranty Plus On Site Service Extended Months(s) (977-8950)	1
ProSupport : Next Business Day Onsite Service After Problem Diagnosis, 15 Months (977-8952)	1
ProSupport : Next Business Day Onsite Service After Problem Diagnosis, 24 Months Extended (977-8953)	1
ProSupport : 7x24 Technical Support, 39 Months (977-8966)	1
Thank you choosing Dell ProSupport. For tech support, visit http://www.dell.com/support or call 1-800- 945-3355 (989-3439)	1
On-Site Installation Declined (900-9997)	1
Declined Remote Consulting Service (973-2426)	1
US Order (332-1286)	1
OEM SHIP,MD1420,DAO (340-AKRE)	1
No Raid, no Controller (405-AAFC)	1
Brand/No-Bezel, 2U, OEM Storage (325-BBIQ)	1
Enclosure Management Module, dual (403-BBGM)	1
Hard Drive Filler 2.5in, single blank (400-AEPR)	10
200GB Solid State Drive SAS Write Intensive MLC 12Gbps 2.5in Hot-plug Drive, PX04SH (400-AMEY)	14
Rack rail, 2Us, Static (770-BBJE)	1
OEM MD1420 Regulatory label, Mexico (389-BEVW)	1
Power supply, AC 600W, Redundant (450-AEBJ)	1
C13 to C14, PDU Style, 12 AMP, 6.5 Feet (2m) Power Cord, North America (492-BBDI)	1
C13 to C14, PDU Style, 12 AMP, 6.5 Feet (2m) Power Cord, North America (492-BBDI)	1
12Gb HD-Mini to HD-Mini SAS cable, 0.5m (470-ABDN)	4
Solutions Program Support (927-3179)	1
	
	
	
Description	Quantity
OEM PowerEdge R630 (210-ADCY)	1
PowerEdge R630 Motherboard MLK (329-BCZI)	1
ProSupport : Next Business Day Onsite Service After Problem Diagnosis, 39 Months (976-9165)	1
ProSupport : 7x24 Technical Support, 39 Months (976-9172)	1
Dell Hardware Limited Warranty Plus On Site Service (976-9191)	1
Thank you choosing Dell ProSupport. For tech support, visit http://www.dell.com/support or call 1-800- 945-3355 (989-3439)	1
US Order (332-1286)	1
OEM PowerEdge R630 Shipping-8 Drive Chassis (340-AKRZ)	1
Intel X710 Quad Port 10Gb DA/SFP+ Ethernet, Network Daughter Card (555-BCKP)	1
SAS 12Gbps HBA External Controller, Low Profile (405-AAFB)	1
iDRAC8 Enterprise, integrated Dell Remote Access Controller, Enterprise (385-BBHO)	1
Chassis with up to 8, 2.5" Hard Drives, 3 PCIe Slots (321-BBKL)	1
Brand/No-Bezel, OEM PowerEdge R630, 8 Drive Chassis (325-BBJC)	1
Performance BIOS Settings (384-BBBL)	1
RAID 1 for H330/H730/H730P (2 HDDs or SSDs) (780-BBJK)	1
PERC H330 Integrated RAID Controller (405-AAEF)	1
Intel Xeon E5-2680 v4 2.4GHz,35M Cache,9.60GT/s QPI,Turbo,HT,14C/28T (120W) Max Mem 2400MHz (338-BJDO)	1
Intel Xeon E5-2680 v4 2.4GHz,35M Cache,9.60GT/s QPI,Turbo,HT,14C/28T (120W) Max Mem 2400MHz (338-BJEE)	1
32GB RDIMM, 2400MT/s, Dual Rank, x4 Data Width (370-ACNS)	8
2400MT/s RDIMMs (370-ACPH)	1
Performance Optimized (370-AAIP)	1
200GB Solid State Drive SATA Mix Use MLC 6Gpbs 2.5in Hot-plug Drive, S3610 (400-AIFV)	2
PE Server FIPS TPM 1.2,CC (461-AADP)	1
No Systems Documentation, No OpenManage DVD Kit (631-AACK)	1
No Optical Drive Internal for 8 HD Chassis (429-AAQN)	1
ReadyRails Sliding Rails Without Cable Management Arm (770-BBBC)	1
Dual, Hot-plug, Redundant Power Supply (1+1), 495W (450-ADWQ)	1
C13 to C14, PDU Style, 12 AMP, 6.5 Feet (2m) Power Cord, North America (492-BBDI)	2
No Operating System (619-ABVR)	1
No Media Required (421-5736)	1
DIMM Blanks for System with 2 Processors (370-ABWE)	1
120W Heatsink for PowerEdge R630 (412-AAEE)	1
120W Heatsink for PowerEdge R630 (412-AAEE)	1
Solutions Program Support (927-3179)	1Description	Quantity
OEM Dell Storage MD1420, 24 HDs 2.5", 2Us (210-ADBU)	1
Dell Hardware Limited Warranty Plus On Site Service Initial 15 Months (977-8949)	1
Dell Hardware Limited Warranty Plus On Site Service Extended Months(s) (977-8950)	1
ProSupport : Next Business Day Onsite Service After Problem Diagnosis, 15 Months (977-8952)	1
ProSupport : Next Business Day Onsite Service After Problem Diagnosis, 24 Months Extended (977-8953)	1
ProSupport : 7x24 Technical Support, 39 Months (977-8966)	1
Thank you choosing Dell ProSupport. For tech support, visit http://www.dell.com/support or call 1-800- 945-3355 (989-3439)	1
On-Site Installation Declined (900-9997)	1
Declined Remote Consulting Service (973-2426)	1
US Order (332-1286)	1
OEM SHIP,MD1420,DAO (340-AKRE)	1
No Raid, no Controller (405-AAFC)	1
Brand/No-Bezel, 2U, OEM Storage (325-BBIQ)	1
Enclosure Management Module, dual (403-BBGM)	1
Hard Drive Filler 2.5in, single blank (400-AEPR)	10
200GB Solid State Drive SAS Write Intensive MLC 12Gbps 2.5in Hot-plug Drive, PX04SH (400-AMEY)	14
Rack rail, 2Us, Static (770-BBJE)	1
OEM MD1420 Regulatory label, Mexico (389-BEVW)	1
Power supply, AC 600W, Redundant (450-AEBJ)	1
C13 to C14, PDU Style, 12 AMP, 6.5 Feet (2m) Power Cord, North America (492-BBDI)	1
C13 to C14, PDU Style, 12 AMP, 6.5 Feet (2m) Power Cord, North America (492-BBDI)	1
12Gb HD-Mini to HD-Mini SAS cable, 0.5m (470-ABDN)	4
Solutions Program Support (927-3179)	1
	
	
	
Description	Quantity
OEM PowerEdge R630 (210-ADCY)	1
PowerEdge R630 Motherboard MLK (329-BCZI)	1
ProSupport : Next Business Day Onsite Service After Problem Diagnosis, 39 Months (976-9165)	1
ProSupport : 7x24 Technical Support, 39 Months (976-9172)	1
Dell Hardware Limited Warranty Plus On Site Service (976-9191)	1
Thank you choosing Dell ProSupport. For tech support, visit http://www.dell.com/support or call 1-800- 945-3355 (989-3439)	1
US Order (332-1286)	1
OEM PowerEdge R630 Shipping-8 Drive Chassis (340-AKRZ)	1
Intel X710 Quad Port 10Gb DA/SFP+ Ethernet, Network Daughter Card (555-BCKP)	1
SAS 12Gbps HBA External Controller, Low Profile (405-AAFB)	1
iDRAC8 Enterprise, integrated Dell Remote Access Controller, Enterprise (385-BBHO)	1
Chassis with up to 8, 2.5" Hard Drives, 3 PCIe Slots (321-BBKL)	1
Brand/No-Bezel, OEM PowerEdge R630, 8 Drive Chassis (325-BBJC)	1
Performance BIOS Settings (384-BBBL)	1
RAID 1 for H330/H730/H730P (2 HDDs or SSDs) (780-BBJK)	1
PERC H330 Integrated RAID Controller (405-AAEF)	1
Intel Xeon E5-2680 v4 2.4GHz,35M Cache,9.60GT/s QPI,Turbo,HT,14C/28T (120W) Max Mem 2400MHz (338-BJDO)	1
Intel Xeon E5-2680 v4 2.4GHz,35M Cache,9.60GT/s QPI,Turbo,HT,14C/28T (120W) Max Mem 2400MHz (338-BJEE)	1
32GB RDIMM, 2400MT/s, Dual Rank, x4 Data Width (370-ACNS)	8
2400MT/s RDIMMs (370-ACPH)	1
Performance Optimized (370-AAIP)	1
200GB Solid State Drive SATA Mix Use MLC 6Gpbs 2.5in Hot-plug Drive, S3610 (400-AIFV)	2
PE Server FIPS TPM 1.2,CC (461-AADP)	1
No Systems Documentation, No OpenManage DVD Kit (631-AACK)	1
No Optical Drive Internal for 8 HD Chassis (429-AAQN)	1
ReadyRails Sliding Rails Without Cable Management Arm (770-BBBC)	1
Dual, Hot-plug, Redundant Power Supply (1+1), 495W (450-ADWQ)	1
C13 to C14, PDU Style, 12 AMP, 6.5 Feet (2m) Power Cord, North America (492-BBDI)	2
No Operating System (619-ABVR)	1
No Media Required (421-5736)	1
DIMM Blanks for System with 2 Processors (370-ABWE)	1
120W Heatsink for PowerEdge R630 (412-AAEE)	1
120W Heatsink for PowerEdge R630 (412-AAEE)	1
Solutions Program Support (927-3179)	1
Comment 22 Matthew L. Martin 2016-07-26 15:14:59 UTC
Created attachment 226501 [details]
grep ACPI /var/log/messages output

It appears that messages were rate limited in /var/log/messages. There were 17094100 objects created and 99% were in use. Over time the percent used drops as shown in previous comments
Comment 23 Robert Moore 2016-07-26 19:25:13 UTC
This output is very useful. It will help us to develop another patch that tells us exactly what control method(s) are executing.
thanks.
Comment 24 David Box 2016-07-27 04:02:51 UTC
Created attachment 226561 [details]
Print ACPI Methods associated with Acpi-State.

This patch prints the ACPI Method that's associated with the PSCOPE and WSCOPE descriptors that are being constantly generated as determined by the previous debug data. This should significantly narrow down the culprit. Again you'll want to clear dmesg just before running your script.
Comment 25 Robert Moore 2016-08-16 17:23:46 UTC
Any news on this?
Comment 26 Matthew L. Martin 2016-08-16 17:29:27 UTC
We have had to wait for an internal re-organization to settle out before we can restart our investigations.
Comment 27 Lv Zheng 2017-01-16 05:48:55 UTC
Ping...
Comment 28 Matthew L. Martin 2017-01-19 15:03:06 UTC
I need to get resources assigned for reproducing this bug.

Is the patch (above) the only thing you want changed?
Comment 29 David Box 2017-01-19 17:26:48 UTC
Yes. This won't fix it. But it will hopefully show us the ASL method at the center of this problem.
Comment 30 Matthew L. Martin 2017-01-24 17:00:56 UTC
I have applied the second patch and have reproduced the fault, but dmesg is silent during the run. Did you want both patches applied?
Comment 31 Lv Zheng 2017-03-23 06:15:42 UTC
Refiling this bug to ACPICA bugzilla, please track it there.

Thanks
Lv

Note You need to log in before you can comment on or make changes to this bug.