Created attachment 305802 [details]
otcpl-dell-7390-icl_mem.html

An issue was introduced in kernel 6.8-rc1 which causes several machines to take 30 seconds on all but the first S3 suspend. The issue is that the HID devices are timing out and causing S3 suspend to have to wait a huge amount of time.

[   75.363468] hid-sensor-hub 001F:8087:0AC2.0004: timeout waiting for response from ISHTP device
[   78.435492] hid-sensor-hub 001F:8087:0AC2.0004: timeout waiting for response from ISHTP device
[   81.507479] hid-sensor-hub 001F:8087:0AC2.0004: timeout waiting for response from ISHTP device
[   81.507486] hid_sensor_magn_3d HID-SENSOR-200083.23.auto: PM: platform_pm_suspend+0x0/0x60 returned 0 after 9175367 usecs

[   84.579227] hid-sensor-hub 001F:8087:0AC2.0003: timeout waiting for response from ISHTP device
[   87.651552] hid-sensor-hub 001F:8087:0AC2.0003: timeout waiting for response from ISHTP device
[   90.723480] hid-sensor-hub 001F:8087:0AC2.0003: timeout waiting for response from ISHTP device
[   90.723488] hid_sensor_accel_3d HID-SENSOR-200073.15.auto: PM: platform_pm_suspend+0x0/0x60 returned 0 after 9215960 usecs

[   90.723530] hid_sensor_als HID-SENSOR-200041.6.auto: PM: calling platform_pm_suspend+0x0/0x60 @ 4033, parent: 001F:8087:0AC2.0002
[   93.795494] hid-sensor-hub 001F:8087:0AC2.0002: timeout waiting for response from ISHTP device
[   96.867514] hid-sensor-hub 001F:8087:0AC2.0002: timeout waiting for response from ISHTP device
[   99.939481] hid-sensor-hub 001F:8087:0AC2.0002: timeout waiting for response from ISHTP device
[   99.939488] hid_sensor_als HID-SENSOR-200041.6.auto: PM: platform_pm_suspend+0x0/0x60 returned 0 after 9215956 usecs

I've bisected this issue to the following commit:

commit f645a90e8ff732c48dd9f18815baef08c44ac8a0 (refs/bisect/bad)
Author: Even Xu <even.xu@intel.com>
Date:   Tue Dec 5 09:50:31 2023 +0800

    HID: intel-ish-hid: ishtp-hid-client: use helper functions for connection
    
    Use helper functions ishtp_cl_establish_connection() and
    ishtp_cl_destroy_connection() to establish and destroy connection
    respectively. These functions are used during initialization, reset and
    deinitialization flows.
    
    No functional changes are expected.
    
    Signed-off-by: Even Xu <even.xu@intel.com>
    Acked-by: Srinivas Pandruvada <srinivas.pandruvada@linux.intel.com>
    Signed-off-by: Jiri Kosina <jkosina@suse.com>

The issue has affected two of our dell machines. The attached sleepgraph timelines include the dmesg logs, just click the dmesg button in the upper right. The log button shows system info.