I forwarded the report to the lists to increase its visibility:
https://lore.kernel.org/all/ed31b6fe-e73d-34af-445b-81c5c644d615@leemhuis.info/

If no developer comes up with a idea what might cause this within the next week or so, I'd assume you likely need to do use "git bisect" to find the change that caused the problem.

This problem is present in v6.1. (Linux mainline tree).
This problem is NOT present in v5.15.99 (Linux-stable tree)

Embedded ARM CPU vendor (NXP) supports only kernels only 5.15 and 6.1. NXP unstreamed its code to Linux mainline sometime after 6.1.

I switched to x86-64 for doing testing on kernels 5.16-6.0.

Running Ubuntu 22.04, I see the following
1. vmlinuz-5.15.0-43-generic kernel works (you can associated and connect to AP)
2. vmlinuz-5.17.0-1029-oem kernel does not work (association is rejected with the same error as above).
3. vmlinuz-5.19.0-38-generic does not work (same as in 5.17)

The main obvious difference is that ath11k driver support 160MHz channel bandwidth in 5.17 & 5.19 while 5.15 version supports only 80MHz maximum channel bandwidth.

Here is bisect log for finding the commit that causes the problem.

$ git bisect log
git bisect start '--' './drivers/net/wireless/ath/ath11k'
# good: [8bb7eca972ad531c9b149c0a51ab43a417385813] Linux 5.15
git bisect good 8bb7eca972ad531c9b149c0a51ab43a417385813
# bad: [df0cc57e057f18e44dac8e6c18aba47ab53202f9] Linux 5.16
git bisect bad df0cc57e057f18e44dac8e6c18aba47ab53202f9
# bad: [79feedfea7793d91293ab72fac5fc66aae0c6a85] ath11k: Avoid "No VIF found" warning message
git bisect bad 79feedfea7793d91293ab72fac5fc66aae0c6a85
# bad: [eb19efed836a51ee30a602abe2dd21a97c47bbcc] ath11k: Wstringop-overread warning
git bisect bad eb19efed836a51ee30a602abe2dd21a97c47bbcc
# good: [64e06b78a92744d43d3993ba623d2686d8f937e7] ath11k: add separate APIs for monitor mode
git bisect good 64e06b78a92744d43d3993ba623d2686d8f937e7
# bad: [cc2ad7541486f1f755949c1ccd17e14a15bf1f4e] ath11k: Refactor spectral FFT bin size
git bisect bad cc2ad7541486f1f755949c1ccd17e14a15bf1f4e
# good: [61fe43e7216df6e9a912d831aafc7142fa20f280] ath11k: add support for setting fixed HE rate/gi/ltf
git bisect good 61fe43e7216df6e9a912d831aafc7142fa20f280
# bad: [f552d6fd2f27ce9430c74482c46272838e2de688] ath11k: add support for 80P80 and 160 MHz bandwidth
git bisect bad f552d6fd2f27ce9430c74482c46272838e2de688
# first bad commit: [f552d6fd2f27ce9430c74482c46272838e2de688] ath11k: add support for 80P80 and 160 MHz bandwidth

(In reply to Michal Smulski from comment #4)
> Here is bisect log for finding the commit that causes the problem.
>
> # first bad commit: [f552d6fd2f27ce9430c74482c46272838e2de688] ath11k: add
> support for 80P80 and 160 MHz bandwidth

Great, thx. Pointed the authors of that commit to this ticket here: https://lore.kernel.org/all/afa92c35-b17a-49de-ac4c-6d60237a2dca@leemhuis.info/

Michal, could you provide more information about the AP? What's the exact make and model? What are the Wi-Fi settings on the AP? This helps with reproducing the bug on our end.

AP is Synology Router RT6600AX

# uname -a
Linux SynologyRouter 4.4.60 #9346 SMP PREEMPT Fri Nov 4 11:53:13 CST 2022 aarch64 GNU/Linux synology_cypress_rt6600ax

/lib/firmware/qcn9000 # more fw_version.txt 
WLAN.HK.2.5.r4-00745-QCAHKSWPL_SILICONZ-1 v1

/etc/hostapd # cat hostapd-wlan100.conf 
ctrl_interface=/var/run/hostapd
driver=nl80211
interface=wlan100
bridge=lbr0
ssid=8e55bb
ignore_broadcast_ssid=0
max_num_sta=256
ap_isolate=0
dtim_period=4
wmm_enabled=1
wpa_group_rekey=3600
send_probe_response=0
ieee80211ax=1
ieee80211ac=1
ieee80211n=1
hw_mode=a
channel=149
he_oper_centr_freq_seg0_idx=155
he_oper_chwidth=1
ht_capab=[LDPC][SMPS-DYNAMIC][TX-STBC][RX-STBC-1][MAX-AMSDU-7935][DSSS_CCK-40][HT40+][SHORT-GI-40]
vht_capab=[MAX-MPDU-11454][VHT160-80PLUS80][RXLDPC][SHORT-GI-80][SHORT-GI-160][TX-STBC-2BY1][RX-STBC1][SU-BEAMFORMER][SOUNDING-DIMENSION-4][SU-BEAMFORMEE][MAX-A-MPDU-LEN-EXP7][MU-BEAMFORMER][RX-ANTENNA-PATTERN][TX-ANTENNA-PATTERN]
auth_algs=1
wpa=2
wpa_key_mgmt=WPA-PSK  
wpa_pairwise=CCMP
ieee80211w=0
wpa_passphrase=244283540
wps_independent=1
eap_server=1
wps_state=2
ap_setup_locked=0
wps_rf_bands=a
config_methods=label display virtual_display virtual_push_button physical_push_button keypad
device_type=6-0050F204-1
manufacturer=Synology Inc.
model_name=Synology Router
model_number=RT6600AX
serial_number=2290UXRTXFQP8
upnp_iface=lbr0
device_name=Synology Router
friendly_name=Synology Router
uuid=e5ea0080-01db-4969-a47e-9009d0310272
macaddr_acl=0
deny_mac_file=/etc/hostapd/hostapd-wlan100-mac.list

Reverting part of the commit f552d6fd2f27ce9430c74482c46272838e2de688, specifically:

@@ -4427,11 +4497,6 @@ ath11k_create_vht_cap(struct ath11k *ar, u32 rate_cap_tx_chainmask,
 
        ath11k_set_vht_txbf_cap(ar, &vht_cap.cap);
 
-       /* TODO: Enable back VHT160 mode once association issues are fixed */
-       /* Disabling VHT160 and VHT80+80 modes */
-       vht_cap.cap &= ~IEEE80211_VHT_CAP_SUPP_CHAN_WIDTH_MASK;
-       vht_cap.cap &= ~IEEE80211_VHT_CAP_SHORT_GI_160;
-
        rxmcs_map = 0;
        txmcs_map = 0;
        for (i = 0; i < 8; i++) {

fixes my problem. Moreover, I know get 160MHz link
#iw dev wlan0 link
Connected to 90:09:d0:31:02:77 (on wlan0)
        SSID: 8e55bb
        freq: 5500
        RX: 392016 bytes (1590 packets)
        TX: 3069 bytes (20 packets)
        signal: -95 dBm
        rx bitrate: 2507.8 MBit/s 160MHz HE-MCS 6 HE-NSS 4 HE-GI 0 HE-DCM 0
        tx bitrate: 2882.6 MBit/s 160MHz HE-MCS 7 HE-NSS 4 HE-GI 0 HE-DCM 0

        bss flags:      short-slot-time
        dtim period:    4
        beacon int:     100

I captured association/management frame from Qualcomm (configured as STA)  via wireshark which triggers error message on AP. I can see the following:

Supported Channel Width = 0x2
Extended NSS BW Support = 0x1
Extended NSS BW Capable = 0x1

I noted from linux/net/wireless/util.c:

	/* This is an invalid combination so pretend nothing is supported */
        if (supp_width == 2 && (ext_nss_bw == 1 || ext_nss_bw == 2))
                return 0;

So it appears that Synology router is correct that channel bw and ext NSS combination is invalid. I presume that other AP device use this lenient approach and ignore the invalid combination but Synology does not accept this.


VHT Capabilities Info: 0x738b79fa
    .... .... .... .... .... .... .... ..10 = Maximum MPDU Length: 11 454 (0x2)
    .... .... .... .... .... .... .... 10.. = Supported Channel Width Set: 160MHz and 80+80 Supported (0x2)
    .... .... .... .... .... .... ...1 .... = Rx LDPC: Supported
    .... .... .... .... .... .... ..1. .... = Short GI for 80MHz/TVHT_MODE_4C: Supported
    .... .... .... .... .... .... .1.. .... = Short GI for 160MHz and 80+80MHz: Supported
    .... .... .... .... .... .... 1... .... = Tx STBC: Supported
    .... .... .... .... .... .001 .... .... = Rx STBC: 1 Spatial Stream Supported (0x1)
    .... .... .... .... .... 1... .... .... = SU Beamformer Capable: Supported
    .... .... .... .... ...1 .... .... .... = SU Beamformee Capable: Supported
    .... .... .... .... 011. .... .... .... = Beamformee STS Capability: 4 (0x3)
    .... .... .... .011 .... .... .... .... = Number of Sounding Dimensions: 4 (0x3)
    .... .... .... 1... .... .... .... .... = MU Beamformer Capable: Supported
    .... .... ...0 .... .... .... .... .... = MU Beamformee Capable: Not supported
    .... .... ..0. .... .... .... .... .... = TXOP PS: Not supported
    .... .... .0.. .... .... .... .... .... = +HTC-VHT Capable: Not supported
    .... ..11 1... .... .... .... .... .... = Max A-MPDU Length Exponent: 1 048 575 (0x7)
    .... 00.. .... .... .... .... .... .... = VHT Link Adaptation: No Feedback (0x0)
    ...1 .... .... .... .... .... .... .... = Rx Antenna Pattern Consistency: Supported
    ..1. .... .... .... .... .... .... .... = Tx Antenna Pattern Consistency: Supported
    01.. .... .... .... .... .... .... .... = Extended NSS BW Support: 0x1




..1. .... .... .... = Extended NSS BW Capable: Capable

Another way I was able to verify this is by setting:

In mac.c (in ath11kcreate_vht_cap() function):
  vht_cap.cap &= ~IEEE80211_VHT_CAP_EXT_NSS_BW_MASK;
instead of removing 160MHz support from vht_cap.cap.

Setting this field allows for full 160MHz connection with AP.

I am not sure where is Extended NSS BW Support=1 set. I don't see this set in ath11k driver. Is this done in firmware?