Bug 217134

Summary: TCP performance regression on loopback interface
Product: Networking Reporter: uint8ptr
Component: OtherAssignee: Stephen Hemminger (stephen)
Status: RESOLVED CODE_FIX    
Severity: normal CC: mike
Priority: P1    
Hardware: All   
OS: Linux   
Kernel Version: 5.10-6.2 Subsystem:
Regression: Yes Bisected commit-id:
Attachments: Packet captures of the GDB communication on Linux 5.9 and 5.10

Description uint8ptr 2023-03-04 15:54:34 UTC
Created attachment 303849 [details]
Packet captures of the GDB communication on Linux 5.9 and 5.10

Somewhere between Linux 5.9-rc1 and Linux 5.10 LTS the TCP performance on the loopback interface slowed down to a crawl. This issue makes doing some operations with gdb a lot slower than they should be, in particular when using Python GDB extensions that require sending and receiving data to/from the GDB server.

The bug affects all versions of the Linux kernel from 5.10 LTS up to 6.2.
Versions 5.4 LTS and 5.9-rc1 are not affected by this bug.

I attached two PCAP files that shows the problem.

The packet captures were obtained by connecting the mGBA (version 0.10.1) GDB server and reading 8 bytes from the memory with the following commands:

```
(gdb) target remote localhost:2345
Remote debugging using localhost:2345
0x080008c6 in WaitForVBlank ()
(gdb) x/8bx 0x08000000
0x8000000:	0x7f	0x00	0x00	0xea	0x24	0xff	0xae	0x51
```

I made a table illustrating the performance difference for reading the first byte. You can trace back the exact packets by referring to the first column of the tables.

---

Linux 5.9 packet capture summary

Seq |   | Timestamp | Payload
168 | > | 4.457569  | $m8000000,1#22
172 | < | 4.457689  | $7f#9d

Time delta: 0.00011s/110μs

---

Linux 5.10 packet capture summary

Seq |   | Timestamp | Payload
164 | > | 3.583188  | $m8000000,1#22
168 | < | 3.623667  | $7f#9d

Time delta: 0.040479s/40ms

---

I already discussed extensively the issue with the emulator developer, and we came to the conclusion that the packet is getting delayed by the OS at some point. The issue is present on any version of the emulator that supports the GDB server.

Additional information:

During the TCP communication, no new messages appeared by running the dmesg command.

```
root # uname -a
Linux manjaro 6.1.12-1-MANJARO #1 SMP PREEMPT_DYNAMIC Tue Feb 14 21:59:10 UTC 2023 x86_64 GNU/Linux
root # lspci | grep "Network controller"
02:00.0 Network controller: Intel Corporation Wireless 8265 / 8275 (rev 78)
```

The non-affected Kernel version I used for testing can be downloaded here: https://kacabenggala.uny.ac.id/manjaro/stable-staging/core/x86_64/linux59-5.9rc1.d0816.g9123e3a-1-x86_64.pkg.tar.zst

Steps to reproduce:

1. Start a game on any version of mGBA that supports the GDB server on Linux 5.10 LTS or newer
2. Go to mGBA > Tools > Start GDB server... > Start
3. Start arm-none-eabi-gdb and run the following commands
   - target remote localhost:2345
   - x/8bx 0x08000000

If you display more bytes on the second command, the issue is more noticeable.
Comment 1 uint8ptr 2023-03-06 20:48:13 UTC
Update: since GDB send lots of small packets most of the time, enabling TCP_NODELAY on the emulator's GDB stub fixes the issue altogether