Bug 217134 - TCP performance regression on loopback interface
Summary: TCP performance regression on loopback interface
Status: RESOLVED CODE_FIX
Alias: None
Product: Networking
Classification: Unclassified
Component: Other (show other bugs)
Hardware: All Linux
: P1 normal
Assignee: Stephen Hemminger
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2023-03-04 15:54 UTC by uint8ptr
Modified: 2023-03-06 20:48 UTC (History)
1 user (show)

See Also:
Kernel Version: 5.10-6.2
Subsystem:
Regression: Yes
Bisected commit-id:


Attachments
Packet captures of the GDB communication on Linux 5.9 and 5.10 (8.86 KB, application/gzip)
2023-03-04 15:54 UTC, uint8ptr
Details

Description uint8ptr 2023-03-04 15:54:34 UTC
Created attachment 303849 [details]
Packet captures of the GDB communication on Linux 5.9 and 5.10

Somewhere between Linux 5.9-rc1 and Linux 5.10 LTS the TCP performance on the loopback interface slowed down to a crawl. This issue makes doing some operations with gdb a lot slower than they should be, in particular when using Python GDB extensions that require sending and receiving data to/from the GDB server.

The bug affects all versions of the Linux kernel from 5.10 LTS up to 6.2.
Versions 5.4 LTS and 5.9-rc1 are not affected by this bug.

I attached two PCAP files that shows the problem.

The packet captures were obtained by connecting the mGBA (version 0.10.1) GDB server and reading 8 bytes from the memory with the following commands:

```
(gdb) target remote localhost:2345
Remote debugging using localhost:2345
0x080008c6 in WaitForVBlank ()
(gdb) x/8bx 0x08000000
0x8000000:	0x7f	0x00	0x00	0xea	0x24	0xff	0xae	0x51
```

I made a table illustrating the performance difference for reading the first byte. You can trace back the exact packets by referring to the first column of the tables.

---

Linux 5.9 packet capture summary

Seq |   | Timestamp | Payload
168 | > | 4.457569  | $m8000000,1#22
172 | < | 4.457689  | $7f#9d

Time delta: 0.00011s/110μs

---

Linux 5.10 packet capture summary

Seq |   | Timestamp | Payload
164 | > | 3.583188  | $m8000000,1#22
168 | < | 3.623667  | $7f#9d

Time delta: 0.040479s/40ms

---

I already discussed extensively the issue with the emulator developer, and we came to the conclusion that the packet is getting delayed by the OS at some point. The issue is present on any version of the emulator that supports the GDB server.

Additional information:

During the TCP communication, no new messages appeared by running the dmesg command.

```
root # uname -a
Linux manjaro 6.1.12-1-MANJARO #1 SMP PREEMPT_DYNAMIC Tue Feb 14 21:59:10 UTC 2023 x86_64 GNU/Linux
root # lspci | grep "Network controller"
02:00.0 Network controller: Intel Corporation Wireless 8265 / 8275 (rev 78)
```

The non-affected Kernel version I used for testing can be downloaded here: https://kacabenggala.uny.ac.id/manjaro/stable-staging/core/x86_64/linux59-5.9rc1.d0816.g9123e3a-1-x86_64.pkg.tar.zst

Steps to reproduce:

1. Start a game on any version of mGBA that supports the GDB server on Linux 5.10 LTS or newer
2. Go to mGBA > Tools > Start GDB server... > Start
3. Start arm-none-eabi-gdb and run the following commands
   - target remote localhost:2345
   - x/8bx 0x08000000

If you display more bytes on the second command, the issue is more noticeable.
Comment 1 uint8ptr 2023-03-06 20:48:13 UTC
Update: since GDB send lots of small packets most of the time, enabling TCP_NODELAY on the emulator's GDB stub fixes the issue altogether

Note You need to log in before you can comment on or make changes to this bug.