Bug 111041

Summary:	random openssh connection failure during connection to server
Product:	Networking	Reporter:	Tomas Mozes (hydrapolic)
Component:	IPV4	Assignee:	Stephen Hemminger (stephen)
Status:	RESOLVED CODE_FIX
Severity:	normal	CC:	ganthore, hannes, szg00000
Priority:	P1
Hardware:	All
OS:	Linux
Kernel Version:	4.4.1	Subsystem:
Regression:	No	Bisected commit-id:

Description Tomas Mozes 2016-01-20 10:14:17 UTC

It's sometimes impossible to connect to a server running Gentoo Linux with kernel 4.4 using ssh. The problem is only the connection, it fails a few times but then it goes fine and ssh works as expected.

OpenSSH_7.1p2-hpn14v10, OpenSSL 1.0.2e 3 Dec 2015
debug1: Reading configuration data /etc/ssh/ssh_config
debug2: ssh_connect: needpriv 0
debug1: Connecting to 10.0.0.5 [10.0.0.5] port 22.
debug1: Connection established.
debug1: Enabling compatibility mode for protocol 2.0
write: Connection reset by peer

Or alternatively it fails with "ssh_exchange_identification: read: Connection reset by peer".

We were able to reproduce this problem with random clients, Gentoo, Ubuntu, Debian, random kernels. It's interesting that only 1 client doesn't have a problem to connect to the server - a system with Gentoo Linux Hardened 4.1 (using the same openssh version).

The server is a Supermicro X10DRW, Intel PCI-Express Gigabit Ethernet (igb) networking. Two ethernet ports were connected to bond and that to a bridge for xen. I've also tried without bond and without any iptables rules, it didn't help. This setup works without any modification on kernel 4.1.15.

A Gentoo forum post: https://forums.gentoo.org/viewtopic.php?p=7868744

Comment 1 hannes 2016-01-20 18:06:42 UTC

Please follow this patch, it might solve the problem:
https://patchwork.ozlabs.org/patch/570514/

Comment 2 hannes 2016-01-20 18:08:48 UTC

Hmm, probably might not help, sorry.

Comment 3 Tomas Mozes 2016-02-17 05:49:15 UTC

(In reply to hannes from comment #1)
> Please follow this patch, it might solve the problem:
> https://patchwork.ozlabs.org/patch/570514/

This happens with/without bonding, so not sure if it really helps.

Comment 4 Tomas Mozes 2016-02-17 05:49:37 UTC

Just tested with 4.4.1, still the same.

Comment 5 Tomas Mozes 2016-03-24 08:46:25 UTC

Just tried with 4.4.6 and the problem seems to go away, cannot reproduce it after numerous login retries.

Comment 6 ganthore 2016-06-24 06:10:32 UTC

Happens for me using kernel 4.6.2.

Comment 7 ganthore 2016-06-24 06:11:13 UTC

> ssh 10.0.0.200 -vvv
OpenSSH_7.2p2lpk, OpenSSL 1.0.2h  3 May 2016
debug1: Reading configuration data /home/ganthore/.ssh/config
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: auto-mux: Trying existing master
debug1: Control socket "/tmp/ganthore@10.0.0.200:22" does not exist
debug2: resolving "10.0.0.200" port 22
debug2: ssh_connect_direct: needpriv 0
debug1: Connecting to 10.0.0.200 [10.0.0.200] port 22.
debug1: Connection established.
debug1: identity file /home/ganthore/.ssh/id_rsa type 1
debug1: key_load_public: No such file or directory
debug1: identity file /home/ganthore/.ssh/id_rsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/ganthore/.ssh/id_dsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/ganthore/.ssh/id_dsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/ganthore/.ssh/id_ecdsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/ganthore/.ssh/id_ecdsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/ganthore/.ssh/id_ed25519 type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/ganthore/.ssh/id_ed25519-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_7.2
ssh_exchange_identification: read: Connection reset by peer

Comment 8 ganthore 2016-06-24 07:20:31 UTC

Disregard my post, it was a different issue entirely (bad config from NetworkManager on a static route).