From owner-freebsd-bugs@FreeBSD.ORG Thu Jan 24 13:11:13 2008 Return-Path: Delivered-To: freebsd-bugs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id CDF2716A417 for ; Thu, 24 Jan 2008 13:11:13 +0000 (UTC) (envelope-from andre@freebsd.org) Received: from c00l3r.networx.ch (c00l3r.networx.ch [62.48.2.2]) by mx1.freebsd.org (Postfix) with ESMTP id 51FB713C448 for ; Thu, 24 Jan 2008 13:11:13 +0000 (UTC) (envelope-from andre@freebsd.org) Received: (qmail 99923 invoked from network); 24 Jan 2008 12:05:45 -0000 Received: from localhost (HELO [127.0.0.1]) ([127.0.0.1]) (envelope-sender ) by c00l3r.networx.ch (qmail-ldap-1.03) with SMTP for ; 24 Jan 2008 12:05:45 -0000 Message-ID: <47988833.8010701@freebsd.org> Date: Thu, 24 Jan 2008 13:44:35 +0100 From: Andre Oppermann User-Agent: Thunderbird 1.5.0.14 (Windows/20071210) MIME-Version: 1.0 To: rob.zietlow@gmail.com References: <200801241054.m0OAs0Gu032350@freefall.freebsd.org> In-Reply-To: <200801241054.m0OAs0Gu032350@freefall.freebsd.org> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-net@freebsd.org, freebsd-bugs@FreeBSD.org, silby@FreeBSD.org Subject: Re: bin/118005: Can No Longer SSH into 7.0 host X-BeenThere: freebsd-bugs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Bug reports List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 24 Jan 2008 13:11:14 -0000 Hi Rob, > Since upgrading to 7.0 I am no longer able to SSH into my server. I > cvsup'ed to 7.0 code and rebuild world and since then I have had this > issue. I have rebuilt multiple times in beta 1, 1.5 and 2. I can SSH into > my host from some hosts within the local LAN. Some machines from outside my > LAN I cannot ssh into this host. Hosts on my lan I have ssh'ed into this > host with are windows(putty), Linux, and Solaris. From outside my LAN I > cannot ssh into my host from Freebsd 6.2, Openbsd 4.1, and Linux(RHEL 4U4). > Freebsd & Openbsd machines are on my home network. However my OSX laptop and > windows machine, from my home network, can SSH into the host without a > problem. This is very strange as the operating systems you cite here correctly implement timestamps and pass all tests that were added to FreeBSD 7BETA. Please see the comments below for an analysis of what may be going on here. > voltron# tcpdump -e -vvnn port 22 and host 192.168.3.132 > tcpdump: listening on em0, link-type EN10MB (Ethernet), capture size 68 > bytes A capture size of 68 bytes is insufficient to see all TCP headers. Please increase it to 100 bytes with "-s 100". > 08:09:55.816411 00:90:5f:0c:00:00 > 00:18:fe:67:54:76, ethertype IPv4 > (0x0800), length 74: (tos 0x0, ttl 61, id 56887, offset 0, flags [DF], proto > TCP (6), length 60) 192.168.3.132.41922 > 192.168.8.163.22: S > 722288481:722288481(0) win 5840 The client sends the SYN to the server including timestamps (full information cut off from too small capture size). > 08:09:55.816432 00:18:fe:67:54:76 > 00:00:0c:07:ac:09, ethertype IPv4 > (0x0800), length 74: (tos 0x0, ttl 64, id 27230, offset 0, flags [DF], proto > TCP (6), length 60) 192.168.8.163.22 > 192.168.3.132.41922: S > 2406244836:2406244836(0) ack 722288482 win 65535 3,nop,nop,timestamp[|tcp]> Server answers SYN with SYN|ACK, mss, window scale and timestamps. Everything perfectly fine so far. > 08:09:55.816925 00:90:5f:0c:00:00 > 00:18:fe:67:54:76, ethertype IPv4 > (0x0800), length 60: (tos 0x0, ttl 58, id 0, offset 0, flags [none], proto > TCP (6), length 40) 192.168.3.132.41922 > 192.168.8.163.22: ., cksum 0x6872 > (correct), 1:1(0) ack 1 win 0 This packet is completely broken. It appears to come from the client and to ACK the SYN|ACK from above. However it advertises a zero window, has a different TTL (58 vs. 61 for the other packets from the client), a IPID of zero and contains no timestamps. This causes the check for missing timestamps to trigger and to abort the connection. RFC1323 requires timestamps to be always present when negotiated during the SYN phase. I've verified that FreeBSD 6.2, OpenBSD 4.1 and Linux RHEL4 implement timestamps correctly and do not turn them off once negotiated. This packet differs significantly from any other packet the client sends. It appears not to come from the real client but some other device that either is in the path between client and server or to share the IP address with the client. Do you have any routers, firewalls, gateways or NAT devices along the path that may cause this strange packet to be sent? > 08:09:55.816933 00:18:fe:67:54:76 > 00:00:0c:07:ac:09, ethertype IPv4 > (0x0800), length 54: (tos 0x0, ttl 64, id 27231, offset 0, flags [DF], proto > TCP (6), length 40) 192.168.8.163.22 > 192.168.3.132.41922: R, cksum 0x47e3 > (incorrect (-> 0xd2ed), 2406244837:2406244837(0) win 0 The server sends the reset and aborts the connection because the timestamp check failed. > 08:09:55.817215 00:90:5f:0c:00:00 > 00:18:fe:67:54:76, ethertype IPv4 > (0x0800), length 66: (tos 0x0, ttl 61, id 56889, offset 0, flags [DF], proto > TCP (6), length 52) 192.168.3.132.41922 > 192.168.8.163.22: ., cksum 0x8036 > (correct), 1:1(0) ack 1 win 1460 Here the real client appears to ACK the SYN|ACK. Note the correct TTL, IPID, window size and timestamps. > 08:09:55.833093 00:18:fe:67:54:76 > 00:00:0c:07:ac:09, ethertype IPv4 > (0x0800), length 105: (tos 0x0, ttl 64, id 27232, offset 0, flags [DF], > proto TCP (6), length 91) 192.168.8.163.22 > 192.168.3.132.41922: P 1:40(39) > ack 1 win 8326 The server re-opens the connection based on the correct ACK because the SYN cookie is correct. The original syncache full entry was deleted above when the reset was sent. However the syncookie is still valid and passed all tests. > 08:09:55.833929 00:90:5f:0c:00:00 > 00:18:fe:67:54:76, ethertype IPv4 > (0x0800), length 60: (tos 0x0, ttl 61, id 8446, offset 0, flags [DF], proto > TCP (6), length 40) 192.168.3.132.41922 > 192.168.8.163.22: R, cksum 0x59d0 > (correct), 722288482:722288482(0) win 0 The client answers with a reset because its socket was shut down due to the reset above caused by the broken (spoofed) ACK packet earlier. It all boils down to where the broken/spoofed ACK (packet #3) comes from. There must be some strange or completely broken device in the packet path between client and server. Please investigate and identify potential candidates. -- Andre