From owner-freebsd-pf@FreeBSD.ORG Sun Jul 22 23:49:08 2012 Return-Path: Delivered-To: freebsd-pf@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 0F8D2106564A for ; Sun, 22 Jul 2012 23:49:08 +0000 (UTC) (envelope-from jmattax@clanspum.net) Received: from mail.clanspum.net (twopir-2-pt.tunnel.tserv9.chi1.ipv6.he.net [IPv6:2001:470:1f10:1b9::2]) by mx1.freebsd.org (Postfix) with ESMTP id C808D8FC12 for ; Sun, 22 Jul 2012 23:49:07 +0000 (UTC) Received: from mail.clanspum.net (localhost.localdomain [IPv6:::1]) by mail.clanspum.net (Postfix) with ESMTP id 6E3D822400C for ; Sun, 22 Jul 2012 18:49:05 -0500 (CDT) Received: from 63.231.116.1 (SquirrelMail authenticated user jmattax) by mail.clanspum.net with HTTP; Sun, 22 Jul 2012 18:49:05 -0500 Message-ID: Date: Sun, 22 Jul 2012 18:49:05 -0500 From: jmattax@clanspum.net To: freebsd-pf@freebsd.org User-Agent: SquirrelMail/1.4.22 MIME-Version: 1.0 Content-Type: text/plain;charset=iso-8859-1 Content-Transfer-Encoding: 8bit X-Priority: 3 (Normal) Importance: Normal Subject: PF suddenly malfunctioned X-BeenThere: freebsd-pf@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "Technical discussion and general questions about packet filter \(pf\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 22 Jul 2012 23:49:08 -0000 A few weeks ago (I've been trying to debug it myself since then) my pf firewall stopped working fully correctly. The symptom is that I can no longer access a variety of websites when I'm behind the firewall. I have verified that I can access all of the affected websites from outside my firewall. I have since stripped down my firewall (and general home server) so that it is no longer running named, sshguard or any useful firewalling rules in an attempt to figure out was broken but have been unable to do so. Attached are my current /etc/pf.conf and /etc/rc.conf, to ensure that these are the configurations being used as of my last test I restarted the system and am still getting the same behavior. This behavior started sometime around a storm at my house, but since the firewall can see the websites that the computers behind it can't I don't believe the hardware is an issue. Also, some websites (like anything google hosts) are just fine. The also, so people can see what my kernel thinks I've attach the output of a couple of commands below [root@ ~]# pfctl -s rules No ALTQ support in kernel ALTQ related functions disabled pass in quick all flags S/SA keep state pass out quick all flags S/SA keep state [root@ ~]# pfctl -s nat No ALTQ support in kernel ALTQ related functions disabled nat on xl0 inet from 10.11.10.0/24 to any -> 192.168.0.200 [root@stilgar ~]# ifconfig re0: flags=8843 metric 0 mtu 1500 options=389b ether 90:e6:ba:60:9a:33 inet 10.11.10.1 netmask 0xffffff00 broadcast 10.11.10.255 media: Ethernet autoselect (100baseTX ) status: active xl0: flags=8843 metric 0 mtu 1500 options=82009 ether 00:01:03:d1:fa:90 inet 192.168.0.200 netmask 0xffffff00 broadcast 192.168.0.255 media: Ethernet autoselect (100baseTX ) status: active plip0: flags=8810 metric 0 mtu 1500 ipfw0: flags=8801 metric 0 mtu 65536 lo0: flags=8049 metric 0 mtu 16384 options=3 inet6 fe80::1%lo0 prefixlen 64 scopeid 0x5 inet6 ::1 prefixlen 128 inet 127.0.0.1 netmask 0xff000000 nd6 options=3 pflog0: flags=141 metric 0 mtu 33152 I would be very appreciative of any suggestions anyone can offer. Jason Mattax From owner-freebsd-pf@FreeBSD.ORG Mon Jul 23 01:30:16 2012 Return-Path: Delivered-To: freebsd-pf@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 321CC106564A for ; Mon, 23 Jul 2012 01:30:16 +0000 (UTC) (envelope-from ml@my.gd) Received: from mail-yw0-f54.google.com (mail-yw0-f54.google.com [209.85.213.54]) by mx1.freebsd.org (Postfix) with ESMTP id DA8EB8FC12 for ; Mon, 23 Jul 2012 01:30:15 +0000 (UTC) Received: by yhfs35 with SMTP id s35so5857797yhf.13 for ; Sun, 22 Jul 2012 18:30:09 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=references:in-reply-to:mime-version:content-transfer-encoding :content-type:message-id:cc:x-mailer:from:subject:date:to :x-gm-message-state; bh=eJUug0iOKjFdE3zBPQJKpzUbk+MmAHVGRp81g/P2+KQ=; b=PwghqRoNtp4DtppKtvjp4AzKEl9yoQZJvBP3KDq8tMr9X/ItYkog9IUgREjGfhbXVh 0527FmTF5jMmcDlTAOe1aQUX1JtchHIK6NIhOhTqfSQpW7xyM23WvZba/iLbnyy9BhUY 1NPQi2ikZBT4w6ubc0eM3Rva4pSsJzt1b+8EB83Nwq353HIQ1poLdsn9SFKWdvQQ5G7j /A+0DJwBKlF9gaYtJpfd/qTPVTIHeUfxlxuAUjTy89R6IC2w6GhDjOe23/StMqyGABUe iSPOEw+LUwDVbEOIYF9gOvEIPZJEBWP264MbKJljOiG/qZWl4ggVwoxY+EuTyuP0MLRy oPAA== Received: by 10.236.151.110 with SMTP id a74mr12669501yhk.35.1343007009651; Sun, 22 Jul 2012 18:30:09 -0700 (PDT) Received: from ?IPv6:2a01:e35:8aac:83c0:8cea:47d5:fb7f:ad? ([2a01:e35:8aac:83c0:8cea:47d5:fb7f:ad]) by mx.google.com with ESMTPS id x4sm22598781yhh.2.2012.07.22.18.30.07 (version=TLSv1/SSLv3 cipher=OTHER); Sun, 22 Jul 2012 18:30:09 -0700 (PDT) References: In-Reply-To: Mime-Version: 1.0 (1.0) Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=us-ascii Message-Id: <2B5A7CC5-0950-47E9-928F-D5909238052C@my.gd> X-Mailer: iPhone Mail (9A405) From: Damien Fleuriot Date: Mon, 23 Jul 2012 03:30:05 +0200 To: "jmattax@clanspum.net" X-Gm-Message-State: ALoCoQnD/ChOupCk4Mmj91CeeD7BhPK6sw+YRrIENqRltQ8GJ0cavNDpUSKBBWiRBTsYMUrE2wFX Cc: "freebsd-pf@freebsd.org" Subject: Re: PF suddenly malfunctioned X-BeenThere: freebsd-pf@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "Technical discussion and general questions about packet filter \(pf\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 23 Jul 2012 01:30:16 -0000 On 23 Jul 2012, at 01:49, jmattax@clanspum.net wrote: > A few weeks ago (I've been trying to debug it myself since then) my pf > firewall stopped working fully correctly. The symptom is that I can no lon= ger > access a variety of websites when I'm behind the firewall. I have verified= > that I can access all of the affected websites from outside my firewall. I= > have since stripped down my firewall (and general home server) so that it i= s > no longer running named, sshguard or any useful firewalling rules in an > attempt to figure out was broken but have been unable to do so. >=20 > Attached are my current /etc/pf.conf and /etc/rc.conf, to ensure that thes= e > are the configurations being used as of my last test I restarted the syste= m > and am still getting the same behavior. This behavior started sometime aro= und > a storm at my house, but since the firewall can see the websites that the > computers behind it can't I don't believe the hardware is an issue. >=20 > Also, some websites (like anything google hosts) are just fine. >=20 > The also, so people can see what my kernel thinks I've attach the output o= f a > couple of commands below >=20 > [root@ ~]# pfctl -s rules > No ALTQ support in kernel > ALTQ related functions disabled > pass in quick all flags S/SA keep state > pass out quick all flags S/SA keep state > [root@ ~]# pfctl -s nat > No ALTQ support in kernel > ALTQ related functions disabled > nat on xl0 inet from 10.11.10.0/24 to any -> 192.168.0.200 > [root@stilgar ~]# ifconfig > re0: flags=3D8843 metric 0 mtu 150= 0 > options=3D389b > ether 90:e6:ba:60:9a:33 > inet 10.11.10.1 netmask 0xffffff00 broadcast 10.11.10.255 > media: Ethernet autoselect (100baseTX ) > status: active > xl0: flags=3D8843 metric 0 mtu 150= 0 > options=3D82009 > ether 00:01:03:d1:fa:90 > inet 192.168.0.200 netmask 0xffffff00 broadcast 192.168.0.255 > media: Ethernet autoselect (100baseTX > ) > status: active > plip0: flags=3D8810 metric 0 mtu 1500 > ipfw0: flags=3D8801 metric 0 mtu 65536 > lo0: flags=3D8049 metric 0 mtu 16384 > options=3D3 > inet6 fe80::1%lo0 prefixlen 64 scopeid 0x5 > inet6 ::1 prefixlen 128 > inet 127.0.0.1 netmask 0xff000000 > nd6 options=3D3 > pflog0: flags=3D141 metric 0 mtu 33152 >=20 > I would be very appreciative of any suggestions anyone can offer. >=20 > Jason Mattax >=20 1/ OS version ? We can't tell from the current info 2/ When the problem appears. Have you tried disabling PF ? (pfctl -d) Does it help ? 3/ The websites wouldn't be using connection recycling per chance ? (linux) We've had a lot of problems with Linux enabled hosts using recycling, having= them turn it off solved the problems. There was not a thing we found on our side to fix it. Disabling scrubbing wouldn't help either.= From owner-freebsd-pf@FreeBSD.ORG Mon Jul 23 05:31:48 2012 Return-Path: Delivered-To: freebsd-pf@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 3FB1F106564A for ; Mon, 23 Jul 2012 05:31:48 +0000 (UTC) (envelope-from jmattax@storytotell.org) Received: from mail.clanspum.net (twopir-2-pt.tunnel.tserv9.chi1.ipv6.he.net [IPv6:2001:470:1f10:1b9::2]) by mx1.freebsd.org (Postfix) with ESMTP id F2F918FC08 for ; Mon, 23 Jul 2012 05:31:47 +0000 (UTC) Received: from [192.168.0.23] (unknown [63.231.116.1]) (using TLSv1 with cipher ECDHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mail.clanspum.net (Postfix) with ESMTPSA id 8B54622400C; Mon, 23 Jul 2012 00:31:39 -0500 (CDT) Message-ID: <500CE1B2.5040303@storytotell.org> Date: Sun, 22 Jul 2012 23:31:30 -0600 From: Jason Mattax User-Agent: Mozilla/5.0 (X11; Linux i686; rv:14.0) Gecko/20120714 Thunderbird/14.0 MIME-Version: 1.0 To: ml@my.gd References: <2B5A7CC5-0950-47E9-928F-D5909238052C@my.gd> In-Reply-To: <2B5A7CC5-0950-47E9-928F-D5909238052C@my.gd> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-pf@freebsd.org Subject: Re: PF suddenly malfunctioned X-BeenThere: freebsd-pf@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "Technical discussion and general questions about packet filter \(pf\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 23 Jul 2012 05:31:48 -0000 On 07/22/2012 07:30 PM, Damien Fleuriot wrote: > > On 23 Jul 2012, at 01:49, jmattax@clanspum.net wrote: > >> A few weeks ago (I've been trying to debug it myself since then) my pf >> firewall stopped working fully correctly. The symptom is that I can no longer >> access a variety of websites when I'm behind the firewall. I have verified >> that I can access all of the affected websites from outside my firewall. I >> have since stripped down my firewall (and general home server) so that it is >> no longer running named, sshguard or any useful firewalling rules in an >> attempt to figure out was broken but have been unable to do so. >> >> Attached are my current /etc/pf.conf and /etc/rc.conf, to ensure that these >> are the configurations being used as of my last test I restarted the system >> and am still getting the same behavior. This behavior started sometime around >> a storm at my house, but since the firewall can see the websites that the >> computers behind it can't I don't believe the hardware is an issue. >> >> Also, some websites (like anything google hosts) are just fine. >> >> The also, so people can see what my kernel thinks I've attach the output of a >> couple of commands below >> >> [root@ ~]# pfctl -s rules >> No ALTQ support in kernel >> ALTQ related functions disabled >> pass in quick all flags S/SA keep state >> pass out quick all flags S/SA keep state >> [root@ ~]# pfctl -s nat >> No ALTQ support in kernel >> ALTQ related functions disabled >> nat on xl0 inet from 10.11.10.0/24 to any -> 192.168.0.200 >> [root@stilgar ~]# ifconfig >> re0: flags=8843 metric 0 mtu 1500 >> options=389b >> ether 90:e6:ba:60:9a:33 >> inet 10.11.10.1 netmask 0xffffff00 broadcast 10.11.10.255 >> media: Ethernet autoselect (100baseTX ) >> status: active >> xl0: flags=8843 metric 0 mtu 1500 >> options=82009 >> ether 00:01:03:d1:fa:90 >> inet 192.168.0.200 netmask 0xffffff00 broadcast 192.168.0.255 >> media: Ethernet autoselect (100baseTX >> ) >> status: active >> plip0: flags=8810 metric 0 mtu 1500 >> ipfw0: flags=8801 metric 0 mtu 65536 >> lo0: flags=8049 metric 0 mtu 16384 >> options=3 >> inet6 fe80::1%lo0 prefixlen 64 scopeid 0x5 >> inet6 ::1 prefixlen 128 >> inet 127.0.0.1 netmask 0xff000000 >> nd6 options=3 >> pflog0: flags=141 metric 0 mtu 33152 >> >> I would be very appreciative of any suggestions anyone can offer. >> >> Jason Mattax >> > > 1/ OS version ? We can't tell from the current info > [jmattax@ ~]$ uname -a FreeBSD hostname 8.2-RELEASE-p9 FreeBSD 8.2-RELEASE-p9 #0: Mon Jun 11 23:00:11 UTC 2012 root@amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC amd64 based on that I could easily upgrade to 8.3, or possibly 9.0 tomorrow if I have the inclination. > 2/ When the problem appears. Have you tried disabling PF ? (pfctl -d) > Does it help ? > Since I can consistently reproduce the problem with en.wikipedia.org I have a good way to test. When I run pfctl -d on the firewall it looks like no traffic is being forwarded, including DNS so I eventually get a notice that the web page timed out because I typed the address wrong. That is as opposed to the web browser saying waiting for en.wikipedia.org (and if I recall correctly occasionally getting the redirect to en.wikipedia.org/wiki/Main_Page.) I just tested and got stuck at the waiting for en.wikipedia.org for a couple of minutes before I called it good enough to report here. > 3/ The websites wouldn't be using connection recycling per chance ? (linux) > We've had a lot of problems with Linux enabled hosts using recycling, having them turn it off solved the problems. > There was not a thing we found on our side to fix it. > Disabling scrubbing wouldn't help either. To be clear wikipedia was working with a full firewall configuration, so although I believe they are hosted on linux I would hope someone else would also see this problem. I know there are other websites that also became broken around the same time, but because they are largely advertisement hosting websites I don't know their names off hand and have been bypassing the firewall for the moment. Thanks Jason From owner-freebsd-pf@FreeBSD.ORG Mon Jul 23 09:12:57 2012 Return-Path: Delivered-To: freebsd-pf@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id DAFF2106564A for ; Mon, 23 Jul 2012 09:12:57 +0000 (UTC) (envelope-from ml@my.gd) Received: from mail-lb0-f182.google.com (mail-lb0-f182.google.com [209.85.217.182]) by mx1.freebsd.org (Postfix) with ESMTP id 55D9A8FC0C for ; Mon, 23 Jul 2012 09:12:57 +0000 (UTC) Received: by lbon10 with SMTP id n10so9608981lbo.13 for ; Mon, 23 Jul 2012 02:12:56 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=message-id:date:from:user-agent:mime-version:to:cc:subject :references:in-reply-to:content-type:content-transfer-encoding :x-gm-message-state; bh=SAd99QfhXzanIr6EU08B6b9S+Vc0lhKgiEDSoYUEENI=; b=mDAg29S/WQcLYqQ0e3AByZDJ+4yKotmT9Qe6eT75uHeLAoYr90uSjtBN+F04IQvPxV AEyKIKr3zQ2ALMsTFc5Au3FgB75u1dl4tnJ51WGHZ2rA8iQWb2l9HFY3qElpz4nRpiRd 4JeECFJmRcEx4caa/PsnvbAlFI4jOngzDXiTgiEMuSJ/pUVwODbgLU4yO77lcWXfOV7r kJh6OzuohbPiF4zQnvGHkXPP48Y4i3Ju3rkTCWLaBp/p4zZC7cR1jHijsGGDJ6GSpc6s JRy5qcRhPeoVT5jjQ5Rdpiv528Yb4YRCkwxc3LW+z2aHfRkAt4bVoXNK0ksMwL3LfdJ0 4GEQ== Received: by 10.152.46.232 with SMTP id y8mr15995107lam.18.1343034776292; Mon, 23 Jul 2012 02:12:56 -0700 (PDT) Received: from dfleuriot-at-hi-media.com ([83.167.62.196]) by mx.google.com with ESMTPS id fy10sm12852170lab.0.2012.07.23.02.12.54 (version=SSLv3 cipher=OTHER); Mon, 23 Jul 2012 02:12:55 -0700 (PDT) Message-ID: <500D1595.4010405@my.gd> Date: Mon, 23 Jul 2012 11:12:53 +0200 From: Damien Fleuriot User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:14.0) Gecko/20120713 Thunderbird/14.0 MIME-Version: 1.0 To: Jason Mattax References: <2B5A7CC5-0950-47E9-928F-D5909238052C@my.gd> <500CE1B2.5040303@storytotell.org> In-Reply-To: <500CE1B2.5040303@storytotell.org> Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit X-Gm-Message-State: ALoCoQngpyU4tU+8p1x0c4IWsSRZXTfbOzwlwpDZLq/WOZpB15kDxqK82TH64YLVzSccjme+ZEZs Cc: freebsd-pf@freebsd.org Subject: Re: PF suddenly malfunctioned X-BeenThere: freebsd-pf@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "Technical discussion and general questions about packet filter \(pf\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 23 Jul 2012 09:12:58 -0000 On 7/23/12 7:31 AM, Jason Mattax wrote: > > > On 07/22/2012 07:30 PM, Damien Fleuriot wrote: >> >> On 23 Jul 2012, at 01:49, jmattax@clanspum.net wrote: >> >>> A few weeks ago (I've been trying to debug it myself since then) my pf >>> firewall stopped working fully correctly. The symptom is that I can >>> no longer >>> access a variety of websites when I'm behind the firewall. I have >>> verified >>> that I can access all of the affected websites from outside my >>> firewall. I >>> have since stripped down my firewall (and general home server) so >>> that it is >>> no longer running named, sshguard or any useful firewalling rules in an >>> attempt to figure out was broken but have been unable to do so. >>> >>> Attached are my current /etc/pf.conf and /etc/rc.conf, to ensure that >>> these >>> are the configurations being used as of my last test I restarted the >>> system >>> and am still getting the same behavior. This behavior started >>> sometime around >>> a storm at my house, but since the firewall can see the websites that >>> the >>> computers behind it can't I don't believe the hardware is an issue. >>> >>> Also, some websites (like anything google hosts) are just fine. >>> >>> The also, so people can see what my kernel thinks I've attach the >>> output of a >>> couple of commands below >>> >>> [root@ ~]# pfctl -s rules >>> No ALTQ support in kernel >>> ALTQ related functions disabled >>> pass in quick all flags S/SA keep state >>> pass out quick all flags S/SA keep state >>> [root@ ~]# pfctl -s nat >>> No ALTQ support in kernel >>> ALTQ related functions disabled >>> nat on xl0 inet from 10.11.10.0/24 to any -> 192.168.0.200 >>> [root@stilgar ~]# ifconfig >>> re0: flags=8843 metric 0 mtu >>> 1500 >>> >>> options=389b >>> >>> ether 90:e6:ba:60:9a:33 >>> inet 10.11.10.1 netmask 0xffffff00 broadcast 10.11.10.255 >>> media: Ethernet autoselect (100baseTX ) >>> status: active >>> xl0: flags=8843 metric 0 mtu >>> 1500 >>> options=82009 >>> ether 00:01:03:d1:fa:90 >>> inet 192.168.0.200 netmask 0xffffff00 broadcast 192.168.0.255 >>> media: Ethernet autoselect (100baseTX >>> ) >>> status: active >>> plip0: flags=8810 metric 0 mtu 1500 >>> ipfw0: flags=8801 metric 0 mtu 65536 >>> lo0: flags=8049 metric 0 mtu 16384 >>> options=3 >>> inet6 fe80::1%lo0 prefixlen 64 scopeid 0x5 >>> inet6 ::1 prefixlen 128 >>> inet 127.0.0.1 netmask 0xff000000 >>> nd6 options=3 >>> pflog0: flags=141 metric 0 mtu 33152 >>> >>> I would be very appreciative of any suggestions anyone can offer. >>> >>> Jason Mattax >>> >> >> 1/ OS version ? We can't tell from the current info >> > [jmattax@ ~]$ uname -a > FreeBSD hostname 8.2-RELEASE-p9 FreeBSD 8.2-RELEASE-p9 #0: Mon Jun 11 > 23:00:11 UTC 2012 > root@amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC amd64 > > based on that I could easily upgrade to 8.3, or possibly 9.0 tomorrow if > I have the inclination. > I can recommend 8.3, we're using it widely in production. >> 2/ When the problem appears. Have you tried disabling PF ? (pfctl -d) >> Does it help ? >> > Since I can consistently reproduce the problem with en.wikipedia.org I > have a good way to test. When I run pfctl -d on the firewall it looks > like no traffic is being forwarded, including DNS so I eventually get a > notice that the web page timed out because I typed the address wrong. > That is as opposed to the web browser saying waiting for > en.wikipedia.org (and if I recall correctly occasionally getting the > redirect to en.wikipedia.org/wiki/Main_Page.) I just tested and got > stuck at the waiting for en.wikipedia.org for a couple of minutes before > I called it good enough to report here. > Keep in mind that after disabling PF you don't get NAT anymore from your workstations through the firewall. So any test you run while PF is disabled has to be run from the PF box itself. >> 3/ The websites wouldn't be using connection recycling per chance ? >> (linux) >> We've had a lot of problems with Linux enabled hosts using recycling, >> having them turn it off solved the problems. >> There was not a thing we found on our side to fix it. >> Disabling scrubbing wouldn't help either. > > To be clear wikipedia was working with a full firewall configuration, so > although I believe they are hosted on linux I would hope someone else > would also see this problem. I know there are other websites that also > became broken around the same time, but because they are largely > advertisement hosting websites I don't know their names off hand and > have been bypassing the firewall for the moment. > > Thanks > Jason From owner-freebsd-pf@FreeBSD.ORG Mon Jul 23 09:37:37 2012 Return-Path: Delivered-To: freebsd-pf@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 4C48610656EB for ; Mon, 23 Jul 2012 09:37:37 +0000 (UTC) (envelope-from tonix@interazioni.it) Received: from mx02.interazioni.net (mx02.interazioni.net [80.94.114.204]) by mx1.freebsd.org (Postfix) with ESMTP id 820B38FC08 for ; Mon, 23 Jul 2012 09:37:36 +0000 (UTC) Received: (qmail 6609 invoked by uid 88); 23 Jul 2012 09:37:29 -0000 Received: from unknown (HELO ?192.168.200.253?) (tonix@interazioni.it@217.19.151.67) by relay.interazioni.net with ESMTPA; 23 Jul 2012 09:37:29 -0000 Message-ID: <500D1B57.8080405@interazioni.it> Date: Mon, 23 Jul 2012 11:37:27 +0200 From: "Tonix (Antonio Nati)" User-Agent: Mozilla/5.0 (Windows NT 5.1; rv:13.0) Gecko/20120614 Thunderbird/13.0.1 MIME-Version: 1.0 To: Daniel Hartmeier , "freebsd-pf@freebsd.org" References: <500826BD.3070602@interazioni.it> <9EB23F6C23A8B6488E8BCC92A48E83264BB4D26F80@PEMEXMBXVS04.jellyfishnet.co.uk.local> <500AB340.2040405@interazioni.it> <9EB23F6C23A8B6488E8BCC92A48E83264BB4D27241@PEMEXMBXVS04.jellyfishnet.co.uk.local> <500AC91F.9090907@interazioni.it> <20120721182316.GA32530@insomnia.benzedrine.cx> In-Reply-To: <20120721182316.GA32530@insomnia.benzedrine.cx> Content-Type: text/plain; charset=ISO-8859-15; format=flowed Content-Transfer-Encoding: 7bit Cc: Subject: Re: Question on packet filter using in and out interfaces X-BeenThere: freebsd-pf@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "Technical discussion and general questions about packet filter \(pf\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 23 Jul 2012 09:37:37 -0000 Il 21/07/2012 20:23, Daniel Hartmeier ha scritto: > On Sat, Jul 21, 2012 at 05:22:07PM +0200, Tonix (Antonio Nati) wrote: > >> If you can provide a link to this PF diagram it would be very useful. > > A copy is preserved on http://www.benzedrine.cx/pf_flow.png > > Yes, there are two phases. > > HTH, > Daniel > Daniel, thanks for pointing at the diagram. What it is not clear to me is related to in/out rules evaluation. Diagram starts obviously from the packet entering the system, until the packet exits the system. When the packet enters the system, which rules are evaluated? All rules related to interface, both for IN and OUT? Or only IN? PF manual says all rules in pf.conf are evaluated, so I suppose all rules applying to that interface are evaluated... or only IN rules are evaluated in this first step, and only OUT rules are evaluated in second step? Sorry, but I'm missing some key points. Regards, Tonino -- ------------------------------------------------------------ Inter@zioni Interazioni di Antonio Nati http://www.interazioni.it tonix@interazioni.it ------------------------------------------------------------ From owner-freebsd-pf@FreeBSD.ORG Mon Jul 23 09:55:17 2012 Return-Path: Delivered-To: freebsd-pf@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id B5A77106564A for ; Mon, 23 Jul 2012 09:55:17 +0000 (UTC) (envelope-from dhartmei@insomnia.benzedrine.cx) Received: from insomnia.benzedrine.cx (106-30.3-213.fix.bluewin.ch [213.3.30.106]) by mx1.freebsd.org (Postfix) with ESMTP id 2CD098FC14 for ; Mon, 23 Jul 2012 09:55:16 +0000 (UTC) Received: from insomnia.benzedrine.cx (localhost.benzedrine.cx [127.0.0.1]) by insomnia.benzedrine.cx (8.14.1/8.13.4) with ESMTP id q6N9t9N8018754 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Mon, 23 Jul 2012 11:55:10 +0200 (MEST) Received: (from dhartmei@localhost) by insomnia.benzedrine.cx (8.14.1/8.12.10/Submit) id q6N9t9cE026308; Mon, 23 Jul 2012 11:55:09 +0200 (MEST) Date: Mon, 23 Jul 2012 11:55:09 +0200 From: Daniel Hartmeier To: "Tonix (Antonio Nati)" Message-ID: <20120723095509.GB32530@insomnia.benzedrine.cx> References: <500826BD.3070602@interazioni.it> <9EB23F6C23A8B6488E8BCC92A48E83264BB4D26F80@PEMEXMBXVS04.jellyfishnet.co.uk.local> <500AB340.2040405@interazioni.it> <9EB23F6C23A8B6488E8BCC92A48E83264BB4D27241@PEMEXMBXVS04.jellyfishnet.co.uk.local> <500AC91F.9090907@interazioni.it> <20120721182316.GA32530@insomnia.benzedrine.cx> <500D1B57.8080405@interazioni.it> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <500D1B57.8080405@interazioni.it> User-Agent: Mutt/1.5.12-2006-07-14 Cc: "freebsd-pf@freebsd.org" Subject: Re: Question on packet filter using in and out interfaces X-BeenThere: freebsd-pf@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "Technical discussion and general questions about packet filter \(pf\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 23 Jul 2012 09:55:17 -0000 On Mon, Jul 23, 2012 at 11:37:27AM +0200, Tonix (Antonio Nati) wrote: > What it is not clear to me is related to in/out rules evaluation. > > Diagram starts obviously from the packet entering the system, until the > packet exits the system. When the packet enters the system, which rules > are evaluated? All rules related to interface, both for IN and OUT? Or > only IN? During both phases (first incoming on one interface, then outgoing on the other interface), all rules are evaluated. Rules can omit the direction (e.g. 'pass from src to dst'), and such rules can match in either phase, or both. If rules do specify a direction (e.g. 'pass in from src to dst'), they are still evaluated during both in and out phase, but they cannot possibly match during the wrong phase. > PF manual says all rules in pf.conf are evaluated, so I suppose all > rules applying to that interface are evaluated... or only IN rules are > evaluated in this first step, and only OUT rules are evaluated in second > step? There isn't really any difference: while all rules are evaluated, only the IN rules can possibly match (in the first step), so there's no way you notice the OUT rules are being evaluated... Daniel From owner-freebsd-pf@FreeBSD.ORG Mon Jul 23 10:05:29 2012 Return-Path: Delivered-To: freebsd-pf@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 0417D106566C for ; Mon, 23 Jul 2012 10:05:29 +0000 (UTC) (envelope-from dhartmei@insomnia.benzedrine.cx) Received: from insomnia.benzedrine.cx (106-30.3-213.fix.bluewin.ch [213.3.30.106]) by mx1.freebsd.org (Postfix) with ESMTP id 856748FC0C for ; Mon, 23 Jul 2012 10:05:28 +0000 (UTC) Received: from insomnia.benzedrine.cx (localhost.benzedrine.cx [127.0.0.1]) by insomnia.benzedrine.cx (8.14.1/8.13.4) with ESMTP id q6NA5MjE024903 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Mon, 23 Jul 2012 12:05:22 +0200 (MEST) Received: (from dhartmei@localhost) by insomnia.benzedrine.cx (8.14.1/8.12.10/Submit) id q6NA5M2m009895; Mon, 23 Jul 2012 12:05:22 +0200 (MEST) Date: Mon, 23 Jul 2012 12:05:21 +0200 From: Daniel Hartmeier To: jmattax@clanspum.net Message-ID: <20120723100521.GC32530@insomnia.benzedrine.cx> References: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.12-2006-07-14 Cc: freebsd-pf@freebsd.org Subject: Re: PF suddenly malfunctioned X-BeenThere: freebsd-pf@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "Technical discussion and general questions about packet filter \(pf\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 23 Jul 2012 10:05:29 -0000 If you can reliably reproduce the problem with en.wikipedia.org, I suggest the following: On the firewall 1) enable verbose logging with pfctl -xm 2) save the output of pfctl -si and netstat -s 3) run the following three tcpdump in parallel, and save the output: tcpdump -s 1600 -nvvvpSi xl0 'host 91.198.174.225' tcpdump -s 1600 -nvvvpSi re0 'host 91.198.174.225' tcpdump -s 1600 -nvvveeepi pflog0 On a client 4) printf "GET /wiki/Main_Page HTTP/1.1\r\nHost: en.wikipedia.org\r\n\r\n" | nc -v 91.198.174.225 80 | wc -c 5) this should hang until some timout occurs, you need only wait 10s. Back on the firewall 6) re-run pfctl -si and netstat -s (again saving the output) 7) stop the tcpdumps 8) check /var/log/messages for anything from pf The post the outputs :) Daniel From owner-freebsd-pf@FreeBSD.ORG Mon Jul 23 10:53:45 2012 Return-Path: Delivered-To: freebsd-pf@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 38083106566B for ; Mon, 23 Jul 2012 10:53:45 +0000 (UTC) (envelope-from tonix@interazioni.it) Received: from mx02.interazioni.net (mx02.interazioni.net [80.94.114.204]) by mx1.freebsd.org (Postfix) with ESMTP id 8831B8FC0C for ; Mon, 23 Jul 2012 10:53:44 +0000 (UTC) Received: (qmail 20059 invoked by uid 88); 23 Jul 2012 10:53:43 -0000 Received: from unknown (HELO ?192.168.200.253?) (tonix@interazioni.it@217.19.151.67) by relay.interazioni.net with ESMTPA; 23 Jul 2012 10:53:43 -0000 Message-ID: <500D2D35.4070608@interazioni.it> Date: Mon, 23 Jul 2012 12:53:41 +0200 From: "Tonix (Antonio Nati)" User-Agent: Mozilla/5.0 (Windows NT 5.1; rv:13.0) Gecko/20120614 Thunderbird/13.0.1 MIME-Version: 1.0 To: Daniel Hartmeier References: <500826BD.3070602@interazioni.it> <9EB23F6C23A8B6488E8BCC92A48E83264BB4D26F80@PEMEXMBXVS04.jellyfishnet.co.uk.local> <500AB340.2040405@interazioni.it> <9EB23F6C23A8B6488E8BCC92A48E83264BB4D27241@PEMEXMBXVS04.jellyfishnet.co.uk.local> <500AC91F.9090907@interazioni.it> <20120721182316.GA32530@insomnia.benzedrine.cx> <500D1B57.8080405@interazioni.it> <20120723095509.GB32530@insomnia.benzedrine.cx> In-Reply-To: <20120723095509.GB32530@insomnia.benzedrine.cx> Content-Type: text/plain; charset=ISO-8859-15; format=flowed Content-Transfer-Encoding: 7bit Cc: "freebsd-pf@freebsd.org" Subject: Re: Question on packet filter using in and out interfaces X-BeenThere: freebsd-pf@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "Technical discussion and general questions about packet filter \(pf\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 23 Jul 2012 10:53:45 -0000 Il 23/07/2012 11:55, Daniel Hartmeier ha scritto: > On Mon, Jul 23, 2012 at 11:37:27AM +0200, Tonix (Antonio Nati) wrote: > >> What it is not clear to me is related to in/out rules evaluation. >> >> Diagram starts obviously from the packet entering the system, until the >> packet exits the system. When the packet enters the system, which rules >> are evaluated? All rules related to interface, both for IN and OUT? Or >> only IN? > > During both phases (first incoming on one interface, then outgoing on > the other interface), all rules are evaluated. > > Rules can omit the direction (e.g. 'pass from src to dst'), and such > rules can match in either phase, or both. > > If rules do specify a direction (e.g. 'pass in from src to dst'), they are > still evaluated during both in and out phase, but they cannot possibly > match during the wrong phase. Daniel, thanks for the detailed explanation. So, does that mean the OUT phase evaluation always occurs when IN phase has been positive (packet should pass)? I'm thinking to management of a lot of interfaces, where one is the WAN, and others are DMZ and/or customers dedicated subnets. I'd love to put basic protections on WAN input, and then permit all other interfaces to define its own rules for packets coming/going from/to the specific subnet. According to what I understand of your explanation, each interface could have its own IN rules, and if the IN rules of a specific INPUT interface are successfull, the OUT rules of the 'outgoing' interface are then evaluated. This would be wonderful, as each interface could have both IN and OUT rules which do not interphere with or break other interfaces rules. And would permit to write the most of rules just once, according to each interface needs. Regards, Tonino > >> PF manual says all rules in pf.conf are evaluated, so I suppose all >> rules applying to that interface are evaluated... or only IN rules are >> evaluated in this first step, and only OUT rules are evaluated in second >> step? > > There isn't really any difference: while all rules are evaluated, only > the IN rules can possibly match (in the first step), so there's no way > you notice the OUT rules are being evaluated... > > Daniel > -- ------------------------------------------------------------ Inter@zioni Interazioni di Antonio Nati http://www.interazioni.it tonix@interazioni.it ------------------------------------------------------------ From owner-freebsd-pf@FreeBSD.ORG Mon Jul 23 11:07:22 2012 Return-Path: Delivered-To: freebsd-pf@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 02D521065688 for ; Mon, 23 Jul 2012 11:07:22 +0000 (UTC) (envelope-from owner-bugmaster@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id E05898FC23 for ; Mon, 23 Jul 2012 11:07:21 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.5/8.14.5) with ESMTP id q6NB7Ll6090107 for ; Mon, 23 Jul 2012 11:07:21 GMT (envelope-from owner-bugmaster@FreeBSD.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.5/8.14.5/Submit) id q6NB7LmP090105 for freebsd-pf@FreeBSD.org; Mon, 23 Jul 2012 11:07:21 GMT (envelope-from owner-bugmaster@FreeBSD.org) Date: Mon, 23 Jul 2012 11:07:21 GMT Message-Id: <201207231107.q6NB7LmP090105@freefall.freebsd.org> X-Authentication-Warning: freefall.freebsd.org: gnats set sender to owner-bugmaster@FreeBSD.org using -f From: FreeBSD bugmaster To: freebsd-pf@FreeBSD.org Cc: Subject: Current problem reports assigned to freebsd-pf@FreeBSD.org X-BeenThere: freebsd-pf@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "Technical discussion and general questions about packet filter \(pf\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 23 Jul 2012 11:07:22 -0000 Note: to view an individual PR, use: http://www.freebsd.org/cgi/query-pr.cgi?pr=(number). The following is a listing of current problems submitted by FreeBSD users. These represent problem reports covering all versions including experimental development code and obsolete releases. S Tracker Resp. Description -------------------------------------------------------------------------------- o kern/169630 pf [pf] [patch] pf fragment reassembly of padded (undersi o kern/168952 pf [pf] direction scrub rules don't work o kern/168190 pf [pf] panic when using pf and route-to (maybe: bad frag s kern/167057 pf [pf] PF firewall version 4.5 in FreeBSD 9.0 & 8.2 nolo o kern/166336 pf [pf] kern.securelevel 3 +pf reload o kern/165315 pf [pf] States never cleared in PF with DEVICE_POLLING o kern/164402 pf [pf] pf crashes with a particular set of rules when fi o kern/164271 pf [pf] not working pf nat on FreeBSD 9.0 [regression] o kern/163208 pf [pf] PF state key linking mismatch o kern/160370 pf [pf] Incorrect pfctl check of pf.conf o kern/155736 pf [pf] [altq] borrow from parent queue does not work wit o kern/153307 pf [pf] Bug with PF firewall o kern/148290 pf [pf] "sticky-address" option of Packet Filter (PF) blo o kern/148260 pf [pf] [patch] pf rdr incompatible with dummynet o kern/147789 pf [pf] Firewall PF no longer drops connections by sendin o kern/143543 pf [pf] [panic] PF route-to causes kernel panic o bin/143504 pf [patch] outgoing states are not killed by authpf(8) o conf/142961 pf [pf] No way to adjust pidfile in pflogd o conf/142817 pf [patch] etc/rc.d/pf: silence pfctl o kern/141905 pf [pf] [panic] pf kernel panic on 7.2-RELEASE with empty o kern/140697 pf [pf] pf behaviour changes - must be documented o kern/137982 pf [pf] when pf can hit state limits, random IP failures o kern/136781 pf [pf] Packets appear to drop with pf scrub and if_bridg o kern/135948 pf [pf] [gre] pf not natting gre protocol o kern/135162 pf [pfsync] pfsync(4) not usable with GENERIC kernel o kern/134996 pf [pf] Anchor tables not included when pfctl(8) is run w o kern/133732 pf [pf] max-src-conn issue o kern/132769 pf [pf] [lor] 2 LOR's with pf task mtx / ifnet and rtent f kern/132176 pf [pf] pf stalls connection when using route-to [regress o conf/130381 pf [rc.d] [pf] [ip6] ipv6 not fully configured when pf st o kern/129861 pf [pf] [patch] Argument names reversed in pf_table.c:_co o kern/127920 pf [pf] ipv6 and synproxy don't play well together o conf/127814 pf [pf] The flush in pf_reload in /etc/rc.d/pf does not w o kern/127439 pf [pf] deadlock in pf o kern/127121 pf [pf] [patch] pf incorrect log priority o kern/127042 pf [pf] [patch] pf recursion panic if interface group is o kern/125467 pf [pf] pf keep state bug while handling sessions between s kern/124933 pf [pf] [ip6] pf does not support (drops) IPv6 fragmented o kern/124364 pf [pf] [panic] Kernel panic with pf + bridge o kern/122773 pf [pf] pf doesn't log uid or pid when configured to o kern/122014 pf [pf] [panic] FreeBSD 6.2 panic in pf o kern/120281 pf [pf] [request] lost returning packets to PF for a rdr o kern/120057 pf [pf] [patch] Allow proper settings of ALTQ_HFSC. The c o bin/118355 pf [pf] [patch] pfctl(8) help message options order false o kern/114567 pf [pf] [lor] pf_ioctl.c + if.c s conf/110838 pf [pf] tagged parameter on nat not working on FreeBSD 5. o kern/103283 pf pfsync fails to sucessfully transfer some sessions o kern/103281 pf pfsync reports bulk update failures o kern/93825 pf [pf] pf reply-to doesn't work o sparc/93530 pf [pf] Incorrect checksums when using pf's route-to on s o kern/92949 pf [pf] PF + ALTQ problems with latency o bin/86635 pf [patch] pfctl(8): allow new page character (^L) in pf. o kern/82271 pf [pf] cbq scheduler cause bad latency 53 problems total. From owner-freebsd-pf@FreeBSD.ORG Mon Jul 23 11:13:51 2012 Return-Path: Delivered-To: freebsd-pf@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id F1778106564A for ; Mon, 23 Jul 2012 11:13:50 +0000 (UTC) (envelope-from dhartmei@insomnia.benzedrine.cx) Received: from insomnia.benzedrine.cx (106-30.3-213.fix.bluewin.ch [213.3.30.106]) by mx1.freebsd.org (Postfix) with ESMTP id 57EFF8FC14 for ; Mon, 23 Jul 2012 11:13:50 +0000 (UTC) Received: from insomnia.benzedrine.cx (localhost.benzedrine.cx [127.0.0.1]) by insomnia.benzedrine.cx (8.14.1/8.13.4) with ESMTP id q6NBDn72004305 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Mon, 23 Jul 2012 13:13:49 +0200 (MEST) Received: (from dhartmei@localhost) by insomnia.benzedrine.cx (8.14.1/8.12.10/Submit) id q6NBDndT008749; Mon, 23 Jul 2012 13:13:49 +0200 (MEST) Date: Mon, 23 Jul 2012 13:13:49 +0200 From: Daniel Hartmeier To: "Tonix (Antonio Nati)" Message-ID: <20120723111348.GD32530@insomnia.benzedrine.cx> References: <500826BD.3070602@interazioni.it> <9EB23F6C23A8B6488E8BCC92A48E83264BB4D26F80@PEMEXMBXVS04.jellyfishnet.co.uk.local> <500AB340.2040405@interazioni.it> <9EB23F6C23A8B6488E8BCC92A48E83264BB4D27241@PEMEXMBXVS04.jellyfishnet.co.uk.local> <500AC91F.9090907@interazioni.it> <20120721182316.GA32530@insomnia.benzedrine.cx> <500D1B57.8080405@interazioni.it> <20120723095509.GB32530@insomnia.benzedrine.cx> <500D2D35.4070608@interazioni.it> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <500D2D35.4070608@interazioni.it> User-Agent: Mutt/1.5.12-2006-07-14 Cc: "freebsd-pf@freebsd.org" Subject: Re: Question on packet filter using in and out interfaces X-BeenThere: freebsd-pf@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "Technical discussion and general questions about packet filter \(pf\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 23 Jul 2012 11:13:51 -0000 On Mon, Jul 23, 2012 at 12:53:41PM +0200, Tonix (Antonio Nati) wrote: > So, does that mean the OUT phase evaluation always occurs when IN phase > has been positive (packet should pass)? Yes. You have to both allow a packet in on the first interface and out on the second interface. If you forget/omit the second part, the packet will get dropped (assuming a default block policy). > I'm thinking to management of a lot of interfaces, where one is the WAN, > and others are DMZ and/or customers dedicated subnets. > > I'd love to put basic protections on WAN input, and then permit all > other interfaces to define its own rules for packets coming/going > from/to the specific subnet. > > According to what I understand of your explanation, each interface could > have its own IN rules, and if the IN rules of a specific INPUT interface > are successfull, the OUT rules of the 'outgoing' interface are then > evaluated. Yes. Example: you want to prevent customers from talking to arbitrary SMTP hosts (prevent spam by forcing the use of a spam filtering proxy). You can block this with OUT rules on the WAN interface, i.e. by only allowing the proxy's source address to connect to external hosts' port 25. Even if customers can add pass rules for their respective interfaces, they cannot circumvent your OUT on WAN rules. > This would be wonderful, as each interface could have both IN and OUT > rules which do not interphere with or break other interfaces rules. And > would permit to write the most of rules just once, according to each > interface needs. Yes, that's the upside of filtering on both directions on all involved interfaces :) The downside is that you might have to add some redundancy in your rules: even if a customer adds 'pass out on DMZ to port 80' you'll also have to add 'pass out on WAN to port 80'. When a customer complains that something isn't working, you'll have to check both his interface's rules AND the WAN rules. Daniel From owner-freebsd-pf@FreeBSD.ORG Mon Jul 23 11:26:53 2012 Return-Path: Delivered-To: freebsd-pf@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 2C8BF106566C for ; Mon, 23 Jul 2012 11:26:53 +0000 (UTC) (envelope-from tonix@interazioni.it) Received: from mx02.interazioni.net (mx02.interazioni.net [80.94.114.204]) by mx1.freebsd.org (Postfix) with ESMTP id 7BA8F8FC19 for ; Mon, 23 Jul 2012 11:26:52 +0000 (UTC) Received: (qmail 24859 invoked by uid 88); 23 Jul 2012 11:26:51 -0000 Received: from unknown (HELO ?82.143.55.20?) (tonix@interazioni.it@82.143.55.20) by relay.interazioni.net with ESMTPA; 23 Jul 2012 11:26:51 -0000 Message-ID: <500D34F9.5000502@interazioni.it> Date: Mon, 23 Jul 2012 13:26:49 +0200 From: "Tonix (Antonio Nati)" User-Agent: Mozilla/5.0 (Windows NT 5.1; rv:13.0) Gecko/20120614 Thunderbird/13.0.1 MIME-Version: 1.0 To: Daniel Hartmeier References: <500826BD.3070602@interazioni.it> <9EB23F6C23A8B6488E8BCC92A48E83264BB4D26F80@PEMEXMBXVS04.jellyfishnet.co.uk.local> <500AB340.2040405@interazioni.it> <9EB23F6C23A8B6488E8BCC92A48E83264BB4D27241@PEMEXMBXVS04.jellyfishnet.co.uk.local> <500AC91F.9090907@interazioni.it> <20120721182316.GA32530@insomnia.benzedrine.cx> <500D1B57.8080405@interazioni.it> <20120723095509.GB32530@insomnia.benzedrine.cx> <500D2D35.4070608@interazioni.it> <20120723111348.GD32530@insomnia.benzedrine.cx> In-Reply-To: <20120723111348.GD32530@insomnia.benzedrine.cx> Content-Type: text/plain; charset=ISO-8859-15; format=flowed Content-Transfer-Encoding: 7bit Cc: "freebsd-pf@freebsd.org" Subject: Re: Question on packet filter using in and out interfaces X-BeenThere: freebsd-pf@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "Technical discussion and general questions about packet filter \(pf\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 23 Jul 2012 11:26:53 -0000 Il 23/07/2012 13:13, Daniel Hartmeier ha scritto: > On Mon, Jul 23, 2012 at 12:53:41PM +0200, Tonix (Antonio Nati) wrote: > >> So, does that mean the OUT phase evaluation always occurs when IN phase >> has been positive (packet should pass)? > > Yes. You have to both allow a packet in on the first interface and out > on the second interface. If you forget/omit the second part, the packet > will get dropped (assuming a default block policy). > >> I'm thinking to management of a lot of interfaces, where one is the WAN, >> and others are DMZ and/or customers dedicated subnets. >> >> I'd love to put basic protections on WAN input, and then permit all >> other interfaces to define its own rules for packets coming/going >> from/to the specific subnet. >> >> According to what I understand of your explanation, each interface could >> have its own IN rules, and if the IN rules of a specific INPUT interface >> are successfull, the OUT rules of the 'outgoing' interface are then >> evaluated. > > Yes. Example: you want to prevent customers from talking to arbitrary > SMTP hosts (prevent spam by forcing the use of a spam filtering proxy). > > You can block this with OUT rules on the WAN interface, i.e. by only > allowing the proxy's source address to connect to external hosts' > port 25. > > Even if customers can add pass rules for their respective interfaces, > they cannot circumvent your OUT on WAN rules. > >> This would be wonderful, as each interface could have both IN and OUT >> rules which do not interphere with or break other interfaces rules. And >> would permit to write the most of rules just once, according to each >> interface needs. > > Yes, that's the upside of filtering on both directions on all involved > interfaces :) > > The downside is that you might have to add some redundancy in your > rules: even if a customer adds 'pass out on DMZ to port 80' you'll also > have to add 'pass out on WAN to port 80'. When a customer complains that > something isn't working, you'll have to check both his interface's rules > AND the WAN rules. I have customers which should be allowed to go whetever they like. So I'd love to make something like this: - deny on INPUT WAN from hackers/abusers - allow any other INPUT on WAN - custom INPUT rules on remaining interfaces - custom OUT rules on remaining interfaces So, if a customer wants to allow anyone to access his port 80, he/she just add that OUT rule to his/her interface. And that avoids me to add the same rule to WAN and all remaining interfaces. Respect to the dominant model (i.e. which puts any rules on INPUT only), do you see any security hole? Or just some more processing? Regards, Tonino > > Daniel > -- ------------------------------------------------------------ Inter@zioni Interazioni di Antonio Nati http://www.interazioni.it tonix@interazioni.it ------------------------------------------------------------ From owner-freebsd-pf@FreeBSD.ORG Mon Jul 23 11:32:10 2012 Return-Path: Delivered-To: freebsd-pf@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id DC5DC1065673 for ; Mon, 23 Jul 2012 11:32:09 +0000 (UTC) (envelope-from tonix@interazioni.it) Received: from mx02.interazioni.net (mx02.interazioni.net [80.94.114.204]) by mx1.freebsd.org (Postfix) with ESMTP id 41D638FC12 for ; Mon, 23 Jul 2012 11:32:09 +0000 (UTC) Received: (qmail 25587 invoked by uid 88); 23 Jul 2012 11:32:08 -0000 Received: from unknown (HELO ?82.143.55.20?) (tonix@interazioni.it@82.143.55.20) by relay.interazioni.net with ESMTPA; 23 Jul 2012 11:32:08 -0000 Message-ID: <500D3637.9070806@interazioni.it> Date: Mon, 23 Jul 2012 13:32:07 +0200 From: "Tonix (Antonio Nati)" User-Agent: Mozilla/5.0 (Windows NT 5.1; rv:13.0) Gecko/20120614 Thunderbird/13.0.1 MIME-Version: 1.0 To: "freebsd-pf@freebsd.org" References: <500826BD.3070602@interazioni.it> <9EB23F6C23A8B6488E8BCC92A48E83264BB4D26F80@PEMEXMBXVS04.jellyfishnet.co.uk.local> <500AB340.2040405@interazioni.it> <9EB23F6C23A8B6488E8BCC92A48E83264BB4D27241@PEMEXMBXVS04.jellyfishnet.co.uk.local> <500AC91F.9090907@interazioni.it> <20120721182316.GA32530@insomnia.benzedrine.cx> <500D1B57.8080405@interazioni.it> <20120723095509.GB32530@insomnia.benzedrine.cx> <500D2D35.4070608@interazioni.it> <20120723111348.GD32530@insomnia.benzedrine.cx> <500D34F9.5000502@interazioni.it> In-Reply-To: <500D34F9.5000502@interazioni.it> Content-Type: text/plain; charset=ISO-8859-15; format=flowed Content-Transfer-Encoding: 7bit Subject: Re: Question on packet filter using in and out interfaces X-BeenThere: freebsd-pf@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "Technical discussion and general questions about packet filter \(pf\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 23 Jul 2012 11:32:10 -0000 Sorry, gorgot a basic rule! Il 23/07/2012 13:26, Tonix (Antonio Nati) ha scritto: > Il 23/07/2012 13:13, Daniel Hartmeier ha scritto: >> On Mon, Jul 23, 2012 at 12:53:41PM +0200, Tonix (Antonio Nati) wrote: >> >>> So, does that mean the OUT phase evaluation always occurs when IN phase >>> has been positive (packet should pass)? >> >> Yes. You have to both allow a packet in on the first interface and out >> on the second interface. If you forget/omit the second part, the packet >> will get dropped (assuming a default block policy). >> >>> I'm thinking to management of a lot of interfaces, where one is the WAN, >>> and others are DMZ and/or customers dedicated subnets. >>> >>> I'd love to put basic protections on WAN input, and then permit all >>> other interfaces to define its own rules for packets coming/going >>> from/to the specific subnet. >>> >>> According to what I understand of your explanation, each interface could >>> have its own IN rules, and if the IN rules of a specific INPUT interface >>> are successfull, the OUT rules of the 'outgoing' interface are then >>> evaluated. >> >> Yes. Example: you want to prevent customers from talking to arbitrary >> SMTP hosts (prevent spam by forcing the use of a spam filtering proxy). >> >> You can block this with OUT rules on the WAN interface, i.e. by only >> allowing the proxy's source address to connect to external hosts' >> port 25. >> >> Even if customers can add pass rules for their respective interfaces, >> they cannot circumvent your OUT on WAN rules. >> >>> This would be wonderful, as each interface could have both IN and OUT >>> rules which do not interphere with or break other interfaces rules. And >>> would permit to write the most of rules just once, according to each >>> interface needs. >> >> Yes, that's the upside of filtering on both directions on all involved >> interfaces :) >> >> The downside is that you might have to add some redundancy in your >> rules: even if a customer adds 'pass out on DMZ to port 80' you'll also >> have to add 'pass out on WAN to port 80'. When a customer complains that >> something isn't working, you'll have to check both his interface's rules >> AND the WAN rules. > > I have customers which should be allowed to go whetever they like and accept from all. So I'd love to make something like this: - deny on INPUT WAN from hackers/abusers - allow any other INPUT on WAN - allow any OUTPUT to WAN - custom INPUT rules on all other interfaces - custom OUT rules on all other interfaces So, if a customer wants to allow anyone to access his port 80, he/she just add that OUT rule to his/her interface. And that avoids me to add the same rule to WAN and all remaining interfaces. Respect to the dominant model (i.e. which puts any rules on INPUT only), do you see any security hole? Or just some more processing? Regards, Tonino -- ------------------------------------------------------------ Inter@zioni Interazioni di Antonio Nati http://www.interazioni.it tonix@interazioni.it ------------------------------------------------------------ From owner-freebsd-pf@FreeBSD.ORG Mon Jul 23 12:05:25 2012 Return-Path: Delivered-To: freebsd-pf@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id CD367106567A for ; Mon, 23 Jul 2012 12:05:25 +0000 (UTC) (envelope-from dhartmei@insomnia.benzedrine.cx) Received: from insomnia.benzedrine.cx (106-30.3-213.fix.bluewin.ch [213.3.30.106]) by mx1.freebsd.org (Postfix) with ESMTP id 290148FC21 for ; Mon, 23 Jul 2012 12:05:24 +0000 (UTC) Received: from insomnia.benzedrine.cx (localhost.benzedrine.cx [127.0.0.1]) by insomnia.benzedrine.cx (8.14.1/8.13.4) with ESMTP id q6NC5Okt003580 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Mon, 23 Jul 2012 14:05:24 +0200 (MEST) Received: (from dhartmei@localhost) by insomnia.benzedrine.cx (8.14.1/8.12.10/Submit) id q6NC5Ov2009720; Mon, 23 Jul 2012 14:05:24 +0200 (MEST) Date: Mon, 23 Jul 2012 14:05:23 +0200 From: Daniel Hartmeier To: "Tonix (Antonio Nati)" Message-ID: <20120723120523.GE32530@insomnia.benzedrine.cx> References: <500AB340.2040405@interazioni.it> <9EB23F6C23A8B6488E8BCC92A48E83264BB4D27241@PEMEXMBXVS04.jellyfishnet.co.uk.local> <500AC91F.9090907@interazioni.it> <20120721182316.GA32530@insomnia.benzedrine.cx> <500D1B57.8080405@interazioni.it> <20120723095509.GB32530@insomnia.benzedrine.cx> <500D2D35.4070608@interazioni.it> <20120723111348.GD32530@insomnia.benzedrine.cx> <500D34F9.5000502@interazioni.it> <500D3637.9070806@interazioni.it> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <500D3637.9070806@interazioni.it> User-Agent: Mutt/1.5.12-2006-07-14 Cc: "freebsd-pf@freebsd.org" Subject: Re: Question on packet filter using in and out interfaces X-BeenThere: freebsd-pf@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "Technical discussion and general questions about packet filter \(pf\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 23 Jul 2012 12:05:26 -0000 On Mon, Jul 23, 2012 at 01:32:07PM +0200, Tonix (Antonio Nati) wrote: > I have customers which should be allowed to go whetever they like and > accept from all. > > So I'd love to make something like this: > > - deny on INPUT WAN from hackers/abusers > - allow any other INPUT on WAN > - allow any OUTPUT to WAN > > - custom INPUT rules on all other interfaces > - custom OUT rules on all other interfaces > > So, if a customer wants to allow anyone to access his port 80, he/she > just add that OUT rule to his/her interface. And that avoids me to add > the same rule to WAN and all remaining interfaces. That will work just fine, assuming you don't run any services on the firewall itself that need to be protected. > Respect to the dominant model (i.e. which puts any rules on INPUT only), > do you see any security hole? Or just some more processing? It's just a matter of personal preference, combined with the fact that other products don't offer the choice. Generally, it makes sense to block packets "as early as possible" (i.e. on input). But if you have specific reasons to do differently, that can make sense, too :) Daniel From owner-freebsd-pf@FreeBSD.ORG Mon Jul 23 13:54:20 2012 Return-Path: Delivered-To: freebsd-pf@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 9EE9C106564A for ; Mon, 23 Jul 2012 13:54:20 +0000 (UTC) (envelope-from jmattax@storytotell.org) Received: from mail.clanspum.net (mail.clanspum.net [69.164.206.246]) by mx1.freebsd.org (Postfix) with ESMTP id 7BC3F8FC16 for ; Mon, 23 Jul 2012 13:54:20 +0000 (UTC) Received: from mail.clanspum.net (localhost.localdomain [IPv6:::1]) by mail.clanspum.net (Postfix) with ESMTP id B0EA722400C; Mon, 23 Jul 2012 08:54:19 -0500 (CDT) Received: from 63.231.116.1 (SquirrelMail authenticated user jmattax) by mail.clanspum.net with HTTP; Mon, 23 Jul 2012 08:54:19 -0500 Message-ID: <5a7781121487392bc1d40f3ed7971692.squirrel@mail.clanspum.net> In-Reply-To: <500CF511.2030305@gmail.com> References: <2B5A7CC5-0950-47E9-928F-D5909238052C@my.gd> <500CE1B2.5040303@storytotell.org> <500CF511.2030305@gmail.com> Date: Mon, 23 Jul 2012 08:54:19 -0500 From: "Jason Mattax" To: freebsd-pf@freebsd.org User-Agent: SquirrelMail/1.4.22 MIME-Version: 1.0 Content-Type: text/plain;charset=iso-8859-1 Content-Transfer-Encoding: 8bit X-Priority: 3 (Normal) Importance: Normal Cc: Subject: Re: PF suddenly malfunctioned X-BeenThere: freebsd-pf@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "Technical discussion and general questions about packet filter \(pf\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 23 Jul 2012 13:54:20 -0000 calderon81@gmail.com wrote > I have the same problem, although i remember having it from the start. I > started with some basic example configuration for gateway. Noticed that > some sites would'nt load ans some do.. exc. googles. > > Added pass all rule for Internal IF on the right spot, and it works. > Would be nice to hear more. > > I actually had a much more complicated set of rules running that did include a pass all for my Internal IF and it was still broken. Also, I believe the pass all in and pass all out rules are applied to my internal interface. Also, FYI you didn't actually send this to the freebsd-pf mailing list, it automatically is in the CC field (at least for me) so hitting reply doesn't send it to the list. From owner-freebsd-pf@FreeBSD.ORG Mon Jul 23 14:01:19 2012 Return-Path: Delivered-To: freebsd-pf@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id E3C891065678 for ; Mon, 23 Jul 2012 14:01:19 +0000 (UTC) (envelope-from jmattax@storytotell.org) Received: from mail.clanspum.net (twopir-2-pt.tunnel.tserv9.chi1.ipv6.he.net [IPv6:2001:470:1f10:1b9::2]) by mx1.freebsd.org (Postfix) with ESMTP id A09E78FC18 for ; Mon, 23 Jul 2012 14:01:19 +0000 (UTC) Received: from mail.clanspum.net (localhost.localdomain [IPv6:::1]) by mail.clanspum.net (Postfix) with ESMTP id 4202A22400C; Mon, 23 Jul 2012 09:01:18 -0500 (CDT) Received: from 63.231.116.1 (SquirrelMail authenticated user jmattax) by mail.clanspum.net with HTTP; Mon, 23 Jul 2012 09:01:18 -0500 Message-ID: <04e3e73987e308c73f65a95e16022573.squirrel@mail.clanspum.net> In-Reply-To: <500D1595.4010405@my.gd> References: <2B5A7CC5-0950-47E9-928F-D5909238052C@my.gd> <500CE1B2.5040303@storytotell.org> <500D1595.4010405@my.gd> Date: Mon, 23 Jul 2012 09:01:18 -0500 From: "Jason Mattax" To: "Damien Fleuriot" User-Agent: SquirrelMail/1.4.22 MIME-Version: 1.0 Content-Type: text/plain;charset=iso-8859-1 Content-Transfer-Encoding: 8bit X-Priority: 3 (Normal) Importance: Normal Cc: Jason Mattax , freebsd-pf@freebsd.org Subject: Re: PF suddenly malfunctioned X-BeenThere: freebsd-pf@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "Technical discussion and general questions about packet filter \(pf\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 23 Jul 2012 14:01:20 -0000 On Mon, July 23, 2012 04:12, Damien Fleuriot wrote: > > > On 7/23/12 7:31 AM, Jason Mattax wrote: >> >> based on that I could easily upgrade to 8.3, or possibly 9.0 tomorrow if >> I have the inclination. >> > > I can recommend 8.3, we're using it widely in production. > Thanks. > >>> 2/ When the problem appears. Have you tried disabling PF ? (pfctl -d) >>> Does it help ? >>> >> Since I can consistently reproduce the problem with en.wikipedia.org I >> have a good way to test. When I run pfctl -d on the firewall it looks >> like no traffic is being forwarded, including DNS so I eventually get a >> notice that the web page timed out because I typed the address wrong. >> That is as opposed to the web browser saying waiting for >> en.wikipedia.org (and if I recall correctly occasionally getting the >> redirect to en.wikipedia.org/wiki/Main_Page.) I just tested and got >> stuck at the waiting for en.wikipedia.org for a couple of minutes before >> I called it good enough to report here. >> > > Keep in mind that after disabling PF you don't get NAT anymore from your > workstations through the firewall. > > So any test you run while PF is disabled has to be run from the PF box > itself. > That's what I thought, but the firewall itself can see the outside network just fine whether pf is running or not (I just rechecked that.) From owner-freebsd-pf@FreeBSD.ORG Tue Jul 24 03:10:02 2012 Return-Path: Delivered-To: freebsd-pf@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id B49CD106566B for ; Tue, 24 Jul 2012 03:10:02 +0000 (UTC) (envelope-from jmattax@storytotell.org) Received: from mail.clanspum.net (mail.clanspum.net [69.164.206.246]) by mx1.freebsd.org (Postfix) with ESMTP id 691388FC16 for ; Tue, 24 Jul 2012 03:10:02 +0000 (UTC) Received: from [192.168.0.14] (unknown [63.231.116.1]) (using TLSv1 with cipher ECDHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mail.clanspum.net (Postfix) with ESMTPSA id AAD5922400C; Mon, 23 Jul 2012 22:10:00 -0500 (CDT) Message-ID: <500E1202.20108@storytotell.org> Date: Mon, 23 Jul 2012 21:09:54 -0600 From: Jason Mattax User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:14.0) Gecko/20120713 Thunderbird/14.0 MIME-Version: 1.0 To: Daniel Hartmeier References: <20120723100521.GC32530@insomnia.benzedrine.cx> In-Reply-To: <20120723100521.GC32530@insomnia.benzedrine.cx> Content-Type: multipart/mixed; boundary="------------060703070603060501020404" Cc: jmattax@clanspum.net, freebsd-pf@freebsd.org Subject: Re: PF suddenly malfunctioned X-BeenThere: freebsd-pf@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "Technical discussion and general questions about packet filter \(pf\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 24 Jul 2012 03:10:02 -0000 This is a multi-part message in MIME format. --------------060703070603060501020404 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit On 7/23/2012 4:05 AM, Daniel Hartmeier wrote: > If you can reliably reproduce the problem with en.wikipedia.org, I > suggest the following: > > On the firewall > > 1) enable verbose logging with pfctl -xm > 2) save the output of pfctl -si and netstat -s > 3) run the following three tcpdump in parallel, and save the output: > tcpdump -s 1600 -nvvvpSi xl0 'host 91.198.174.225' > tcpdump -s 1600 -nvvvpSi re0 'host 91.198.174.225' > tcpdump -s 1600 -nvvveeepi pflog0 > > On a client > > 4) printf "GET /wiki/Main_Page HTTP/1.1\r\nHost: en.wikipedia.org\r\n\r\n" | > nc -v 91.198.174.225 80 | wc -c > 5) this should hang until some timout occurs, you need only wait 10s. > > Back on the firewall > > 6) re-run pfctl -si and netstat -s (again saving the output) > 7) stop the tcpdumps > 8) check /var/log/messages for anything from pf > > The post the outputs :) > > Daniel > The files are attached, it should be noted that I did the run I'm posting around 21:00 according to my servers clock. There were no messages about the above in /var/log/messages but there were some messages from earlier in the day. The reason it took me so long to get this posted is that I was (and still am) getting unexpected output from the netcat above, when I run the netcat I nearly immediately get a notice that the connection succeeded, so I decided to look at what the server was sending me, as it turns out it was only sending me whitespace if anything. You can see a copy and pate of the command line below. Thanks for looking at this. Jason Mattax --------------060703070603060501020404 Content-Type: text/plain; charset=windows-1252; name="messages" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="messages" Jul 23 16:24:58 stilgar kernel: pf: state reuse TCP 192.168.0.200:139 192.168.0.200:139 24.123.237.238:34820 [lo=3243560508 high=3243560510 win=15088 modulator=0] [lo=0 high=15088 win=1 modulator=0] 10:10 S Jul 23 16:24:58 stilgar kernel: pf: state reuse TCP 192.168.0.200:139 192.168.0.200:139 24.123.237.238:34820 [lo=3243560508 high=3243560510 win=15088 modulator=0] [lo=0 high=15088 win=1 modulator=0] 10:10 S Jul 23 16:25:04 stilgar kernel: pf: state reuse TCP 192.168.0.200:445 192.168.0.200:445 24.123.237.238:34871 [lo=3247592298 high=3247592300 win=15088 modulator=0] [lo=0 high=15088 win=1 modulator=0] 10:10 S Jul 23 16:25:04 stilgar kernel: pf: state reuse TCP 192.168.0.200:445 192.168.0.200:445 24.123.237.238:34871 [lo=3247592298 high=3247592300 win=15088 modulator=0] [lo=0 high=15088 win=1 modulator=0] 10:10 S Jul 23 17:53:04 stilgar kernel: pf: state reuse TCP 192.168.0.200:4899 192.168.0.200:4899 80.32.31.160:2205 [lo=47482671 high=47482673 win=65535 modulator=0] [lo=0 high=65535 win=1 modulator=0] 10:10 S Jul 23 17:53:05 stilgar kernel: pf: state reuse TCP 192.168.0.200:4899 192.168.0.200:4899 80.32.31.160:2205 [lo=47482671 high=47482673 win=65535 modulator=0] [lo=0 high=65535 win=1 modulator=0] 10:10 S --------------060703070603060501020404 Content-Type: text/plain; charset=windows-1252; name="netcat" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="netcat" jmattax@chani:~$ printf "GET /wiki/Main_Page HTTP/1.1\r\nHost: en.wikipedia.org\r\n\r\n" | nc -v 91.198.174.225 80 Connection to 91.198.174.225 80 port [tcp/http] succeeded! --------------060703070603060501020404 Content-Type: text/plain; charset=windows-1252; name="netstat_after" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="netstat_after" tcp: 3880 packets sent 1339 data packets (297910 bytes) 41 data packets (13121 bytes) retransmitted 0 data packets unnecessarily retransmitted 3 resends initiated by MTU discovery 2374 ack-only packets (141 delayed) 0 URG only packets 0 window probe packets 63 window update packets 63 control packets 6316 packets received 1219 acks (for 300091 bytes) 46 duplicate acks 0 acks for unsent data 5390 packets (6205996 bytes) received in-sequence 5 completely duplicate packets (2920 bytes) 0 old duplicate packets 0 packets with some dup. data (0 bytes duped) 24 out-of-order packets (19313 bytes) 0 packets (0 bytes) of data after window 0 window probes 6 window update packets 4 packets received after close 0 discarded for bad checksums 0 discarded for bad header offset fields 0 discarded because packet too short 0 discarded due to memory problems 17 connection requests 29 connection accepts 0 bad connection attempts 0 listen queue overflows 1 ignored RSTs in the window 45 connections established (including accepts) 55 connections closed (including 4 drops) 34 connections updated cached RTT on close 36 connections updated cached RTT variance on close 5 connections updated cached ssthresh on close 1 embryonic connection dropped 1213 segments updated rtt (of 1181 attempts) 47 retransmit timeouts 3 connections dropped by rexmit timeout 0 persist timeouts 0 connections dropped by persist timeout 0 Connections (fin_wait_2) dropped because of timeout 9 keepalive timeouts 8 keepalive probes sent 1 connection dropped by keepalive 1 correct ACK header prediction 4887 correct data packet header predictions 32 syncache entries added 0 retransmitted 0 dupsyn 0 dropped 29 completed 0 bucket overflow 0 cache overflow 3 reset 0 stale 0 aborted 0 badack 0 unreach 0 zone failures 32 cookies sent 0 cookies received 0 SACK recovery episodes 0 segment rexmits in SACK recovery episodes 0 byte rexmits in SACK recovery episodes 3 SACK options (SACK blocks) received 23 SACK options (SACK blocks) sent 0 SACK scoreboard overflow 0 packets with ECN CE bit set 0 packets with ECN ECT(0) bit set 0 packets with ECN ECT(1) bit set 0 successful ECN handshakes 0 times ECN reduced the congestion window udp: 2751 datagrams received 0 with incomplete header 0 with bad data length field 0 with bad checksum 1 with no checksum 146 dropped due to no socket 2474 broadcast/multicast datagrams undelivered 0 dropped due to full socket buffers 0 not for hashed pcb 131 delivered 248 datagrams output 0 times multicast source filter matched sctp: 0 input packets 0 datagrams 0 packets that had data 0 input SACK chunks 0 input DATA chunks 0 duplicate DATA chunks 0 input HB chunks 0 HB-ACK chunks 0 input ECNE chunks 0 input AUTH chunks 0 chunks missing AUTH 0 invalid HMAC ids received 0 invalid secret ids received 0 auth failed 0 fast path receives all one chunk 0 fast path multi-part data 0 output packets 0 output SACKs 0 output DATA chunks 0 retransmitted DATA chunks 0 fast retransmitted DATA chunks 0 FR's that happened more than once to same chunk 0 intput HB chunks 0 output ECNE chunks 0 output AUTH chunks 0 ip_output error counter Packet drop statistics: 0 from middle box 0 from end host 0 with data 0 non-data, non-endhost 0 non-endhost, bandwidth rep only 0 not enough for chunk header 0 not enough data to confirm 0 where process_chunk_drop said break 0 failed to find TSN 0 attempt reverse TSN lookup 0 e-host confirms zero-rwnd 0 midbox confirms no space 0 data did not match TSN 0 TSN's marked for Fast Retran Timeouts: 0 iterator timers fired 0 T3 data time outs 0 window probe (T3) timers fired 0 INIT timers fired 0 sack timers fired 0 shutdown timers fired 0 heartbeat timers fired 0 a cookie timeout fired 0 an endpoint changed its cookiesecret 0 PMTU timers fired 0 shutdown ack timers fired 0 shutdown guard timers fired 0 stream reset timers fired 0 early FR timers fired 0 an asconf timer fired 0 auto close timer fired 0 asoc free timers expired 0 inp free timers expired 0 packet shorter than header 0 checksum error 0 no endpoint for port 0 bad v-tag 0 bad SID 0 no memory 0 number of multiple FR in a RTT window 0 RFC813 allowed sending 0 RFC813 does not allow sending 0 times max burst prohibited sending 0 look ahead tells us no memory in interface 0 numbers of window probes sent 0 times an output error to clamp down on next user send 0 times sctp_senderrors were caused from a user 0 number of in data drops due to chunk limit reached 0 number of in data drops due to rwnd limit reached 0 times a ECN reduced the cwnd 0 used express lookup via vtag 0 collision in express lookup 0 times the sender ran dry of user data on primary 0 same for above 0 sacks the slow way 0 window update only sacks sent 0 sends with sinfo_flags !=0 0 unordered sends 0 sends with EOF flag set 0 sends with ABORT flag set 0 times protocol drain called 0 times we did a protocol drain 0 times recv was called with peek 0 cached chunks used 0 cached stream oq's used 0 unread messages abandonded by close 0 send burst avoidance, already max burst inflight to net 0 send cwnd full avoidance, already max burst inflight to net 0 number of map array over-runs via fwd-tsn's ip: 30044 total packets received 0 bad header checksums 0 with size smaller than minimum 0 with data size < data length 0 with ip length > max ip packet size 0 with header length < data size 0 with data length < header length 0 with bad options 0 with incorrect version number 0 fragments received 0 fragments dropped (dup or out of space) 0 fragments dropped after timeout 0 packets reassembled ok 9082 packets for this host 111 packets for unknown/unsupported protocol 20818 packets forwarded (0 packets fast forwarded) 33 packets not forwardable 0 packets received for unknown multicast group 0 redirects sent 4387 packets sent from this host 0 packets sent with fabricated ip header 0 output packets dropped due to no bufs, etc. 0 output packets discarded due to no route 0 output datagrams fragmented 0 fragments created 0 datagrams that can't be fragmented 0 tunneling packets that can't find gif 0 datagrams with bad address in header icmp: 148 calls to icmp_error 0 errors not generated in response to an icmp message Output histogram: echo reply: 15 destination unreachable: 148 0 messages with bad code fields 0 messages less than the minimum length 0 messages with bad checksum 0 messages with bad length 0 multicast echo requests ignored 0 multicast timestamp requests ignored Input histogram: destination unreachable: 111 echo: 15 15 message responses generated 0 invalid return addresses 0 no return routes ICMP address mask responses are disabled igmp: 0 messages received 0 messages received with too few bytes 0 messages received with wrong TTL 0 messages received with bad checksum 0 V1/V2 membership queries received 0 V3 membership queries received 0 membership queries received with invalid field(s) 0 general queries received 0 group queries received 0 group-source queries received 0 group-source queries dropped 0 membership reports received 0 membership reports received with invalid field(s) 0 membership reports received for groups to which we belong 0 V3 reports received without Router Alert 0 membership reports sent arp: 146 ARP requests sent 1627 ARP replies sent 22184 ARP requests received 7 ARP replies received 22191 ARP packets received 84 total packets dropped due to no ARP entry 69 ARP entrys timed out 0 Duplicate IPs seen ip6: 0 total packets received 0 with size smaller than minimum 0 with data size < data length 0 with bad options 0 with incorrect version number 0 fragments received 0 fragments dropped (dup or out of space) 0 fragments dropped after timeout 0 fragments that exceeded limit 0 packets reassembled ok 0 packets for this host 0 packets forwarded 0 packets not forwardable 0 redirects sent 0 packets sent from this host 0 packets sent with fabricated ip header 0 output packets dropped due to no bufs, etc. 7 output packets discarded due to no route 0 output datagrams fragmented 0 fragments created 0 datagrams that can't be fragmented 0 packets that violated scope rules 0 multicast packets which we don't join Mbuf statistics: 0 one mbuf 0 one ext mbuf 0 two or more ext mbuf 0 packets whose headers are not continuous 0 tunneling packets that can't find gif 0 packets discarded because of too many headers 0 failures of source address selection Source addresses selection rule applied: icmp6: 0 calls to icmp6_error 0 errors not generated in response to an icmp6 message 0 errors not generated because of rate limitation 0 messages with bad code fields 0 messages < minimum length 0 bad checksums 0 messages with bad length Histogram of error messages to be generated: 0 no route 0 administratively prohibited 0 beyond scope 0 address unreachable 0 port unreachable 0 packet too big 0 time exceed transit 0 time exceed reassembly 0 erroneous header field 0 unrecognized next header 0 unrecognized option 0 redirect 0 unknown 0 message responses generated 0 messages with too many ND options 0 messages with bad ND options 0 bad neighbor solicitation messages 0 bad neighbor advertisement messages 0 bad router solicitation messages 0 bad router advertisement messages 0 bad redirect messages 0 path MTU changes rip6: 0 messages received 0 checksum calculations on inbound 0 messages with bad checksum 0 messages dropped due to no socket 0 multicast messages dropped due to no socket 0 messages dropped due to full socket buffers 0 delivered 0 datagrams output --------------060703070603060501020404 Content-Type: text/plain; charset=windows-1252; name="netstat_before" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="netstat_before" tcp: 3786 packets sent 1255 data packets (275510 bytes) 41 data packets (13121 bytes) retransmitted 0 data packets unnecessarily retransmitted 3 resends initiated by MTU discovery 2364 ack-only packets (132 delayed) 0 URG only packets 0 window probe packets 63 window update packets 63 control packets 6192 packets received 1156 acks (for 277691 bytes) 46 duplicate acks 0 acks for unsent data 5329 packets (6202824 bytes) received in-sequence 5 completely duplicate packets (2920 bytes) 0 old duplicate packets 0 packets with some dup. data (0 bytes duped) 24 out-of-order packets (19313 bytes) 0 packets (0 bytes) of data after window 0 window probes 6 window update packets 4 packets received after close 0 discarded for bad checksums 0 discarded for bad header offset fields 0 discarded because packet too short 0 discarded due to memory problems 17 connection requests 29 connection accepts 0 bad connection attempts 0 listen queue overflows 1 ignored RSTs in the window 45 connections established (including accepts) 55 connections closed (including 4 drops) 34 connections updated cached RTT on close 36 connections updated cached RTT variance on close 5 connections updated cached ssthresh on close 1 embryonic connection dropped 1151 segments updated rtt (of 1119 attempts) 47 retransmit timeouts 3 connections dropped by rexmit timeout 0 persist timeouts 0 connections dropped by persist timeout 0 Connections (fin_wait_2) dropped because of timeout 9 keepalive timeouts 8 keepalive probes sent 1 connection dropped by keepalive 1 correct ACK header prediction 4826 correct data packet header predictions 32 syncache entries added 0 retransmitted 0 dupsyn 0 dropped 29 completed 0 bucket overflow 0 cache overflow 3 reset 0 stale 0 aborted 0 badack 0 unreach 0 zone failures 32 cookies sent 0 cookies received 0 SACK recovery episodes 0 segment rexmits in SACK recovery episodes 0 byte rexmits in SACK recovery episodes 3 SACK options (SACK blocks) received 23 SACK options (SACK blocks) sent 0 SACK scoreboard overflow 0 packets with ECN CE bit set 0 packets with ECN ECT(0) bit set 0 packets with ECN ECT(1) bit set 0 successful ECN handshakes 0 times ECN reduced the congestion window udp: 2751 datagrams received 0 with incomplete header 0 with bad data length field 0 with bad checksum 1 with no checksum 146 dropped due to no socket 2474 broadcast/multicast datagrams undelivered 0 dropped due to full socket buffers 0 not for hashed pcb 131 delivered 248 datagrams output 0 times multicast source filter matched sctp: 0 input packets 0 datagrams 0 packets that had data 0 input SACK chunks 0 input DATA chunks 0 duplicate DATA chunks 0 input HB chunks 0 HB-ACK chunks 0 input ECNE chunks 0 input AUTH chunks 0 chunks missing AUTH 0 invalid HMAC ids received 0 invalid secret ids received 0 auth failed 0 fast path receives all one chunk 0 fast path multi-part data 0 output packets 0 output SACKs 0 output DATA chunks 0 retransmitted DATA chunks 0 fast retransmitted DATA chunks 0 FR's that happened more than once to same chunk 0 intput HB chunks 0 output ECNE chunks 0 output AUTH chunks 0 ip_output error counter Packet drop statistics: 0 from middle box 0 from end host 0 with data 0 non-data, non-endhost 0 non-endhost, bandwidth rep only 0 not enough for chunk header 0 not enough data to confirm 0 where process_chunk_drop said break 0 failed to find TSN 0 attempt reverse TSN lookup 0 e-host confirms zero-rwnd 0 midbox confirms no space 0 data did not match TSN 0 TSN's marked for Fast Retran Timeouts: 0 iterator timers fired 0 T3 data time outs 0 window probe (T3) timers fired 0 INIT timers fired 0 sack timers fired 0 shutdown timers fired 0 heartbeat timers fired 0 a cookie timeout fired 0 an endpoint changed its cookiesecret 0 PMTU timers fired 0 shutdown ack timers fired 0 shutdown guard timers fired 0 stream reset timers fired 0 early FR timers fired 0 an asconf timer fired 0 auto close timer fired 0 asoc free timers expired 0 inp free timers expired 0 packet shorter than header 0 checksum error 0 no endpoint for port 0 bad v-tag 0 bad SID 0 no memory 0 number of multiple FR in a RTT window 0 RFC813 allowed sending 0 RFC813 does not allow sending 0 times max burst prohibited sending 0 look ahead tells us no memory in interface 0 numbers of window probes sent 0 times an output error to clamp down on next user send 0 times sctp_senderrors were caused from a user 0 number of in data drops due to chunk limit reached 0 number of in data drops due to rwnd limit reached 0 times a ECN reduced the cwnd 0 used express lookup via vtag 0 collision in express lookup 0 times the sender ran dry of user data on primary 0 same for above 0 sacks the slow way 0 window update only sacks sent 0 sends with sinfo_flags !=0 0 unordered sends 0 sends with EOF flag set 0 sends with ABORT flag set 0 times protocol drain called 0 times we did a protocol drain 0 times recv was called with peek 0 cached chunks used 0 cached stream oq's used 0 unread messages abandonded by close 0 send burst avoidance, already max burst inflight to net 0 send cwnd full avoidance, already max burst inflight to net 0 number of map array over-runs via fwd-tsn's ip: 29911 total packets received 0 bad header checksums 0 with size smaller than minimum 0 with data size < data length 0 with ip length > max ip packet size 0 with header length < data size 0 with data length < header length 0 with bad options 0 with incorrect version number 0 fragments received 0 fragments dropped (dup or out of space) 0 fragments dropped after timeout 0 packets reassembled ok 8958 packets for this host 111 packets for unknown/unsupported protocol 20809 packets forwarded (0 packets fast forwarded) 33 packets not forwardable 0 packets received for unknown multicast group 0 redirects sent 4293 packets sent from this host 0 packets sent with fabricated ip header 0 output packets dropped due to no bufs, etc. 0 output packets discarded due to no route 0 output datagrams fragmented 0 fragments created 0 datagrams that can't be fragmented 0 tunneling packets that can't find gif 0 datagrams with bad address in header icmp: 148 calls to icmp_error 0 errors not generated in response to an icmp message Output histogram: echo reply: 15 destination unreachable: 148 0 messages with bad code fields 0 messages less than the minimum length 0 messages with bad checksum 0 messages with bad length 0 multicast echo requests ignored 0 multicast timestamp requests ignored Input histogram: destination unreachable: 111 echo: 15 15 message responses generated 0 invalid return addresses 0 no return routes ICMP address mask responses are disabled igmp: 0 messages received 0 messages received with too few bytes 0 messages received with wrong TTL 0 messages received with bad checksum 0 V1/V2 membership queries received 0 V3 membership queries received 0 membership queries received with invalid field(s) 0 general queries received 0 group queries received 0 group-source queries received 0 group-source queries dropped 0 membership reports received 0 membership reports received with invalid field(s) 0 membership reports received for groups to which we belong 0 V3 reports received without Router Alert 0 membership reports sent arp: 146 ARP requests sent 1626 ARP replies sent 22177 ARP requests received 7 ARP replies received 22184 ARP packets received 84 total packets dropped due to no ARP entry 69 ARP entrys timed out 0 Duplicate IPs seen ip6: 0 total packets received 0 with size smaller than minimum 0 with data size < data length 0 with bad options 0 with incorrect version number 0 fragments received 0 fragments dropped (dup or out of space) 0 fragments dropped after timeout 0 fragments that exceeded limit 0 packets reassembled ok 0 packets for this host 0 packets forwarded 0 packets not forwardable 0 redirects sent 0 packets sent from this host 0 packets sent with fabricated ip header 0 output packets dropped due to no bufs, etc. 7 output packets discarded due to no route 0 output datagrams fragmented 0 fragments created 0 datagrams that can't be fragmented 0 packets that violated scope rules 0 multicast packets which we don't join Mbuf statistics: 0 one mbuf 0 one ext mbuf 0 two or more ext mbuf 0 packets whose headers are not continuous 0 tunneling packets that can't find gif 0 packets discarded because of too many headers 0 failures of source address selection Source addresses selection rule applied: icmp6: 0 calls to icmp6_error 0 errors not generated in response to an icmp6 message 0 errors not generated because of rate limitation 0 messages with bad code fields 0 messages < minimum length 0 bad checksums 0 messages with bad length Histogram of error messages to be generated: 0 no route 0 administratively prohibited 0 beyond scope 0 address unreachable 0 port unreachable 0 packet too big 0 time exceed transit 0 time exceed reassembly 0 erroneous header field 0 unrecognized next header 0 unrecognized option 0 redirect 0 unknown 0 message responses generated 0 messages with too many ND options 0 messages with bad ND options 0 bad neighbor solicitation messages 0 bad neighbor advertisement messages 0 bad router solicitation messages 0 bad router advertisement messages 0 bad redirect messages 0 path MTU changes rip6: 0 messages received 0 checksum calculations on inbound 0 messages with bad checksum 0 messages dropped due to no socket 0 multicast messages dropped due to no socket 0 messages dropped due to full socket buffers 0 delivered 0 datagrams output --------------060703070603060501020404 Content-Type: text/plain; charset=windows-1252; name="pfctl_after" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="pfctl_after" No ALTQ support in kernel ALTQ related functions disabled Status: Enabled for 0 days 21:47:22 Debug: Misc State Table Total Rate current entries 20 searches 55249 0.7/s inserts 1901 0.0/s removals 1881 0.0/s Counters match 1917 0.0/s bad-offset 0 0.0/s fragment 0 0.0/s short 0 0.0/s normalize 0 0.0/s memory 0 0.0/s bad-timestamp 0 0.0/s congestion 0 0.0/s ip-option 0 0.0/s proto-cksum 0 0.0/s state-mismatch 0 0.0/s state-insert 0 0.0/s state-limit 0 0.0/s src-limit 0 0.0/s synproxy 0 0.0/s --------------060703070603060501020404 Content-Type: text/plain; charset=windows-1252; name="pfctl_before" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="pfctl_before" No ALTQ support in kernel ALTQ related functions disabled Status: Enabled for 0 days 21:46:41 Debug: Misc State Table Total Rate current entries 21 searches 55023 0.7/s inserts 1899 0.0/s removals 1878 0.0/s Counters match 1915 0.0/s bad-offset 0 0.0/s fragment 0 0.0/s short 0 0.0/s normalize 0 0.0/s memory 0 0.0/s bad-timestamp 0 0.0/s congestion 0 0.0/s ip-option 0 0.0/s proto-cksum 0 0.0/s state-mismatch 0 0.0/s state-insert 0 0.0/s state-limit 0 0.0/s src-limit 0 0.0/s synproxy 0 0.0/s --------------060703070603060501020404 Content-Type: text/plain; charset=windows-1252; name="pflog0_tcpdump" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="pflog0_tcpdump" --------------060703070603060501020404 Content-Type: text/plain; charset=windows-1252; name="re0_tcpdump" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="re0_tcpdump" 20:56:23.455030 IP (tos 0x0, ttl 64, id 50886, offset 0, flags [DF], proto TCP (6), length 60) 10.11.10.45.51996 > 91.198.174.225.80: Flags [S], cksum 0x34cc (correct), seq 3868567477, win 14600, options [mss 1460,sackOK,TS val 2384243 ecr 0,nop,wscale 4], length 0 20:56:23.633425 IP (tos 0x0, ttl 52, id 0, offset 0, flags [DF], proto TCP (6), length 60, bad cksum 0 (->27dd)!) 91.198.174.225.80 > 10.11.10.45.51996: Flags [S.], cksum 0x95a1 (correct), seq 2727041994, ack 3868567478, win 5792, options [mss 1460,sackOK,TS val 669489983 ecr 2384243,nop,wscale 9], length 0 20:56:23.634947 IP (tos 0x0, ttl 64, id 50887, offset 0, flags [DF], proto TCP (6), length 52) 10.11.10.45.51996 > 91.198.174.225.80: Flags [.], cksum 0xd751 (correct), seq 3868567478, ack 2727041995, win 913, options [nop,nop,TS val 2384288 ecr 669489983], length 0 20:56:23.635166 IP (tos 0x0, ttl 64, id 50888, offset 0, flags [DF], proto TCP (6), length 108) 10.11.10.45.51996 > 91.198.174.225.80: Flags [P.], cksum 0x6f6b (correct), seq 3868567478:3868567534, ack 2727041995, win 913, options [nop,nop,TS val 2384288 ecr 669489983], length 56 20:56:23.635810 IP (tos 0x0, ttl 64, id 50889, offset 0, flags [DF], proto TCP (6), length 52) 10.11.10.45.51996 > 91.198.174.225.80: Flags [F.], cksum 0xd718 (correct), seq 3868567534, ack 2727041995, win 913, options [nop,nop,TS val 2384288 ecr 669489983], length 0 20:56:23.813427 IP (tos 0x0, ttl 52, id 49306, offset 0, flags [DF], proto TCP (6), length 64, bad cksum 0 (->673e)!) 91.198.174.225.80 > 10.11.10.45.51996: Flags [.], cksum 0x87a3 (correct), seq 2727041995, ack 3868567478, win 12, options [nop,nop,TS val 669490001 ecr 2384288,nop,nop,sack 1 {3868567534:3868567535}], length 0 20:56:23.814752 IP (tos 0x0, ttl 52, id 49307, offset 0, flags [DF], proto TCP (6), length 52, bad cksum 0 (->6749)!) 91.198.174.225.80 > 10.11.10.45.51996: Flags [.], cksum 0xda8b (correct), seq 2727041995, ack 3868567535, win 12, options [nop,nop,TS val 669490001 ecr 2384288], length 0 20:56:23.815233 IP (tos 0x0, ttl 52, id 49308, offset 0, flags [DF], proto TCP (6), length 52, bad cksum 0 (->6748)!) 91.198.174.225.80 > 10.11.10.45.51996: Flags [F.], cksum 0xda8a (correct), seq 2727041995, ack 3868567535, win 12, options [nop,nop,TS val 669490001 ecr 2384288], length 0 20:56:23.816529 IP (tos 0x0, ttl 64, id 50890, offset 0, flags [DF], proto TCP (6), length 52) 10.11.10.45.51996 > 91.198.174.225.80: Flags [.], cksum 0xd6d8 (correct), seq 3868567535, ack 2727041996, win 913, options [nop,nop,TS val 2384333 ecr 669490001], length 0 --------------060703070603060501020404 Content-Type: text/plain; charset=windows-1252; name="xl0_tcpdump" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="xl0_tcpdump" 20:56:23.455415 IP (tos 0x0, ttl 63, id 50886, offset 0, flags [DF], proto TCP (6), length 60) 192.168.0.200.64834 > 91.198.174.225.80: Flags [S], cksum 0x556d (correct), seq 3868567477, win 14600, options [mss 1460,sackOK,TS val 2384243 ecr 0,nop,wscale 4], length 0 20:56:23.633234 IP (tos 0x0, ttl 53, id 0, offset 0, flags [DF], proto TCP (6), length 60) 91.198.174.225.80 > 192.168.0.200.64834: Flags [S.], cksum 0xb642 (correct), seq 2727041994, ack 3868567478, win 5792, options [mss 1460,sackOK,TS val 669489983 ecr 2384243,nop,wscale 9], length 0 20:56:23.635087 IP (tos 0x0, ttl 63, id 50887, offset 0, flags [DF], proto TCP (6), length 52) 192.168.0.200.64834 > 91.198.174.225.80: Flags [.], cksum 0xf7f2 (correct), seq 3868567478, ack 2727041995, win 913, options [nop,nop,TS val 2384288 ecr 669489983], length 0 20:56:23.635277 IP (tos 0x0, ttl 63, id 50888, offset 0, flags [DF], proto TCP (6), length 108) 192.168.0.200.64834 > 91.198.174.225.80: Flags [P.], cksum 0x900c (correct), seq 3868567478:3868567534, ack 2727041995, win 913, options [nop,nop,TS val 2384288 ecr 669489983], length 56 20:56:23.635923 IP (tos 0x0, ttl 63, id 50889, offset 0, flags [DF], proto TCP (6), length 52) 192.168.0.200.64834 > 91.198.174.225.80: Flags [F.], cksum 0xf7b9 (correct), seq 3868567534, ack 2727041995, win 913, options [nop,nop,TS val 2384288 ecr 669489983], length 0 20:56:23.813258 IP (tos 0x0, ttl 53, id 49306, offset 0, flags [DF], proto TCP (6), length 64) 91.198.174.225.80 > 192.168.0.200.64834: Flags [.], cksum 0xa844 (correct), seq 2727041995, ack 3868567478, win 12, options [nop,nop,TS val 669490001 ecr 2384288,nop,nop,sack 1 {3868567534:3868567535}], length 0 20:56:23.814638 IP (tos 0x0, ttl 53, id 49307, offset 0, flags [DF], proto TCP (6), length 52) 91.198.174.225.80 > 192.168.0.200.64834: Flags [.], cksum 0xfb2c (correct), seq 2727041995, ack 3868567535, win 12, options [nop,nop,TS val 669490001 ecr 2384288], length 0 20:56:23.815114 IP (tos 0x0, ttl 53, id 49308, offset 0, flags [DF], proto TCP (6), length 52) 91.198.174.225.80 > 192.168.0.200.64834: Flags [F.], cksum 0xfb2b (correct), seq 2727041995, ack 3868567535, win 12, options [nop,nop,TS val 669490001 ecr 2384288], length 0 20:56:23.816677 IP (tos 0x0, ttl 63, id 50890, offset 0, flags [DF], proto TCP (6), length 52) 192.168.0.200.64834 > 91.198.174.225.80: Flags [.], cksum 0xf779 (correct), seq 3868567535, ack 2727041996, win 913, options [nop,nop,TS val 2384333 ecr 669490001], length 0 --------------060703070603060501020404-- From owner-freebsd-pf@FreeBSD.ORG Tue Jul 24 07:07:11 2012 Return-Path: Delivered-To: freebsd-pf@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 26CC4106564A for ; Tue, 24 Jul 2012 07:07:11 +0000 (UTC) (envelope-from dhartmei@insomnia.benzedrine.cx) Received: from insomnia.benzedrine.cx (106-30.3-213.fix.bluewin.ch [213.3.30.106]) by mx1.freebsd.org (Postfix) with ESMTP id A68C48FC12 for ; Tue, 24 Jul 2012 07:07:09 +0000 (UTC) Received: from insomnia.benzedrine.cx (localhost.benzedrine.cx [127.0.0.1]) by insomnia.benzedrine.cx (8.14.1/8.13.4) with ESMTP id q6O771qI022174 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Tue, 24 Jul 2012 09:07:02 +0200 (MEST) Received: (from dhartmei@localhost) by insomnia.benzedrine.cx (8.14.1/8.12.10/Submit) id q6O770BQ018935; Tue, 24 Jul 2012 09:07:00 +0200 (MEST) Date: Tue, 24 Jul 2012 09:07:00 +0200 From: Daniel Hartmeier To: Jason Mattax Message-ID: <20120724070700.GF32530@insomnia.benzedrine.cx> References: <20120723100521.GC32530@insomnia.benzedrine.cx> <500E1202.20108@storytotell.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <500E1202.20108@storytotell.org> User-Agent: Mutt/1.5.12-2006-07-14 Cc: jmattax@clanspum.net, freebsd-pf@freebsd.org Subject: Re: PF suddenly malfunctioned X-BeenThere: freebsd-pf@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "Technical discussion and general questions about packet filter \(pf\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 24 Jul 2012 07:07:11 -0000 What's the client OS? It looks like it might be an incompatibility between the client and the peculiar wikipedia server (or loadbalancer or proxy or whatever there is). Like the GET request gets lost, but the FIN arrives, and the server selectively ACKs the FIN, and the client doesn't retransmit the request. You ran the tcpdump for several seconds after the netcat was started? Maybe repeat it and wait longer, in case the output is buffered. The client should re-transmit. If I tcpdump the same request here, I see the server selectively ACKing FINs even when the plain ACK does so, too. I've never seen this before. Can you try disabling SACK in the client? OpenBSD: sysctl net.inet.tcp.sack=0 FreeBSD: sysctl net.inet.tcp.sack.enable=0 Linux: sysctl net.ipv4.tcp_sack=0 Daniel From owner-freebsd-pf@FreeBSD.ORG Tue Jul 24 14:42:18 2012 Return-Path: Delivered-To: freebsd-pf@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 3A36A106566B for ; Tue, 24 Jul 2012 14:42:18 +0000 (UTC) (envelope-from jmattax@storytotell.org) Received: from mail.clanspum.net (mail.clanspum.net [69.164.206.246]) by mx1.freebsd.org (Postfix) with ESMTP id 01D928FC0A for ; Tue, 24 Jul 2012 14:42:17 +0000 (UTC) Received: from [192.168.0.23] (unknown [63.231.116.1]) (using TLSv1 with cipher ECDHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mail.clanspum.net (Postfix) with ESMTPSA id E7B3722400C; Tue, 24 Jul 2012 09:42:09 -0500 (CDT) Message-ID: <500EB432.6050803@storytotell.org> Date: Tue, 24 Jul 2012 08:41:54 -0600 From: Jason Mattax User-Agent: Mozilla/5.0 (X11; Linux i686; rv:14.0) Gecko/20120714 Thunderbird/14.0 MIME-Version: 1.0 To: Daniel Hartmeier References: <20120723100521.GC32530@insomnia.benzedrine.cx> <500E1202.20108@storytotell.org> <20120724070700.GF32530@insomnia.benzedrine.cx> In-Reply-To: <20120724070700.GF32530@insomnia.benzedrine.cx> Content-Type: multipart/mixed; boundary="------------060101000404050001020902" Cc: jmattax@clanspum.net, freebsd-pf@freebsd.org Subject: Re: PF suddenly malfunctioned X-BeenThere: freebsd-pf@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "Technical discussion and general questions about packet filter \(pf\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 24 Jul 2012 14:42:18 -0000 This is a multi-part message in MIME format. --------------060101000404050001020902 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit On 07/24/2012 01:07 AM, Daniel Hartmeier wrote: > What's the client OS? > The client OS for this test is Ubuntu 12.04 LTS jmattax@chani:~/pf_debugging$ uname -a Linux chani 3.2.0-26-generic #41-Ubuntu SMP Thu Jun 14 16:26:01 UTC 2012 i686 i686 i386 GNU/Linux > It looks like it might be an incompatibility between the client and the > peculiar wikipedia server (or loadbalancer or proxy or whatever there > is). > > Like the GET request gets lost, but the FIN arrives, and the server > selectively ACKs the FIN, and the client doesn't retransmit the request. > You ran the tcpdump for several seconds after the netcat was started? > Maybe repeat it and wait longer, in case the output is buffered. The > client should re-transmit. > I initially ran the tcpdumps until the client had nc return and give me a new prompt in my shell (that took maybe a second). I just repeated it as above letting the tcpdumps run longer and it captured the same number of packets. > If I tcpdump the same request here, I see the server selectively ACKing > FINs even when the plain ACK does so, too. I've never seen this before. > > Can you try disabling SACK in the client? > > OpenBSD: sysctl net.inet.tcp.sack=0 > FreeBSD: sysctl net.inet.tcp.sack.enable=0 > Linux: sysctl net.ipv4.tcp_sack=0 I just did this, it did not change the behavior of nc it still returns in a non-noticeable amount of time. The attached files with a 2 prefix are the captures (plus like 30 seconds after) for this test. The other thing I did was I accessed the wikipedia server at 208.80.154.225 on the firewall. I did this so that I could do the nc command on the firewall, the output of the tcpdump of which is attached as xl0_tcpdump_nc and seems to have the same behavior as the nc behind the firewall. Then I could run links en.wikipedia.org/wiki/Main_Page (which works about half the time) and capture the tcpdump of that into the attached xl0_tcpd_links so that people here could look at those outputs in case they reveal something useful. In this case the links worked, in the cases where it doesn't I get a message along the bottom that reads "Received 0 B of 57 kB, avg 0 B/s, cur 0 B/s" Thanks for continuing to look at this Jason --------------060101000404050001020902 Content-Type: text/plain; charset=UTF-8; name="pflog0_tcpdump2" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="pflog0_tcpdump2" --------------060101000404050001020902 Content-Type: text/plain; charset=UTF-8; name="re0_tcpdump2" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="re0_tcpdump2" 08:28:19.083955 IP (tos 0x0, ttl 64, id 4539, offset 0, flags [DF], proto TCP (6), length 60) 10.11.10.45.52111 > 91.198.174.225.80: Flags [S], cksum 0xc9e4 (correct), seq 1860497361, win 14600, options [mss 1460,nop,nop,TS val 3669610 ecr 0,nop,wscale 4], length 0 08:28:19.260717 IP (tos 0x0, ttl 52, id 0, offset 0, flags [DF], proto TCP (6), length 60) 91.198.174.225.80 > 10.11.10.45.52111: Flags [S.], cksum 0xfea4 (correct), seq 3314014153, ack 1860497362, win 5792, options [mss 1460,nop,nop,TS val 657598734 ecr 3669610,nop,wscale 9], length 0 08:28:19.262142 IP (tos 0x0, ttl 64, id 4540, offset 0, flags [DF], proto TCP (6), length 52) 10.11.10.45.52111 > 91.198.174.225.80: Flags [.], cksum 0x3d55 (correct), seq 1860497362, ack 3314014154, win 913, options [nop,nop,TS val 3669654 ecr 657598734], length 0 08:28:19.262738 IP (tos 0x0, ttl 64, id 4541, offset 0, flags [DF], proto TCP (6), length 108) 10.11.10.45.52111 > 91.198.174.225.80: Flags [P.], cksum 0xd56e (correct), seq 1860497362:1860497418, ack 3314014154, win 913, options [nop,nop,TS val 3669654 ecr 657598734], length 56 08:28:19.262880 IP (tos 0x0, ttl 64, id 4542, offset 0, flags [DF], proto TCP (6), length 52) 10.11.10.45.52111 > 91.198.174.225.80: Flags [F.], cksum 0x3d1c (correct), seq 1860497418, ack 3314014154, win 913, options [nop,nop,TS val 3669654 ecr 657598734], length 0 08:28:19.438800 IP (tos 0x0, ttl 52, id 65258, offset 0, flags [DF], proto TCP (6), length 52) 91.198.174.225.80 > 10.11.10.45.52111: Flags [.], cksum 0x40c8 (correct), seq 3314014154, ack 1860497362, win 12, options [nop,nop,TS val 657598752 ecr 3669654], length 0 08:28:19.440698 IP (tos 0x0, ttl 52, id 65259, offset 0, flags [DF], proto TCP (6), length 52) 91.198.174.225.80 > 10.11.10.45.52111: Flags [.], cksum 0x408f (correct), seq 3314014154, ack 1860497419, win 12, options [nop,nop,TS val 657598752 ecr 3669654], length 0 08:28:19.440930 IP (tos 0x0, ttl 52, id 65260, offset 0, flags [DF], proto TCP (6), length 52) 91.198.174.225.80 > 10.11.10.45.52111: Flags [F.], cksum 0x408e (correct), seq 3314014154, ack 1860497419, win 12, options [nop,nop,TS val 657598752 ecr 3669654], length 0 08:28:19.442272 IP (tos 0x0, ttl 64, id 4543, offset 0, flags [DF], proto TCP (6), length 52) 10.11.10.45.52111 > 91.198.174.225.80: Flags [.], cksum 0x3cdc (correct), seq 1860497419, ack 3314014155, win 913, options [nop,nop,TS val 3669699 ecr 657598752], length 0 --------------060101000404050001020902 Content-Type: text/plain; charset=UTF-8; name="xl0_tcpdump2" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="xl0_tcpdump2" 08:28:19.084320 IP (tos 0x0, ttl 63, id 4539, offset 0, flags [DF], proto TCP (6), length 60) 192.168.0.200.52574 > 91.198.174.225.80: Flags [S], cksum 0x1add (correct), seq 1860497361, win 14600, options [mss 1460,nop,nop,TS val 3669610 ecr 0,nop,wscale 4], length 0 08:28:19.260526 IP (tos 0x0, ttl 53, id 0, offset 0, flags [DF], proto TCP (6), length 60) 91.198.174.225.80 > 192.168.0.200.52574: Flags [S.], cksum 0x4f9d (correct), seq 3314014153, ack 1860497362, win 5792, options [mss 1460,nop,nop,TS val 657598734 ecr 3669610,nop,wscale 9], length 0 08:28:19.262291 IP (tos 0x0, ttl 63, id 4540, offset 0, flags [DF], proto TCP (6), length 52) 192.168.0.200.52574 > 91.198.174.225.80: Flags [.], cksum 0x8e4d (correct), seq 1860497362, ack 3314014154, win 913, options [nop,nop,TS val 3669654 ecr 657598734], length 0 08:28:19.262852 IP (tos 0x0, ttl 63, id 4541, offset 0, flags [DF], proto TCP (6), length 108) 192.168.0.200.52574 > 91.198.174.225.80: Flags [P.], cksum 0x2667 (correct), seq 1860497362:1860497418, ack 3314014154, win 913, options [nop,nop,TS val 3669654 ecr 657598734], length 56 08:28:19.262978 IP (tos 0x0, ttl 63, id 4542, offset 0, flags [DF], proto TCP (6), length 52) 192.168.0.200.52574 > 91.198.174.225.80: Flags [F.], cksum 0x8e14 (correct), seq 1860497418, ack 3314014154, win 913, options [nop,nop,TS val 3669654 ecr 657598734], length 0 08:28:19.438626 IP (tos 0x0, ttl 53, id 65258, offset 0, flags [DF], proto TCP (6), length 52) 91.198.174.225.80 > 192.168.0.200.52574: Flags [.], cksum 0x91c0 (correct), seq 3314014154, ack 1860497362, win 12, options [nop,nop,TS val 657598752 ecr 3669654], length 0 08:28:19.440585 IP (tos 0x0, ttl 53, id 65259, offset 0, flags [DF], proto TCP (6), length 52) 91.198.174.225.80 > 192.168.0.200.52574: Flags [.], cksum 0x9187 (correct), seq 3314014154, ack 1860497419, win 12, options [nop,nop,TS val 657598752 ecr 3669654], length 0 08:28:19.440823 IP (tos 0x0, ttl 53, id 65260, offset 0, flags [DF], proto TCP (6), length 52) 91.198.174.225.80 > 192.168.0.200.52574: Flags [F.], cksum 0x9186 (correct), seq 3314014154, ack 1860497419, win 12, options [nop,nop,TS val 657598752 ecr 3669654], length 0 08:28:19.442415 IP (tos 0x0, ttl 63, id 4543, offset 0, flags [DF], proto TCP (6), length 52) 192.168.0.200.52574 > 91.198.174.225.80: Flags [.], cksum 0x8dd4 (correct), seq 1860497419, ack 3314014155, win 913, options [nop,nop,TS val 3669699 ecr 657598752], length 0 --------------060101000404050001020902 Content-Type: text/plain; charset=UTF-8; name="xl0_tcpdump_links" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="xl0_tcpdump_links" 08:12:52.860041 IP (tos 0x0, ttl 64, id 4775, offset 0, flags [DF], proto TCP (6), length 60) 192.168.0.200.27797 > 208.80.154.225.80: Flags [S], cksum 0x74ff (correct), seq 1652546302, win 65535, options [mss 1460,nop,wscale 3,sackOK,TS val 35518945 ecr 0], length 0 08:12:52.954499 IP (tos 0x0, ttl 58, id 0, offset 0, flags [DF], proto TCP (6), length 60) 208.80.154.225.80 > 192.168.0.200.27797: Flags [S.], cksum 0x2dc8 (correct), seq 3002976197, ack 1652546303, win 5792, options [mss 1460,sackOK,TS val 1544640939 ecr 35518945,nop,wscale 9], length 0 08:12:52.954796 IP (tos 0x0, ttl 64, id 4777, offset 0, flags [DF], proto TCP (6), length 52) 192.168.0.200.27797 > 208.80.154.225.80: Flags [.], cksum 0x5251 (correct), seq 1652546303, ack 3002976198, win 8326, options [nop,nop,TS val 35519040 ecr 1544640939], length 0 08:12:52.956096 IP (tos 0x0, ttl 64, id 4778, offset 0, flags [DF], proto TCP (6), length 502) 192.168.0.200.27797 > 208.80.154.225.80: Flags [P.], cksum 0xe230 (correct), seq 1652546303:1652546753, ack 3002976198, win 8326, options [nop,nop,TS val 35519041 ecr 1544640939], length 450 08:12:53.055505 IP (tos 0x0, ttl 58, id 40563, offset 0, flags [DF], proto TCP (6), length 52) 208.80.154.225.80 > 192.168.0.200.27797: Flags [.], cksum 0x70fc (correct), seq 3002976198, ack 1652546753, win 14, options [nop,nop,TS val 1544640949 ecr 35519041], length 0 08:12:53.058005 IP (tos 0x0, ttl 58, id 40564, offset 0, flags [DF], proto TCP (6), length 591) 208.80.154.225.80 > 192.168.0.200.27797: Flags [P.], cksum 0x61ba (correct), seq 3002976198:3002976737, ack 1652546753, win 14, options [nop,nop,TS val 1544640949 ecr 35519041], length 539 08:12:53.157463 IP (tos 0x0, ttl 64, id 4781, offset 0, flags [DF], proto TCP (6), length 52) 192.168.0.200.27797 > 208.80.154.225.80: Flags [.], cksum 0x4d9f (correct), seq 1652546753, ack 3002976737, win 8326, options [nop,nop,TS val 35519243 ecr 1544640949], length 0 08:12:55.830341 IP (tos 0x0, ttl 58, id 40572, offset 0, flags [DF], proto TCP (6), length 1492) 208.80.154.225.80 > 192.168.0.200.27797: Flags [.], cksum 0x1c84 (correct), seq 3002976737:3002978177, ack 1652546753, win 14, options [nop,nop,TS val 1544641226 ecr 35519243], length 1440 08:12:55.830541 IP (tos 0x0, ttl 58, id 40573, offset 0, flags [DF], proto TCP (6), length 60) 208.80.154.225.80 > 192.168.0.200.27797: Flags [.], cksum 0xc5b8 (correct), seq 3002978177:3002978185, ack 1652546753, win 14, options [nop,nop,TS val 1544641226 ecr 35519243], length 8 08:12:55.830759 IP (tos 0x0, ttl 64, id 4783, offset 0, flags [DF], proto TCP (6), length 52) 192.168.0.200.27797 > 208.80.154.225.80: Flags [.], cksum 0x3d26 (correct), seq 1652546753, ack 3002978185, win 8145, options [nop,nop,TS val 35521916 ecr 1544641226], length 0 08:12:55.836196 IP (tos 0x0, ttl 58, id 40574, offset 0, flags [DF], proto TCP (6), length 1492) 208.80.154.225.80 > 192.168.0.200.27797: Flags [.], cksum 0x62d4 (correct), seq 3002978185:3002979625, ack 1652546753, win 14, options [nop,nop,TS val 1544641226 ecr 35519243], length 1440 08:12:55.836325 IP (tos 0x0, ttl 58, id 40575, offset 0, flags [DF], proto TCP (6), length 60) 208.80.154.225.80 > 192.168.0.200.27797: Flags [.], cksum 0x82ce (correct), seq 3002979625:3002979633, ack 1652546753, win 14, options [nop,nop,TS val 1544641226 ecr 35519243], length 8 08:12:55.836468 IP (tos 0x0, ttl 64, id 4784, offset 0, flags [DF], proto TCP (6), length 52) 192.168.0.200.27797 > 208.80.154.225.80: Flags [.], cksum 0x3778 (correct), seq 1652546753, ack 3002979633, win 8145, options [nop,nop,TS val 35521922 ecr 1544641226], length 0 08:12:55.930079 IP (tos 0x0, ttl 58, id 40576, offset 0, flags [DF], proto TCP (6), length 1492) 208.80.154.225.80 > 192.168.0.200.27797: Flags [.], cksum 0xa94d (correct), seq 3002979633:3002981073, ack 1652546753, win 14, options [nop,nop,TS val 1544641236 ecr 35521916], length 1440 08:12:55.935479 IP (tos 0x0, ttl 58, id 40577, offset 0, flags [DF], proto TCP (6), length 1492) 208.80.154.225.80 > 192.168.0.200.27797: Flags [.], cksum 0xdb42 (correct), seq 3002981073:3002982513, ack 1652546753, win 14, options [nop,nop,TS val 1544641236 ecr 35521916], length 1440 08:12:55.935677 IP (tos 0x0, ttl 64, id 4786, offset 0, flags [DF], proto TCP (6), length 52) 192.168.0.200.27797 > 208.80.154.225.80: Flags [.], cksum 0x2bca (correct), seq 1652546753, ack 3002982513, win 8146, options [nop,nop,TS val 35522021 ecr 1544641236], length 0 08:12:55.935712 IP (tos 0x0, ttl 58, id 40578, offset 0, flags [DF], proto TCP (6), length 68) 208.80.154.225.80 > 192.168.0.200.27797: Flags [.], cksum 0x8e80 (correct), seq 3002982513:3002982529, ack 1652546753, win 14, options [nop,nop,TS val 1544641237 ecr 35521922], length 16 08:12:55.941149 IP (tos 0x0, ttl 58, id 40579, offset 0, flags [DF], proto TCP (6), length 1492) 208.80.154.225.80 > 192.168.0.200.27797: Flags [.], cksum 0x0393 (correct), seq 3002982529:3002983969, ack 1652546753, win 14, options [nop,nop,TS val 1544641237 ecr 35521922], length 1440 08:12:55.941348 IP (tos 0x0, ttl 64, id 4787, offset 0, flags [DF], proto TCP (6), length 52) 192.168.0.200.27797 > 208.80.154.225.80: Flags [.], cksum 0x2614 (correct), seq 1652546753, ack 3002983969, win 8146, options [nop,nop,TS val 35522026 ecr 1544641237], length 0 08:12:56.034811 IP (tos 0x0, ttl 58, id 40580, offset 0, flags [DF], proto TCP (6), length 1492) 208.80.154.225.80 > 192.168.0.200.27797: Flags [.], cksum 0x9f57 (correct), seq 3002983969:3002985409, ack 1652546753, win 14, options [nop,nop,TS val 1544641247 ecr 35522021], length 1440 08:12:56.040200 IP (tos 0x0, ttl 58, id 40581, offset 0, flags [DF], proto TCP (6), length 1492) 208.80.154.225.80 > 192.168.0.200.27797: Flags [.], cksum 0xbe37 (correct), seq 3002985409:3002986849, ack 1652546753, win 14, options [nop,nop,TS val 1544641247 ecr 35522021], length 1440 08:12:56.040417 IP (tos 0x0, ttl 64, id 4789, offset 0, flags [DF], proto TCP (6), length 52) 192.168.0.200.27797 > 208.80.154.225.80: Flags [.], cksum 0x1a66 (correct), seq 1652546753, ack 3002986849, win 8146, options [nop,nop,TS val 35522126 ecr 1544641247], length 0 08:12:56.045596 IP (tos 0x0, ttl 58, id 40582, offset 0, flags [DF], proto TCP (6), length 1492) 208.80.154.225.80 > 192.168.0.200.27797: Flags [.], cksum 0xdcc6 (correct), seq 3002986849:3002988289, ack 1652546753, win 14, options [nop,nop,TS val 1544641247 ecr 35522026], length 1440 08:12:56.050802 IP (tos 0x0, ttl 58, id 40583, offset 0, flags [DF], proto TCP (6), length 1492) 208.80.154.225.80 > 192.168.0.200.27797: Flags [.], cksum 0xb79d (correct), seq 3002988289:3002989729, ack 1652546753, win 14, options [nop,nop,TS val 1544641247 ecr 35522026], length 1440 08:12:56.050982 IP (tos 0x0, ttl 64, id 4790, offset 0, flags [DF], proto TCP (6), length 52) 192.168.0.200.27797 > 208.80.154.225.80: Flags [.], cksum 0x0f1c (correct), seq 1652546753, ack 3002989729, win 8146, options [nop,nop,TS val 35522136 ecr 1544641247], length 0 08:12:56.139742 IP (tos 0x0, ttl 58, id 40584, offset 0, flags [DF], proto TCP (6), length 1492) 208.80.154.225.80 > 192.168.0.200.27797: Flags [.], cksum 0x9c13 (correct), seq 3002989729:3002991169, ack 1652546753, win 14, options [nop,nop,TS val 1544641257 ecr 35522126], length 1440 08:12:56.144918 IP (tos 0x0, ttl 58, id 40585, offset 0, flags [DF], proto TCP (6), length 1492) 208.80.154.225.80 > 192.168.0.200.27797: Flags [.], cksum 0x20c1 (correct), seq 3002991169:3002992609, ack 1652546753, win 14, options [nop,nop,TS val 1544641257 ecr 35522126], length 1440 08:12:56.145144 IP (tos 0x0, ttl 64, id 4795, offset 0, flags [DF], proto TCP (6), length 52) 192.168.0.200.27797 > 208.80.154.225.80: Flags [.], cksum 0x0374 (correct), seq 1652546753, ack 3002992609, win 8146, options [nop,nop,TS val 35522230 ecr 1544641257], length 0 08:12:56.150073 IP (tos 0x0, ttl 58, id 40586, offset 0, flags [DF], proto TCP (6), length 1492) 208.80.154.225.80 > 192.168.0.200.27797: Flags [.], cksum 0x8bbc (correct), seq 3002992609:3002994049, ack 1652546753, win 14, options [nop,nop,TS val 1544641258 ecr 35522136], length 1440 08:12:56.155498 IP (tos 0x0, ttl 58, id 40587, offset 0, flags [DF], proto TCP (6), length 1492) 208.80.154.225.80 > 192.168.0.200.27797: Flags [.], cksum 0x1d16 (correct), seq 3002994049:3002995489, ack 1652546753, win 14, options [nop,nop,TS val 1544641258 ecr 35522136], length 1440 08:12:56.155675 IP (tos 0x0, ttl 64, id 4796, offset 0, flags [DF], proto TCP (6), length 52) 192.168.0.200.27797 > 208.80.154.225.80: Flags [.], cksum 0xf827 (correct), seq 1652546753, ack 3002995489, win 8146, options [nop,nop,TS val 35522241 ecr 1544641258], length 0 08:12:56.244469 IP (tos 0x0, ttl 58, id 40588, offset 0, flags [DF], proto TCP (6), length 1492) 208.80.154.225.80 > 192.168.0.200.27797: Flags [.], cksum 0x984a (correct), seq 3002995489:3002996929, ack 1652546753, win 14, options [nop,nop,TS val 1544641268 ecr 35522230], length 1440 08:12:56.249614 IP (tos 0x0, ttl 58, id 40589, offset 0, flags [DF], proto TCP (6), length 1492) 208.80.154.225.80 > 192.168.0.200.27797: Flags [.], cksum 0x47c6 (correct), seq 3002996929:3002998369, ack 1652546753, win 14, options [nop,nop,TS val 1544641268 ecr 35522230], length 1440 08:12:56.249813 IP (tos 0x0, ttl 64, id 4798, offset 0, flags [DF], proto TCP (6), length 52) 192.168.0.200.27797 > 208.80.154.225.80: Flags [.], cksum 0xec7f (correct), seq 1652546753, ack 3002998369, win 8146, options [nop,nop,TS val 35522335 ecr 1544641268], length 0 08:12:56.255024 IP (tos 0x0, ttl 58, id 40590, offset 0, flags [DF], proto TCP (6), length 1492) 208.80.154.225.80 > 192.168.0.200.27797: Flags [.], cksum 0x3e96 (correct), seq 3002998369:3002999809, ack 1652546753, win 14, options [nop,nop,TS val 1544641269 ecr 35522241], length 1440 08:12:56.260450 IP (tos 0x0, ttl 58, id 40591, offset 0, flags [DF], proto TCP (6), length 1492) 208.80.154.225.80 > 192.168.0.200.27797: Flags [.], cksum 0x7dd8 (correct), seq 3002999809:3003001249, ack 1652546753, win 14, options [nop,nop,TS val 1544641269 ecr 35522241], length 1440 08:12:56.260629 IP (tos 0x0, ttl 64, id 4799, offset 0, flags [DF], proto TCP (6), length 52) 192.168.0.200.27797 > 208.80.154.225.80: Flags [.], cksum 0xe133 (correct), seq 1652546753, ack 3003001249, win 8146, options [nop,nop,TS val 35522346 ecr 1544641269], length 0 08:12:56.348670 IP (tos 0x0, ttl 58, id 40592, offset 0, flags [DF], proto TCP (6), length 1492) 208.80.154.225.80 > 192.168.0.200.27797: Flags [.], cksum 0x0797 (correct), seq 3003001249:3003002689, ack 1652546753, win 14, options [nop,nop,TS val 1544641278 ecr 35522335], length 1440 08:12:56.353834 IP (tos 0x0, ttl 58, id 40593, offset 0, flags [DF], proto TCP (6), length 1492) 208.80.154.225.80 > 192.168.0.200.27797: Flags [.], cksum 0x224a (correct), seq 3003002689:3003004129, ack 1652546753, win 14, options [nop,nop,TS val 1544641278 ecr 35522335], length 1440 08:12:56.354021 IP (tos 0x0, ttl 64, id 4801, offset 0, flags [DF], proto TCP (6), length 52) 192.168.0.200.27797 > 208.80.154.225.80: Flags [.], cksum 0xd58d (correct), seq 1652546753, ack 3003004129, win 8146, options [nop,nop,TS val 35522439 ecr 1544641278], length 0 08:12:56.359255 IP (tos 0x0, ttl 58, id 40594, offset 0, flags [DF], proto TCP (6), length 1492) 208.80.154.225.80 > 192.168.0.200.27797: Flags [.], cksum 0x72e7 (correct), seq 3003004129:3003005569, ack 1652546753, win 14, options [nop,nop,TS val 1544641278 ecr 35522335], length 1440 08:12:56.364446 IP (tos 0x0, ttl 58, id 40595, offset 0, flags [DF], proto TCP (6), length 1492) 208.80.154.225.80 > 192.168.0.200.27797: Flags [.], cksum 0x2305 (correct), seq 3003005569:3003007009, ack 1652546753, win 14, options [nop,nop,TS val 1544641279 ecr 35522346], length 1440 08:12:56.364653 IP (tos 0x0, ttl 64, id 4802, offset 0, flags [DF], proto TCP (6), length 52) 192.168.0.200.27797 > 208.80.154.225.80: Flags [.], cksum 0xca42 (correct), seq 1652546753, ack 3003007009, win 8146, options [nop,nop,TS val 35522450 ecr 1544641278], length 0 08:12:56.369841 IP (tos 0x0, ttl 58, id 40596, offset 0, flags [DF], proto TCP (6), length 1492) 208.80.154.225.80 > 192.168.0.200.27797: Flags [.], cksum 0x2fa0 (correct), seq 3003007009:3003008449, ack 1652546753, win 14, options [nop,nop,TS val 1544641279 ecr 35522346], length 1440 08:12:56.453412 IP (tos 0x0, ttl 58, id 40597, offset 0, flags [DF], proto TCP (6), length 1492) 208.80.154.225.80 > 192.168.0.200.27797: Flags [.], cksum 0xbd99 (correct), seq 3003008449:3003009889, ack 1652546753, win 14, options [nop,nop,TS val 1544641289 ecr 35522439], length 1440 08:12:56.453619 IP (tos 0x0, ttl 64, id 4804, offset 0, flags [DF], proto TCP (6), length 52) 192.168.0.200.27797 > 208.80.154.225.80: Flags [.], cksum 0xbea8 (correct), seq 1652546753, ack 3003009889, win 8146, options [nop,nop,TS val 35522539 ecr 1544641279], length 0 08:12:56.458548 IP (tos 0x0, ttl 58, id 40598, offset 0, flags [DF], proto TCP (6), length 1492) 208.80.154.225.80 > 192.168.0.200.27797: Flags [.], cksum 0xeb3d (correct), seq 3003009889:3003011329, ack 1652546753, win 14, options [nop,nop,TS val 1544641289 ecr 35522439], length 1440 08:12:56.464208 IP (tos 0x0, ttl 58, id 40599, offset 0, flags [DF], proto TCP (6), length 1492) 208.80.154.225.80 > 192.168.0.200.27797: Flags [.], cksum 0x4237 (correct), seq 3003011329:3003012769, ack 1652546753, win 14, options [nop,nop,TS val 1544641290 ecr 35522450], length 1440 08:12:56.464395 IP (tos 0x0, ttl 64, id 4805, offset 0, flags [DF], proto TCP (6), length 52) 192.168.0.200.27797 > 208.80.154.225.80: Flags [.], cksum 0xb353 (correct), seq 1652546753, ack 3003012769, win 8146, options [nop,nop,TS val 35522550 ecr 1544641289], length 0 08:12:56.464929 IP (tos 0x0, ttl 58, id 40600, offset 0, flags [DF], proto TCP (6), length 220) 208.80.154.225.80 > 192.168.0.200.27797: Flags [.], cksum 0x3783 (correct), seq 3003012769:3003012937, ack 1652546753, win 14, options [nop,nop,TS val 1544641290 ecr 35522450], length 168 08:12:56.552926 IP (tos 0x0, ttl 58, id 40601, offset 0, flags [DF], proto TCP (6), length 1492) 208.80.154.225.80 > 192.168.0.200.27797: Flags [.], cksum 0x9d72 (correct), seq 3003012937:3003014377, ack 1652546753, win 14, options [nop,nop,TS val 1544641298 ecr 35522539], length 1440 08:12:56.553124 IP (tos 0x0, ttl 64, id 4807, offset 0, flags [DF], proto TCP (6), length 52) 192.168.0.200.27797 > 208.80.154.225.80: Flags [.], cksum 0xacb2 (correct), seq 1652546753, ack 3003014377, win 8146, options [nop,nop,TS val 35522638 ecr 1544641290], length 0 08:12:56.558079 IP (tos 0x0, ttl 58, id 40602, offset 0, flags [DF], proto TCP (6), length 1492) 208.80.154.225.80 > 192.168.0.200.27797: Flags [.], cksum 0x1835 (correct), seq 3003014377:3003015817, ack 1652546753, win 14, options [nop,nop,TS val 1544641298 ecr 35522539], length 1440 08:12:56.564009 IP (tos 0x0, ttl 58, id 40603, offset 0, flags [DF], proto TCP (6), length 1492) 208.80.154.225.80 > 192.168.0.200.27797: Flags [.], cksum 0xb1c3 (correct), seq 3003015817:3003017257, ack 1652546753, win 14, options [nop,nop,TS val 1544641300 ecr 35522550], length 1440 08:12:56.564189 IP (tos 0x0, ttl 64, id 4808, offset 0, flags [DF], proto TCP (6), length 52) 192.168.0.200.27797 > 208.80.154.225.80: Flags [.], cksum 0xa15f (correct), seq 1652546753, ack 3003017257, win 8146, options [nop,nop,TS val 35522649 ecr 1544641298], length 0 08:12:56.569152 IP (tos 0x0, ttl 58, id 40604, offset 0, flags [DF], proto TCP (6), length 1492) 208.80.154.225.80 > 192.168.0.200.27797: Flags [.], cksum 0x24d8 (correct), seq 3003017257:3003018697, ack 1652546753, win 14, options [nop,nop,TS val 1544641300 ecr 35522550], length 1440 08:12:56.651945 IP (tos 0x0, ttl 58, id 40605, offset 0, flags [DF], proto TCP (6), length 1492) 208.80.154.225.80 > 192.168.0.200.27797: Flags [.], cksum 0x52ad (correct), seq 3003018697:3003020137, ack 1652546753, win 14, options [nop,nop,TS val 1544641308 ecr 35522638], length 1440 08:12:56.652164 IP (tos 0x0, ttl 64, id 4810, offset 0, flags [DF], proto TCP (6), length 52) 192.168.0.200.27797 > 208.80.154.225.80: Flags [.], cksum 0x95c5 (correct), seq 1652546753, ack 3003020137, win 8146, options [nop,nop,TS val 35522737 ecr 1544641300], length 0 08:12:56.657127 IP (tos 0x0, ttl 58, id 40606, offset 0, flags [DF], proto TCP (6), length 1492) 208.80.154.225.80 > 192.168.0.200.27797: Flags [.], cksum 0xf2e9 (correct), seq 3003020137:3003021577, ack 1652546753, win 14, options [nop,nop,TS val 1544641308 ecr 35522638], length 1440 08:12:56.663285 IP (tos 0x0, ttl 58, id 40607, offset 0, flags [DF], proto TCP (6), length 1492) 208.80.154.225.80 > 192.168.0.200.27797: Flags [.], cksum 0x5f8e (correct), seq 3003021577:3003023017, ack 1652546753, win 14, options [nop,nop,TS val 1544641310 ecr 35522649], length 1440 08:12:56.663455 IP (tos 0x0, ttl 64, id 4811, offset 0, flags [DF], proto TCP (6), length 52) 192.168.0.200.27797 > 208.80.154.225.80: Flags [.], cksum 0x8a71 (correct), seq 1652546753, ack 3003023017, win 8146, options [nop,nop,TS val 35522749 ecr 1544641308], length 0 08:12:56.668915 IP (tos 0x0, ttl 58, id 40608, offset 0, flags [DF], proto TCP (6), length 1492) 208.80.154.225.80 > 192.168.0.200.27797: Flags [.], cksum 0xf0e8 (correct), seq 3003023017:3003024457, ack 1652546753, win 14, options [nop,nop,TS val 1544641310 ecr 35522649], length 1440 08:12:56.751236 IP (tos 0x0, ttl 58, id 40609, offset 0, flags [DF], proto TCP (6), length 1492) 208.80.154.225.80 > 192.168.0.200.27797: Flags [.], cksum 0x742c (correct), seq 3003024457:3003025897, ack 1652546753, win 14, options [nop,nop,TS val 1544641318 ecr 35522737], length 1440 08:12:56.751439 IP (tos 0x0, ttl 64, id 4813, offset 0, flags [DF], proto TCP (6), length 52) 192.168.0.200.27797 > 208.80.154.225.80: Flags [.], cksum 0x7ed7 (correct), seq 1652546753, ack 3003025897, win 8146, options [nop,nop,TS val 35522837 ecr 1544641310], length 0 08:12:56.756406 IP (tos 0x0, ttl 58, id 40610, offset 0, flags [DF], proto TCP (6), length 1492) 208.80.154.225.80 > 192.168.0.200.27797: Flags [.], cksum 0xcf00 (correct), seq 3003025897:3003027337, ack 1652546753, win 14, options [nop,nop,TS val 1544641318 ecr 35522737], length 1440 08:12:56.762075 IP (tos 0x0, ttl 58, id 40611, offset 0, flags [DF], proto TCP (6), length 1492) 208.80.154.225.80 > 192.168.0.200.27797: Flags [.], cksum 0x5a81 (correct), seq 3003027337:3003028777, ack 1652546753, win 14, options [nop,nop,TS val 1544641319 ecr 35522749], length 1440 08:12:56.762248 IP (tos 0x0, ttl 64, id 4814, offset 0, flags [DF], proto TCP (6), length 52) 192.168.0.200.27797 > 208.80.154.225.80: Flags [.], cksum 0x7385 (correct), seq 1652546753, ack 3003028777, win 8146, options [nop,nop,TS val 35522847 ecr 1544641318], length 0 08:12:56.767478 IP (tos 0x0, ttl 58, id 40612, offset 0, flags [DF], proto TCP (6), length 1492) 208.80.154.225.80 > 192.168.0.200.27797: Flags [.], cksum 0x0bf2 (correct), seq 3003028777:3003030217, ack 1652546753, win 14, options [nop,nop,TS val 1544641319 ecr 35522749], length 1440 08:12:56.850310 IP (tos 0x0, ttl 58, id 40613, offset 0, flags [DF], proto TCP (6), length 1492) 208.80.154.225.80 > 192.168.0.200.27797: Flags [.], cksum 0xaabe (correct), seq 3003030217:3003031657, ack 1652546753, win 14, options [nop,nop,TS val 1544641328 ecr 35522837], length 1440 08:12:56.850531 IP (tos 0x0, ttl 64, id 4816, offset 0, flags [DF], proto TCP (6), length 52) 192.168.0.200.27797 > 208.80.154.225.80: Flags [.], cksum 0x67eb (correct), seq 1652546753, ack 3003031657, win 8146, options [nop,nop,TS val 35522936 ecr 1544641319], length 0 08:12:56.855658 IP (tos 0x0, ttl 58, id 40614, offset 0, flags [DF], proto TCP (6), length 1492) 208.80.154.225.80 > 192.168.0.200.27797: Flags [.], cksum 0xb1cb (correct), seq 3003031657:3003033097, ack 1652546753, win 14, options [nop,nop,TS val 1544641328 ecr 35522837], length 1440 08:12:56.859244 IP (tos 0x0, ttl 58, id 40615, offset 0, flags [DF], proto TCP (6), length 1005) 208.80.154.225.80 > 192.168.0.200.27797: Flags [P.], cksum 0xb8fe (correct), seq 3003033097:3003034050, ack 1652546753, win 14, options [nop,nop,TS val 1544641328 ecr 35522837], length 953 08:12:56.859422 IP (tos 0x0, ttl 64, id 4817, offset 0, flags [DF], proto TCP (6), length 52) 192.168.0.200.27797 > 208.80.154.225.80: Flags [.], cksum 0x5e44 (correct), seq 1652546753, ack 3003034050, win 8206, options [nop,nop,TS val 35522945 ecr 1544641328], length 0 08:13:15.867863 IP (tos 0x0, ttl 58, id 40616, offset 0, flags [DF], proto TCP (6), length 52) 208.80.154.225.80 > 192.168.0.200.27797: Flags [F.], cksum 0x76d4 (correct), seq 3003034050, ack 1652546753, win 14, options [nop,nop,TS val 1544643231 ecr 35522945], length 0 08:13:15.868091 IP (tos 0x0, ttl 64, id 5173, offset 0, flags [DF], proto TCP (6), length 52) 192.168.0.200.27797 > 208.80.154.225.80: Flags [.], cksum 0x0c1c (correct), seq 1652546753, ack 3003034051, win 8326, options [nop,nop,TS val 35541953 ecr 1544643231], length 0 08:13:16.701079 IP (tos 0x0, ttl 64, id 5174, offset 0, flags [DF], proto TCP (6), length 52) 192.168.0.200.27797 > 208.80.154.225.80: Flags [F.], cksum 0x08da (correct), seq 1652546753, ack 3003034051, win 8326, options [nop,nop,TS val 35542786 ecr 1544643231], length 0 08:13:16.794496 IP (tos 0x0, ttl 58, id 0, offset 0, flags [DF], proto TCP (6), length 52) 208.80.154.225.80 > 192.168.0.200.27797: Flags [.], cksum 0x28f6 (correct), seq 3003034051, ack 1652546754, win 14, options [nop,nop,TS val 1544643323 ecr 35542786], length 0 --------------060101000404050001020902 Content-Type: text/plain; charset=UTF-8; name="xl0_tcpdump_nc" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="xl0_tcpdump_nc" 08:11:44.351453 IP (tos 0x0, ttl 64, id 4689, offset 0, flags [DF], proto TCP (6), length 60) 192.168.0.200.21231 > 208.80.154.225.80: Flags [S], cksum 0x8450 (correct), seq 2864621746, win 65535, options [mss 1460,nop,wscale 3,sackOK,TS val 35450436 ecr 0], length 0 08:11:44.447520 IP (tos 0x0, ttl 58, id 0, offset 0, flags [DF], proto TCP (6), length 60) 208.80.154.225.80 > 192.168.0.200.21231: Flags [S.], cksum 0x125a (correct), seq 1253516475, ack 2864621747, win 5792, options [mss 1460,sackOK,TS val 1034923549 ecr 35450436,nop,wscale 9], length 0 08:11:44.447806 IP (tos 0x0, ttl 64, id 4691, offset 0, flags [DF], proto TCP (6), length 52) 192.168.0.200.21231 > 208.80.154.225.80: Flags [.], cksum 0x36e1 (correct), seq 2864621747, ack 1253516476, win 8326, options [nop,nop,TS val 35450533 ecr 1034923549], length 0 08:11:44.455698 IP (tos 0x0, ttl 64, id 4692, offset 0, flags [DF], proto TCP (6), length 108) 192.168.0.200.21231 > 208.80.154.225.80: Flags [P.], cksum 0xcef2 (correct), seq 2864621747:2864621803, ack 1253516476, win 8326, options [nop,nop,TS val 35450541 ecr 1034923549], length 56 08:11:44.455898 IP (tos 0x0, ttl 64, id 4693, offset 0, flags [DF], proto TCP (6), length 52) 192.168.0.200.21231 > 208.80.154.225.80: Flags [F.], cksum 0x36a0 (correct), seq 2864621803, ack 1253516476, win 8326, options [nop,nop,TS val 35450541 ecr 1034923549], length 0 08:11:44.551876 IP (tos 0x0, ttl 58, id 35483, offset 0, flags [DF], proto TCP (6), length 64) 208.80.154.225.80 > 192.168.0.200.21231: Flags [.], cksum 0x79e3 (correct), seq 1253516476, ack 2864621747, win 12, options [nop,nop,TS val 1034923560 ecr 35450533,nop,nop,sack 1 {2864621803:2864621804}], length 0 08:11:44.553442 IP (tos 0x0, ttl 58, id 35484, offset 0, flags [DF], proto TCP (6), length 52) 208.80.154.225.80 > 192.168.0.200.21231: Flags [.], cksum 0x570f (correct), seq 1253516476, ack 2864621804, win 12, options [nop,nop,TS val 1034923560 ecr 35450541], length 0 08:11:44.553677 IP (tos 0x0, ttl 58, id 35485, offset 0, flags [DF], proto TCP (6), length 52) 208.80.154.225.80 > 192.168.0.200.21231: Flags [F.], cksum 0x570e (correct), seq 1253516476, ack 2864621804, win 12, options [nop,nop,TS val 1034923560 ecr 35450541], length 0 08:11:44.553913 IP (tos 0x0, ttl 64, id 4695, offset 0, flags [DF], proto TCP (6), length 52) 192.168.0.200.21231 > 208.80.154.225.80: Flags [.], cksum 0x3633 (correct), seq 2864621804, ack 1253516477, win 8325, options [nop,nop,TS val 35450639 ecr 1034923560], length 0 --------------060101000404050001020902-- From owner-freebsd-pf@FreeBSD.ORG Tue Jul 24 17:12:37 2012 Return-Path: Delivered-To: freebsd-pf@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 088621065673 for ; Tue, 24 Jul 2012 17:12:37 +0000 (UTC) (envelope-from dhartmei@insomnia.benzedrine.cx) Received: from insomnia.benzedrine.cx (106-30.3-213.fix.bluewin.ch [213.3.30.106]) by mx1.freebsd.org (Postfix) with ESMTP id 681E48FC08 for ; Tue, 24 Jul 2012 17:12:35 +0000 (UTC) Received: from insomnia.benzedrine.cx (localhost.benzedrine.cx [127.0.0.1]) by insomnia.benzedrine.cx (8.14.1/8.13.4) with ESMTP id q6OHCPqW008280 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Tue, 24 Jul 2012 19:12:26 +0200 (MEST) Received: (from dhartmei@localhost) by insomnia.benzedrine.cx (8.14.1/8.12.10/Submit) id q6OHCPp7028980; Tue, 24 Jul 2012 19:12:25 +0200 (MEST) Date: Tue, 24 Jul 2012 19:12:25 +0200 From: Daniel Hartmeier To: Jason Mattax Message-ID: <20120724171225.GA27107@insomnia.benzedrine.cx> References: <20120723100521.GC32530@insomnia.benzedrine.cx> <500E1202.20108@storytotell.org> <20120724070700.GF32530@insomnia.benzedrine.cx> <500EB432.6050803@storytotell.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <500EB432.6050803@storytotell.org> User-Agent: Mutt/1.5.12-2006-07-14 Cc: jmattax@clanspum.net, freebsd-pf@freebsd.org Subject: Re: PF suddenly malfunctioned X-BeenThere: freebsd-pf@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "Technical discussion and general questions about packet filter \(pf\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 24 Jul 2012 17:12:37 -0000 On Tue, Jul 24, 2012 at 08:41:54AM -0600, Jason Mattax wrote: > The other thing I did was I accessed the wikipedia server at > 208.80.154.225 on the firewall. I did this so that I could do the nc > command on the firewall, the output of the tcpdump of which is attached > as xl0_tcpdump_nc and seems to have the same behavior as the nc behind > the firewall. Then I could run links en.wikipedia.org/wiki/Main_Page > (which works about half the time) and capture the tcpdump of that into > the attached xl0_tcpd_links so that people here could look at those > outputs in case they reveal something useful. In this case the links > worked, in the cases where it doesn't I get a message along the bottom > that reads "Received 0 B of 57 kB, avg 0 B/s, cur 0 B/s" This begins to look like the problem is with the upstream NAT device (which NATs source 192.168.0.200 again, right?) So, the simple nc invokation reliably returns an empty HTTP response, no matter if you run it on a client or on the firewall itself? But if you run links, it sometimes works? One difference is that links sends a longer HTTP GET request (User-Agent: and Encoding: headers and such). If the upstream router does HTTP inspection, it might be buggy (since the thunderstorm? :) and react to different HTTP headers. Or it might run an (broken) antivirus patterns on the HTTP result? Can you disable any layer 7 inspection? I'd tcpdump with -s 1600 -X to capture a working links connection. Then extract the exact HTTP GET request from the hex dump. Then try to send that with printf | nc. That should work equally well. If so, remove headers until you hit the bug again. Or just replace the upstream device (router, ISP modem?) and see if it goes away. The more far-fetched theories would be that wikipedia has some DOS prevention or scraping protection going on that you somehow trigger. Do you have a simpler server that reproduces the problem reliably, possibly one that you have control over yourself, to tcpdump there? Daniel From owner-freebsd-pf@FreeBSD.ORG Tue Jul 24 21:38:03 2012 Return-Path: Delivered-To: freebsd-pf@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id ECC2D106566B for ; Tue, 24 Jul 2012 21:38:03 +0000 (UTC) (envelope-from Greg.Hennessy@nviz.net) Received: from mail1.jellyfishnet.co.uk (mail1.jellyfishnet.co.uk [93.91.20.9]) by mx1.freebsd.org (Postfix) with ESMTP id 856C08FC0C for ; Tue, 24 Jul 2012 21:38:03 +0000 (UTC) Received: from pemexhub01.jellyfishnet.co.uk.local (93.91.20.3) by mail1.jellyfishnet.co.uk (93.91.20.9) with Microsoft SMTP Server (TLS) id 8.1.393.1; Tue, 24 Jul 2012 22:37:56 +0100 Received: from PEMEXMBXVS04.jellyfishnet.co.uk.local ([192.168.65.52]) by pemexhub01.jellyfishnet.co.uk.local ([192.168.65.7]) with mapi; Tue, 24 Jul 2012 22:37:15 +0100 From: Greg Hennessy To: Jason Mattax Date: Tue, 24 Jul 2012 22:37:52 +0100 Thread-Topic: PF suddenly malfunctioned Thread-Index: Ac1pqoX+eH91zKzQTe2nEsTd6S1YwwAObqCg Message-ID: <9EB23F6C23A8B6488E8BCC92A48E83264BB4D27913@PEMEXMBXVS04.jellyfishnet.co.uk.local> References: <20120723100521.GC32530@insomnia.benzedrine.cx> <500E1202.20108@storytotell.org> <20120724070700.GF32530@insomnia.benzedrine.cx> <500EB432.6050803@storytotell.org> In-Reply-To: <500EB432.6050803@storytotell.org> Accept-Language: en-US, en-GB Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: acceptlanguage: en-US, en-GB Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Cc: "jmattax@clanspum.net" , "freebsd-pf@freebsd.org" Subject: RE: PF suddenly malfunctioned X-BeenThere: freebsd-pf@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "Technical discussion and general questions about packet filter \(pf\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 24 Jul 2012 21:38:04 -0000 >=20 > On 07/24/2012 01:07 AM, Daniel Hartmeier wrote: > > What's the client OS? > > > The client OS for this test is Ubuntu 12.04 LTS >=20 > jmattax@chani:~/pf_debugging$ uname -a > Linux chani 3.2.0-26-generic #41-Ubuntu SMP Thu Jun 14 16:26:01 UTC 2012 > i686 i686 i386 GNU/Linux >=20 > > It looks like it might be an incompatibility between the client and > > the peculiar wikipedia server (or loadbalancer or proxy or whatever > > there is). > > > > Like the GET request gets lost, but the FIN arrives, and the server > > selectively ACKs the FIN, and the client doesn't retransmit the request= . > > You ran the tcpdump for several seconds after the netcat was started? > > Maybe repeat it and wait longer, in case the output is buffered. The > > client should re-transmit. > > >=20 > I initially ran the tcpdumps until the client had nc return and give me a= new > prompt in my shell (that took maybe a second). I just repeated it as abov= e > letting the tcpdumps run longer and it captured the same number of packet= s. >=20 Hi Jason,=20 Try mss clamping the outside interface using the relevant 'scrub' option to= rule out a Path MTU issue.=20 Greg From owner-freebsd-pf@FreeBSD.ORG Wed Jul 25 05:05:41 2012 Return-Path: Delivered-To: freebsd-pf@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 372BB106564A for ; Wed, 25 Jul 2012 05:05:41 +0000 (UTC) (envelope-from jmattax@storytotell.org) Received: from mail.clanspum.net (mail.clanspum.net [69.164.206.246]) by mx1.freebsd.org (Postfix) with ESMTP id 138E58FC0C for ; Wed, 25 Jul 2012 05:05:41 +0000 (UTC) Received: from [10.11.10.45] (unknown [63.231.116.1]) (using TLSv1 with cipher ECDHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mail.clanspum.net (Postfix) with ESMTPSA id DA6A222400C; Wed, 25 Jul 2012 00:05:39 -0500 (CDT) Message-ID: <500F7EA2.6050707@storytotell.org> Date: Tue, 24 Jul 2012 23:05:38 -0600 From: Jason Mattax User-Agent: Mozilla/5.0 (X11; Linux i686; rv:14.0) Gecko/20120714 Thunderbird/14.0 MIME-Version: 1.0 To: Daniel Hartmeier References: <20120723100521.GC32530@insomnia.benzedrine.cx> <500E1202.20108@storytotell.org> <20120724070700.GF32530@insomnia.benzedrine.cx> <500EB432.6050803@storytotell.org> <20120724171225.GA27107@insomnia.benzedrine.cx> In-Reply-To: <20120724171225.GA27107@insomnia.benzedrine.cx> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-pf@freebsd.org Subject: Re: PF suddenly malfunctioned X-BeenThere: freebsd-pf@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "Technical discussion and general questions about packet filter \(pf\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 25 Jul 2012 05:05:41 -0000 On 07/24/2012 11:12 AM, Daniel Hartmeier wrote: > On Tue, Jul 24, 2012 at 08:41:54AM -0600, Jason Mattax wrote: > If the upstream router does HTTP inspection, it might be buggy (since > the thunderstorm? :) and react to different HTTP headers. Or it might > run an (broken) antivirus patterns on the HTTP result? Can you disable > any layer 7 inspection? > I'd tcpdump with -s 1600 -X to capture a working links connection. Then > extract the exact HTTP GET request from the hex dump. Then try to send > that with printf | nc. That should work equally well. If so, remove > headers until you hit the bug again. > > Or just replace the upstream device (router, ISP modem?) and see if it > goes away. > I was going to go through these in order, but decided I could do some of the faster items first. As it turns out I had a "spare" DSL modem around because Qwest told me that it would work the the new faster internet I purchasing from them, plugged that in and it seems to work fine. The thing about my network that I forgot was that the DSL modem is not protected from lightning by a UPS on the phone line, and unfortunately it can't be (protecting it causes it to lose all DSL signal because of the crappy phone lines in my house.) The phone line comes from the wall strait to the DSL modem, then the ethernet comes from the DSL modem to my UPS to protect the rest of the network. Sorry for the long and complicated thread when there wasn't actually an issue with the PF filter. Also, thank you all for donating your time to help resolve this issue. -- Jason Mattax From owner-freebsd-pf@FreeBSD.ORG Wed Jul 25 08:45:20 2012 Return-Path: Delivered-To: freebsd-pf@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 2C881106566C for ; Wed, 25 Jul 2012 08:45:20 +0000 (UTC) (envelope-from tonix@interazioni.it) Received: from mx02.interazioni.net (mx02.interazioni.net [80.94.114.204]) by mx1.freebsd.org (Postfix) with ESMTP id 5CD518FC0A for ; Wed, 25 Jul 2012 08:45:19 +0000 (UTC) Received: (qmail 11233 invoked by uid 88); 25 Jul 2012 08:45:12 -0000 Received: from unknown (HELO ?82.143.55.20?) (tonix@interazioni.it@82.143.55.20) by relay.interazioni.net with ESMTPA; 25 Jul 2012 08:45:12 -0000 Message-ID: <500FB217.7070603@interazioni.it> Date: Wed, 25 Jul 2012 10:45:11 +0200 From: "Tonix (Antonio Nati)" User-Agent: Mozilla/5.0 (Windows NT 5.1; rv:13.0) Gecko/20120614 Thunderbird/13.0.1 MIME-Version: 1.0 To: "freebsd-pf@freebsd.org" References: <500AB340.2040405@interazioni.it> <9EB23F6C23A8B6488E8BCC92A48E83264BB4D27241@PEMEXMBXVS04.jellyfishnet.co.uk.local> <500AC91F.9090907@interazioni.it> <20120721182316.GA32530@insomnia.benzedrine.cx> <500D1B57.8080405@interazioni.it> <20120723095509.GB32530@insomnia.benzedrine.cx> <500D2D35.4070608@interazioni.it> <20120723111348.GD32530@insomnia.benzedrine.cx> <500D34F9.5000502@interazioni.it> <500D3637.9070806@interazioni.it> <20120723120523.GE32530@insomnia.benzedrine.cx> In-Reply-To: <20120723120523.GE32530@insomnia.benzedrine.cx> Content-Type: text/plain; charset=ISO-8859-15; format=flowed Content-Transfer-Encoding: 7bit Subject: Re: Question on packet filter using in and out interfaces X-BeenThere: freebsd-pf@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "Technical discussion and general questions about packet filter \(pf\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 25 Jul 2012 08:45:20 -0000 Daniel. thanks for detailed explanations! Regards, Tonino ------------------------------------------------------------ Inter@zioni Interazioni di Antonio Nati http://www.interazioni.it tonix@interazioni.it ------------------------------------------------------------