From owner-freebsd-pf@FreeBSD.ORG Tue Dec 15 20:33:26 2009 Return-Path: Delivered-To: freebsd-pf@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id A988D1065670 for ; Tue, 15 Dec 2009 20:33:26 +0000 (UTC) (envelope-from allicient3141@googlemail.com) Received: from mail-bw0-f213.google.com (mail-bw0-f213.google.com [209.85.218.213]) by mx1.freebsd.org (Postfix) with ESMTP id 3153A8FC1E for ; Tue, 15 Dec 2009 20:33:25 +0000 (UTC) Received: by bwz5 with SMTP id 5so267828bwz.3 for ; Tue, 15 Dec 2009 12:33:25 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlemail.com; s=gamma; h=domainkey-signature:mime-version:sender:received:in-reply-to :references:date:x-google-sender-auth:message-id:subject:from:to :content-type:content-transfer-encoding; bh=ZsP8yElGDARq/b10Ks/b/HExPNcM4CBK3chhwNcWtxc=; b=wSQ7Xpt9ccIu54/2yKXLDzCfva320MJxlaQFWPNgJH5IE9SClD6qGBKEyWr+YLT6lb NfIrK9vG49PeJ23Wa99+OWc2kfMW4jEkJNU2OgsfCaHPlNPzF24P5z4ret0sZmoO8ztX 5LBKFTqa0ccx46XEsh8vSH5pepVZfOa0As+6E= DomainKey-Signature: a=rsa-sha1; c=nofws; d=googlemail.com; s=gamma; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:content-type :content-transfer-encoding; b=L+GlZxGKFEti27D+qIBhv2MogoIZmQdH+ylO47dZ/kxOviEiAebBFvpOtgkkH71gTj us+oI0252yeGJXgMtuXkZDqC1LKNmIUexmby+F6Js6Rp8NQP/hIGM1cPwIyZpsY0zGds ElYX/8xUjBnmxyKIVAGgPWA41EUznR8/8Avak= MIME-Version: 1.0 Sender: allicient3141@googlemail.com Received: by 10.204.13.215 with SMTP id d23mr30638bka.18.1260909204762; Tue, 15 Dec 2009 12:33:24 -0800 (PST) In-Reply-To: <237c27100912151114w6912dd63l80523d02831b2588@mail.gmail.com> References: <237c27100912142221k6cf62b7ay97c8ae20bd0e7eb2@mail.gmail.com> <7731938b0912150808y5cfa7cbs3bb31b80f222159c@mail.gmail.com> <237c27100912151114w6912dd63l80523d02831b2588@mail.gmail.com> Date: Tue, 15 Dec 2009 20:33:24 +0000 X-Google-Sender-Auth: 70aea519e07f3259 Message-ID: <7731938b0912151233u6d6a4ca6pc430ca0db01c73c0@mail.gmail.com> From: Peter Maxwell To: Linda Messerschmidt , freebsd-pf@freebsd.org Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Cc: Subject: Re: Lots of weird PF behavior on 7.2-STABLE X-BeenThere: freebsd-pf@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "Technical discussion and general questions about packet filter \(pf\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 15 Dec 2009 20:33:26 -0000 2009/12/15 Linda Messerschmidt : > On Tue, Dec 15, 2009 at 11:08 AM, Peter Maxwell w= rote: >> I'm pretty sure you can run tcpdump against a packet capture from the >> pflog interface on the pf box; that will include fields like >> block/pass and rule number for each packet filtered. > > I have done that with "log" on all block rules. =A0The packets shown as > missing are *not* present in the dump when I do, so as far as I can > tell they are not being dropped by a filter rule. =A0Which makes sense, > since none of the few block rules we have would touch packets in the > middle of an established connection that was permitted. Although it's not likely to be causing the problem (the default "flags" on tcp rules are S/SA, which should exclude this possibility), I'd check that the implicit pass rule isn't getting hit by the web traffic. Add in an explicit "pass all" rule at the start and set the log keyword on it. Make sure *none* of the web traffic is hitting this rule. > > For comparison, endless streams of packets from those DNS guys we > block *are* present in the tcpdump output, exactly as you describe, so > I know pflog is working. > > So it really seems like something wrong in the internals of pf. =A0I > just don't know how to pursue it further. If the box isn't too loaded, you may try using "log (all)" on the pass rules (so that ALL packets are logged, not just the initial SYN) - that way at least you'd find out which rules those data packets are hitting, which would likely pin down the problem quite a bit.. those missing packets must have went somewhere ;-) If it were me, that would be my preferable option if it was available. Barring that, I'd suggest simplifying your setup as much as possible - is there too much traffic to remove the "route-to" and try it against a single webserver? If it's not possible, you could try setting up a simple TCP service on internal hosts and get something that works, then build up (ECHO or TIME are not bad for this). I'd also suggest removing the "scrub" directive until you have it working properly. Is the "state-policy" floating or if-bound? Does the problem affect other services in a similar manner, can you replicate the exact same problem with the web servers with sshd, for example? What's annoying me is that I'm fairly sure I've seen this problem before, but for the life of me I can't remember what caused it :-( > >> However one thing that I would strongly suggest is using proper packet >> filter design: either decide upon a default deny then pass what you >> want, or decide on a default pass and only deny what is bad. > > We are doing the latter; default pass and denying only what is bad. > This isn't even really a firewall, it's for load balancing web > connections. =A0We just threw a couple of block rules in there because > it was a good place to stop a particular attack. =A0There are other > "default deny" firewalls on other machines that handle all the traffic > that isn't getting diverted by the load balance rule. > >> If you're having to use the "quick" keyword, you've probably* >> done something wrong. > > Since we are using load balancing, the "pass" rule that wouldn't pass > all the traffic we've just gone to the trouble to block would be > outrageously complex. =A0Hence, quick. > pf uses the last rule in the ruleset that matches for a given packet, so for a first instance setup I'd suggest putting an explicit "pass all" as the first rule, then any pass rules that do load-balancing and the like, then a list of block rules. It makes it a lot easier to read and debug. Then once it's working as you want, you can move the block rules up to the top and add in the "quick" keyword for performance purposes. > If a case can be made that the use of "quick" is causing our packets > to disappear, we'd probably be willing to tackle trying to restructure > the rules. > > Thanks! >