From owner-freebsd-net@FreeBSD.ORG Mon Apr 7 22:23:06 2014 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 7C07EB06 for ; Mon, 7 Apr 2014 22:23:06 +0000 (UTC) Received: from mail-ob0-x22a.google.com (mail-ob0-x22a.google.com [IPv6:2607:f8b0:4003:c01::22a]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 3FEB4C0D for ; Mon, 7 Apr 2014 22:23:06 +0000 (UTC) Received: by mail-ob0-f170.google.com with SMTP id uz6so100694obc.15 for ; Mon, 07 Apr 2014 15:23:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=9z8yf8sZP6tkqxydJU/oGsyKeemKnWoD//Hy2de2SmY=; b=tbhtnE6fSfexp7cTCu402CLfdJOSuCc/OyQlBIl4trTOKBvZF7d9/RsWIpnwa8jNxO +sDOruDs6wEuS3dI7yWHZsPUXjQX7kHEpk7UYcu5N+JaX/ouBb8GJb/Azn2v4zmBc2Qs c1rgPMZ1lmYPpwA+XKjS67yaZ8XI6XnDguJkfHm9My0N5IbDtbdl6EBB2jYKMAXR5k9C iUTKR3Egm6BN1iycyc3n7Ve9Z21FtyDAjSO/qLBanXiSTUMKUg9p9zv3xaIwmn4j1JRO Tya03XeriYDBhwj8VSWTln6rttBEv+5tPAQWsiI7hLdYDRq2sC6kRsVx+MXBsQSbr0Pl GLFQ== MIME-Version: 1.0 X-Received: by 10.182.65.199 with SMTP id z7mr25172obs.7.1396909385553; Mon, 07 Apr 2014 15:23:05 -0700 (PDT) Received: by 10.76.12.34 with HTTP; Mon, 7 Apr 2014 15:23:05 -0700 (PDT) In-Reply-To: References: <531771C8.1040207@yandex.ru> <20140306145231.Q75313@sola.nimnet.asn.au> <20140307024724.T75313@sola.nimnet.asn.au> Date: Tue, 8 Apr 2014 00:23:05 +0200 Message-ID: Subject: Re: ipfw / routing issue on 9.2-RELEASE From: Andreas Nilsson To: Ian Smith , FreeBSD Net Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.17 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 07 Apr 2014 22:23:06 -0000 On Wed, Mar 26, 2014 at 5:42 PM, Andreas Nilsson wrote: > ... snip ... > > >>> I'm wondering what's happening on the outbound path, most of your rules >>> handle inbound (to kernel) and it seems that rule 65535 deals with most >>> outbound, except those specifically acting on both paths. >>> >> So do I :) >> >>> >>> Maybe try adding to the above: >>> ipfw add 63510 count log ip from table(1) to any out recv table(8) >>> and >>> ipfw add 64510 count log ip from table(1) to any out recv table(8) >>> >>> which should catch them on the outbound path before the default rule; >>> outbound packets have both receive and transmit interfaces available. >>> >>> They never get that far :( Those rules log nothing. So I see the packet >> coming in, triggering the skipto, triggering the count log ... in recv but >> not the count log ... out. >> >> >>> I guess you're sure that these packets are actually going out to some >>> external address, not a localhost or alias address (which of course are >>> received and not sent out anywhere)? >>> >> Yes, when the go through they go to external address, which is in the >> routing table. >> >>> >>> I guess the above new log lines should show the interface (if any) >>> these packets are leaving on, after routing (maybe a routing issue?) >>> >>> Good luck hunting. If in doubt, throw ever more logging at it! :) And >>> please let the list know if you solve it - or not! >>> >>> cheers, Ian >>> >> >> To make it even more interesting, it tested this on another machine, >> where I'm unable to reproduce this "problem". I'm thinking that a reboot >> might help, but this is kind of fascinating so I would like to actually >> find the cause. I will investigate further. >> >> Best regards >> Andreas >> > > And now it happened on another machine, but different type of traffic. > Traffic triggering the divert rule work fine, some traffic not triggering > the divert rule does not. After everything works as expected. > > Before reboot I flushed the firewall so that only default rule was in play > ( allow all from any to any ), and then *no* traffic for that destination > went out of the box. > > There are something like ~1000 routes in the routing table. Routes are > added and removed according to some triggers, so during uptime of the > machine there is a bit of messing with the routing table. At this stage I'm > at a loss on how to continue investigating this, so I'll just implement the > forwarding via fwd rules in ipfw and altogether ignore the routing table. > > Best regards > Andreas > Ok, found the culprit: -# $FreeBSD: releng/9.1/etc/rc.d/netif 212579 2010-09-13 19:55:40Z hrs $ +# $FreeBSD: releng/9.2/etc/rc.d/netif 253238 2013-07-12 01:34:24Z hrs $ + if [ -f /etc/rc.d/routing -a -n "$cmdifn" ] ; then + for _if in $cmdifn; do + /etc/rc.d/routing start any $_if + done + fi etc/rc.d/routing enforces gateway_enable setting in rc.conf which does not play well with setting net.inet.ip.forwarding=1 in sysctl.conf directly. Very annoying. I can find nothing i UPDATING on the subject. Even more annoying. Best regards Andreas