From owner-freebsd-ipfw@FreeBSD.ORG Mon Sep 17 03:39:37 2012 Return-Path: Delivered-To: freebsd-ipfw@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 92E01106566B for ; Mon, 17 Sep 2012 03:39:37 +0000 (UTC) (envelope-from dreijer@echobit.net) Received: from mail-oa0-f54.google.com (mail-oa0-f54.google.com [209.85.219.54]) by mx1.freebsd.org (Postfix) with ESMTP id 45F318FC0C for ; Mon, 17 Sep 2012 03:39:36 +0000 (UTC) Received: by oagm1 with SMTP id m1so5415292oag.13 for ; Sun, 16 Sep 2012 20:39:36 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type :x-gm-message-state; bh=yp8+H9IYvMwMCbKfI8Db9wS/HTc/YA0OOGfz5nKpeMY=; b=AJBKKCqZN0dNj1K0wAvRNcpqMBPeP45Nd0UT1coXBy0GvCYsOW+EGR+6eJ6Zftdw8N 1ZZ7522ZmmwsCVb50azSo1wnHJhEUaWxY2zwiYiFIRQFpN/bUZviyUwrfhKU1lwwG2yy Q+6fEQbmVzw51HKJx4CVHpWGKRuxy/bEH5WHBt7TGqI2FHo0vT168wrL6od0OgUm6jBE X15pU4QM6ZbRQWuuwU5bx0JvXhPD/QKXVmge0AJqf6qqWc7a1otQ4rV6Bx02qqbzWqZ8 7A+QDjZWE0LqnJvcSbJ4VcqUftnmhxcyrPvqTtPLmNI4ExuPMtQUwtXw0w4/9UWEiwmR pkAg== MIME-Version: 1.0 Received: by 10.182.174.9 with SMTP id bo9mr10471930obc.19.1347853176295; Sun, 16 Sep 2012 20:39:36 -0700 (PDT) Sender: dreijer@echobit.net Received: by 10.76.99.75 with HTTP; Sun, 16 Sep 2012 20:39:36 -0700 (PDT) In-Reply-To: References: <20120913221758.E51539@sola.nimnet.asn.au> <20120913163013.GA22049@onelab2.iet.unipi.it> <20120913174612.GB22571@onelab2.iet.unipi.it> <20120914144529.R51539@sola.nimnet.asn.au> <20120915034627.V51539@sola.nimnet.asn.au> Date: Sun, 16 Sep 2012 22:39:36 -0500 X-Google-Sender-Auth: DsJBhMaDoxbcsYeCWlwETtTJ29Y Message-ID: From: Soren Dreijer To: Ian Smith Content-Type: text/plain; charset=ISO-8859-1 X-Gm-Message-State: ALoCoQm93ZjIJuQ1ufj6+4zrCpivpEgPR0rl3YRaHdDixsc0bhX8Sy0WENCZt/jZos8nETkhlXwB Cc: freebsd-ipfw@freebsd.org, Luigi Rizzo Subject: Re: Significant network latency when using ipfw and in-kernel NAT X-BeenThere: freebsd-ipfw@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: IPFW Technical Discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 17 Sep 2012 03:39:37 -0000 Some more updates: I went ahead and disabled a few options on the ixgbe network interface today (most notably rxcsum and txcsum), which improved ping times to the FreeBSD box. I'm now able to reliably ping it with ~40ms from my house. TCP traffic in general also seems to be slightly "better" as I can actually 'wget google.com' now, although it's still horribly slow and takes maybe 20 seconds or so to download. The ifconfig for the public adapter now looks like this: ix1: flags=8843 metric 0 mtu 1500 options=b8 I also changed all "out via ix1" rules to "out xmit ix1", and have updated a few other "in via ix1" to "in recv ix1". None of these changes seemed to have any effect on traffic originating from the FreeBSD box, though, and I still have ping times of >3 seconds to google.com. Like I mentioned earlier, I tried putting "allow icmp from any to any via ix1" at the top of the ipfw ruleset (to avoid any NAT'ing whatsoever) to see if that had any effect on the ping times from the box and it didn't. What I did notice, however, (and I don't know if this is related to the overall network latency), was that the outgoing ping packets were severely delayed in tcpdump from when the ping utility sent a packet. The output in tcpdump would be so delayed that after having killed the ping utility, I'd still see a packet or two go out on the interface! I'm running out of ideas of what to do here... / Soren On Sun, Sep 16, 2012 at 11:46 AM, Soren Dreijer wrote: > Just to follow up on this a bit: > > I haven't disabled any other options on the NICs yet due to high > server load over the weekend, but I'll give it a go in the next few > days. Also, it looks like pings to the box are now no longer as fast > as I had previously stated. Pinging it from my home connection now > yields >3 second roundtrip times, which neatly matches the ping time > from the box itself to google.com. > > As I mentioned before, I'm not sure how e.g. rxcsum and txcsum have > anything to do with high latency on ICMP traffic, so I'm wondering if > we're perhaps barking up the wrong tree here (especially since > forwarded traffic *through* the FreeBSD box seems to work just fine)? > > Thanks again for helping out here, guys. I'm in pretty deep water when > it comes to issues like this one. > > / Soren > > On Fri, Sep 14, 2012 at 12:59 PM, Ian Smith wrote: >> On Fri, 14 Sep 2012 09:12:27 -0500, Soren Dreijer wrote: >> >> > Can anybody confirm that disabling these other options (rxcsum, >> > txcsum, vlanmtu, vlanhwtag, vlanhwfilter, vlanhwtso) won't cause my >> > adapter to lose its connectivity? This is a server in production and >> > I'd rather not cause an outage if I can prevent it. :) >> >> Fair question Soren. I've configured no VLANs; out of my depth, again! >> >> cheers, Ian >> >> > On Fri, Sep 14, 2012 at 12:00 AM, Ian Smith wrote: >> > > On Thu, 13 Sep 2012 12:37:23 -0500, Soren Dreijer wrote: >> > > [Luigi Rizzo wrote:] >> > > > > i'd start by disabling all accelerations (and jumobgrams) >> > > > > and then move on from the results to figure out where is the problem. >> > > > >> > > > So, I went ahead and disabled TSO on ix0. That seemed to fix the >> > > > intermittent connection issues I had been experiencing with keeping an >> > > > XMPP connection alive to one of our internal boxes. It hasn't done >> > > > anything for the ICMPs or TCP traffic originating from the FreeBSD >> > > > box, of course. >> > > >> > > Please show ifconfig for ix0 and ix1 again after disabling tso, >> > > rxcsum, txcsum, vlanmtu, vlanhwtag, vlanhwfilter, vlanhwtso >> > > and any other configured accelerations, as Luigi recommended? >> > > >> > > Then we'd know if your problem was related to any of that, or not. >> > > >> > > cheers, Ian >> >