From owner-freebsd-net@freebsd.org Sun Aug 23 02:41:16 2015 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 7C40B9C14A6 for ; Sun, 23 Aug 2015 02:41:16 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 68A0D1BC2 for ; Sun, 23 Aug 2015 02:41:16 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id t7N2fGMM092317 for ; Sun, 23 Aug 2015 02:41:16 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-net@FreeBSD.org Subject: [Bug 202510] [CARP] advertisements sourced from CARP IP cause double MASTER situations Date: Sun, 23 Aug 2015 02:41:16 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 10.0-STABLE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Many People X-Bugzilla-Who: linimon@FreeBSD.org X-Bugzilla-Status: New X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-net@FreeBSD.org X-Bugzilla-Target-Milestone: --- X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: assigned_to Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 7bit X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 23 Aug 2015 02:41:16 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=202510 Mark Linimon changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|freebsd-bugs@FreeBSD.org |freebsd-net@FreeBSD.org -- You are receiving this mail because: You are the assignee for the bug. From owner-freebsd-net@freebsd.org Sun Aug 23 06:00:28 2015 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 39E019C0060 for ; Sun, 23 Aug 2015 06:00:28 +0000 (UTC) (envelope-from smithi@nimnet.asn.au) Received: from sola.nimnet.asn.au (paqi.nimnet.asn.au [115.70.110.159]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 3A1B61B15; Sun, 23 Aug 2015 06:00:26 +0000 (UTC) (envelope-from smithi@nimnet.asn.au) Received: from localhost (localhost [127.0.0.1]) by sola.nimnet.asn.au (8.14.2/8.14.2) with ESMTP id t7N60Fju091181; Sun, 23 Aug 2015 16:00:16 +1000 (EST) (envelope-from smithi@nimnet.asn.au) Date: Sun, 23 Aug 2015 16:00:15 +1000 (EST) From: Ian Smith To: Hiroki Sato cc: truckman@freebsd.org, freebsd-net@freebsd.org Subject: Re: a couple /etc/rc.firewall questions In-Reply-To: <20150823.084453.1715908115913144015.hrs@allbsd.org> Message-ID: <20150823151421.G8515@sola.nimnet.asn.au> References: <201508222103.t7ML3gAx000794@gw.catspoiler.org> <20150823.084453.1715908115913144015.hrs@allbsd.org> MIME-Version: 1.0 Content-Type: MULTIPART/MIXED; BOUNDARY="0-315991574-1440309615=:8515" X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 23 Aug 2015 06:00:28 -0000 This message is in MIME format. The first part should be readable text, while the remaining parts are likely unreadable without MIME-aware tools. --0-315991574-1440309615=:8515 Content-Type: TEXT/PLAIN; charset=US-ASCII On Sun, 23 Aug 2015 08:44:53 +0900, Hiroki Sato wrote: > Don Lewis wrote > in <201508222103.t7ML3gAx000794@gw.catspoiler.org>: > > tr> The example /etc/rc.firewall has provisions to use either in-kernel NAT > tr> or natd for the open and client firewall types, but the simple filewall > tr> type only has code for natd. Is there any reason that in-kernel NAT > tr> could not be used with the simple firewall type? > > I think there is no particular reason. Simple rule was just not updated. I did send you and -ipfw@ a patch for that on several occasions since Feb. 2013, though I did fail to push it into the 3-4 PRs it affected. The attached patch addresses that, chooses kernel NAT over natd(8) if both were enabled in rc.conf, updates some commentary and fixes an overwordy line in 'workstation'. Just now checked it against HEAD. > tr> After allowing connections to selected TCP ports and then denying all > tr> other incoming TCP setup connections from ${oif}, the simple firewall > tr> code in /etc/rc.firewall then permits all other TCP setup connections: > tr> # Allow setup of any other TCP connection > tr> ${fwcmd} add pass tcp from any to any setup > tr> This is potentially undesirable since it allows unrestricted TCP > tr> connections between "me" and the inside network. When I changed this to > tr> ${fwcmd} add pass tcp from any to any out via ${oif} setup > tr> I was able to open TCP connections from the firewall box to the outside, > tr> but NATed connections from inside network to the outside were blocked. > tr> If I run "ipfw show", it appears that the TCP setup packets are falling > tr> through to the final implicit deny all rule, but I don't see any obvious > tr> reason. > > A TCP setup packet coming from a host on the internal LAN to the NAPT > router falls into the last deny-all rule because it does not match if > you added "out via ${oif}" to that rule. Does the following > additional rule work for you? > > ${fwcmd} add pass tcp from any to any out via ${oif} setup That looks ok, maybe some UDP too? Adding some stateful rules is another option for dealing with inside hosts' external requests. > ${fwcmd} add pass tcp from any to not me in via ${iif} setup If you want to deny inside hosts access to host services, that'll do it. The other long-term issue with 'simple' is that it permits no ICMPv4 at all. Neither inside nor outside, no pings, no PMTU, nothing .. although curiously allows selected ICMP for ipv6. I usually add something like: ${fwcmd} add pass icmp from any to any icmptype 0,3,8,11 If you don't want to allow pings from outside your net, preceded with: ${fwcmd} add deny icmp from any to any in recv ${oif} icmptype 8 cheers, Ian > -- Hiroki --0-315991574-1440309615=:8515 Content-Type: TEXT/PLAIN; charset=US-ASCII; name=rc.firewall.patch2 Content-Transfer-Encoding: BASE64 Content-ID: <20150823160015.Y8515@sola.nimnet.asn.au> Content-Description: Content-Disposition: attachment; filename=rc.firewall.patch2 LS0tIHJjLmZpcmV3YWxsLnIyMzg0MTYJMjAxNS0wOC0yMyAxNDo0ODoyNi4w MDAwMDAwMDAgKzEwMDANCisrKyByYy5maXJld2FsbAkyMDE0LTAzLTI2IDE1 OjQyOjI1LjAwMDAwMDAwMCArMTEwMA0KQEAgLTUyLDcgKzUyLDcgQEANCiAj ICAgZmlsZW5hbWUgICAgLSB3aWxsIGxvYWQgdGhlIHJ1bGVzIGluIHRoZSBn aXZlbiBmaWxlbmFtZSAoZnVsbCBwYXRoIHJlcXVpcmVkKQ0KICMNCiAjIEZv ciBgYGNsaWVudCcnIGFuZCBgYHNpbXBsZScnIHRoZSBlbnRyaWVzIGJlbG93 IHNob3VsZCBiZSBjdXN0b21pemVkDQotIyBhcHByb3ByaWF0ZWx5Lg0KKyMg YXBwcm9wcmlhdGVseSB3aXRoIHJjLmNvbmYgdmFyaWFibGVzLg0KIA0KICMj IyMjIyMjIyMjIw0KICMNCkBAIC0xMTIsNiArMTEyLDI5IEBADQogCSR7Zndj bWR9IGFkZCBwYXNzIGlwdjYtaWNtcCBmcm9tIGFueSB0byBhbnkgaWNtcDZ0 eXBlcyAyLDEzNSwxMzYNCiB9DQogDQorc2V0dXBfbmF0ICgpIHsNCisJbG9j YWwgaWZsYWcNCisJaWYgY2hlY2t5ZXNubyBmaXJld2FsbF9uYXRfZW5hYmxl OyB0aGVuDQorCQlpZiBbIC1uICIke2ZpcmV3YWxsX25hdF9pbnRlcmZhY2V9 IiBdOyB0aGVuDQorCQkJaWYgZWNobyAiJHtmaXJld2FsbF9uYXRfaW50ZXJm YWNlfSIgfCBcDQorCQkJCWdyZXAgLXEgLUUgJ15bMC05XSsoXC5bMC05XSsp ezAsM30kJzsgdGhlbg0KKwkJCQlpZmxhZz0iaXAgJHtmaXJld2FsbF9uYXRf aW50ZXJmYWNlfSINCisJCQllbHNlDQorCQkJCWlmbGFnPSJpZiAke2ZpcmV3 YWxsX25hdF9pbnRlcmZhY2V9Ig0KKwkJCWZpDQorCQkJZmlyZXdhbGxfbmF0 X2ZsYWdzPSIkaWZsYWcgJHtmaXJld2FsbF9uYXRfZmxhZ3N9Ig0KKwkJCSR7 ZndjbWR9IG5hdCAxMjMgY29uZmlnIGxvZyAke2ZpcmV3YWxsX25hdF9mbGFn c30NCisJCQkke2Z3Y21kfSBhZGQgJDEgbmF0IDEyMyBpcDQgZnJvbSBhbnkg dG8gYW55IFwNCisJCQkJdmlhICR7ZmlyZXdhbGxfbmF0X2ludGVyZmFjZX0N CisJCWZpDQorCWVsaWYgY2hlY2t5ZXNubyBuYXRkX2VuYWJsZTsgdGhlbg0K KwkJaWYgWyAtbiAiJHtuYXRkX2ludGVyZmFjZX0iIF07IHRoZW4NCisJCQkk e2Z3Y21kfSBhZGQgJDEgZGl2ZXJ0IG5hdGQgaXA0IGZyb20gYW55IHRvIGFu eSBcDQorCQkJCXZpYSAke25hdGRfaW50ZXJmYWNlfQ0KKwkJZmkNCisJZmkN Cit9DQorDQogaWYgWyAtbiAiJHsxfSIgXTsgdGhlbg0KIAlmaXJld2FsbF90 eXBlPSIkezF9Ig0KIGZpDQpAQCAtMTQyLDM3ICsxNjUsMTcgQEANCiBzZXR1 cF9pcHY2X21hbmRhdG9yeQ0KIA0KICMjIyMjIyMjIyMjIw0KLSMgTmV0d29y ayBBZGRyZXNzIFRyYW5zbGF0aW9uLiAgQWxsIHBhY2tldHMgYXJlIHBhc3Nl ZCB0byBuYXRkKDgpDQotIyBiZWZvcmUgdGhleSBlbmNvdW50ZXIgeW91ciBy ZW1haW5pbmcgcnVsZXMuICBUaGUgZmlyZXdhbGwgcnVsZXMNCi0jIHdpbGwg dGhlbiBiZSBydW4gYWdhaW4gb24gZWFjaCBwYWNrZXQgYWZ0ZXIgdHJhbnNs YXRpb24gYnkgbmF0ZA0KLSMgc3RhcnRpbmcgYXQgdGhlIHJ1bGUgbnVtYmVy IGZvbGxvd2luZyB0aGUgZGl2ZXJ0IHJ1bGUuDQorIyBOZXR3b3JrIEFkZHJl c3MgVHJhbnNsYXRpb24uICBBbGwgcGFja2V0cyBhcmUgcGFzc2VkIHRvIGtl cm5lbCBuYXQNCisjIG9yIG5hdGQoOCkgYmVmb3JlIHRoZXkgZW5jb3VudGVy IHlvdXIgcmVtYWluaW5nIHJ1bGVzLiAgVGhlIGZpcmV3YWxsDQorIyBydWxl cyB3aWxsIHRoZW4gYmUgcnVuIGFnYWluIG9uIGVhY2ggcGFja2V0IGFmdGVy IE5BVCB0cmFuc2xhdGlvbg0KKyMgc3RhcnRpbmcgYXQgdGhlIHJ1bGUgbnVt YmVyIGZvbGxvd2luZyB0aGUgbmF0IG9yIGRpdmVydCBydWxlLg0KICMNCi0j IEZvciBgYHNpbXBsZScnIGZpcmV3YWxsIHR5cGUgdGhlIGRpdmVydCBydWxl IHNob3VsZCBiZSBwdXQgdG8gYQ0KLSMgZGlmZmVyZW50IHBsYWNlIHRvIG5v dCBpbnRlcmZlcmUgd2l0aCBhZGRyZXNzLWNoZWNraW5nIHJ1bGVzLg0KKyMg Rm9yIGBgc2ltcGxlJycgZmlyZXdhbGwgdHlwZSB0aGUgbmF0IG9yIGRpdmVy dCBydWxlIGlzIGluc3RhbGxlZCBpbg0KKyMgYSBkaWZmZXJlbnQgcGxhY2Ug dG8gYXZvaWQgaW50ZXJmZXJpbmcgd2l0aCBhZGRyZXNzLWNoZWNraW5nIHJ1 bGVzLg0KICMNCiBjYXNlICR7ZmlyZXdhbGxfdHlwZX0gaW4NCiBbT29dW1Bw XVtFZV1bTm5dfFtDY11bTGxdW0lpXVtFZV1bTm5dW1R0XSkNCi0JY2FzZSAk e25hdGRfZW5hYmxlfSBpbg0KLQlbWXldW0VlXVtTc10pDQotCQlpZiBbIC1u ICIke25hdGRfaW50ZXJmYWNlfSIgXTsgdGhlbg0KLQkJCSR7ZndjbWR9IGFk ZCA1MCBkaXZlcnQgbmF0ZCBpcDQgZnJvbSBhbnkgdG8gYW55IHZpYSAke25h dGRfaW50ZXJmYWNlfQ0KLQkJZmkNCi0JCTs7DQotCWVzYWMNCi0JY2FzZSAk e2ZpcmV3YWxsX25hdF9lbmFibGV9IGluDQotCVtZeV1bRWVdW1NzXSkNCi0J CWlmIFsgLW4gIiR7ZmlyZXdhbGxfbmF0X2ludGVyZmFjZX0iIF07IHRoZW4N Ci0JCQlpZiBlY2hvICIke2ZpcmV3YWxsX25hdF9pbnRlcmZhY2V9IiB8IFwN Ci0JCQkJZ3JlcCAtcSAtRSAnXlswLTldKyhcLlswLTldKyl7MCwzfSQnOyB0 aGVuDQotCQkJCWZpcmV3YWxsX25hdF9mbGFncz0iaXAgJHtmaXJld2FsbF9u YXRfaW50ZXJmYWNlfSAke2ZpcmV3YWxsX25hdF9mbGFnc30iDQotCQkJZWxz ZQ0KLQkJCQlmaXJld2FsbF9uYXRfZmxhZ3M9ImlmICR7ZmlyZXdhbGxfbmF0 X2ludGVyZmFjZX0gJHtmaXJld2FsbF9uYXRfZmxhZ3N9Ig0KLQkJCWZpDQot CQkJJHtmd2NtZH0gbmF0IDEyMyBjb25maWcgbG9nICR7ZmlyZXdhbGxfbmF0 X2ZsYWdzfQ0KLQkJCSR7ZndjbWR9IGFkZCA1MCBuYXQgMTIzIGlwNCBmcm9t IGFueSB0byBhbnkgdmlhICR7ZmlyZXdhbGxfbmF0X2ludGVyZmFjZX0NCi0J CWZpDQotCQk7Ow0KLQllc2FjDQorCXNldHVwX25hdCA1MA0KIGVzYWMNCiAN CiAjIyMjIyMjIyMjIyMNCkBAIC0zMTEsMTMgKzMxNCw3IEBADQogCSMgdHJh bnNsYXRlZCBieSBuYXRkKDgpIHdvdWxkIG1hdGNoIHRoZSBgZGVueScgcnVs ZSBhYm92ZS4gIFNpbWlsYXJseQ0KIAkjIGFuIG91dGdvaW5nIHBhY2tldCBv cmlnaW5hdGVkIGZyb20gaXQgYmVmb3JlIGJlaW5nIHRyYW5zbGF0ZWQgd291 bGQNCiAJIyBtYXRjaCB0aGUgYGRlbnknIHJ1bGUgYmVsb3cuDQotCWNhc2Ug JHtuYXRkX2VuYWJsZX0gaW4NCi0JW1l5XVtFZV1bU3NdKQ0KLQkJaWYgWyAt biAiJHtuYXRkX2ludGVyZmFjZX0iIF07IHRoZW4NCi0JCQkke2Z3Y21kfSBh ZGQgZGl2ZXJ0IG5hdGQgaXA0IGZyb20gYW55IHRvIGFueSB2aWEgJHtuYXRk X2ludGVyZmFjZX0NCi0JCWZpDQotCQk7Ow0KLQllc2FjDQorCXNldHVwX25h dA0KIA0KIAkjIFN0b3AgUkZDMTkxOCBuZXRzIG9uIHRoZSBvdXRzaWRlIGlu dGVyZmFjZQ0KIAkke2Z3Y21kfSBhZGQgZGVueSBhbGwgZnJvbSAxMC4wLjAu MC84IHRvIGFueSB2aWEgJHtvaWZ9DQpAQCAtNTE5LDcgKzUxNiw3IEBADQog DQogCSMgRGVueSBhbmQgKGlmIHdhbnRlZCkgbG9nIHRoZSByZXN0IHVuY29u ZGl0aW9uYWxseS4NCiAJbG9nPSIiDQotCWlmIFsgJHtmaXJld2FsbF9sb2dk ZW55Oi14fSA9ICJZRVMiIC1vICR7ZmlyZXdhbGxfbG9nZGVueToteH0gPSAi eWVzIiBdIDsgdGhlbg0KKwlpZiBjaGVja3llc25vIGZpcmV3YWxsX2xvZ2Rl bnk7IHRoZW4NCiAJICBsb2c9ImxvZyBsb2dhbW91bnQgNTAwIgkjIFRoZSBk ZWZhdWx0IG9mIDEwMCBpcyB0b28gbG93Lg0KIAkgIHN5c2N0bCBuZXQuaW5l dC5pcC5mdy52ZXJib3NlPTEgPi9kZXYvbnVsbA0KIAlmaQ0K --0-315991574-1440309615=:8515-- From owner-freebsd-net@freebsd.org Sun Aug 23 06:50:00 2015 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id BBFC59C085F for ; Sun, 23 Aug 2015 06:50:00 +0000 (UTC) (envelope-from kob6558@gmail.com) Received: from mail-ob0-x22b.google.com (mail-ob0-x22b.google.com [IPv6:2607:f8b0:4003:c01::22b]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 88F80CB9; Sun, 23 Aug 2015 06:50:00 +0000 (UTC) (envelope-from kob6558@gmail.com) Received: by obkg7 with SMTP id g7so89183310obk.3; Sat, 22 Aug 2015 23:49:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=g4ebMzFO5gaEYynGw2A9ojpZHUN9/OVDRxydrKa7YYw=; b=OVgmcbWl9ltzrxZRtJuUoiQpKwq8g9CUqcWY7u9ZxpixASOVyzrtH3Ucu9HukmpGjC F0oU6RKeTxhe8H/mIOCd/f8bdjQ92E/6NOFCdat+Jedt3qsPg1xqNhj6uARzblPq2FDy Hf3nInHnJQA2SNeP/+UIwHmqhbZUzvNKxVDPmXoenfbxr9esNb5u8oZ13fzeLVjF8Sps 1hKHbt+l3Lgq7GU4LhbAm24hc+0vf58Ez3xJLmenr1jOI19A8qk+U/ahUW6ob5kOSL06 XKzGWZqol5XkRreKvorMtgbcLtbZR7Ik2Y5bgWEJ9mB1SKIenklTlq9Vcmknxd0kvWPk BKdA== MIME-Version: 1.0 X-Received: by 10.60.94.210 with SMTP id de18mr15628218oeb.75.1440312599697; Sat, 22 Aug 2015 23:49:59 -0700 (PDT) Sender: kob6558@gmail.com Received: by 10.202.173.83 with HTTP; Sat, 22 Aug 2015 23:49:59 -0700 (PDT) In-Reply-To: <20150823151421.G8515@sola.nimnet.asn.au> References: <201508222103.t7ML3gAx000794@gw.catspoiler.org> <20150823.084453.1715908115913144015.hrs@allbsd.org> <20150823151421.G8515@sola.nimnet.asn.au> Date: Sat, 22 Aug 2015 20:49:59 -1000 X-Google-Sender-Auth: ZZixis7yFo3euYNk6gdfsNfnVZs Message-ID: Subject: Re: a couple /etc/rc.firewall questions From: Kevin Oberman To: Ian Smith Cc: Hiroki Sato , "freebsd-net@freebsd.org" , Don Lewis Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 23 Aug 2015 06:50:00 -0000 On Sat, Aug 22, 2015 at 8:00 PM, Ian Smith wrote: > On Sun, 23 Aug 2015 08:44:53 +0900, Hiroki Sato wrote: > > Don Lewis wrote > > in <201508222103.t7ML3gAx000794@gw.catspoiler.org>: > > > > tr> The example /etc/rc.firewall has provisions to use either in-kernel > NAT > > tr> or natd for the open and client firewall types, but the simple > filewall > > tr> type only has code for natd. Is there any reason that in-kernel NAT > > tr> could not be used with the simple firewall type? > > > > I think there is no particular reason. Simple rule was just not > updated. > > I did send you and -ipfw@ a patch for that on several occasions since > Feb. 2013, though I did fail to push it into the 3-4 PRs it affected. > > The attached patch addresses that, chooses kernel NAT over natd(8) if > both were enabled in rc.conf, updates some commentary and fixes an > overwordy line in 'workstation'. Just now checked it against HEAD. > > > tr> After allowing connections to selected TCP ports and then denying > all > > tr> other incoming TCP setup connections from ${oif}, the simple > firewall > > tr> code in /etc/rc.firewall then permits all other TCP setup > connections: > > tr> # Allow setup of any other TCP connection > > tr> ${fwcmd} add pass tcp from any to any setup > > tr> This is potentially undesirable since it allows unrestricted TCP > > tr> connections between "me" and the inside network. When I changed > this to > > tr> ${fwcmd} add pass tcp from any to any out via ${oif} setup > > tr> I was able to open TCP connections from the firewall box to the > outside, > > tr> but NATed connections from inside network to the outside were > blocked. > > tr> If I run "ipfw show", it appears that the TCP setup packets are > falling > > tr> through to the final implicit deny all rule, but I don't see any > obvious > > tr> reason. > > > > A TCP setup packet coming from a host on the internal LAN to the NAPT > > router falls into the last deny-all rule because it does not match if > > you added "out via ${oif}" to that rule. Does the following > > additional rule work for you? > > > > ${fwcmd} add pass tcp from any to any out via ${oif} setup > > That looks ok, maybe some UDP too? Adding some stateful rules is > another option for dealing with inside hosts' external requests. > > > ${fwcmd} add pass tcp from any to not me in via ${iif} setup > > If you want to deny inside hosts access to host services, that'll do it. > > The other long-term issue with 'simple' is that it permits no ICMPv4 at > all. Neither inside nor outside, no pings, no PMTU, nothing .. although > curiously allows selected ICMP for ipv6. I usually add something like: > Not so curious. Without ICMPv6, IPv6 is effectively off. unlike IPv4 which will work fairly well without ICMP, though it will technically be broken and, for some cases will be REALLY broken. That's why network rc.firewall contains: setup_ipv6_mandatory() { [ $ipv6_available -eq 0 ] || return 0 ############ # Only in rare cases do you want to change these rules with the rules that allow normal IPv6 networking start when default deny is specified. Most importantly, NDP is required to get IPv6 to start (though there are work-arounds) and DAD is required to start it safely. This is a slightly touchy point with me as I filed a PR on this many years ago when such had not been added to rc.firewall. It was summarily closed with a message that proper security mandated that there be nothing allowed until they were put in place by firewall.conf (which ran WAY too late to do the job.) I don't recall who shot it down, but I was quite annoyed that someone who was clearly clueless about IPv6 would just toss a PR on a subject he didn't know. It was about a year before the standard rc.firewall started opening the required holes. I added this to mine as my old network started running IPv6 around the turn of the millennium and made it a production service in 2004, IIRC. -- Kevin Oberman, Network Engineer, Retired E-mail: rkoberman@gmail.com PGP Fingerprint: D03FB98AFA78E3B78C1694B318AB39EF1B055683 From owner-freebsd-net@freebsd.org Sun Aug 23 11:09:12 2015 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id A98199C14BE; Sun, 23 Aug 2015 11:09:12 +0000 (UTC) (envelope-from danny@cs.huji.ac.il) Received: from kabab.cs.huji.ac.il (kabab.cs.huji.ac.il [132.65.116.210]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 299A818D0; Sun, 23 Aug 2015 11:09:11 +0000 (UTC) (envelope-from danny@cs.huji.ac.il) Received: from chamsa.cs.huji.ac.il ([132.65.80.19]) by kabab.cs.huji.ac.il with esmtp id 1ZTT8z-000KPV-3n; Sun, 23 Aug 2015 14:08:57 +0300 Mime-Version: 1.0 (Mac OS X Mail 8.2 \(2104\)) Subject: Re: ix(intel) vs mlxen(mellanox) 10Gb performance From: Daniel Braniss In-Reply-To: <818666007.28930310.1440244756872.JavaMail.zimbra@uoguelph.ca> Date: Sun, 23 Aug 2015 14:08:56 +0300 Cc: pyunyh@gmail.com, Hans Petter Selasky , FreeBSD stable , FreeBSD Net , Christopher Forgeron , Gleb Smirnoff , Slawa Olhovchenkov Message-Id: <49173B1F-7B5E-4D59-8651-63D97B0CB5AC@cs.huji.ac.il> References: <1D52028A-B39F-4F9B-BD38-CB1D73BF5D56@cs.huji.ac.il> <55D429A4.3010407@selasky.org> <20150819074212.GB964@michelle.fasterthan.com> <55D43615.1030401@selasky.org> <2013503980.25726607.1439989235806.JavaMail.zimbra@uoguelph.ca> <20150820023024.GB996@michelle.fasterthan.com> <1153838447.28656490.1440193567940.JavaMail.zimbra@uoguelph.ca> <15D19823-08F7-4E55-BBD0-CE230F67D26E@cs.huji.ac.il> <818666007.28930310.1440244756872.JavaMail.zimbra@uoguelph.ca> To: Rick Macklem X-Mailer: Apple Mail (2.2104) Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 23 Aug 2015 11:09:12 -0000 > On 22 Aug 2015, at 14:59, Rick Macklem wrote: >=20 > Daniel Braniss wrote: >>=20 >>> On Aug 22, 2015, at 12:46 AM, Rick Macklem = wrote: >>>=20 >>> Yonghyeon PYUN wrote: >>>> On Wed, Aug 19, 2015 at 09:00:35AM -0400, Rick Macklem wrote: >>>>> Hans Petter Selasky wrote: >>>>>> On 08/19/15 09:42, Yonghyeon PYUN wrote: >>>>>>> On Wed, Aug 19, 2015 at 09:00:52AM +0200, Hans Petter Selasky = wrote: >>>>>>>> On 08/18/15 23:54, Rick Macklem wrote: >>>>>>>>> Ouch! Yes, I now see that the code that counts the # of mbufs = is >>>>>>>>> before >>>>>>>>> the >>>>>>>>> code that adds the tcp/ip header mbuf. >>>>>>>>>=20 >>>>>>>>> In my opinion, this should be fixed by setting = if_hw_tsomaxsegcount >>>>>>>>> to >>>>>>>>> whatever >>>>>>>>> the driver provides - 1. It is not the driver's responsibility = to >>>>>>>>> know if >>>>>>>>> a tcp/ip >>>>>>>>> header mbuf will be added and is a lot less confusing that = expecting >>>>>>>>> the >>>>>>>>> driver >>>>>>>>> author to know to subtract one. (I had mistakenly thought that >>>>>>>>> tcp_output() had >>>>>>>>> added the tc/ip header mbuf before the loop that counts mbufs = in the >>>>>>>>> list. >>>>>>>>> Btw, >>>>>>>>> this tcp/ip header mbuf also has leading space for the MAC = layer >>>>>>>>> header.) >>>>>>>>>=20 >>>>>>>>=20 >>>>>>>> Hi Rick, >>>>>>>>=20 >>>>>>>> Your question is good. With the Mellanox hardware we have = separate >>>>>>>> so-called inline data space for the TCP/IP headers, so if the = TCP >>>>>>>> stack >>>>>>>> subtracts something, then we would need to add something to the = limit, >>>>>>>> because then the scatter gather list is only used for the data = part. >>>>>>>>=20 >>>>>>>=20 >>>>>>> I think all drivers in tree don't subtract 1 for >>>>>>> if_hw_tsomaxsegcount. Probably touching Mellanox driver would = be >>>>>>> simpler than fixing all other drivers in tree. >>>>>>>=20 >>>>>>>> Maybe it can be controlled by some kind of flag, if all the = three TSO >>>>>>>> limits should include the TCP/IP/ethernet headers too. I'm = pretty sure >>>>>>>> we want both versions. >>>>>>>>=20 >>>>>>>=20 >>>>>>> Hmm, I'm afraid it's already complex. Drivers have to tell = almost >>>>>>> the same information to both bus_dma(9) and network stack. >>>>>>=20 >>>>>> Don't forget that not all drivers in the tree set the TSO limits = before >>>>>> if_attach(), so possibly the subtraction of one TSO fragment = needs to go >>>>>> into ip_output() .... >>>>>>=20 >>>>> Ok, I realized that some drivers may not know the answers before >>>>> ether_ifattach(), >>>>> due to the way they are configured/written (I saw the use of >>>>> if_hw_tsomax_update() >>>>> in the patch). >>>>=20 >>>> I was not able to find an interface that configures TSO parameters >>>> after if_t conversion. I'm under the impression >>>> if_hw_tsomax_update() is not designed to use this way. Probably we >>>> need a better one?(CCed to Gleb). >>>>=20 >>>>>=20 >>>>> If it is subtracted as a part of the assignment to = if_hw_tsomaxsegcount >>>>> in >>>>> tcp_output() >>>>> at line#791 in tcp_output() like the following, I don't think it = should >>>>> matter if the >>>>> values are set before ether_ifattach()? >>>>> /* >>>>> * Subtract 1 for the tcp/ip header mbuf that >>>>> * will be prepended to the mbuf chain in this >>>>> * function in the code below this block. >>>>> */ >>>>> if_hw_tsomaxsegcount =3D tp->t_tsomaxsegcount - = 1; >>>>>=20 >>>>> I don't have a good solution for the case where a driver doesn't = plan on >>>>> using the >>>>> tcp/ip header provided by tcp_output() except to say the driver = can add >>>>> one >>>>> to the >>>>> setting to compensate for that (and if they fail to do so, it = still >>>>> works, >>>>> although >>>>> somewhat suboptimally). When I now read the comment in = sys/net/if_var.h >>>>> it >>>>> is clear >>>>> what it means, but for some reason I didn't read it that way = before? (I >>>>> think it was >>>>> the part that said the driver didn't have to subtract for the = headers >>>>> that >>>>> confused me?) >>>>> In any case, we need to try and come up with a clear definition of = what >>>>> they need to >>>>> be set to. >>>>>=20 >>>>> I can now think of two ways to deal with this: >>>>> 1 - Leave tcp_output() as is, but provide a macro for the device = driver >>>>> authors to use >>>>> that sets if_hw_tsomaxsegcount with a flag for "driver uses = tcp/ip >>>>> header mbuf", >>>>> documenting that this flag should normally be true. >>>>> OR >>>>> 2 - Change tcp_output() as above, noting that this is a workaround = for >>>>> confusion w.r.t. >>>>> whether or not if_hw_tsomaxsegcount should include the tcp/ip = header >>>>> mbuf and >>>>> update the comment in if_var.h to reflect this. Then drivers = that >>>>> don't >>>>> use the >>>>> tcp/ip header mbuf can increase their value for = if_hw_tsomaxsegcount >>>>> by >>>>> 1. >>>>> (The comment should also mention that a value of 35 or greater = is much >>>>> preferred to >>>>> 32 if the hardware will support that.) >>>>>=20 >>>>=20 >>>> Both works for me. My preference is 2 just because it's very >>>> common for most drivers that use tcp/ip header mbuf. >>> Thanks for this comment. I tend to agree, both for the reason you = state and >>> also >>> because the patch is simple enough that it might qualify as an = errata for >>> 10.2. >>>=20 >>> I am hoping Daniel Braniss will be able to test the patch and let us = know >>> if it >>> improves performance with TSO enabled? >>=20 >> send me the patch and I=E2=80=99ll test it ASAP. >> danny >>=20 > Patch is attached. The one for head will also include an update to the = comment > in sys/net/if_var.h, but that isn't needed for testing. well, the plot thickens. Yesterday, before running the new kernel, I decided to re run my test, = and to my surprise i was getting good numbers, about 300MGB/s with and without TSO. this morning, the numbers were again bad, around 70MGB/s,what the ^%$#@! so, after some coffee, I run some more tests, and some conclusions: using a netapp(*) as the nfs client: - doing=20 ifconfig ix0 tso or -tso does some magic and numbers are back to normal - for a while using another Fbsd/zfs as client all is nifty, actually a bit faster = than the netapp (not a fair comparison, since the zfs client is not heavily used) and I can=E2=80=99t = see any degradation. =20 btw, this is with the patch applied, but was seeing similar numbers = before the patch. running with tso, initially I get around 300MGB/s, but after a = while(sorry can=E2=80=99t be more scientific) it drops down to about half, and finally to a pathetic 70MGB/s *: while running the tests I monitored the Netapp, and nothing out of = the ordinary there. cheers, danny From owner-freebsd-net@freebsd.org Sun Aug 23 12:03:13 2015 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 0D7D09BA28F for ; Sun, 23 Aug 2015 12:03:13 +0000 (UTC) (envelope-from bounce_Ayw26yaeJnSA8M82_pdBFDE2AJsmknBVO_df8eb5e06b037b98_1@pm1009.com) Received: from b5.mtb-2.com (b5.mtb-2.com [69.63.146.174]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id A8AA31679 for ; Sun, 23 Aug 2015 12:03:12 +0000 (UTC) (envelope-from bounce_Ayw26yaeJnSA8M82_pdBFDE2AJsmknBVO_df8eb5e06b037b98_1@pm1009.com) DKIM-Signature: v=1; a=rsa-sha1; c=relaxed/relaxed; s=mykey2; d=mtb-2.com; h=Reply-To:List-Unsubscribe:From:To:Subject:MIME-Version:Date:Message-Id:Content-Type; bh=EKmAVbuFXQ+nxHpEcDr3gw4MSOI=; b=i5DnX80gizqN8nksj2WYmpw6iBSHCuzrStuuUf58n3ARFqwjGHr9QGK06F4fYTdqHR9ddL0Axeho XzQ2kCtBTAGk4MoZaQhnCmkk0HAkc1TXI3+GlZHT5Kqr3HRRR1rbJ/OA9v9vKq5kMlpTMY/x5pxt fJTBCjnTdTHvpZMva7aCS+fqNEg7qgGOl7A/ZNTGb0MiHRbzlcyNsEJ/a1xJiazVm+L6sLaejamS 7vKIyMGYbtfoPLtN8jzRIsR/ocFBh84ZjeOuWHu5s8kPNUpCUSFGbEABa1Fbth/4le+C2JIBw9P7 KQeeITiv0kKhK/KoNDIQ1gUqmghhwlXR+IVceg== Received: from localhost (10.10.204.18) by b5.mtb-2.com id hr6r80157o40 for ; Sun, 23 Aug 2015 13:53:05 +0200 (envelope-from ) Reply-To: digital@cgconsulting.co.za X-Priority: 3 X-Report-Abuse: X-Data: pdBFDE2AJsmknBVO.df8eb5e06b037b98 X-Data-EUID: 38c72f7ae0162144901e855c78e5cf06 X-Data-Rating: -2 From: Philips Healthcare To: freebsd-net@freebsd.org Subject: Accelerating Healthcare Innovation MIME-Version: 1.0 Date: Sun, 23 Aug 2015 12:53:05 +0100 Message-Id: <2015082312122305.25455.57@digitalfire.co.za> Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 23 Aug 2015 12:03:13 -0000 Philips Healthcare Issue: August 2015 Dramatic changes in store for the hospital of tomorrow “Big data” and information and communications technology set to revolutionize healthcare. Find out more ( http://dfire.ensighthq.com/content/philips/P020/media/Philips_WSJ-02_Dramatic-changes-in-store-for-the-hospital-of-tomorrow.pdf?origin=3_me_en_healthcare_media_campaign_digitalfire_eb_jul__kenya ) Precious details See how Philips MRI technology helped a team of MRI clinicians and surgeons remove a boy’s brain tumor. Follow Mustafa into the operating room in this touching story of hope. Find out more ( https://www.youtube.com/watch?v=k5KZsxLw1Bg?origin=3_me_en_healthcare_media_campaign_philips_eb_jul__kenya ) Precious details No baby forgotten No baby forgotten Gertrude’s Children’s Hospital used the Philips IntelliVue X2 technology to monitor an abandoned baby’s vital signs and help her thrive. Find out more ( https://www.youtube.com/watch?v=O2OMV-7xgts?origin=3_me_en_healthcare_media_campaign_philips_eb_jul__kenya ) Also in this issue: Leveraging key intersections to accelerate healthcare innovation Sharing risk, responsibility and reward Technology and process re-engineering open the door to coordinated care Leveraging key intersections to accelerate healthcare innovation Sharing risk, responsibility and reward Technology and process re-engineering open the door to coordinated care Find out more ( http://dfire.ensighthq.com/content/philips/P020/media/Philips_WSJ-05_Leveraging-key-intersections-to-accelerate-healthcare-innovation.pdf?origin=3_me_en_healthcare_media_campaign_digitalfire_eb_jul__kenya ) Find out more ( http://dfire.ensighthq.com/content/philips/P020/media/Philips_WSJ-13_Sharing-risk-responsibility-and-reward.pdf?origin=3_me_en_healthcare_media_campaign_digitalfire_eb_jul__kenya ) Find out more ( http://dfire.ensighthq.com/content/philips/P020/media/Philips_WSJ-16_Technology-and-process-re-engineering-open-the-door-to-coordinated-care.pdf?origin=3_me_en_healthcare_media_campaign_digitalfire_eb_jul__kenya ) Subscribe to the Philips Health newsletter ( http://dfire.ensighthq.com/live/content.php?Item_ID=7911 ) Footer From owner-freebsd-net@freebsd.org Sun Aug 23 12:17:07 2015 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 6559C9BA4B8; Sun, 23 Aug 2015 12:17:07 +0000 (UTC) (envelope-from slw@zxy.spb.ru) Received: from zxy.spb.ru (zxy.spb.ru [195.70.199.98]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 190851A56; Sun, 23 Aug 2015 12:17:07 +0000 (UTC) (envelope-from slw@zxy.spb.ru) Received: from slw by zxy.spb.ru with local (Exim 4.84 (FreeBSD)) (envelope-from ) id 1ZTUCi-0005eT-Cl; Sun, 23 Aug 2015 15:16:52 +0300 Date: Sun, 23 Aug 2015 15:16:52 +0300 From: Slawa Olhovchenkov To: Daniel Braniss Cc: Rick Macklem , pyunyh@gmail.com, Hans Petter Selasky , FreeBSD stable , FreeBSD Net , Gleb Smirnoff , Christopher Forgeron Subject: Re: ix(intel) vs mlxen(mellanox) 10Gb performance Message-ID: <20150823121652.GJ3158@zxy.spb.ru> References: <1D52028A-B39F-4F9B-BD38-CB1D73BF5D56@cs.huji.ac.il> <55D429A4.3010407@selasky.org> <20150819074212.GB964@michelle.fasterthan.com> <55D43615.1030401@selasky.org> <2013503980.25726607.1439989235806.JavaMail.zimbra@uoguelph.ca> <20150820023024.GB996@michelle.fasterthan.com> <1153838447.28656490.1440193567940.JavaMail.zimbra@uoguelph.ca> <15D19823-08F7-4E55-BBD0-CE230F67D26E@cs.huji.ac.il> <818666007.28930310.1440244756872.JavaMail.zimbra@uoguelph.ca> <49173B1F-7B5E-4D59-8651-63D97B0CB5AC@cs.huji.ac.il> MIME-Version: 1.0 Content-Type: text/plain; charset=koi8-r Content-Disposition: inline In-Reply-To: <49173B1F-7B5E-4D59-8651-63D97B0CB5AC@cs.huji.ac.il> User-Agent: Mutt/1.5.23 (2014-03-12) X-SA-Exim-Connect-IP: X-SA-Exim-Mail-From: slw@zxy.spb.ru X-SA-Exim-Scanned: No (on zxy.spb.ru); SAEximRunCond expanded to false X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 23 Aug 2015 12:17:07 -0000 On Sun, Aug 23, 2015 at 02:08:56PM +0300, Daniel Braniss wrote: > >> send me the patch and I'll test it ASAP. > >> danny > >> > > Patch is attached. The one for head will also include an update to the comment > > in sys/net/if_var.h, but that isn't needed for testing. > > > well, the plot thickens. > > Yesterday, before running the new kernel, I decided to re run my test, and to my surprise > i was getting good numbers, about 300MGB/s with and without TSO. > > this morning, the numbers were again bad, around 70MGB/s,what the ^%$#@! > > so, after some coffee, I run some more tests, and some conclusions: > using a netapp(*) as the nfs client: > - doing > ifconfig ix0 tso or -tso > does some magic and numbers are back to normal - for a while > > using another Fbsd/zfs as client all is nifty, actually a bit faster than the netapp (not a fair > comparison, since the zfs client is not heavily used) and I can't see any degradation. > > btw, this is with the patch applied, but was seeing similar numbers before the patch. > > running with tso, initially I get around 300MGB/s, but after a while(sorry can't be more scientific) > it drops down to about half, and finally to a pathetic 70MGB/s > > *: while running the tests I monitored the Netapp, and nothing out of the ordinary there. Can you do this https://lists.freebsd.org/pipermail/freebsd-stable/2015-August/083138.html ? From owner-freebsd-net@freebsd.org Sun Aug 23 15:04:12 2015 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 34F1B9C0371 for ; Sun, 23 Aug 2015 15:04:12 +0000 (UTC) (envelope-from gpalmer@freebsd.org) Received: from mail.in-addr.com (mail.in-addr.com [IPv6:2a01:4f8:191:61e8::2525:2525]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id EE4E7A2E for ; Sun, 23 Aug 2015 15:04:11 +0000 (UTC) (envelope-from gpalmer@freebsd.org) Received: from gjp by mail.in-addr.com with local (Exim 4.86 (FreeBSD)) (envelope-from ) id 1ZTWoa-000Baa-Od for freebsd-net@freebsd.org; Sun, 23 Aug 2015 16:04:08 +0100 Date: Sun, 23 Aug 2015 16:04:08 +0100 From: Gary Palmer To: freebsd-net@freebsd.org Subject: Routing IPv6 over tun0 (PPPoE) issue Message-ID: <20150823150408.GE13503@in-addr.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline X-SA-Exim-Connect-IP: X-SA-Exim-Mail-From: gpalmer@freebsd.org X-SA-Exim-Scanned: No (on mail.in-addr.com); SAEximRunCond expanded to false X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 23 Aug 2015 15:04:12 -0000 Hi, I'm trying to set up IPv6 now that my ISP has decided to start offering native V6. I've been using a tunnelbroker setup until now. I have ipv6_gateway_enable="YES" ipv6_cpe_wanif="tun0" set in /etc/rc.conf and PPP has "enable ipv6cp" set. OS is FreeBSD 9.3-RELEASE-p21 When the system boots up I get tun0: flags=8051 metric 0 mtu 1492 options=80000 inet6 fe80::200:24ff:fec9:5bbc%tun0 prefixlen 64 scopeid 0xa inet 217.155.53.182 --> 62.3.83.6 netmask 0xffffffff inet6 xxxx:yyyy:zzzz:2:200:24ff:fec9:5bbc prefixlen 64 autoconf nd6 options=23 Opened by PID 1038 Routing is # netstat -nr -f inet6 Routing tables Internet6: Destination Gateway Flags Netif Expire ::/96 ::1 UGRS lo0 => default fe80::230:88ff:fe16:ec4f%tun0 UG tun0 ::1 link#9 UH lo0 traceroute6 www.freebsd.org works when the traffic is sourced from the tun0 interface IP # traceroute6 www.freebsd.org traceroute6 to wfe0.ysv.freebsd.org (2001:1900:2254:206a::50:0) from xxxx:yyyy:zzzz:2:200:24ff:fec9:5bbc, 64 hops max, 12 byte packets 1 xxxx:yyyy::3:0:0:2 29.030 ms 28.782 ms 29.205 ms 2 xxxx:yyyy:0:301:: 29.414 ms 28.967 ms 29.232 ms 3 xxxx:yyyy:0:4::1 28.750 ms 29.253 ms 82.200 ms 4 xxxx:yyyy:0:3::1 36.181 ms 35.352 ms 35.330 ms However if I configure other IPs on other interfaces from the netblock that has been delegated to me and either source the traffic from those IPs or try the traceroute from another computer using IPs in that netblock, I don't even see the traffic leaving tun0 with tcpdump, let alone get any replies. I do have PF running, but all my rules that stop traffic are logged and I don't see any hits in pflog. Also, I tried turning pf off once and didn't have any luck either, although I must admit I didn't leave it off long for obvious reasons, so maybe I missed something in my test. Any ideas? Is it because I am routing to a link local address rather than a routable IP? Unfortunately the returned packets from the first hop aren't in the subnet I was given for the link so I can't use that as a gateway. Thanks, Gary From owner-freebsd-net@freebsd.org Sun Aug 23 15:10:00 2015 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id CBBF79C040D; Sun, 23 Aug 2015 15:10:00 +0000 (UTC) (envelope-from kp@vega.codepro.be) Received: from venus.codepro.be (venus.codepro.be [IPv6:2a01:4f8:162:1127::2]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "*.codepro.be", Issuer "Gandi Standard SSL CA" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 6468FCEC; Sun, 23 Aug 2015 15:10:00 +0000 (UTC) (envelope-from kp@vega.codepro.be) Received: from vega.codepro.be (unknown [172.16.1.3]) by venus.codepro.be (Postfix) with ESMTP id CB50CE295; Sun, 23 Aug 2015 17:09:57 +0200 (CEST) Received: by vega.codepro.be (Postfix, from userid 1001) id B528B25B39; Sun, 23 Aug 2015 17:09:57 +0200 (CEST) Date: Sun, 23 Aug 2015 17:09:57 +0200 From: Kristof Provost To: freebsd-net@freebsd.org, freebsd-pf@freebsd.org Subject: Near-term pf plans Message-ID: <20150823150957.GK48727@vega.codepro.be> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline X-Checked-By-NSA: Probably User-Agent: Mutt/1.5.23 (2014-03-12) X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 23 Aug 2015 15:10:01 -0000 Hi, Some of you may have noticed that I fixed a couple of pf issues (or in some cases broke things. Sorry Allan.) recently. Here's a quick list of my current priorities: - PR 127042, 202178: This is a panic when an interface an ifgroup have the same name. There's a fix for the immediate problem in https://reviews.freebsd.org/D3435 I'm inclined to say that ifgroups and interfaces should share a namespace. That would certainly help pf, because it uses both interchangeably, which just doesn't work unless they're in the same namespace. That affects more in the network stack though, so I'd like opinions for people with more experience with ifgroups. - PR 202351 This is a panic after ip6 reassembly in pf. We set the rcvif to NULL when refragmenting. That seems to go OK execpt when we're refragmenting broadcast/multicast packets in the forwarding path. It's not at all clear to me how that could happen. - Removal of frag drop / frag crop support in pf. pf has two (or really three, but crop and drop mode are very similar) ways to handle fragmented packets. I've recently extended the 'reassemble' mode to also work for ip6. The crop/drop modes really only verify that fragments make sense (i.e. don't claim to deliver data beyond the last fragment, or duplicate data). I'd like to remove support for the crop/drop modes for two reasons: * the code is relatively large and complex. I've already run into a couple of problems related to it. * It doesn't do what users expect. Enabling scrub fragment crop doesn't actually mean pf will filter on the entire fragment. You still end up having to let all fragments pass through the firewall. There's a patch to do just that here: https://reviews.freebsd.org/D3466 - PR 198868, 193579 (TSO issues) pf has issues on certain network interfaces with TSO enabled. The most important of these are amazon VMs. I believe the problem is that pf tries to work with packets with full checksums. Usually output packets have pseudo-header checksums in the IP and TCP headers. pf doesn't know about those. It always tries to do updates on checksums assuming there's a full checksum. (Which is the case, pf always calculates a full checksum in pf_check_out()) I believe the fix for this issue is to have pf be aware of the pseudo-header checksums. The type of checksum can be determined from the CSUM_TCP and CSUM_TCP_IPV6 flags. Whenever pf changes a packet header it will have to check for those flags to figure out if the checksum should be updated or not. - PR 188511 divert-reply implementation for pf I need to review the use case and implementation of the work done by PiBa.NL.dev@gmail.com - PR 172648 This bug started out with an issue with TCP reassembly, but I've not been able to reproduce that. There is a problem with rdr targets for ipv6 though. With rdr to ::1 we fail the scope check in ip6_input() (right after the pfil hook) because we have a packet to localhost with a m->m_pkthdr.rcvif which is not a loopback interface. That can be fixed by having pf rewrite the rcvif, but that'd special-case rdr to ::1. We've got a similar problem for the reply. There we've got a packet from ::1 to something else. This fails the scope lookup too. In essence the problem is that we've already made the routing decision before pf gets the chance to rewrite the destination address. I'm not quite sure how to fix this though. If anyone has any comments or questions, or disagrees with any of the above, please let me know. If anyone knows of critical problems not on this list please let me know, and I'll see what I can do. Also, pf is currently an (unpaid) side project for me. I'm reasonably limited in the amount of time I can invest in it. Regards, Kristof From owner-freebsd-net@freebsd.org Sun Aug 23 15:38:06 2015 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 558239C0B5C for ; Sun, 23 Aug 2015 15:38:06 +0000 (UTC) (envelope-from matthew@FreeBSD.org) Received: from smtp.infracaninophile.co.uk (smtp.infracaninophile.co.uk [IPv6:2001:8b0:151:1:3cd3:cd67:fafa:3d78]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.infracaninophile.co.uk", Issuer "infracaninophile.co.uk" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id EBBCD6B6 for ; Sun, 23 Aug 2015 15:38:05 +0000 (UTC) (envelope-from matthew@FreeBSD.org) Received: from liminal.local ([192.168.100.2]) (authenticated bits=0) by smtp.infracaninophile.co.uk (8.15.2/8.15.2) with ESMTPSA id t7NFc14Z032650 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO) for ; Sun, 23 Aug 2015 16:38:01 +0100 (BST) (envelope-from matthew@FreeBSD.org) Authentication-Results: smtp.infracaninophile.co.uk; dmarc=none header.from=FreeBSD.org DKIM-Filter: OpenDKIM Filter v2.10.3 smtp.infracaninophile.co.uk t7NFc14Z032650 Authentication-Results: smtp.infracaninophile.co.uk/t7NFc14Z032650; dkim=none; dkim-atps=neutral X-Authentication-Warning: lucid-nonsense.infracaninophile.co.uk: Host [192.168.100.2] claimed to be liminal.local Subject: Re: Routing IPv6 over tun0 (PPPoE) issue To: freebsd-net@freebsd.org References: <20150823150408.GE13503@in-addr.com> From: Matthew Seaman X-Enigmail-Draft-Status: N1110 Message-ID: <55D9E8D4.1050700@FreeBSD.org> Date: Sun, 23 Aug 2015 16:37:56 +0100 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:38.0) Gecko/20100101 Thunderbird/38.2.0 MIME-Version: 1.0 In-Reply-To: <20150823150408.GE13503@in-addr.com> Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="L5mkpI2m91wKkft7VHfXbmJSnd9csOWrT" X-Virus-Scanned: clamav-milter 0.98.7 at lucid-nonsense.infracaninophile.co.uk X-Virus-Status: Clean X-Spam-Status: No, score=-2.9 required=5.0 tests=ALL_TRUSTED,BAYES_00 autolearn=ham autolearn_force=no version=3.4.1 X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on lucid-nonsense.infracaninophile.co.uk X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 23 Aug 2015 15:38:06 -0000 This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --L5mkpI2m91wKkft7VHfXbmJSnd9csOWrT Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: quoted-printable On 23/08/2015 16:04, Gary Palmer wrote: > However if I configure other IPs on other interfaces from the netblock = that > has been delegated to me and either source the traffic from those IPs o= r > try the traceroute from another computer using IPs in that netblock, I > don't even see the traffic leaving tun0 with tcpdump, let alone get any= > replies. I have a similar setup. Looks to me as if there's a problem with your routing internally. My routing table looks like this (excluding the ff01::, ff02:: and ff03:: routes and anything that's a host specific route): % netstat -rn -f inet6 | grep -vE '(UH|ff0)' Routing tables Internet6: Destination Gateway Flags Netif Expire ::/96 ::1 UGRS lo0 default fe80::203:97ff:fe19:8000%tun0 UGS tun0 ::ffff:0.0.0.0/96 ::1 UGRS lo0 2001:8b0:151:1::/64 link#1 U em0 <<<---** fe80::/10 ::1 UGRS lo0 fe80::%em0/64 link#1 U em0 fe80::%re0/64 link#2 U re0 fe80::%lo0/64 link#3 U lo0 fe80::%tun0/64 link#5 U tun0 Here em0 is the interface onto my internal network, and any addresses from my assigned IPv6 netblock are configured on that interface or the network directly attached to it. You should have a route equivalent to the one marked with the arrow. Note that tun0 uses link-local addresses for the IPv6 tunnelling, not addresses from my assigned range. Depending on how your ISP has configured things you may need a "real" IPv6 address on your tun0 interface, but this should be from a distinct subnet to the block you're using internally. Hmmm.... you do have 'gateway_enable=3D"YES"' and 'ipv6_gateway_enable=3D"YES"' in your /etc/rc.conf ? Cheers, Matthew --L5mkpI2m91wKkft7VHfXbmJSnd9csOWrT Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG/MacGPG2 v2 iQJ8BAEBCgBmBQJV2ejUXxSAAAAAAC4AKGlzc3Vlci1mcHJAbm90YXRpb25zLm9w ZW5wZ3AuZmlmdGhob3JzZW1hbi5uZXQ2NTNBNjhCOTEzQTRFNkNGM0UxRTEzMjZC QjIzQUY1MThFMUE0MDEzAAoJELsjr1GOGkAT9aQQAJAL3zDc8KXv40CbuDb6AxtH Al7/6HwLxVbshiWJr1q0HbsDHW9PHsfxAL2J8ER9qVdZmc5dm/EG80GKMNb8v+IG ecilWPN+fP4H/FurC/Nxsz1ihSo+Zo8Hf9zn8GHOuJrnP1s9lx7NSixhDt/2/8Gg T3W3JJhrJXovYfD4+cs3DVlEOT8xnDsHZRt4rdOXWpK6IXJF86HxINDORnx01AcM yyyEwcsNOrfog3+hA+6QHGELe9oqaPEJeTl0ZEWsq9CKkj5HQNzsnd1KcnaJjfku UpSR9G7QPOv0e8htRgtXHzsr0oyRaYCkhmwmrC5n7oe0UTrWMxqLdMmdLiHyVzsQ i9dplUDvv5xAcqPIeccVQfS08aOELDji8ldt9zOgiT0jE/omUg247RJ9N9w6ODHb uVZgq1IrZnwfKbXtsYnrtMoMKvvO8yZzy25sAEbqRjpBrzjGd5554qbudrrr2n9U GlfGAmuEnHsEP6WU+50wr7f3YAY+1/+8aLvAaI04eTFN0dOPsy8Fom4iacE96QEn Mlgyi2laxnC0F2nv7tIBxDf0bs3zC74KyuZHsFCTmmxSLl8Sn7rnmKikznjLitxp x5nhXV8fEikVkPW7I+BJjYO0FKrkjY0NQZHWOkKs4gaTEnKzIxaAJMjtHwszMN7/ EBI6nmYzxdXdJBp+VgIB =gsIB -----END PGP SIGNATURE----- --L5mkpI2m91wKkft7VHfXbmJSnd9csOWrT-- From owner-freebsd-net@freebsd.org Sun Aug 23 16:48:30 2015 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id D4D7C9C094B for ; Sun, 23 Aug 2015 16:48:30 +0000 (UTC) (envelope-from gpalmer@freebsd.org) Received: from mail.in-addr.com (mail.in-addr.com [IPv6:2a01:4f8:191:61e8::2525:2525]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 93A3512D3; Sun, 23 Aug 2015 16:48:30 +0000 (UTC) (envelope-from gpalmer@freebsd.org) Received: from gjp by mail.in-addr.com with local (Exim 4.86 (FreeBSD)) (envelope-from ) id 1ZTYRY-000Bnq-M0; Sun, 23 Aug 2015 17:48:28 +0100 Date: Sun, 23 Aug 2015 17:48:28 +0100 From: Gary Palmer To: Matthew Seaman Cc: freebsd-net@freebsd.org Subject: Re: Routing IPv6 over tun0 (PPPoE) issue Message-ID: <20150823164828.GF13503@in-addr.com> References: <20150823150408.GE13503@in-addr.com> <55D9E8D4.1050700@FreeBSD.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <55D9E8D4.1050700@FreeBSD.org> X-SA-Exim-Connect-IP: X-SA-Exim-Mail-From: gpalmer@freebsd.org X-SA-Exim-Scanned: No (on mail.in-addr.com); SAEximRunCond expanded to false X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 23 Aug 2015 16:48:30 -0000 On Sun, Aug 23, 2015 at 04:37:56PM +0100, Matthew Seaman wrote: > On 23/08/2015 16:04, Gary Palmer wrote: > > However if I configure other IPs on other interfaces from the netblock that > > has been delegated to me and either source the traffic from those IPs or > > try the traceroute from another computer using IPs in that netblock, I > > don't even see the traffic leaving tun0 with tcpdump, let alone get any > > replies. > > I have a similar setup. Looks to me as if there's a problem with your > routing internally. > > My routing table looks like this (excluding the ff01::, ff02:: and > ff03:: routes and anything that's a host specific route): > > % netstat -rn -f inet6 | grep -vE '(UH|ff0)' > Routing tables > > Internet6: > Destination Gateway Flags Netif Expire > ::/96 ::1 UGRS lo0 > default fe80::203:97ff:fe19:8000%tun0 UGS tun0 > ::ffff:0.0.0.0/96 ::1 UGRS lo0 > 2001:8b0:151:1::/64 link#1 U em0 <<<---** > fe80::/10 ::1 UGRS lo0 > fe80::%em0/64 link#1 U em0 > fe80::%re0/64 link#2 U re0 > fe80::%lo0/64 link#3 U lo0 > fe80::%tun0/64 link#5 U tun0 > > Here em0 is the interface onto my internal network, and any addresses > from my assigned IPv6 netblock are configured on that interface or the > network directly attached to it. You should have a route equivalent to > the one marked with the arrow. > > Note that tun0 uses link-local addresses for the IPv6 tunnelling, not > addresses from my assigned range. Depending on how your ISP has > configured things you may need a "real" IPv6 address on your tun0 > interface, but this should be from a distinct subnet to the block you're > using internally. Hi Matthew, Thanks for the reply. I may have messed up manually masking the network data so let me do it by sed this time so I don't mess up. aaaa:bbbb:cccc:dddd is the /64 prefix used for the connection xxxx:yyyy:zzzz is the /48 used for internal IPs The tunnelbroker IPs are also configured but I've removed them as they shouldn't be relevant. I've checked gif0 and none of the traffic is going out that interface either. tun0 shows: tun0: flags=8051 metric 0 mtu 1492 options=80000 inet6 fe80::200:24ff:fec9:5bbc%tun0 prefixlen 64 scopeid 0xa inet a.b.c.d --> e.f.g.h netmask 0xffffffff inet6 aaaa:bbbb:cccc:dddd:200:24ff:fec9:5bbc prefixlen 64 autoconf nd6 options=23 Opened by PID 1038 vr0 shows: vr0: flags=8843 metric 0 mtu 1500 options=8280b ether 00:00:24:c9:5b:bc inet i.j.k.l netmask 0xffffff00 broadcast i.j.k.m inet6 fe80::200:24ff:fec9:5bbc%vr0 prefixlen 64 scopeid 0x1 inet6 xxxx:yyyy:zzzz:1::1 prefixlen 64 nd6 options=21 media: Ethernet autoselect (100baseTX ) status: active IPv6 routing table: Routing tables Internet6: Destination Gateway Flags Netif Expire ::/96 ::1 UGRS lo0 => default fe80::230:88ff:fe16:ec4f%tun0 UG tun0 ::1 link#9 UH lo0 ::ffff:0.0.0.0/96 ::1 UGRS lo0 xxxx:yyyy:zzzz:1::/64 link#1 U vr0 xxxx:yyyy:zzzz:1::1 link#1 UHS lo0 xxxx:yyyy:zzzz:2::/64 link#3 U vr2 xxxx:yyyy:zzzz:2::1 link#3 UHS lo0 aaaa:bbbb:cccc:dddd::/64 link#10 U tun0 aaaa:bbbb:cccc:dddd:200:24ff:fec9:5bbc link#10 UHS lo0 traceroute from tun0 IP (first 4 hops only shown) traceroute6 to wfe0.ysv.freebsd.org (2001:1900:2254:206a::50:0) from aaaa:bbbb:cccc:dddd:200:24ff:fec9:5bbc, 4 hops max, 12 byte packets 1 aaaa:bbbb::3:0:0:2 29.318 ms 29.860 ms 28.065 ms 2 aaaa:bbbb:0:301:: 28.724 ms 29.064 ms 29.421 ms 3 aaaa:bbbb:0:4::1 29.881 ms 29.189 ms 28.254 ms 4 aaaa:bbbb:0:3::1 35.764 ms 36.488 ms 36.054 ms traceroute from vr0 IP using 'traceroute6 -s' traceroute6 to wfe0.ysv.freebsd.org (2001:1900:2254:206a::50:0) from xxxx:yyyy:zzzz:1::1, 4 hops max, 12 byte packets 1 * * * 2 * * * > Hmmm.... you do have 'gateway_enable="YES"' and > 'ipv6_gateway_enable="YES"' in your /etc/rc.conf ? gateway_enable="YES" ipv6_gateway_enable="YES" Yes. v4 continues to work fine. Thanks, Gary From owner-freebsd-net@freebsd.org Sun Aug 23 16:53:58 2015 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 150C19C0AB0 for ; Sun, 23 Aug 2015 16:53:58 +0000 (UTC) (envelope-from daemon-user@freebsd.org) Received: from phabric-backend.isc.freebsd.org (phabric-backend.isc.freebsd.org [IPv6:2001:4f8:3:ffe0:406a:0:50:2]) by mx1.freebsd.org (Postfix) with ESMTP id F036D172D for ; Sun, 23 Aug 2015 16:53:57 +0000 (UTC) (envelope-from daemon-user@freebsd.org) Received: by phabric-backend.isc.freebsd.org (Postfix, from userid 1346) id EA27618324; Sun, 23 Aug 2015 16:53:57 +0000 (UTC) Date: Sun, 23 Aug 2015 16:53:57 +0000 To: freebsd-net@freebsd.org From: "javier_ovi_yahoo.com (Javier Villavicencio)" Reply-to: D1944+325+8925873bdc96dfc2@reviews.freebsd.org Subject: [Differential] [Changed Subscribers] D1944: PF and VIMAGE fixes Message-ID: <7a0581aed4635bdf9d3bbfb69b215679@localhost.localdomain> X-Priority: 3 X-Phabricator-Sent-This-Message: Yes X-Mail-Transport-Agent: MetaMTA X-Auto-Response-Suppress: All X-Phabricator-Mail-Tags: Thread-Topic: D1944: PF and VIMAGE fixes X-Herald-Rules: none X-Phabricator-To: X-Phabricator-To: X-Phabricator-To: X-Phabricator-To: X-Phabricator-To: X-Phabricator-To: X-Phabricator-To: X-Phabricator-To: X-Phabricator-To: X-Phabricator-Cc: X-Phabricator-Cc: X-Phabricator-Cc: X-Phabricator-Cc: X-Phabricator-Cc: X-Phabricator-Cc: X-Phabricator-Cc: Precedence: bulk In-Reply-To: References: Thread-Index: NDc2NzM0MzY4OTdiYThiNTU1MjY2ZDZmMTJiIFXZ+qU= MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Content-Type: text/plain; charset="utf-8" X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 23 Aug 2015 16:53:58 -0000 javier_ovi_yahoo.com added a subscriber: javier_ovi_yahoo.com. REVISION DETAIL https://reviews.freebsd.org/D1944 EMAIL PREFERENCES https://reviews.freebsd.org/settings/panel/emailpreferences/ To: nvass-gmx.com, bz, trociny, kristof, gnn, zec, rodrigc, glebius, eri Cc: javier_ovi_yahoo.com, farrokhi, julian, robak, freebsd-virtualization-list, freebsd-pf-list, freebsd-net-list From owner-freebsd-net@freebsd.org Sun Aug 23 21:00:34 2015 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id C7BA59C0945 for ; Sun, 23 Aug 2015 21:00:34 +0000 (UTC) (envelope-from bugzilla-noreply@FreeBSD.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id A2FA81E43 for ; Sun, 23 Aug 2015 21:00:34 +0000 (UTC) (envelope-from bugzilla-noreply@FreeBSD.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id t7NL0Y1E096553 for ; Sun, 23 Aug 2015 21:00:34 GMT (envelope-from bugzilla-noreply@FreeBSD.org) Message-Id: <201508232100.t7NL0Y1E096553@kenobi.freebsd.org> From: bugzilla-noreply@FreeBSD.org To: freebsd-net@FreeBSD.org Subject: Problem reports for freebsd-net@FreeBSD.org that need special attention X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 Date: Sun, 23 Aug 2015 21:00:34 +0000 Content-Type: text/plain; charset="UTF-8" X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 23 Aug 2015 21:00:34 -0000 To view an individual PR, use: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=(Bug Id). The following is a listing of current problems submitted by FreeBSD users, which need special attention. These represent problem reports covering all versions including experimental development code and obsolete releases. Status | Bug Id | Description ------------+-----------+--------------------------------------------------- Open | 194515 | Fatal Trap 12 Kernel with vimage Open | 199136 | [if_tap] Added down_on_close sysctl variable to t 2 problems total for which you should take action. From owner-freebsd-net@freebsd.org Sun Aug 23 23:03:06 2015 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 34C119C1011; Sun, 23 Aug 2015 23:03:06 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from esa-jnhn.mail.uoguelph.ca (esa-jnhn.mail.uoguelph.ca [131.104.91.44]) by mx1.freebsd.org (Postfix) with ESMTP id E714C1B8F; Sun, 23 Aug 2015 23:03:04 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) IronPort-PHdr: 9a23:8U61ah+7CLjffP9uRHKM819IXTAuvvDOBiVQ1KB91O0cTK2v8tzYMVDF4r011RmSDd6dtq4P0rCempujcFJDyK7JiGoFfp1IWk1NouQttCtkPvS4D1bmJuXhdS0wEZcKflZk+3amLRodQ56mNBXsq3G/pQQfBg/4fVIsYL+lQciP04/ujaibwN76XUZhvHKFe7R8LRG7/036l/I9ps9cEJs30QbDuXBSeu5blitCLFOXmAvgtI/rpMYwuwwZgf8q9tZBXKPmZOx4COUAVHV1e1wyseHisxjOSUOl/HIaU34N2k5ECg7D/TnxRdHxryn78ON2niiea57YV7cxDA6j5KQjbRbjiyMKMnZt6mTegc90gadzvRWuuhF7246Sa4jDZ6k2Rb/UYd5PHTkJZc1WTSEUR9rkN4Y= X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: A2BkBADDUNpV/61jaINdDoNhaQaDH7pEgW0KhTFKAoFcEgEBAQEBAQEBgQmCHYIGAQEBAwEBAQEgBCcgCwULAgEIDgoCAg0ZAgInAQkmAgQIBwQBGgIEiAUIDbA/lS4BAQEBAQEEAQEBAQEBGASBIoo1hDIGAQEFFwEuBQeCaYFDBZU0hQaFCoQth0yJBoRJg2gCJoIOHIEWWiIzB38IFyOBBAEBAQ X-IronPort-AV: E=Sophos;i="5.15,734,1432612800"; d="scan'208";a="232638254" Received: from nipigon.cs.uoguelph.ca (HELO zcs1.mail.uoguelph.ca) ([131.104.99.173]) by esa-jnhn.mail.uoguelph.ca with ESMTP; 23 Aug 2015 19:02:54 -0400 Received: from localhost (localhost [127.0.0.1]) by zcs1.mail.uoguelph.ca (Postfix) with ESMTP id 7467B15F55D; Sun, 23 Aug 2015 19:02:54 -0400 (EDT) Received: from zcs1.mail.uoguelph.ca ([127.0.0.1]) by localhost (zcs1.mail.uoguelph.ca [127.0.0.1]) (amavisd-new, port 10032) with ESMTP id TqB1uw5-B9Uv; Sun, 23 Aug 2015 19:02:53 -0400 (EDT) Received: from localhost (localhost [127.0.0.1]) by zcs1.mail.uoguelph.ca (Postfix) with ESMTP id 31E5915F563; Sun, 23 Aug 2015 19:02:53 -0400 (EDT) X-Virus-Scanned: amavisd-new at zcs1.mail.uoguelph.ca Received: from zcs1.mail.uoguelph.ca ([127.0.0.1]) by localhost (zcs1.mail.uoguelph.ca [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id OMuYpwye10f5; Sun, 23 Aug 2015 19:02:53 -0400 (EDT) Received: from zcs1.mail.uoguelph.ca (zcs1.mail.uoguelph.ca [172.17.95.18]) by zcs1.mail.uoguelph.ca (Postfix) with ESMTP id 0A4D415F55D; Sun, 23 Aug 2015 19:02:53 -0400 (EDT) Date: Sun, 23 Aug 2015 19:02:53 -0400 (EDT) From: Rick Macklem To: Daniel Braniss Cc: pyunyh@gmail.com, Hans Petter Selasky , FreeBSD stable , FreeBSD Net , Slawa Olhovchenkov , Gleb Smirnoff , Christopher Forgeron Message-ID: <1815942485.29539597.1440370972998.JavaMail.zimbra@uoguelph.ca> In-Reply-To: <49173B1F-7B5E-4D59-8651-63D97B0CB5AC@cs.huji.ac.il> References: <1D52028A-B39F-4F9B-BD38-CB1D73BF5D56@cs.huji.ac.il> <55D43615.1030401@selasky.org> <2013503980.25726607.1439989235806.JavaMail.zimbra@uoguelph.ca> <20150820023024.GB996@michelle.fasterthan.com> <1153838447.28656490.1440193567940.JavaMail.zimbra@uoguelph.ca> <15D19823-08F7-4E55-BBD0-CE230F67D26E@cs.huji.ac.il> <818666007.28930310.1440244756872.JavaMail.zimbra@uoguelph.ca> <49173B1F-7B5E-4D59-8651-63D97B0CB5AC@cs.huji.ac.il> Subject: Re: ix(intel) vs mlxen(mellanox) 10Gb performance MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Originating-IP: [172.17.95.10] X-Mailer: Zimbra 8.0.9_GA_6191 (ZimbraWebClient - FF34 (Win)/8.0.9_GA_6191) Thread-Topic: ix(intel) vs mlxen(mellanox) 10Gb performance Thread-Index: WDLDHT1fctGcdpZQVUJYgbbvlrrT4Q== X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 23 Aug 2015 23:03:06 -0000 Daniel Braniss wrote: >=20 > > On 22 Aug 2015, at 14:59, Rick Macklem wrote: > >=20 > > Daniel Braniss wrote: > >>=20 > >>> On Aug 22, 2015, at 12:46 AM, Rick Macklem wro= te: > >>>=20 > >>> Yonghyeon PYUN wrote: > >>>> On Wed, Aug 19, 2015 at 09:00:35AM -0400, Rick Macklem wrote: > >>>>> Hans Petter Selasky wrote: > >>>>>> On 08/19/15 09:42, Yonghyeon PYUN wrote: > >>>>>>> On Wed, Aug 19, 2015 at 09:00:52AM +0200, Hans Petter Selasky wro= te: > >>>>>>>> On 08/18/15 23:54, Rick Macklem wrote: > >>>>>>>>> Ouch! Yes, I now see that the code that counts the # of mbufs i= s > >>>>>>>>> before > >>>>>>>>> the > >>>>>>>>> code that adds the tcp/ip header mbuf. > >>>>>>>>>=20 > >>>>>>>>> In my opinion, this should be fixed by setting if_hw_tsomaxsegc= ount > >>>>>>>>> to > >>>>>>>>> whatever > >>>>>>>>> the driver provides - 1. It is not the driver's responsibility = to > >>>>>>>>> know if > >>>>>>>>> a tcp/ip > >>>>>>>>> header mbuf will be added and is a lot less confusing that > >>>>>>>>> expecting > >>>>>>>>> the > >>>>>>>>> driver > >>>>>>>>> author to know to subtract one. (I had mistakenly thought that > >>>>>>>>> tcp_output() had > >>>>>>>>> added the tc/ip header mbuf before the loop that counts mbufs i= n > >>>>>>>>> the > >>>>>>>>> list. > >>>>>>>>> Btw, > >>>>>>>>> this tcp/ip header mbuf also has leading space for the MAC laye= r > >>>>>>>>> header.) > >>>>>>>>>=20 > >>>>>>>>=20 > >>>>>>>> Hi Rick, > >>>>>>>>=20 > >>>>>>>> Your question is good. With the Mellanox hardware we have separa= te > >>>>>>>> so-called inline data space for the TCP/IP headers, so if the TC= P > >>>>>>>> stack > >>>>>>>> subtracts something, then we would need to add something to the > >>>>>>>> limit, > >>>>>>>> because then the scatter gather list is only used for the data p= art. > >>>>>>>>=20 > >>>>>>>=20 > >>>>>>> I think all drivers in tree don't subtract 1 for > >>>>>>> if_hw_tsomaxsegcount. Probably touching Mellanox driver would be > >>>>>>> simpler than fixing all other drivers in tree. > >>>>>>>=20 > >>>>>>>> Maybe it can be controlled by some kind of flag, if all the thre= e > >>>>>>>> TSO > >>>>>>>> limits should include the TCP/IP/ethernet headers too. I'm prett= y > >>>>>>>> sure > >>>>>>>> we want both versions. > >>>>>>>>=20 > >>>>>>>=20 > >>>>>>> Hmm, I'm afraid it's already complex. Drivers have to tell almos= t > >>>>>>> the same information to both bus_dma(9) and network stack. > >>>>>>=20 > >>>>>> Don't forget that not all drivers in the tree set the TSO limits > >>>>>> before > >>>>>> if_attach(), so possibly the subtraction of one TSO fragment needs= to > >>>>>> go > >>>>>> into ip_output() .... > >>>>>>=20 > >>>>> Ok, I realized that some drivers may not know the answers before > >>>>> ether_ifattach(), > >>>>> due to the way they are configured/written (I saw the use of > >>>>> if_hw_tsomax_update() > >>>>> in the patch). > >>>>=20 > >>>> I was not able to find an interface that configures TSO parameters > >>>> after if_t conversion. I'm under the impression > >>>> if_hw_tsomax_update() is not designed to use this way. Probably we > >>>> need a better one?(CCed to Gleb). > >>>>=20 > >>>>>=20 > >>>>> If it is subtracted as a part of the assignment to if_hw_tsomaxsegc= ount > >>>>> in > >>>>> tcp_output() > >>>>> at line#791 in tcp_output() like the following, I don't think it sh= ould > >>>>> matter if the > >>>>> values are set before ether_ifattach()? > >>>>> =09=09=09/* > >>>>> =09=09=09 * Subtract 1 for the tcp/ip header mbuf that > >>>>> =09=09=09 * will be prepended to the mbuf chain in this > >>>>> =09=09=09 * function in the code below this block. > >>>>> =09=09=09 */ > >>>>> =09=09=09if_hw_tsomaxsegcount =3D tp->t_tsomaxsegcount - 1; > >>>>>=20 > >>>>> I don't have a good solution for the case where a driver doesn't pl= an > >>>>> on > >>>>> using the > >>>>> tcp/ip header provided by tcp_output() except to say the driver can= add > >>>>> one > >>>>> to the > >>>>> setting to compensate for that (and if they fail to do so, it still > >>>>> works, > >>>>> although > >>>>> somewhat suboptimally). When I now read the comment in sys/net/if_v= ar.h > >>>>> it > >>>>> is clear > >>>>> what it means, but for some reason I didn't read it that way before= ? (I > >>>>> think it was > >>>>> the part that said the driver didn't have to subtract for the heade= rs > >>>>> that > >>>>> confused me?) > >>>>> In any case, we need to try and come up with a clear definition of = what > >>>>> they need to > >>>>> be set to. > >>>>>=20 > >>>>> I can now think of two ways to deal with this: > >>>>> 1 - Leave tcp_output() as is, but provide a macro for the device dr= iver > >>>>> authors to use > >>>>> that sets if_hw_tsomaxsegcount with a flag for "driver uses tcp/i= p > >>>>> header mbuf", > >>>>> documenting that this flag should normally be true. > >>>>> OR > >>>>> 2 - Change tcp_output() as above, noting that this is a workaround = for > >>>>> confusion w.r.t. > >>>>> whether or not if_hw_tsomaxsegcount should include the tcp/ip hea= der > >>>>> mbuf and > >>>>> update the comment in if_var.h to reflect this. Then drivers that > >>>>> don't > >>>>> use the > >>>>> tcp/ip header mbuf can increase their value for if_hw_tsomaxsegco= unt > >>>>> by > >>>>> 1. > >>>>> (The comment should also mention that a value of 35 or greater is > >>>>> much > >>>>> preferred to > >>>>> 32 if the hardware will support that.) > >>>>>=20 > >>>>=20 > >>>> Both works for me. My preference is 2 just because it's very > >>>> common for most drivers that use tcp/ip header mbuf. > >>> Thanks for this comment. I tend to agree, both for the reason you sta= te > >>> and > >>> also > >>> because the patch is simple enough that it might qualify as an errata= for > >>> 10.2. > >>>=20 > >>> I am hoping Daniel Braniss will be able to test the patch and let us = know > >>> if it > >>> improves performance with TSO enabled? > >>=20 > >> send me the patch and I=E2=80=99ll test it ASAP. > >> =09danny > >>=20 > > Patch is attached. The one for head will also include an update to the > > comment > > in sys/net/if_var.h, but that isn't needed for testing. >=20 >=20 > well, the plot thickens. >=20 > Yesterday, before running the new kernel, I decided to re run my test, an= d to > my surprise > i was getting good numbers, about 300MGB/s with and without TSO. >=20 > this morning, the numbers were again bad, around 70MGB/s,what the ^%$#@! >=20 > so, after some coffee, I run some more tests, and some conclusions: > using a netapp(*) as the nfs client: > - doing > =09ifconfig ix0 tso or -tso > does some magic and numbers are back to normal - for a while >=20 > using another Fbsd/zfs as client all is nifty, actually a bit faster than= the > netapp (not a fair > comparison, since the zfs client is not heavily used) and I can=E2=80=99t= see any > degradation. > =20 I assume you meant "server" and not "client" above. > btw, this is with the patch applied, but was seeing similar numbers befor= e > the patch. >=20 > running with tso, initially I get around 300MGB/s, but after a while(sorr= y > can=E2=80=99t be more scientific) > it drops down to about half, and finally to a pathetic 70MGB/s >=20 Ok, so it sounds like tso isn't the issue. (At least it seems the patch, which I believe is needed, doesn't cause a regression.) All I can suggest is: - looking at the ix stats (I know nothing about them), but if you post them maybe someone conversant with the chip can help? (Before and after degred= ation.) - if you captured packets for a short period of time when degraded and then after doing "ifconfig", looking at the packet capture in wireshark might = give some indication of what changes? - For this I'd be focused on the TCP layer (window sizes, etc) and timing= of packets. --> I don't know if there is a packet capture tool like tcpdump on a Netapp= , but that might be better than capturing them on the client, in case tcpdump= affects the outcome. However, tcpdump run on the client would be a fallback, I = think. The other thing is the degradation seems to cut the rate by about half each= time. 300-->150-->70 I have no idea if this helps to explain it. Have fun with it, rick > *: while running the tests I monitored the Netapp, and nothing out of the > ordinary there. >=20 > cheers, > =09danny >=20 > _______________________________________________ > freebsd-stable@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-stable > To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org" From owner-freebsd-net@freebsd.org Mon Aug 24 01:01:29 2015 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id CEDC39C1606 for ; Mon, 24 Aug 2015 01:01:29 +0000 (UTC) (envelope-from truckman@FreeBSD.org) Received: from gw.catspoiler.org (unknown [IPv6:2602:304:b010:ef20::f2]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "gw.catspoiler.org", Issuer "gw.catspoiler.org" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 9DE921AE6; Mon, 24 Aug 2015 01:01:29 +0000 (UTC) (envelope-from truckman@FreeBSD.org) Received: from FreeBSD.org (mousie.catspoiler.org [192.168.101.2]) by gw.catspoiler.org (8.15.2/8.15.2) with ESMTP id t7O11Kgu002655; Sun, 23 Aug 2015 18:01:25 -0700 (PDT) (envelope-from truckman@FreeBSD.org) Message-Id: <201508240101.t7O11Kgu002655@gw.catspoiler.org> Date: Sun, 23 Aug 2015 18:01:20 -0700 (PDT) From: Don Lewis Subject: Re: a couple /etc/rc.firewall questions To: smithi@nimnet.asn.au cc: hrs@freebsd.org, freebsd-net@freebsd.org In-Reply-To: <20150823151421.G8515@sola.nimnet.asn.au> MIME-Version: 1.0 Content-Type: TEXT/plain; charset=us-ascii X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 24 Aug 2015 01:01:30 -0000 On 23 Aug, Ian Smith wrote: > On Sun, 23 Aug 2015 08:44:53 +0900, Hiroki Sato wrote: > > Don Lewis wrote > > in <201508222103.t7ML3gAx000794@gw.catspoiler.org>: > > > > tr> The example /etc/rc.firewall has provisions to use either in-kernel NAT > > tr> or natd for the open and client firewall types, but the simple filewall > > tr> type only has code for natd. Is there any reason that in-kernel NAT > > tr> could not be used with the simple firewall type? > > > > I think there is no particular reason. Simple rule was just not updated. > > I did send you and -ipfw@ a patch for that on several occasions since > Feb. 2013, though I did fail to push it into the 3-4 PRs it affected. > > The attached patch addresses that, chooses kernel NAT over natd(8) if > both were enabled in rc.conf, updates some commentary and fixes an > overwordy line in 'workstation'. Just now checked it against HEAD. > > > tr> After allowing connections to selected TCP ports and then denying all > > tr> other incoming TCP setup connections from ${oif}, the simple firewall > > tr> code in /etc/rc.firewall then permits all other TCP setup connections: > > tr> # Allow setup of any other TCP connection > > tr> ${fwcmd} add pass tcp from any to any setup > > tr> This is potentially undesirable since it allows unrestricted TCP > > tr> connections between "me" and the inside network. When I changed this to > > tr> ${fwcmd} add pass tcp from any to any out via ${oif} setup > > tr> I was able to open TCP connections from the firewall box to the outside, > > tr> but NATed connections from inside network to the outside were blocked. > > tr> If I run "ipfw show", it appears that the TCP setup packets are falling > > tr> through to the final implicit deny all rule, but I don't see any obvious > > tr> reason. > > > > A TCP setup packet coming from a host on the internal LAN to the NAPT > > router falls into the last deny-all rule because it does not match if > > you added "out via ${oif}" to that rule. Does the following > > additional rule work for you? > > > > ${fwcmd} add pass tcp from any to any out via ${oif} setup > > That looks ok, maybe some UDP too? Adding some stateful rules is > another option for dealing with inside hosts' external requests. I don't have a specific need for UDP between inside and outside, so I didn't bother with that. One end all my UDP connections is currently always the firewall box itself. I did just add stateful rules for TCPv6 between the inside and outside to replicate the stateful behaviour of TCPv4 NAT. > > ${fwcmd} add pass tcp from any to not me in via ${iif} setup > > If you want to deny inside hosts access to host services, that'll do it. > > The other long-term issue with 'simple' is that it permits no ICMPv4 at > all. Neither inside nor outside, no pings, no PMTU, nothing .. although > curiously allows selected ICMP for ipv6. I usually add something like: > > ${fwcmd} add pass icmp from any to any icmptype 0,3,8,11 > > If you don't want to allow pings from outside your net, preceded with: > > ${fwcmd} add deny icmp from any to any in recv ${oif} icmptype 8 Yeah, I alway end up adding ICMPv4 rules. From owner-freebsd-net@freebsd.org Mon Aug 24 01:06:12 2015 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 421C79C166E for ; Mon, 24 Aug 2015 01:06:12 +0000 (UTC) (envelope-from truckman@FreeBSD.org) Received: from gw.catspoiler.org (unknown [IPv6:2602:304:b010:ef20::f2]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "gw.catspoiler.org", Issuer "gw.catspoiler.org" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 08E0A1CEC; Mon, 24 Aug 2015 01:06:12 +0000 (UTC) (envelope-from truckman@FreeBSD.org) Received: from FreeBSD.org (mousie.catspoiler.org [192.168.101.2]) by gw.catspoiler.org (8.15.2/8.15.2) with ESMTP id t7O0qsFF002623; Sun, 23 Aug 2015 17:52:58 -0700 (PDT) (envelope-from truckman@FreeBSD.org) Message-Id: <201508240052.t7O0qsFF002623@gw.catspoiler.org> Date: Sun, 23 Aug 2015 17:52:54 -0700 (PDT) From: Don Lewis Subject: Re: a couple /etc/rc.firewall questions To: hrs@FreeBSD.org cc: freebsd-net@FreeBSD.org In-Reply-To: <20150823.084453.1715908115913144015.hrs@allbsd.org> MIME-Version: 1.0 Content-Type: TEXT/plain; charset=us-ascii X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 24 Aug 2015 01:06:12 -0000 On 23 Aug, Hiroki Sato wrote: > Don Lewis wrote > in <201508222103.t7ML3gAx000794@gw.catspoiler.org>: > > tr> The example /etc/rc.firewall has provisions to use either in-kernel NAT > tr> or natd for the open and client firewall types, but the simple filewall > tr> type only has code for natd. Is there any reason that in-kernel NAT > tr> could not be used with the simple firewall type? > > I think there is no particular reason. Simple rule was just not updated. Yeah, it seems to work if I add the rule for it in the appropriate place. > tr> After allowing connections to selected TCP ports and then denying all > tr> other incoming TCP setup connections from ${oif}, the simple firewall > tr> code in /etc/rc.firewall then permits all other TCP setup connections: > tr> # Allow setup of any other TCP connection > tr> ${fwcmd} add pass tcp from any to any setup > tr> This is potentially undesirable since it allows unrestricted TCP > tr> connections between "me" and the inside network. When I changed this to > tr> ${fwcmd} add pass tcp from any to any out via ${oif} setup > tr> I was able to open TCP connections from the firewall box to the outside, > tr> but NATed connections from inside network to the outside were blocked. > tr> If I run "ipfw show", it appears that the TCP setup packets are falling > tr> through to the final implicit deny all rule, but I don't see any obvious > tr> reason. > > A TCP setup packet coming from a host on the internal LAN to the NAPT > router falls into the last deny-all rule because it does not match if > you added "out via ${oif}" to that rule. Does the following > additional rule work for you? > > ${fwcmd} add pass tcp from any to any out via ${oif} setup > ${fwcmd} add pass tcp from any to not me in via ${iif} setup That works for now, but won't do the correct thing when I subdivide my internal network because it will allow unrestricted connections between the internal subnets. What I'd really like is something like: ${fwcmd} add pass tcp from any to not me,${inet} setup but that isn't a valid rule. I ended up adding a couple of deny rules for me and ${inet} before the wildcard pass allow rule. I had to make sure that some other more specific rules allowing connections between me and the inside were before the new deny rules. From owner-freebsd-net@freebsd.org Mon Aug 24 04:28:11 2015 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id A54309C1A8F for ; Mon, 24 Aug 2015 04:28:11 +0000 (UTC) (envelope-from hrs@FreeBSD.org) Received: from mail.allbsd.org (gatekeeper.allbsd.org [IPv6:2001:2f0:104:e001::32]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "*.allbsd.org", Issuer "RapidSSL CA" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id AA835F78; Mon, 24 Aug 2015 04:28:10 +0000 (UTC) (envelope-from hrs@FreeBSD.org) Received: from alph.d.allbsd.org (alph.d.allbsd.org [IPv6:2001:2f0:104:e010:862b:2bff:febc:8956] (may be forged)) (authenticated bits=56) by mail.allbsd.org (8.14.9/8.14.9) with ESMTP id t7O4Ru0B067508 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Mon, 24 Aug 2015 13:27:58 +0900 (JST) (envelope-from hrs@FreeBSD.org) Received: from localhost (localhost [IPv6:::1]) (authenticated bits=0) by alph.d.allbsd.org (8.14.9/8.14.9) with ESMTP id t7O4Rso1090905; Mon, 24 Aug 2015 13:27:56 +0900 (JST) (envelope-from hrs@FreeBSD.org) Date: Mon, 24 Aug 2015 13:25:31 +0900 (JST) Message-Id: <20150824.132531.1687906630049554750.hrs@allbsd.org> To: truckman@FreeBSD.org Cc: freebsd-net@FreeBSD.org Subject: Re: a couple /etc/rc.firewall questions From: Hiroki Sato In-Reply-To: <201508240052.t7O0qsFF002623@gw.catspoiler.org> References: <20150823.084453.1715908115913144015.hrs@allbsd.org> <201508240052.t7O0qsFF002623@gw.catspoiler.org> X-PGPkey-fingerprint: BDB3 443F A5DD B3D0 A530 FFD7 4F2C D3D8 2793 CF2D X-Mailer: Mew version 6.7 on Emacs 24.5 / Mule 6.0 (HANACHIRUSATO) Mime-Version: 1.0 Content-Type: Multipart/Signed; protocol="application/pgp-signature"; micalg=pgp-sha1; boundary="--Security_Multipart(Mon_Aug_24_13_25_31_2015_174)--" Content-Transfer-Encoding: 7bit X-Virus-Scanned: clamav-milter 0.98.6 at gatekeeper.allbsd.org X-Virus-Status: Clean X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.4.3 (mail.allbsd.org [IPv6:2001:2f0:104:e001::32]); Mon, 24 Aug 2015 13:28:03 +0900 (JST) X-Spam-Status: No, score=-98.0 required=13.0 tests=CONTENT_TYPE_PRESENT, RCVD_IN_AHBL, RCVD_IN_AHBL_PROXY, RCVD_IN_AHBL_SPAM, RDNS_NONE, USER_IN_WHITELIST autolearn=no autolearn_force=no version=3.4.0 X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on gatekeeper.allbsd.org X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 24 Aug 2015 04:28:11 -0000 ----Security_Multipart(Mon_Aug_24_13_25_31_2015_174)-- Content-Type: Text/Plain; charset=us-ascii Content-Transfer-Encoding: 7bit Don Lewis wrote in <201508240052.t7O0qsFF002623@gw.catspoiler.org>: tr> > A TCP setup packet coming from a host on the internal LAN to the NAPT tr> > router falls into the last deny-all rule because it does not match if tr> > you added "out via ${oif}" to that rule. Does the following tr> > additional rule work for you? tr> > tr> > ${fwcmd} add pass tcp from any to any out via ${oif} setup tr> > ${fwcmd} add pass tcp from any to not me in via ${iif} setup tr> tr> That works for now, but won't do the correct thing when I subdivide my tr> internal network because it will allow unrestricted connections between tr> the internal subnets. What I'd really like is something like: tr> tr> ${fwcmd} add pass tcp from any to not me,${inet} setup tr> tr> but that isn't a valid rule. I ended up adding a couple of deny tr> rules for me and ${inet} before the wildcard pass allow rule. I had to tr> make sure that some other more specific rules allowing connections tr> between me and the inside were before the new deny rules. Hmmm, I think "table" would be useful to restrict connections between the internal subnets in that case like: ## allow TCP setup going to outside network: ${fwcmd} add pass tcp from any to any out via ${oif} setup ## list of all internal subnets including NAPT router itself: ${fwcmd} table 1 flush ${fwcmd} table 1 add 192.168.1.1/32 # NAPT router ${fwcmd} table 1 add 192.168.3.0/24 ${fwcmd} table 1 add 192.168.4.0/24 ... ## allow TCP setup from the internal subnets to outside network: ${fwcmd} add pass tcp from "table(1)" to not "table(1)" in via ${iif} setup ## ## list of internal subnets which can connect to me: ${fwcmd} table 2 flush ${fwcmd} table 2 add 192.168.3.0/24 ... ## allow TCP setup from some of the internal subnets to me: ${fwcmd} add pass tcp from "table(2)" to me in via ${iif} setup -- Hiroki ----Security_Multipart(Mon_Aug_24_13_25_31_2015_174)-- Content-Type: application/pgp-signature Content-Transfer-Encoding: 7bit -----BEGIN PGP SIGNATURE----- Version: GnuPG v1 iEYEABECAAYFAlXanLsACgkQTyzT2CeTzy1gvwCcCOaEwtSkDugtWHhyhte8K/Hw GG0AnRZ1AlVFuxQIP7KHqlnOexS7c0of =v8xY -----END PGP SIGNATURE----- ----Security_Multipart(Mon_Aug_24_13_25_31_2015_174)---- From owner-freebsd-net@freebsd.org Mon Aug 24 06:06:12 2015 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 3976A9C192E for ; Mon, 24 Aug 2015 06:06:12 +0000 (UTC) (envelope-from truckman@FreeBSD.org) Received: from gw.catspoiler.org (unknown [IPv6:2602:304:b010:ef20::f2]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "gw.catspoiler.org", Issuer "gw.catspoiler.org" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id F3D47199A; Mon, 24 Aug 2015 06:06:11 +0000 (UTC) (envelope-from truckman@FreeBSD.org) Received: from FreeBSD.org (mousie.catspoiler.org [192.168.101.2]) by gw.catspoiler.org (8.15.2/8.15.2) with ESMTP id t7O5wkEZ003893; Sun, 23 Aug 2015 22:58:50 -0700 (PDT) (envelope-from truckman@FreeBSD.org) Message-Id: <201508240558.t7O5wkEZ003893@gw.catspoiler.org> Date: Sun, 23 Aug 2015 22:58:46 -0700 (PDT) From: Don Lewis Subject: Re: a couple /etc/rc.firewall questions To: hrs@FreeBSD.org cc: freebsd-net@FreeBSD.org In-Reply-To: <20150824.132531.1687906630049554750.hrs@allbsd.org> MIME-Version: 1.0 Content-Type: TEXT/plain; charset=us-ascii X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 24 Aug 2015 06:06:12 -0000 On 24 Aug, Hiroki Sato wrote: > Don Lewis wrote > in <201508240052.t7O0qsFF002623@gw.catspoiler.org>: > > tr> > A TCP setup packet coming from a host on the internal LAN to the NAPT > tr> > router falls into the last deny-all rule because it does not match if > tr> > you added "out via ${oif}" to that rule. Does the following > tr> > additional rule work for you? > tr> > > tr> > ${fwcmd} add pass tcp from any to any out via ${oif} setup > tr> > ${fwcmd} add pass tcp from any to not me in via ${iif} setup > tr> > tr> That works for now, but won't do the correct thing when I subdivide my > tr> internal network because it will allow unrestricted connections between > tr> the internal subnets. What I'd really like is something like: > tr> > tr> ${fwcmd} add pass tcp from any to not me,${inet} setup > tr> > tr> but that isn't a valid rule. I ended up adding a couple of deny > tr> rules for me and ${inet} before the wildcard pass allow rule. I had to > tr> make sure that some other more specific rules allowing connections > tr> between me and the inside were before the new deny rules. > > Hmmm, I think "table" would be useful to restrict connections between > the internal subnets in that case like: > > ## allow TCP setup going to outside network: > ${fwcmd} add pass tcp from any to any out via ${oif} setup > ## list of all internal subnets including NAPT router itself: > ${fwcmd} table 1 flush > ${fwcmd} table 1 add 192.168.1.1/32 # NAPT router > ${fwcmd} table 1 add 192.168.3.0/24 > ${fwcmd} table 1 add 192.168.4.0/24 > ... > ## allow TCP setup from the internal subnets to outside network: > ${fwcmd} add pass tcp from "table(1)" to not "table(1)" in via ${iif} setup Using the interface name here does not work if the internal subnets are connected via distinct interfaces. Fortunately this isn't necessary if each interface has anti-spoofing rules associated with it, so something like this should work: ${fwcmd} add pass tcp from "table(1)" to not "table(1)" setup I realized a short while ago that we don't need all of the addresses associated with "me" here, so only the outside address of the router needs to be added to the table. Rather than using a table, it would also be possible to just use address lists: oip=192.168.1.1/32 # router external address inet1=192.168.3.0/24 inet2=192.168.3.0/24 inet=${inet1},${inet2} ... ${fwcmd} add pass tcp from ${oip},${inet} to not ${oip},${inet} setup And then ${inet1}, ${inet2}, "me", etc. can be used to add more fine-grained rules for allowing connections between subnets, and between the subnets and the router. Unfortunately inet6 is rather badly named for this scheme. > ## > ## list of internal subnets which can connect to me: > ${fwcmd} table 2 flush > ${fwcmd} table 2 add 192.168.3.0/24 > ... > ## allow TCP setup from some of the internal subnets to me: > ${fwcmd} add pass tcp from "table(2)" to me in via ${iif} setup From owner-freebsd-net@freebsd.org Mon Aug 24 07:09:45 2015 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 733519C129F for ; Mon, 24 Aug 2015 07:09:45 +0000 (UTC) (envelope-from john@maxnet.ru) Received: from basic.maxnet.ru (mx.maxnet.ru [195.112.97.17]) by mx1.freebsd.org (Postfix) with ESMTP id 3441C11E8; Mon, 24 Aug 2015 07:09:43 +0000 (UTC) (envelope-from john@maxnet.ru) Received: from [217.15.204.72] (John.Office.Obninsk.MAXnet.ru [217.15.204.72] (may be forged)) by basic.maxnet.ru (8.14.6/8.14.6) with ESMTP id t7O79YrE055810; Mon, 24 Aug 2015 10:09:34 +0300 (MSK) (envelope-from john@maxnet.ru) Message-ID: <55DAC32F.4010003@maxnet.ru> Date: Mon, 24 Aug 2015 10:09:35 +0300 From: Evgeny Khorokhorin User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:31.0) Gecko/20100101 Thunderbird/31.7.0 MIME-Version: 1.0 To: Eric Joyner , Adrian Chadd CC: hiren panchasara , FreeBSD Net Subject: Re: FreeBSD 10.2-STABLE + Intel XL710 - free queues References: <55D49611.40603@maxnet.ru> <20150819180051.GM94440@strugglingcoder.info> <55D4DAB3.1020401@maxnet.ru> In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 24 Aug 2015 07:09:45 -0000 Hi Eric, Did you manage to try this problem? - Evgeny 19.08.2015 23:16, Eric Joyner пишет: > Yeah; it should be able to do up to 64 queues for the PF's. It's > possible for the NVM to limit the RSS table size and entry width, but > that seems unlikely. > > - Eric > > On Wed, Aug 19, 2015 at 12:41 PM Adrian Chadd > wrote: > > no, it's not the RSS option - it's the RSS configuration in the NIC > for steering traffic into different queues based on header contents. > > The RSS kernel option includes the framework that ties it all together > into the network stack - if you don't use it (which is the default), > the NICs are free to do whatever they want and there's no affinity in > the network stack. > > Eric - does the intel driver / hardware here support receive traffic > distribution into > 16 queues? > > > > -adrian > > > On 19 August 2015 at 12:36, Evgeny Khorokhorin > wrote: > > Eric, > > I updated this driver in kernel, not as module. And I removed > > #include "opt_rss.h" > > > > from if_ixl.c and ixl_txrx.c: > > > > #ifndef IXL_STANDALONE_BUILD > > #include "opt_inet.h" > > #include "opt_inet6.h" > > #include "opt_rss.h" > > #endif > > > > because RSS for is only in HEAD > > Could I break smth by doing this? > > > > Best regards, > > Evgeny Khorokhorin > > > > 19.08.2015 21:17, Eric Joyner пишет: > >> > >> The IXLV_MAX_QUEUES value is for the VF driver; the standard > driver should > >> be able to allocate and properly use up to 64 queues. > >> > >> That said, you're only getting rx traffic on the first 16 > queues, so that > >> looks like a bug in the driver. I'll take a look at it. > >> > >> - Eric > >> > >> On Wed, Aug 19, 2015 at 11:00 AM hiren panchasara > >> > >> wrote: > >> > >> On 08/19/15 at 05:43P, Evgeny Khorokhorin wrote: > >> > Hi All, > >> > > >> > FreeBSD 10.2-STABLE > >> > 2*CPU Intel E5-2643v3 with HyperThreading enabled > >> > Intel XL710 network adapter > >> > I updated the ixl driver to version 1.4.0 from > >> download.intel.com > > >> > >> > Every ixl interface create 24 queues (6 cores *2 HT *2 > CPUs) but > >> > utilizes only 16-17 of them. Where is the reason of such > behavior or > >> > driver bug? > >> > >> Not sure what is the h/w limit but this may be a possible > cause: > >> #define IXLV_MAX_QUEUES 16 > >> in sys/dev/ixl/ixlv.h > >> > >> and ixlv_init_msix() doing: > >> if (queues > IXLV_MAX_QUEUES) > >> queues = IXLV_MAX_QUEUES; > >> > >> Adding eric from intel to confirm. > >> > >> Cheers, > >> Hiren > >> > > >> > irq284: ixl0:q0 177563088 2054 > >> > irq285: ixl0:q1 402668179 4659 > >> > irq286: ixl0:q2 408885088 4731 > >> > irq287: ixl0:q3 397744300 4602 > >> > irq288: ixl0:q4 403040766 4663 > >> > irq289: ixl0:q5 402499314 4657 > >> > irq290: ixl0:q6 392693663 4543 > >> > irq291: ixl0:q7 389364966 4505 > >> > irq292: ixl0:q8 243244346 2814 > >> > irq293: ixl0:q9 216834450 2509 > >> > irq294: ixl0:q10 229460056 2655 > >> > irq295: ixl0:q11 219591953 2540 > >> > irq296: ixl0:q12 228944960 2649 > >> > irq297: ixl0:q13 226385454 2619 > >> > irq298: ixl0:q14 219174953 2536 > >> > irq299: ixl0:q15 222151378 2570 > >> > irq300: ixl0:q16 82799713 958 > >> > irq301: ixl0:q17 6131 0 > >> > irq302: ixl0:q18 5586 0 > >> > irq303: ixl0:q19 6975 0 > >> > irq304: ixl0:q20 6243 0 > >> > irq305: ixl0:q21 6729 0 > >> > irq306: ixl0:q22 6623 0 > >> > irq307: ixl0:q23 7306 0 > >> > irq309: ixl1:q0 174074462 2014 > >> > irq310: ixl1:q1 435716449 5041 > >> > irq311: ixl1:q2 431030443 4987 > >> > irq312: ixl1:q3 424156413 4907 > >> > irq313: ixl1:q4 414791657 4799 > >> > irq314: ixl1:q5 420260382 4862 > >> > irq315: ixl1:q6 415645708 4809 > >> > irq316: ixl1:q7 422783859 4892 > >> > irq317: ixl1:q8 252737383 2924 > >> > irq318: ixl1:q9 269655708 3120 > >> > irq319: ixl1:q10 252397826 2920 > >> > irq320: ixl1:q11 255649144 2958 > >> > irq321: ixl1:q12 246025621 2846 > >> > irq322: ixl1:q13 240176554 2779 > >> > irq323: ixl1:q14 254882418 2949 > >> > irq324: ixl1:q15 236846536 2740 > >> > irq325: ixl1:q16 86794467 1004 > >> > irq326: ixl1:q17 83 0 > >> > irq327: ixl1:q18 74 0 > >> > irq328: ixl1:q19 202 0 > >> > irq329: ixl1:q20 99 0 > >> > irq330: ixl1:q21 96 0 > >> > irq331: ixl1:q22 91 0 > >> > irq332: ixl1:q23 89 0 > >> > > >> > last pid: 28710; load averages: 7.16, 6.76, 6.49 up > >> 1+00:00:41 17:40:46 > >> > 391 processes: 32 running, 215 sleeping, 144 waiting > >> > CPU 0: 0.0% user, 0.0% nice, 0.0% system, 49.2% > interrupt, > >> 50.8% idle > >> > CPU 1: 0.0% user, 0.0% nice, 0.4% system, 41.3% > interrupt, > >> 58.3% idle > >> > CPU 2: 0.0% user, 0.0% nice, 0.0% system, 39.0% > interrupt, > >> 61.0% idle > >> > CPU 3: 0.0% user, 0.0% nice, 0.0% system, 46.5% > interrupt, > >> 53.5% idle > >> > CPU 4: 0.0% user, 0.0% nice, 0.0% system, 37.4% > interrupt, > >> 62.6% idle > >> > CPU 5: 0.0% user, 0.0% nice, 0.0% system, 40.9% > interrupt, > >> 59.1% idle > >> > CPU 6: 0.0% user, 0.0% nice, 0.0% system, 40.2% > interrupt, > >> 59.8% idle > >> > CPU 7: 0.0% user, 0.0% nice, 0.0% system, 45.3% > interrupt, > >> 54.7% idle > >> > CPU 8: 0.0% user, 0.0% nice, 0.0% system, 20.5% > interrupt, > >> 79.5% idle > >> > CPU 9: 0.0% user, 0.0% nice, 0.0% system, 25.2% > interrupt, > >> 74.8% idle > >> > CPU 10: 0.0% user, 0.0% nice, 0.0% system, 23.2% > interrupt, > >> 76.8% idle > >> > CPU 11: 0.0% user, 0.0% nice, 0.0% system, 19.3% > interrupt, > >> 80.7% idle > >> > CPU 12: 0.0% user, 0.0% nice, 0.0% system, 28.7% > interrupt, > >> 71.3% idle > >> > CPU 13: 0.0% user, 0.0% nice, 0.0% system, 20.5% > interrupt, > >> 79.5% idle > >> > CPU 14: 0.0% user, 0.0% nice, 0.0% system, 35.0% > interrupt, > >> 65.0% idle > >> > CPU 15: 0.0% user, 0.0% nice, 0.0% system, 23.2% > interrupt, > >> 76.8% idle > >> > CPU 16: 0.0% user, 0.0% nice, 0.4% system, 1.2% > interrupt, > >> 98.4% idle > >> > CPU 17: 0.0% user, 0.0% nice, 2.0% system, 0.0% > interrupt, > >> 98.0% idle > >> > CPU 18: 0.0% user, 0.0% nice, 2.4% system, 0.0% > interrupt, > >> 97.6% idle > >> > CPU 19: 0.0% user, 0.0% nice, 2.8% system, 0.0% > interrupt, > >> 97.2% idle > >> > CPU 20: 0.0% user, 0.0% nice, 2.4% system, 0.0% > interrupt, > >> 97.6% idle > >> > CPU 21: 0.0% user, 0.0% nice, 1.6% system, 0.0% > interrupt, > >> 98.4% idle > >> > CPU 22: 0.0% user, 0.0% nice, 2.8% system, 0.0% > interrupt, > >> 97.2% idle > >> > CPU 23: 0.0% user, 0.0% nice, 0.4% system, 0.0% > interrupt, > >> 99.6% idle > >> > > >> > # netstat -I ixl0 -w1 -h > >> > input ixl0 output > >> > packets errs idrops bytes packets errs bytes colls > >> > 253K 0 0 136M 311K 0 325M 0 > >> > 251K 0 0 129M 314K 0 334M 0 > >> > 250K 0 0 135M 313K 0 333M 0 > >> > > >> > hw.ixl.tx_itr: 122 > >> > hw.ixl.rx_itr: 62 > >> > hw.ixl.dynamic_tx_itr: 0 > >> > hw.ixl.dynamic_rx_itr: 0 > >> > hw.ixl.max_queues: 0 > >> > hw.ixl.ring_size: 4096 > >> > hw.ixl.enable_msix: 1 > >> > dev.ixl.3.mac.xoff_recvd: 0 > >> > dev.ixl.3.mac.xoff_txd: 0 > >> > dev.ixl.3.mac.xon_recvd: 0 > >> > dev.ixl.3.mac.xon_txd: 0 > >> > dev.ixl.3.mac.tx_frames_big: 0 > >> > dev.ixl.3.mac.tx_frames_1024_1522: 0 > >> > dev.ixl.3.mac.tx_frames_512_1023: 0 > >> > dev.ixl.3.mac.tx_frames_256_511: 0 > >> > dev.ixl.3.mac.tx_frames_128_255: 0 > >> > dev.ixl.3.mac.tx_frames_65_127: 0 > >> > dev.ixl.3.mac.tx_frames_64: 0 > >> > dev.ixl.3.mac.checksum_errors: 0 > >> > dev.ixl.3.mac.rx_jabber: 0 > >> > dev.ixl.3.mac.rx_oversized: 0 > >> > dev.ixl.3.mac.rx_fragmented: 0 > >> > dev.ixl.3.mac.rx_undersize: 0 > >> > dev.ixl.3.mac.rx_frames_big: 0 > >> > dev.ixl.3.mac.rx_frames_1024_1522: 0 > >> > dev.ixl.3.mac.rx_frames_512_1023: 0 > >> > dev.ixl.3.mac.rx_frames_256_511: 0 > >> > dev.ixl.3.mac.rx_frames_128_255: 0 > >> > dev.ixl.3.mac.rx_frames_65_127: 0 > >> > dev.ixl.3.mac.rx_frames_64: 0 > >> > dev.ixl.3.mac.rx_length_errors: 0 > >> > dev.ixl.3.mac.remote_faults: 0 > >> > dev.ixl.3.mac.local_faults: 0 > >> > dev.ixl.3.mac.illegal_bytes: 0 > >> > dev.ixl.3.mac.crc_errors: 0 > >> > dev.ixl.3.mac.bcast_pkts_txd: 0 > >> > dev.ixl.3.mac.mcast_pkts_txd: 0 > >> > dev.ixl.3.mac.ucast_pkts_txd: 0 > >> > dev.ixl.3.mac.good_octets_txd: 0 > >> > dev.ixl.3.mac.rx_discards: 0 > >> > dev.ixl.3.mac.bcast_pkts_rcvd: 0 > >> > dev.ixl.3.mac.mcast_pkts_rcvd: 0 > >> > dev.ixl.3.mac.ucast_pkts_rcvd: 0 > >> > dev.ixl.3.mac.good_octets_rcvd: 0 > >> > dev.ixl.3.pf.que23.rx_bytes: 0 > >> > dev.ixl.3.pf.que23.rx_packets: 0 > >> > dev.ixl.3.pf.que23.tx_bytes: 0 > >> > dev.ixl.3.pf.que23.tx_packets: 0 > >> > dev.ixl.3.pf.que23.no_desc_avail: 0 > >> > dev.ixl.3.pf.que23.tx_dma_setup: 0 > >> > dev.ixl.3.pf.que23.tso_tx: 0 > >> > dev.ixl.3.pf.que23.irqs: 0 > >> > dev.ixl.3.pf.que23.dropped: 0 > >> > dev.ixl.3.pf.que23.mbuf_defrag_failed: 0 > >> > dev.ixl.3.pf.que22.rx_bytes: 0 > >> > dev.ixl.3.pf.que22.rx_packets: 0 > >> > dev.ixl.3.pf.que22.tx_bytes: 0 > >> > dev.ixl.3.pf.que22.tx_packets: 0 > >> > dev.ixl.3.pf.que22.no_desc_avail: 0 > >> > dev.ixl.3.pf.que22.tx_dma_setup: 0 > >> > dev.ixl.3.pf.que22.tso_tx: 0 > >> > dev.ixl.3.pf.que22.irqs: 0 > >> > dev.ixl.3.pf.que22.dropped: 0 > >> > dev.ixl.3.pf.que22.mbuf_defrag_failed: 0 > >> > dev.ixl.3.pf.que21.rx_bytes: 0 > >> > dev.ixl.3.pf.que21.rx_packets: 0 > >> > dev.ixl.3.pf.que21.tx_bytes: 0 > >> > dev.ixl.3.pf.que21.tx_packets: 0 > >> > dev.ixl.3.pf.que21.no_desc_avail: 0 > >> > dev.ixl.3.pf.que21.tx_dma_setup: 0 > >> > dev.ixl.3.pf.que21.tso_tx: 0 > >> > dev.ixl.3.pf.que21.irqs: 0 > >> > dev.ixl.3.pf.que21.dropped: 0 > >> > dev.ixl.3.pf.que21.mbuf_defrag_failed: 0 > >> > dev.ixl.3.pf.que20.rx_bytes: 0 > >> > dev.ixl.3.pf.que20.rx_packets: 0 > >> > dev.ixl.3.pf.que20.tx_bytes: 0 > >> > dev.ixl.3.pf.que20.tx_packets: 0 > >> > dev.ixl.3.pf.que20.no_desc_avail: 0 > >> > dev.ixl.3.pf.que20.tx_dma_setup: 0 > >> > dev.ixl.3.pf.que20.tso_tx: 0 > >> > dev.ixl.3.pf.que20.irqs: 0 > >> > dev.ixl.3.pf.que20.dropped: 0 > >> > dev.ixl.3.pf.que20.mbuf_defrag_failed: 0 > >> > dev.ixl.3.pf.que19.rx_bytes: 0 > >> > dev.ixl.3.pf.que19.rx_packets: 0 > >> > dev.ixl.3.pf.que19.tx_bytes: 0 > >> > dev.ixl.3.pf.que19.tx_packets: 0 > >> > dev.ixl.3.pf.que19.no_desc_avail: 0 > >> > dev.ixl.3.pf.que19.tx_dma_setup: 0 > >> > dev.ixl.3.pf.que19.tso_tx: 0 > >> > dev.ixl.3.pf.que19.irqs: 0 > >> > dev.ixl.3.pf.que19.dropped: 0 > >> > dev.ixl.3.pf.que19.mbuf_defrag_failed: 0 > >> > dev.ixl.3.pf.que18.rx_bytes: 0 > >> > dev.ixl.3.pf.que18.rx_packets: 0 > >> > dev.ixl.3.pf.que18.tx_bytes: 0 > >> > dev.ixl.3.pf.que18.tx_packets: 0 > >> > dev.ixl.3.pf.que18.no_desc_avail: 0 > >> > dev.ixl.3.pf.que18.tx_dma_setup: 0 > >> > dev.ixl.3.pf.que18.tso_tx: 0 > >> > dev.ixl.3.pf.que18.irqs: 0 > >> > dev.ixl.3.pf.que18.dropped: 0 > >> > dev.ixl.3.pf.que18.mbuf_defrag_failed: 0 > >> > dev.ixl.3.pf.que17.rx_bytes: 0 > >> > dev.ixl.3.pf.que17.rx_packets: 0 > >> > dev.ixl.3.pf.que17.tx_bytes: 0 > >> > dev.ixl.3.pf.que17.tx_packets: 0 > >> > dev.ixl.3.pf.que17.no_desc_avail: 0 > >> > dev.ixl.3.pf.que17.tx_dma_setup: 0 > >> > dev.ixl.3.pf.que17.tso_tx: 0 > >> > dev.ixl.3.pf.que17.irqs: 0 > >> > dev.ixl.3.pf.que17.dropped: 0 > >> > dev.ixl.3.pf.que17.mbuf_defrag_failed: 0 > >> > dev.ixl.3.pf.que16.rx_bytes: 0 > >> > dev.ixl.3.pf.que16.rx_packets: 0 > >> > dev.ixl.3.pf.que16.tx_bytes: 0 > >> > dev.ixl.3.pf.que16.tx_packets: 0 > >> > dev.ixl.3.pf.que16.no_desc_avail: 0 > >> > dev.ixl.3.pf.que16.tx_dma_setup: 0 > >> > dev.ixl.3.pf.que16.tso_tx: 0 > >> > dev.ixl.3.pf.que16.irqs: 0 > >> > dev.ixl.3.pf.que16.dropped: 0 > >> > dev.ixl.3.pf.que16.mbuf_defrag_failed: 0 > >> > dev.ixl.3.pf.que15.rx_bytes: 0 > >> > dev.ixl.3.pf.que15.rx_packets: 0 > >> > dev.ixl.3.pf.que15.tx_bytes: 0 > >> > dev.ixl.3.pf.que15.tx_packets: 0 > >> > dev.ixl.3.pf.que15.no_desc_avail: 0 > >> > dev.ixl.3.pf.que15.tx_dma_setup: 0 > >> > dev.ixl.3.pf.que15.tso_tx: 0 > >> > dev.ixl.3.pf.que15.irqs: 0 > >> > dev.ixl.3.pf.que15.dropped: 0 > >> > dev.ixl.3.pf.que15.mbuf_defrag_failed: 0 > >> > dev.ixl.3.pf.que14.rx_bytes: 0 > >> > dev.ixl.3.pf.que14.rx_packets: 0 > >> > dev.ixl.3.pf.que14.tx_bytes: 0 > >> > dev.ixl.3.pf.que14.tx_packets: 0 > >> > dev.ixl.3.pf.que14.no_desc_avail: 0 > >> > dev.ixl.3.pf.que14.tx_dma_setup: 0 > >> > dev.ixl.3.pf.que14.tso_tx: 0 > >> > dev.ixl.3.pf.que14.irqs: 0 > >> > dev.ixl.3.pf.que14.dropped: 0 > >> > dev.ixl.3.pf.que14.mbuf_defrag_failed: 0 > >> > dev.ixl.3.pf.que13.rx_bytes: 0 > >> > dev.ixl.3.pf.que13.rx_packets: 0 > >> > dev.ixl.3.pf.que13.tx_bytes: 0 > >> > dev.ixl.3.pf.que13.tx_packets: 0 > >> > dev.ixl.3.pf.que13.no_desc_avail: 0 > >> > dev.ixl.3.pf.que13.tx_dma_setup: 0 > >> > dev.ixl.3.pf.que13.tso_tx: 0 > >> > dev.ixl.3.pf.que13.irqs: 0 > >> > dev.ixl.3.pf.que13.dropped: 0 > >> > dev.ixl.3.pf.que13.mbuf_defrag_failed: 0 > >> > dev.ixl.3.pf.que12.rx_bytes: 0 > >> > dev.ixl.3.pf.que12.rx_packets: 0 > >> > dev.ixl.3.pf.que12.tx_bytes: 0 > >> > dev.ixl.3.pf.que12.tx_packets: 0 > >> > dev.ixl.3.pf.que12.no_desc_avail: 0 > >> > dev.ixl.3.pf.que12.tx_dma_setup: 0 > >> > dev.ixl.3.pf.que12.tso_tx: 0 > >> > dev.ixl.3.pf.que12.irqs: 0 > >> > dev.ixl.3.pf.que12.dropped: 0 > >> > dev.ixl.3.pf.que12.mbuf_defrag_failed: 0 > >> > dev.ixl.3.pf.que11.rx_bytes: 0 > >> > dev.ixl.3.pf.que11.rx_packets: 0 > >> > dev.ixl.3.pf.que11.tx_bytes: 0 > >> > dev.ixl.3.pf.que11.tx_packets: 0 > >> > dev.ixl.3.pf.que11.no_desc_avail: 0 > >> > dev.ixl.3.pf.que11.tx_dma_setup: 0 > >> > dev.ixl.3.pf.que11.tso_tx: 0 > >> > dev.ixl.3.pf.que11.irqs: 0 > >> > dev.ixl.3.pf.que11.dropped: 0 > >> > dev.ixl.3.pf.que11.mbuf_defrag_failed: 0 > >> > dev.ixl.3.pf.que10.rx_bytes: 0 > >> > dev.ixl.3.pf.que10.rx_packets: 0 > >> > dev.ixl.3.pf.que10.tx_bytes: 0 > >> > dev.ixl.3.pf.que10.tx_packets: 0 > >> > dev.ixl.3.pf.que10.no_desc_avail: 0 > >> > dev.ixl.3.pf.que10.tx_dma_setup: 0 > >> > dev.ixl.3.pf.que10.tso_tx: 0 > >> > dev.ixl.3.pf.que10.irqs: 0 > >> > dev.ixl.3.pf.que10.dropped: 0 > >> > dev.ixl.3.pf.que10.mbuf_defrag_failed: 0 > >> > dev.ixl.3.pf.que9.rx_bytes: 0 > >> > dev.ixl.3.pf.que9.rx_packets: 0 > >> > dev.ixl.3.pf.que9.tx_bytes: 0 > >> > dev.ixl.3.pf.que9.tx_packets: 0 > >> > dev.ixl.3.pf.que9.no_desc_avail: 0 > >> > dev.ixl.3.pf.que9.tx_dma_setup: 0 > >> > dev.ixl.3.pf.que9.tso_tx: 0 > >> > dev.ixl.3.pf.que9.irqs: 0 > >> > dev.ixl.3.pf.que9.dropped: 0 > >> > dev.ixl.3.pf.que9.mbuf_defrag_failed: 0 > >> > dev.ixl.3.pf.que8.rx_bytes: 0 > >> > dev.ixl.3.pf.que8.rx_packets: 0 > >> > dev.ixl.3.pf.que8.tx_bytes: 0 > >> > dev.ixl.3.pf.que8.tx_packets: 0 > >> > dev.ixl.3.pf.que8.no_desc_avail: 0 > >> > dev.ixl.3.pf.que8.tx_dma_setup: 0 > >> > dev.ixl.3.pf.que8.tso_tx: 0 > >> > dev.ixl.3.pf.que8.irqs: 0 > >> > dev.ixl.3.pf.que8.dropped: 0 > >> > dev.ixl.3.pf.que8.mbuf_defrag_failed: 0 > >> > dev.ixl.3.pf.que7.rx_bytes: 0 > >> > dev.ixl.3.pf.que7.rx_packets: 0 > >> > dev.ixl.3.pf.que7.tx_bytes: 0 > >> > dev.ixl.3.pf.que7.tx_packets: 0 > >> > dev.ixl.3.pf.que7.no_desc_avail: 0 > >> > dev.ixl.3.pf.que7.tx_dma_setup: 0 > >> > dev.ixl.3.pf.que7.tso_tx: 0 > >> > dev.ixl.3.pf.que7.irqs: 0 > >> > dev.ixl.3.pf.que7.dropped: 0 > >> > dev.ixl.3.pf.que7.mbuf_defrag_failed: 0 > >> > dev.ixl.3.pf.que6.rx_bytes: 0 > >> > dev.ixl.3.pf.que6.rx_packets: 0 > >> > dev.ixl.3.pf.que6.tx_bytes: 0 > >> > dev.ixl.3.pf.que6.tx_packets: 0 > >> > dev.ixl.3.pf.que6.no_desc_avail: 0 > >> > dev.ixl.3.pf.que6.tx_dma_setup: 0 > >> > dev.ixl.3.pf.que6.tso_tx: 0 > >> > dev.ixl.3.pf.que6.irqs: 0 > >> > dev.ixl.3.pf.que6.dropped: 0 > >> > dev.ixl.3.pf.que6.mbuf_defrag_failed: 0 > >> > dev.ixl.3.pf.que5.rx_bytes: 0 > >> > dev.ixl.3.pf.que5.rx_packets: 0 > >> > dev.ixl.3.pf.que5.tx_bytes: 0 > >> > dev.ixl.3.pf.que5.tx_packets: 0 > >> > dev.ixl.3.pf.que5.no_desc_avail: 0 > >> > dev.ixl.3.pf.que5.tx_dma_setup: 0 > >> > dev.ixl.3.pf.que5.tso_tx: 0 > >> > dev.ixl.3.pf.que5.irqs: 0 > >> > dev.ixl.3.pf.que5.dropped: 0 > >> > dev.ixl.3.pf.que5.mbuf_defrag_failed: 0 > >> > dev.ixl.3.pf.que4.rx_bytes: 0 > >> > dev.ixl.3.pf.que4.rx_packets: 0 > >> > dev.ixl.3.pf.que4.tx_bytes: 0 > >> > dev.ixl.3.pf.que4.tx_packets: 0 > >> > dev.ixl.3.pf.que4.no_desc_avail: 0 > >> > dev.ixl.3.pf.que4.tx_dma_setup: 0 > >> > dev.ixl.3.pf.que4.tso_tx: 0 > >> > dev.ixl.3.pf.que4.irqs: 0 > >> > dev.ixl.3.pf.que4.dropped: 0 > >> > dev.ixl.3.pf.que4.mbuf_defrag_failed: 0 > >> > dev.ixl.3.pf.que3.rx_bytes: 0 > >> > dev.ixl.3.pf.que3.rx_packets: 0 > >> > dev.ixl.3.pf.que3.tx_bytes: 0 > >> > dev.ixl.3.pf.que3.tx_packets: 0 > >> > dev.ixl.3.pf.que3.no_desc_avail: 0 > >> > dev.ixl.3.pf.que3.tx_dma_setup: 0 > >> > dev.ixl.3.pf.que3.tso_tx: 0 > >> > dev.ixl.3.pf.que3.irqs: 0 > >> > dev.ixl.3.pf.que3.dropped: 0 > >> > dev.ixl.3.pf.que3.mbuf_defrag_failed: 0 > >> > dev.ixl.3.pf.que2.rx_bytes: 0 > >> > dev.ixl.3.pf.que2.rx_packets: 0 > >> > dev.ixl.3.pf.que2.tx_bytes: 0 > >> > dev.ixl.3.pf.que2.tx_packets: 0 > >> > dev.ixl.3.pf.que2.no_desc_avail: 0 > >> > dev.ixl.3.pf.que2.tx_dma_setup: 0 > >> > dev.ixl.3.pf.que2.tso_tx: 0 > >> > dev.ixl.3.pf.que2.irqs: 0 > >> > dev.ixl.3.pf.que2.dropped: 0 > >> > dev.ixl.3.pf.que2.mbuf_defrag_failed: 0 > >> > dev.ixl.3.pf.que1.rx_bytes: 0 > >> > dev.ixl.3.pf.que1.rx_packets: 0 > >> > dev.ixl.3.pf.que1.tx_bytes: 0 > >> > dev.ixl.3.pf.que1.tx_packets: 0 > >> > dev.ixl.3.pf.que1.no_desc_avail: 0 > >> > dev.ixl.3.pf.que1.tx_dma_setup: 0 > >> > dev.ixl.3.pf.que1.tso_tx: 0 > >> > dev.ixl.3.pf.que1.irqs: 0 > >> > dev.ixl.3.pf.que1.dropped: 0 > >> > dev.ixl.3.pf.que1.mbuf_defrag_failed: 0 > >> > dev.ixl.3.pf.que0.rx_bytes: 0 > >> > dev.ixl.3.pf.que0.rx_packets: 0 > >> > dev.ixl.3.pf.que0.tx_bytes: 0 > >> > dev.ixl.3.pf.que0.tx_packets: 0 > >> > dev.ixl.3.pf.que0.no_desc_avail: 0 > >> > dev.ixl.3.pf.que0.tx_dma_setup: 0 > >> > dev.ixl.3.pf.que0.tso_tx: 0 > >> > dev.ixl.3.pf.que0.irqs: 0 > >> > dev.ixl.3.pf.que0.dropped: 0 > >> > dev.ixl.3.pf.que0.mbuf_defrag_failed: 0 > >> > dev.ixl.3.pf.bcast_pkts_txd: 0 > >> > dev.ixl.3.pf.mcast_pkts_txd: 0 > >> > dev.ixl.3.pf.ucast_pkts_txd: 0 > >> > dev.ixl.3.pf.good_octets_txd: 0 > >> > dev.ixl.3.pf.rx_discards: 0 > >> > dev.ixl.3.pf.bcast_pkts_rcvd: 0 > >> > dev.ixl.3.pf.mcast_pkts_rcvd: 0 > >> > dev.ixl.3.pf.ucast_pkts_rcvd: 0 > >> > dev.ixl.3.pf.good_octets_rcvd: 0 > >> > dev.ixl.3.vc_debug_level: 1 > >> > dev.ixl.3.admin_irq: 0 > >> > dev.ixl.3.watchdog_events: 0 > >> > dev.ixl.3.debug: 0 > >> > dev.ixl.3.dynamic_tx_itr: 0 > >> > dev.ixl.3.tx_itr: 122 > >> > dev.ixl.3.dynamic_rx_itr: 0 > >> > dev.ixl.3.rx_itr: 62 > >> > dev.ixl.3.fw_version: f4.33 a1.2 n04.42 e8000191d > >> > dev.ixl.3.current_speed: Unknown > >> > dev.ixl.3.advertise_speed: 0 > >> > dev.ixl.3.fc: 0 > >> > dev.ixl.3.%parent: pci129 > >> > dev.ixl.3.%pnpinfo: vendor=0x8086 device=0x1572 > subvendor=0x8086 > >> > subdevice=0x0000 class=0x020000 > >> > dev.ixl.3.%location: slot=0 function=3 > handle=\_SB_.PCI1.QR3A.H003 > >> > dev.ixl.3.%driver: ixl > >> > dev.ixl.3.%desc: Intel(R) Ethernet Connection XL710 Driver, > >> Version - 1.4.0 > >> > dev.ixl.2.mac.xoff_recvd: 0 > >> > dev.ixl.2.mac.xoff_txd: 0 > >> > dev.ixl.2.mac.xon_recvd: 0 > >> > dev.ixl.2.mac.xon_txd: 0 > >> > dev.ixl.2.mac.tx_frames_big: 0 > >> > dev.ixl.2.mac.tx_frames_1024_1522: 0 > >> > dev.ixl.2.mac.tx_frames_512_1023: 0 > >> > dev.ixl.2.mac.tx_frames_256_511: 0 > >> > dev.ixl.2.mac.tx_frames_128_255: 0 > >> > dev.ixl.2.mac.tx_frames_65_127: 0 > >> > dev.ixl.2.mac.tx_frames_64: 0 > >> > dev.ixl.2.mac.checksum_errors: 0 > >> > dev.ixl.2.mac.rx_jabber: 0 > >> > dev.ixl.2.mac.rx_oversized: 0 > >> > dev.ixl.2.mac.rx_fragmented: 0 > >> > dev.ixl.2.mac.rx_undersize: 0 > >> > dev.ixl.2.mac.rx_frames_big: 0 > >> > dev.ixl.2.mac.rx_frames_1024_1522: 0 > >> > dev.ixl.2.mac.rx_frames_512_1023: 0 > >> > dev.ixl.2.mac.rx_frames_256_511: 0 > >> > dev.ixl.2.mac.rx_frames_128_255: 0 > >> > dev.ixl.2.mac.rx_frames_65_127: 0 > >> > dev.ixl.2.mac.rx_frames_64: 0 > >> > dev.ixl.2.mac.rx_length_errors: 0 > >> > dev.ixl.2.mac.remote_faults: 0 > >> > dev.ixl.2.mac.local_faults: 0 > >> > dev.ixl.2.mac.illegal_bytes: 0 > >> > dev.ixl.2.mac.crc_errors: 0 > >> > dev.ixl.2.mac.bcast_pkts_txd: 0 > >> > dev.ixl.2.mac.mcast_pkts_txd: 0 > >> > dev.ixl.2.mac.ucast_pkts_txd: 0 > >> > dev.ixl.2.mac.good_octets_txd: 0 > >> > dev.ixl.2.mac.rx_discards: 0 > >> > dev.ixl.2.mac.bcast_pkts_rcvd: 0 > >> > dev.ixl.2.mac.mcast_pkts_rcvd: 0 > >> > dev.ixl.2.mac.ucast_pkts_rcvd: 0 > >> > dev.ixl.2.mac.good_octets_rcvd: 0 > >> > dev.ixl.2.pf.que23.rx_bytes: 0 > >> > dev.ixl.2.pf.que23.rx_packets: 0 > >> > dev.ixl.2.pf.que23.tx_bytes: 0 > >> > dev.ixl.2.pf.que23.tx_packets: 0 > >> > dev.ixl.2.pf.que23.no_desc_avail: 0 > >> > dev.ixl.2.pf.que23.tx_dma_setup: 0 > >> > dev.ixl.2.pf.que23.tso_tx: 0 > >> > dev.ixl.2.pf.que23.irqs: 0 > >> > dev.ixl.2.pf.que23.dropped: 0 > >> > dev.ixl.2.pf.que23.mbuf_defrag_failed: 0 > >> > dev.ixl.2.pf.que22.rx_bytes: 0 > >> > dev.ixl.2.pf.que22.rx_packets: 0 > >> > dev.ixl.2.pf.que22.tx_bytes: 0 > >> > dev.ixl.2.pf.que22.tx_packets: 0 > >> > dev.ixl.2.pf.que22.no_desc_avail: 0 > >> > dev.ixl.2.pf.que22.tx_dma_setup: 0 > >> > dev.ixl.2.pf.que22.tso_tx: 0 > >> > dev.ixl.2.pf.que22.irqs: 0 > >> > dev.ixl.2.pf.que22.dropped: 0 > >> > dev.ixl.2.pf.que22.mbuf_defrag_failed: 0 > >> > dev.ixl.2.pf.que21.rx_bytes: 0 > >> > dev.ixl.2.pf.que21.rx_packets: 0 > >> > dev.ixl.2.pf.que21.tx_bytes: 0 > >> > dev.ixl.2.pf.que21.tx_packets: 0 > >> > dev.ixl.2.pf.que21.no_desc_avail: 0 > >> > dev.ixl.2.pf.que21.tx_dma_setup: 0 > >> > dev.ixl.2.pf.que21.tso_tx: 0 > >> > dev.ixl.2.pf.que21.irqs: 0 > >> > dev.ixl.2.pf.que21.dropped: 0 > >> > dev.ixl.2.pf.que21.mbuf_defrag_failed: 0 > >> > dev.ixl.2.pf.que20.rx_bytes: 0 > >> > dev.ixl.2.pf.que20.rx_packets: 0 > >> > dev.ixl.2.pf.que20.tx_bytes: 0 > >> > dev.ixl.2.pf.que20.tx_packets: 0 > >> > dev.ixl.2.pf.que20.no_desc_avail: 0 > >> > dev.ixl.2.pf.que20.tx_dma_setup: 0 > >> > dev.ixl.2.pf.que20.tso_tx: 0 > >> > dev.ixl.2.pf.que20.irqs: 0 > >> > dev.ixl.2.pf.que20.dropped: 0 > >> > dev.ixl.2.pf.que20.mbuf_defrag_failed: 0 > >> > dev.ixl.2.pf.que19.rx_bytes: 0 > >> > dev.ixl.2.pf.que19.rx_packets: 0 > >> > dev.ixl.2.pf.que19.tx_bytes: 0 > >> > dev.ixl.2.pf.que19.tx_packets: 0 > >> > dev.ixl.2.pf.que19.no_desc_avail: 0 > >> > dev.ixl.2.pf.que19.tx_dma_setup: 0 > >> > dev.ixl.2.pf.que19.tso_tx: 0 > >> > dev.ixl.2.pf.que19.irqs: 0 > >> > dev.ixl.2.pf.que19.dropped: 0 > >> > dev.ixl.2.pf.que19.mbuf_defrag_failed: 0 > >> > dev.ixl.2.pf.que18.rx_bytes: 0 > >> > dev.ixl.2.pf.que18.rx_packets: 0 > >> > dev.ixl.2.pf.que18.tx_bytes: 0 > >> > dev.ixl.2.pf.que18.tx_packets: 0 > >> > dev.ixl.2.pf.que18.no_desc_avail: 0 > >> > dev.ixl.2.pf.que18.tx_dma_setup: 0 > >> > dev.ixl.2.pf.que18.tso_tx: 0 > >> > dev.ixl.2.pf.que18.irqs: 0 > >> > dev.ixl.2.pf.que18.dropped: 0 > >> > dev.ixl.2.pf.que18.mbuf_defrag_failed: 0 > >> > dev.ixl.2.pf.que17.rx_bytes: 0 > >> > dev.ixl.2.pf.que17.rx_packets: 0 > >> > dev.ixl.2.pf.que17.tx_bytes: 0 > >> > dev.ixl.2.pf.que17.tx_packets: 0 > >> > dev.ixl.2.pf.que17.no_desc_avail: 0 > >> > dev.ixl.2.pf.que17.tx_dma_setup: 0 > >> > dev.ixl.2.pf.que17.tso_tx: 0 > >> > dev.ixl.2.pf.que17.irqs: 0 > >> > dev.ixl.2.pf.que17.dropped: 0 > >> > dev.ixl.2.pf.que17.mbuf_defrag_failed: 0 > >> > dev.ixl.2.pf.que16.rx_bytes: 0 > >> > dev.ixl.2.pf.que16.rx_packets: 0 > >> > dev.ixl.2.pf.que16.tx_bytes: 0 > >> > dev.ixl.2.pf.que16.tx_packets: 0 > >> > dev.ixl.2.pf.que16.no_desc_avail: 0 > >> > dev.ixl.2.pf.que16.tx_dma_setup: 0 > >> > dev.ixl.2.pf.que16.tso_tx: 0 > >> > dev.ixl.2.pf.que16.irqs: 0 > >> > dev.ixl.2.pf.que16.dropped: 0 > >> > dev.ixl.2.pf.que16.mbuf_defrag_failed: 0 > >> > dev.ixl.2.pf.que15.rx_bytes: 0 > >> > dev.ixl.2.pf.que15.rx_packets: 0 > >> > dev.ixl.2.pf.que15.tx_bytes: 0 > >> > dev.ixl.2.pf.que15.tx_packets: 0 > >> > dev.ixl.2.pf.que15.no_desc_avail: 0 > >> > dev.ixl.2.pf.que15.tx_dma_setup: 0 > >> > dev.ixl.2.pf.que15.tso_tx: 0 > >> > dev.ixl.2.pf.que15.irqs: 0 > >> > dev.ixl.2.pf.que15.dropped: 0 > >> > dev.ixl.2.pf.que15.mbuf_defrag_failed: 0 > >> > dev.ixl.2.pf.que14.rx_bytes: 0 > >> > dev.ixl.2.pf.que14.rx_packets: 0 > >> > dev.ixl.2.pf.que14.tx_bytes: 0 > >> > dev.ixl.2.pf.que14.tx_packets: 0 > >> > dev.ixl.2.pf.que14.no_desc_avail: 0 > >> > dev.ixl.2.pf.que14.tx_dma_setup: 0 > >> > dev.ixl.2.pf.que14.tso_tx: 0 > >> > dev.ixl.2.pf.que14.irqs: 0 > >> > dev.ixl.2.pf.que14.dropped: 0 > >> > dev.ixl.2.pf.que14.mbuf_defrag_failed: 0 > >> > dev.ixl.2.pf.que13.rx_bytes: 0 > >> > dev.ixl.2.pf.que13.rx_packets: 0 > >> > dev.ixl.2.pf.que13.tx_bytes: 0 > >> > dev.ixl.2.pf.que13.tx_packets: 0 > >> > dev.ixl.2.pf.que13.no_desc_avail: 0 > >> > dev.ixl.2.pf.que13.tx_dma_setup: 0 > >> > dev.ixl.2.pf.que13.tso_tx: 0 > >> > dev.ixl.2.pf.que13.irqs: 0 > >> > dev.ixl.2.pf.que13.dropped: 0 > >> > dev.ixl.2.pf.que13.mbuf_defrag_failed: 0 > >> > dev.ixl.2.pf.que12.rx_bytes: 0 > >> > dev.ixl.2.pf.que12.rx_packets: 0 > >> > dev.ixl.2.pf.que12.tx_bytes: 0 > >> > dev.ixl.2.pf.que12.tx_packets: 0 > >> > dev.ixl.2.pf.que12.no_desc_avail: 0 > >> > dev.ixl.2.pf.que12.tx_dma_setup: 0 > >> > dev.ixl.2.pf.que12.tso_tx: 0 > >> > dev.ixl.2.pf.que12.irqs: 0 > >> > dev.ixl.2.pf.que12.dropped: 0 > >> > dev.ixl.2.pf.que12.mbuf_defrag_failed: 0 > >> > dev.ixl.2.pf.que11.rx_bytes: 0 > >> > dev.ixl.2.pf.que11.rx_packets: 0 > >> > dev.ixl.2.pf.que11.tx_bytes: 0 > >> > dev.ixl.2.pf.que11.tx_packets: 0 > >> > dev.ixl.2.pf.que11.no_desc_avail: 0 > >> > dev.ixl.2.pf.que11.tx_dma_setup: 0 > >> > dev.ixl.2.pf.que11.tso_tx: 0 > >> > dev.ixl.2.pf.que11.irqs: 0 > >> > dev.ixl.2.pf.que11.dropped: 0 > >> > dev.ixl.2.pf.que11.mbuf_defrag_failed: 0 > >> > dev.ixl.2.pf.que10.rx_bytes: 0 > >> > dev.ixl.2.pf.que10.rx_packets: 0 > >> > dev.ixl.2.pf.que10.tx_bytes: 0 > >> > dev.ixl.2.pf.que10.tx_packets: 0 > >> > dev.ixl.2.pf.que10.no_desc_avail: 0 > >> > dev.ixl.2.pf.que10.tx_dma_setup: 0 > >> > dev.ixl.2.pf.que10.tso_tx: 0 > >> > dev.ixl.2.pf.que10.irqs: 0 > >> > dev.ixl.2.pf.que10.dropped: 0 > >> > dev.ixl.2.pf.que10.mbuf_defrag_failed: 0 > >> > dev.ixl.2.pf.que9.rx_bytes: 0 > >> > dev.ixl.2.pf.que9.rx_packets: 0 > >> > dev.ixl.2.pf.que9.tx_bytes: 0 > >> > dev.ixl.2.pf.que9.tx_packets: 0 > >> > dev.ixl.2.pf.que9.no_desc_avail: 0 > >> > dev.ixl.2.pf.que9.tx_dma_setup: 0 > >> > dev.ixl.2.pf.que9.tso_tx: 0 > >> > dev.ixl.2.pf.que9.irqs: 0 > >> > dev.ixl.2.pf.que9.dropped: 0 > >> > dev.ixl.2.pf.que9.mbuf_defrag_failed: 0 > >> > dev.ixl.2.pf.que8.rx_bytes: 0 > >> > dev.ixl.2.pf.que8.rx_packets: 0 > >> > dev.ixl.2.pf.que8.tx_bytes: 0 > >> > dev.ixl.2.pf.que8.tx_packets: 0 > >> > dev.ixl.2.pf.que8.no_desc_avail: 0 > >> > dev.ixl.2.pf.que8.tx_dma_setup: 0 > >> > dev.ixl.2.pf.que8.tso_tx: 0 > >> > dev.ixl.2.pf.que8.irqs: 0 > >> > dev.ixl.2.pf.que8.dropped: 0 > >> > dev.ixl.2.pf.que8.mbuf_defrag_failed: 0 > >> > dev.ixl.2.pf.que7.rx_bytes: 0 > >> > dev.ixl.2.pf.que7.rx_packets: 0 > >> > dev.ixl.2.pf.que7.tx_bytes: 0 > >> > dev.ixl.2.pf.que7.tx_packets: 0 > >> > dev.ixl.2.pf.que7.no_desc_avail: 0 > >> > dev.ixl.2.pf.que7.tx_dma_setup: 0 > >> > dev.ixl.2.pf.que7.tso_tx: 0 > >> > dev.ixl.2.pf.que7.irqs: 0 > >> > dev.ixl.2.pf.que7.dropped: 0 > >> > dev.ixl.2.pf.que7.mbuf_defrag_failed: 0 > >> > dev.ixl.2.pf.que6.rx_bytes: 0 > >> > dev.ixl.2.pf.que6.rx_packets: 0 > >> > dev.ixl.2.pf.que6.tx_bytes: 0 > >> > dev.ixl.2.pf.que6.tx_packets: 0 > >> > dev.ixl.2.pf.que6.no_desc_avail: 0 > >> > dev.ixl.2.pf.que6.tx_dma_setup: 0 > >> > dev.ixl.2.pf.que6.tso_tx: 0 > >> > dev.ixl.2.pf.que6.irqs: 0 > >> > dev.ixl.2.pf.que6.dropped: 0 > >> > dev.ixl.2.pf.que6.mbuf_defrag_failed: 0 > >> > dev.ixl.2.pf.que5.rx_bytes: 0 > >> > dev.ixl.2.pf.que5.rx_packets: 0 > >> > dev.ixl.2.pf.que5.tx_bytes: 0 > >> > dev.ixl.2.pf.que5.tx_packets: 0 > >> > dev.ixl.2.pf.que5.no_desc_avail: 0 > >> > dev.ixl.2.pf.que5.tx_dma_setup: 0 > >> > dev.ixl.2.pf.que5.tso_tx: 0 > >> > dev.ixl.2.pf.que5.irqs: 0 > >> > dev.ixl.2.pf.que5.dropped: 0 > >> > dev.ixl.2.pf.que5.mbuf_defrag_failed: 0 > >> > dev.ixl.2.pf.que4.rx_bytes: 0 > >> > dev.ixl.2.pf.que4.rx_packets: 0 > >> > dev.ixl.2.pf.que4.tx_bytes: 0 > >> > dev.ixl.2.pf.que4.tx_packets: 0 > >> > dev.ixl.2.pf.que4.no_desc_avail: 0 > >> > dev.ixl.2.pf.que4.tx_dma_setup: 0 > >> > dev.ixl.2.pf.que4.tso_tx: 0 > >> > dev.ixl.2.pf.que4.irqs: 0 > >> > dev.ixl.2.pf.que4.dropped: 0 > >> > dev.ixl.2.pf.que4.mbuf_defrag_failed: 0 > >> > dev.ixl.2.pf.que3.rx_bytes: 0 > >> > dev.ixl.2.pf.que3.rx_packets: 0 > >> > dev.ixl.2.pf.que3.tx_bytes: 0 > >> > dev.ixl.2.pf.que3.tx_packets: 0 > >> > dev.ixl.2.pf.que3.no_desc_avail: 0 > >> > dev.ixl.2.pf.que3.tx_dma_setup: 0 > >> > dev.ixl.2.pf.que3.tso_tx: 0 > >> > dev.ixl.2.pf.que3.irqs: 0 > >> > dev.ixl.2.pf.que3.dropped: 0 > >> > dev.ixl.2.pf.que3.mbuf_defrag_failed: 0 > >> > dev.ixl.2.pf.que2.rx_bytes: 0 > >> > dev.ixl.2.pf.que2.rx_packets: 0 > >> > dev.ixl.2.pf.que2.tx_bytes: 0 > >> > dev.ixl.2.pf.que2.tx_packets: 0 > >> > dev.ixl.2.pf.que2.no_desc_avail: 0 > >> > dev.ixl.2.pf.que2.tx_dma_setup: 0 > >> > dev.ixl.2.pf.que2.tso_tx: 0 > >> > dev.ixl.2.pf.que2.irqs: 0 > >> > dev.ixl.2.pf.que2.dropped: 0 > >> > dev.ixl.2.pf.que2.mbuf_defrag_failed: 0 > >> > dev.ixl.2.pf.que1.rx_bytes: 0 > >> > dev.ixl.2.pf.que1.rx_packets: 0 > >> > dev.ixl.2.pf.que1.tx_bytes: 0 > >> > dev.ixl.2.pf.que1.tx_packets: 0 > >> > dev.ixl.2.pf.que1.no_desc_avail: 0 > >> > dev.ixl.2.pf.que1.tx_dma_setup: 0 > >> > dev.ixl.2.pf.que1.tso_tx: 0 > >> > dev.ixl.2.pf.que1.irqs: 0 > >> > dev.ixl.2.pf.que1.dropped: 0 > >> > dev.ixl.2.pf.que1.mbuf_defrag_failed: 0 > >> > dev.ixl.2.pf.que0.rx_bytes: 0 > >> > dev.ixl.2.pf.que0.rx_packets: 0 > >> > dev.ixl.2.pf.que0.tx_bytes: 0 > >> > dev.ixl.2.pf.que0.tx_packets: 0 > >> > dev.ixl.2.pf.que0.no_desc_avail: 0 > >> > dev.ixl.2.pf.que0.tx_dma_setup: 0 > >> > dev.ixl.2.pf.que0.tso_tx: 0 > >> > dev.ixl.2.pf.que0.irqs: 0 > >> > dev.ixl.2.pf.que0.dropped: 0 > >> > dev.ixl.2.pf.que0.mbuf_defrag_failed: 0 > >> > dev.ixl.2.pf.bcast_pkts_txd: 0 > >> > dev.ixl.2.pf.mcast_pkts_txd: 0 > >> > dev.ixl.2.pf.ucast_pkts_txd: 0 > >> > dev.ixl.2.pf.good_octets_txd: 0 > >> > dev.ixl.2.pf.rx_discards: 0 > >> > dev.ixl.2.pf.bcast_pkts_rcvd: 0 > >> > dev.ixl.2.pf.mcast_pkts_rcvd: 0 > >> > dev.ixl.2.pf.ucast_pkts_rcvd: 0 > >> > dev.ixl.2.pf.good_octets_rcvd: 0 > >> > dev.ixl.2.vc_debug_level: 1 > >> > dev.ixl.2.admin_irq: 0 > >> > dev.ixl.2.watchdog_events: 0 > >> > dev.ixl.2.debug: 0 > >> > dev.ixl.2.dynamic_tx_itr: 0 > >> > dev.ixl.2.tx_itr: 122 > >> > dev.ixl.2.dynamic_rx_itr: 0 > >> > dev.ixl.2.rx_itr: 62 > >> > dev.ixl.2.fw_version: f4.33 a1.2 n04.42 e8000191d > >> > dev.ixl.2.current_speed: Unknown > >> > dev.ixl.2.advertise_speed: 0 > >> > dev.ixl.2.fc: 0 > >> > dev.ixl.2.%parent: pci129 > >> > dev.ixl.2.%pnpinfo: vendor=0x8086 device=0x1572 > subvendor=0x8086 > >> > subdevice=0x0000 class=0x020000 > >> > dev.ixl.2.%location: slot=0 function=2 > handle=\_SB_.PCI1.QR3A.H002 > >> > dev.ixl.2.%driver: ixl > >> > dev.ixl.2.%desc: Intel(R) Ethernet Connection XL710 Driver, > >> Version - 1.4.0 > >> > dev.ixl.1.mac.xoff_recvd: 0 > >> > dev.ixl.1.mac.xoff_txd: 0 > >> > dev.ixl.1.mac.xon_recvd: 0 > >> > dev.ixl.1.mac.xon_txd: 0 > >> > dev.ixl.1.mac.tx_frames_big: 0 > >> > dev.ixl.1.mac.tx_frames_1024_1522: 1565670684 > >> > dev.ixl.1.mac.tx_frames_512_1023: 101286418 > >> > dev.ixl.1.mac.tx_frames_256_511: 49713129 > >> > dev.ixl.1.mac.tx_frames_128_255: 231617277 > >> > dev.ixl.1.mac.tx_frames_65_127: 2052767669 > >> > dev.ixl.1.mac.tx_frames_64: 1318689044 > >> > dev.ixl.1.mac.checksum_errors: 0 > >> > dev.ixl.1.mac.rx_jabber: 0 > >> > dev.ixl.1.mac.rx_oversized: 0 > >> > dev.ixl.1.mac.rx_fragmented: 0 > >> > dev.ixl.1.mac.rx_undersize: 0 > >> > dev.ixl.1.mac.rx_frames_big: 0 > >> > dev.ixl.1.mac.rx_frames_1024_1522: 4960403414 > >> > dev.ixl.1.mac.rx_frames_512_1023: 113675084 > >> > dev.ixl.1.mac.rx_frames_256_511: 253904920 > >> > dev.ixl.1.mac.rx_frames_128_255: 196369726 > >> > dev.ixl.1.mac.rx_frames_65_127: 1436626211 > >> > dev.ixl.1.mac.rx_frames_64: 242768681 > >> > dev.ixl.1.mac.rx_length_errors: 0 > >> > dev.ixl.1.mac.remote_faults: 0 > >> > dev.ixl.1.mac.local_faults: 0 > >> > dev.ixl.1.mac.illegal_bytes: 0 > >> > dev.ixl.1.mac.crc_errors: 0 > >> > dev.ixl.1.mac.bcast_pkts_txd: 277 > >> > dev.ixl.1.mac.mcast_pkts_txd: 0 > >> > dev.ixl.1.mac.ucast_pkts_txd: 5319743942 > >> > dev.ixl.1.mac.good_octets_txd: 2642351885737 > >> > dev.ixl.1.mac.rx_discards: 0 > >> > dev.ixl.1.mac.bcast_pkts_rcvd: 5 > >> > dev.ixl.1.mac.mcast_pkts_rcvd: 144 > >> > dev.ixl.1.mac.ucast_pkts_rcvd: 7203747879 > >> > dev.ixl.1.mac.good_octets_rcvd: 7770230492434 > >> > dev.ixl.1.pf.que23.rx_bytes: 0 > >> > dev.ixl.1.pf.que23.rx_packets: 0 > >> > dev.ixl.1.pf.que23.tx_bytes: 7111 > >> > dev.ixl.1.pf.que23.tx_packets: 88 > >> > dev.ixl.1.pf.que23.no_desc_avail: 0 > >> > dev.ixl.1.pf.que23.tx_dma_setup: 0 > >> > dev.ixl.1.pf.que23.tso_tx: 0 > >> > dev.ixl.1.pf.que23.irqs: 88 > >> > dev.ixl.1.pf.que23.dropped: 0 > >> > dev.ixl.1.pf.que23.mbuf_defrag_failed: 0 > >> > dev.ixl.1.pf.que22.rx_bytes: 0 > >> > dev.ixl.1.pf.que22.rx_packets: 0 > >> > dev.ixl.1.pf.que22.tx_bytes: 6792 > >> > dev.ixl.1.pf.que22.tx_packets: 88 > >> > dev.ixl.1.pf.que22.no_desc_avail: 0 > >> > dev.ixl.1.pf.que22.tx_dma_setup: 0 > >> > dev.ixl.1.pf.que22.tso_tx: 0 > >> > dev.ixl.1.pf.que22.irqs: 89 > >> > dev.ixl.1.pf.que22.dropped: 0 > >> > dev.ixl.1.pf.que22.mbuf_defrag_failed: 0 > >> > dev.ixl.1.pf.que21.rx_bytes: 0 > >> > dev.ixl.1.pf.que21.rx_packets: 0 > >> > dev.ixl.1.pf.que21.tx_bytes: 7486 > >> > dev.ixl.1.pf.que21.tx_packets: 93 > >> > dev.ixl.1.pf.que21.no_desc_avail: 0 > >> > dev.ixl.1.pf.que21.tx_dma_setup: 0 > >> > dev.ixl.1.pf.que21.tso_tx: 0 > >> > dev.ixl.1.pf.que21.irqs: 95 > >> > dev.ixl.1.pf.que21.dropped: 0 > >> > dev.ixl.1.pf.que21.mbuf_defrag_failed: 0 > >> > dev.ixl.1.pf.que20.rx_bytes: 0 > >> > dev.ixl.1.pf.que20.rx_packets: 0 > >> > dev.ixl.1.pf.que20.tx_bytes: 7850 > >> > dev.ixl.1.pf.que20.tx_packets: 98 > >> > dev.ixl.1.pf.que20.no_desc_avail: 0 > >> > dev.ixl.1.pf.que20.tx_dma_setup: 0 > >> > dev.ixl.1.pf.que20.tso_tx: 0 > >> > dev.ixl.1.pf.que20.irqs: 99 > >> > dev.ixl.1.pf.que20.dropped: 0 > >> > dev.ixl.1.pf.que20.mbuf_defrag_failed: 0 > >> > dev.ixl.1.pf.que19.rx_bytes: 0 > >> > dev.ixl.1.pf.que19.rx_packets: 0 > >> > dev.ixl.1.pf.que19.tx_bytes: 64643 > >> > dev.ixl.1.pf.que19.tx_packets: 202 > >> > dev.ixl.1.pf.que19.no_desc_avail: 0 > >> > dev.ixl.1.pf.que19.tx_dma_setup: 0 > >> > dev.ixl.1.pf.que19.tso_tx: 0 > >> > dev.ixl.1.pf.que19.irqs: 202 > >> > dev.ixl.1.pf.que19.dropped: 0 > >> > dev.ixl.1.pf.que19.mbuf_defrag_failed: 0 > >> > dev.ixl.1.pf.que18.rx_bytes: 0 > >> > dev.ixl.1.pf.que18.rx_packets: 0 > >> > dev.ixl.1.pf.que18.tx_bytes: 5940 > >> > dev.ixl.1.pf.que18.tx_packets: 74 > >> > dev.ixl.1.pf.que18.no_desc_avail: 0 > >> > dev.ixl.1.pf.que18.tx_dma_setup: 0 > >> > dev.ixl.1.pf.que18.tso_tx: 0 > >> > dev.ixl.1.pf.que18.irqs: 74 > >> > dev.ixl.1.pf.que18.dropped: 0 > >> > dev.ixl.1.pf.que18.mbuf_defrag_failed: 0 > >> > dev.ixl.1.pf.que17.rx_bytes: 0 > >> > dev.ixl.1.pf.que17.rx_packets: 0 > >> > dev.ixl.1.pf.que17.tx_bytes: 11675 > >> > dev.ixl.1.pf.que17.tx_packets: 83 > >> > dev.ixl.1.pf.que17.no_desc_avail: 0 > >> > dev.ixl.1.pf.que17.tx_dma_setup: 0 > >> > dev.ixl.1.pf.que17.tso_tx: 0 > >> > dev.ixl.1.pf.que17.irqs: 83 > >> > dev.ixl.1.pf.que17.dropped: 0 > >> > dev.ixl.1.pf.que17.mbuf_defrag_failed: 0 > >> > dev.ixl.1.pf.que16.rx_bytes: 0 > >> > dev.ixl.1.pf.que16.rx_packets: 0 > >> > dev.ixl.1.pf.que16.tx_bytes: 105750457831 > >> > dev.ixl.1.pf.que16.tx_packets: 205406766 > >> > dev.ixl.1.pf.que16.no_desc_avail: 0 > >> > dev.ixl.1.pf.que16.tx_dma_setup: 0 > >> > dev.ixl.1.pf.que16.tso_tx: 0 > >> > dev.ixl.1.pf.que16.irqs: 87222978 > >> > dev.ixl.1.pf.que16.dropped: 0 > >> > dev.ixl.1.pf.que16.mbuf_defrag_failed: 0 > >> > dev.ixl.1.pf.que15.rx_bytes: 289558174088 > >> > dev.ixl.1.pf.que15.rx_packets: 272466190 > >> > dev.ixl.1.pf.que15.tx_bytes: 106152524681 > >> > dev.ixl.1.pf.que15.tx_packets: 205379247 > >> > dev.ixl.1.pf.que15.no_desc_avail: 0 > >> > dev.ixl.1.pf.que15.tx_dma_setup: 0 > >> > dev.ixl.1.pf.que15.tso_tx: 0 > >> > dev.ixl.1.pf.que15.irqs: 238145862 > >> > dev.ixl.1.pf.que15.dropped: 0 > >> > dev.ixl.1.pf.que15.mbuf_defrag_failed: 0 > >> > dev.ixl.1.pf.que14.rx_bytes: 301934533473 > >> > dev.ixl.1.pf.que14.rx_packets: 298452930 > >> > dev.ixl.1.pf.que14.tx_bytes: 111420393725 > >> > dev.ixl.1.pf.que14.tx_packets: 215722532 > >> > dev.ixl.1.pf.que14.no_desc_avail: 0 > >> > dev.ixl.1.pf.que14.tx_dma_setup: 0 > >> > dev.ixl.1.pf.que14.tso_tx: 0 > >> > dev.ixl.1.pf.que14.irqs: 256291617 > >> > dev.ixl.1.pf.que14.dropped: 0 > >> > dev.ixl.1.pf.que14.mbuf_defrag_failed: 0 > >> > dev.ixl.1.pf.que13.rx_bytes: 291380746253 > >> > dev.ixl.1.pf.que13.rx_packets: 273037957 > >> > dev.ixl.1.pf.que13.tx_bytes: 112417776222 > >> > dev.ixl.1.pf.que13.tx_packets: 217500943 > >> > dev.ixl.1.pf.que13.no_desc_avail: 0 > >> > dev.ixl.1.pf.que13.tx_dma_setup: 0 > >> > dev.ixl.1.pf.que13.tso_tx: 0 > >> > dev.ixl.1.pf.que13.irqs: 241422331 > >> > dev.ixl.1.pf.que13.dropped: 0 > >> > dev.ixl.1.pf.que13.mbuf_defrag_failed: 0 > >> > dev.ixl.1.pf.que12.rx_bytes: 301105585425 > >> > dev.ixl.1.pf.que12.rx_packets: 286137817 > >> > dev.ixl.1.pf.que12.tx_bytes: 95851784579 > >> > dev.ixl.1.pf.que12.tx_packets: 199715765 > >> > dev.ixl.1.pf.que12.no_desc_avail: 0 > >> > dev.ixl.1.pf.que12.tx_dma_setup: 0 > >> > dev.ixl.1.pf.que12.tso_tx: 0 > >> > dev.ixl.1.pf.que12.irqs: 247322880 > >> > dev.ixl.1.pf.que12.dropped: 0 > >> > dev.ixl.1.pf.que12.mbuf_defrag_failed: 0 > >> > dev.ixl.1.pf.que11.rx_bytes: 307105398143 > >> > dev.ixl.1.pf.que11.rx_packets: 281046463 > >> > dev.ixl.1.pf.que11.tx_bytes: 110710957789 > >> > dev.ixl.1.pf.que11.tx_packets: 211784031 > >> > dev.ixl.1.pf.que11.no_desc_avail: 0 > >> > dev.ixl.1.pf.que11.tx_dma_setup: 0 > >> > dev.ixl.1.pf.que11.tso_tx: 0 > >> > dev.ixl.1.pf.que11.irqs: 256987179 > >> > dev.ixl.1.pf.que11.dropped: 0 > >> > dev.ixl.1.pf.que11.mbuf_defrag_failed: 0 > >> > dev.ixl.1.pf.que10.rx_bytes: 304288000453 > >> > dev.ixl.1.pf.que10.rx_packets: 278987858 > >> > dev.ixl.1.pf.que10.tx_bytes: 93022244338 > >> > dev.ixl.1.pf.que10.tx_packets: 195869210 > >> > dev.ixl.1.pf.que10.no_desc_avail: 0 > >> > dev.ixl.1.pf.que10.tx_dma_setup: 0 > >> > dev.ixl.1.pf.que10.tso_tx: 0 > >> > dev.ixl.1.pf.que10.irqs: 253622192 > >> > dev.ixl.1.pf.que10.dropped: 0 > >> > dev.ixl.1.pf.que10.mbuf_defrag_failed: 0 > >> > dev.ixl.1.pf.que9.rx_bytes: 320340203822 > >> > dev.ixl.1.pf.que9.rx_packets: 302309010 > >> > dev.ixl.1.pf.que9.tx_bytes: 116604776460 > >> > dev.ixl.1.pf.que9.tx_packets: 223949025 > >> > dev.ixl.1.pf.que9.no_desc_avail: 0 > >> > dev.ixl.1.pf.que9.tx_dma_setup: 0 > >> > dev.ixl.1.pf.que9.tso_tx: 0 > >> > dev.ixl.1.pf.que9.irqs: 271165440 > >> > dev.ixl.1.pf.que9.dropped: 0 > >> > dev.ixl.1.pf.que9.mbuf_defrag_failed: 0 > >> > dev.ixl.1.pf.que8.rx_bytes: 291403725592 > >> > dev.ixl.1.pf.que8.rx_packets: 267859568 > >> > dev.ixl.1.pf.que8.tx_bytes: 205745654558 > >> > dev.ixl.1.pf.que8.tx_packets: 443349835 > >> > dev.ixl.1.pf.que8.no_desc_avail: 0 > >> > dev.ixl.1.pf.que8.tx_dma_setup: 0 > >> > dev.ixl.1.pf.que8.tso_tx: 0 > >> > dev.ixl.1.pf.que8.irqs: 254116755 > >> > dev.ixl.1.pf.que8.dropped: 0 > >> > dev.ixl.1.pf.que8.mbuf_defrag_failed: 0 > >> > dev.ixl.1.pf.que7.rx_bytes: 673363127346 > >> > dev.ixl.1.pf.que7.rx_packets: 617269774 > >> > dev.ixl.1.pf.que7.tx_bytes: 203162891886 > >> > dev.ixl.1.pf.que7.tx_packets: 443709339 > >> > dev.ixl.1.pf.que7.no_desc_avail: 0 > >> > dev.ixl.1.pf.que7.tx_dma_setup: 0 > >> > dev.ixl.1.pf.que7.tso_tx: 0 > >> > dev.ixl.1.pf.que7.irqs: 424706771 > >> > dev.ixl.1.pf.que7.dropped: 0 > >> > dev.ixl.1.pf.que7.mbuf_defrag_failed: 0 > >> > dev.ixl.1.pf.que6.rx_bytes: 644709094218 > >> > dev.ixl.1.pf.que6.rx_packets: 601892919 > >> > dev.ixl.1.pf.que6.tx_bytes: 221661735032 > >> > dev.ixl.1.pf.que6.tx_packets: 460127064 > >> > dev.ixl.1.pf.que6.no_desc_avail: 0 > >> > dev.ixl.1.pf.que6.tx_dma_setup: 0 > >> > dev.ixl.1.pf.que6.tso_tx: 0 > >> > dev.ixl.1.pf.que6.irqs: 417748074 > >> > dev.ixl.1.pf.que6.dropped: 0 > >> > dev.ixl.1.pf.que6.mbuf_defrag_failed: 0 > >> > dev.ixl.1.pf.que5.rx_bytes: 661904432231 > >> > dev.ixl.1.pf.que5.rx_packets: 622012837 > >> > dev.ixl.1.pf.que5.tx_bytes: 230514282876 > >> > dev.ixl.1.pf.que5.tx_packets: 458571100 > >> > dev.ixl.1.pf.que5.no_desc_avail: 0 > >> > dev.ixl.1.pf.que5.tx_dma_setup: 0 > >> > dev.ixl.1.pf.que5.tso_tx: 0 > >> > dev.ixl.1.pf.que5.irqs: 422305039 > >> > dev.ixl.1.pf.que5.dropped: 0 > >> > dev.ixl.1.pf.que5.mbuf_defrag_failed: 0 > >> > dev.ixl.1.pf.que4.rx_bytes: 653522179234 > >> > dev.ixl.1.pf.que4.rx_packets: 603345546 > >> > dev.ixl.1.pf.que4.tx_bytes: 216761219483 > >> > dev.ixl.1.pf.que4.tx_packets: 450329641 > >> > dev.ixl.1.pf.que4.no_desc_avail: 0 > >> > dev.ixl.1.pf.que4.tx_dma_setup: 0 > >> > dev.ixl.1.pf.que4.tso_tx: 3 > >> > dev.ixl.1.pf.que4.irqs: 416920533 > >> > dev.ixl.1.pf.que4.dropped: 0 > >> > dev.ixl.1.pf.que4.mbuf_defrag_failed: 0 > >> > dev.ixl.1.pf.que3.rx_bytes: 676494225882 > >> > dev.ixl.1.pf.que3.rx_packets: 620605168 > >> > dev.ixl.1.pf.que3.tx_bytes: 233854020454 > >> > dev.ixl.1.pf.que3.tx_packets: 464425616 > >> > dev.ixl.1.pf.que3.no_desc_avail: 0 > >> > dev.ixl.1.pf.que3.tx_dma_setup: 0 > >> > dev.ixl.1.pf.que3.tso_tx: 0 > >> > dev.ixl.1.pf.que3.irqs: 426349030 > >> > dev.ixl.1.pf.que3.dropped: 0 > >> > dev.ixl.1.pf.que3.mbuf_defrag_failed: 0 > >> > dev.ixl.1.pf.que2.rx_bytes: 677779337711 > >> > dev.ixl.1.pf.que2.rx_packets: 620883699 > >> > dev.ixl.1.pf.que2.tx_bytes: 211297141668 > >> > dev.ixl.1.pf.que2.tx_packets: 450501525 > >> > dev.ixl.1.pf.que2.no_desc_avail: 0 > >> > dev.ixl.1.pf.que2.tx_dma_setup: 0 > >> > dev.ixl.1.pf.que2.tso_tx: 0 > >> > dev.ixl.1.pf.que2.irqs: 433146278 > >> > dev.ixl.1.pf.que2.dropped: 0 > >> > dev.ixl.1.pf.que2.mbuf_defrag_failed: 0 > >> > dev.ixl.1.pf.que1.rx_bytes: 661360798018 > >> > dev.ixl.1.pf.que1.rx_packets: 619700636 > >> > dev.ixl.1.pf.que1.tx_bytes: 238264220772 > >> > dev.ixl.1.pf.que1.tx_packets: 473425354 > >> > dev.ixl.1.pf.que1.no_desc_avail: 0 > >> > dev.ixl.1.pf.que1.tx_dma_setup: 0 > >> > dev.ixl.1.pf.que1.tso_tx: 0 > >> > dev.ixl.1.pf.que1.irqs: 437959829 > >> > dev.ixl.1.pf.que1.dropped: 0 > >> > dev.ixl.1.pf.que1.mbuf_defrag_failed: 0 > >> > dev.ixl.1.pf.que0.rx_bytes: 685201226330 > >> > dev.ixl.1.pf.que0.rx_packets: 637772348 > >> > dev.ixl.1.pf.que0.tx_bytes: 124808 > >> > dev.ixl.1.pf.que0.tx_packets: 1782 > >> > dev.ixl.1.pf.que0.no_desc_avail: 0 > >> > dev.ixl.1.pf.que0.tx_dma_setup: 0 > >> > dev.ixl.1.pf.que0.tso_tx: 0 > >> > dev.ixl.1.pf.que0.irqs: 174905480 > >> > dev.ixl.1.pf.que0.dropped: 0 > >> > dev.ixl.1.pf.que0.mbuf_defrag_failed: 0 > >> > dev.ixl.1.pf.bcast_pkts_txd: 277 > >> > dev.ixl.1.pf.mcast_pkts_txd: 0 > >> > dev.ixl.1.pf.ucast_pkts_txd: 5319743945 > >> > dev.ixl.1.pf.good_octets_txd: 2613178367282 > >> > dev.ixl.1.pf.rx_discards: 0 > >> > dev.ixl.1.pf.bcast_pkts_rcvd: 1 > >> > dev.ixl.1.pf.mcast_pkts_rcvd: 0 > >> > dev.ixl.1.pf.ucast_pkts_rcvd: 7203747890 > >> > dev.ixl.1.pf.good_octets_rcvd: 7770230490224 > >> > dev.ixl.1.vc_debug_level: 1 > >> > dev.ixl.1.admin_irq: 0 > >> > dev.ixl.1.watchdog_events: 0 > >> > dev.ixl.1.debug: 0 > >> > dev.ixl.1.dynamic_tx_itr: 0 > >> > dev.ixl.1.tx_itr: 122 > >> > dev.ixl.1.dynamic_rx_itr: 0 > >> > dev.ixl.1.rx_itr: 62 > >> > dev.ixl.1.fw_version: f4.33 a1.2 n04.42 e8000191d > >> > dev.ixl.1.current_speed: 10G > >> > dev.ixl.1.advertise_speed: 0 > >> > dev.ixl.1.fc: 0 > >> > dev.ixl.1.%parent: pci129 > >> > dev.ixl.1.%pnpinfo: vendor=0x8086 device=0x1572 > subvendor=0x8086 > >> > subdevice=0x0000 class=0x020000 > >> > dev.ixl.1.%location: slot=0 function=1 > handle=\_SB_.PCI1.QR3A.H001 > >> > dev.ixl.1.%driver: ixl > >> > dev.ixl.1.%desc: Intel(R) Ethernet Connection XL710 Driver, > >> Version - 1.4.0 > >> > dev.ixl.0.mac.xoff_recvd: 0 > >> > dev.ixl.0.mac.xoff_txd: 0 > >> > dev.ixl.0.mac.xon_recvd: 0 > >> > dev.ixl.0.mac.xon_txd: 0 > >> > dev.ixl.0.mac.tx_frames_big: 0 > >> > dev.ixl.0.mac.tx_frames_1024_1522: 4961134019 > >> > dev.ixl.0.mac.tx_frames_512_1023: 113082136 > >> > dev.ixl.0.mac.tx_frames_256_511: 123538450 > >> > dev.ixl.0.mac.tx_frames_128_255: 185051082 > >> > dev.ixl.0.mac.tx_frames_65_127: 1332798493 > >> > dev.ixl.0.mac.tx_frames_64: 243338964 > >> > dev.ixl.0.mac.checksum_errors: 0 > >> > dev.ixl.0.mac.rx_jabber: 0 > >> > dev.ixl.0.mac.rx_oversized: 0 > >> > dev.ixl.0.mac.rx_fragmented: 0 > >> > dev.ixl.0.mac.rx_undersize: 0 > >> > dev.ixl.0.mac.rx_frames_big: 0 > >> > dev.ixl.0.mac.rx_frames_1024_1522: 1566499069 > >> > dev.ixl.0.mac.rx_frames_512_1023: 101390143 > >> > dev.ixl.0.mac.rx_frames_256_511: 49831970 > >> > dev.ixl.0.mac.rx_frames_128_255: 231738168 > >> > dev.ixl.0.mac.rx_frames_65_127: 2123185819 > >> > dev.ixl.0.mac.rx_frames_64: 1320404300 > >> > dev.ixl.0.mac.rx_length_errors: 0 > >> > dev.ixl.0.mac.remote_faults: 0 > >> > dev.ixl.0.mac.local_faults: 0 > >> > dev.ixl.0.mac.illegal_bytes: 0 > >> > dev.ixl.0.mac.crc_errors: 0 > >> > dev.ixl.0.mac.bcast_pkts_txd: 302 > >> > dev.ixl.0.mac.mcast_pkts_txd: 33965 > >> > dev.ixl.0.mac.ucast_pkts_txd: 6958908862 > >> > dev.ixl.0.mac.good_octets_txd: 7698936138858 > >> > dev.ixl.0.mac.rx_discards: 0 > >> > dev.ixl.0.mac.bcast_pkts_rcvd: 1 > >> > dev.ixl.0.mac.mcast_pkts_rcvd: 49693 > >> > dev.ixl.0.mac.ucast_pkts_rcvd: 5392999771 > >> > dev.ixl.0.mac.good_octets_rcvd: 2648906893811 > >> > dev.ixl.0.pf.que23.rx_bytes: 0 > >> > dev.ixl.0.pf.que23.rx_packets: 0 > >> > dev.ixl.0.pf.que23.tx_bytes: 2371273 > >> > dev.ixl.0.pf.que23.tx_packets: 7313 > >> > dev.ixl.0.pf.que23.no_desc_avail: 0 > >> > dev.ixl.0.pf.que23.tx_dma_setup: 0 > >> > dev.ixl.0.pf.que23.tso_tx: 0 > >> > dev.ixl.0.pf.que23.irqs: 7313 > >> > dev.ixl.0.pf.que23.dropped: 0 > >> > dev.ixl.0.pf.que23.mbuf_defrag_failed: 0 > >> > dev.ixl.0.pf.que22.rx_bytes: 0 > >> > dev.ixl.0.pf.que22.rx_packets: 0 > >> > dev.ixl.0.pf.que22.tx_bytes: 1908468 > >> > dev.ixl.0.pf.que22.tx_packets: 6626 > >> > dev.ixl.0.pf.que22.no_desc_avail: 0 > >> > dev.ixl.0.pf.que22.tx_dma_setup: 0 > >> > dev.ixl.0.pf.que22.tso_tx: 0 > >> > dev.ixl.0.pf.que22.irqs: 6627 > >> > dev.ixl.0.pf.que22.dropped: 0 > >> > dev.ixl.0.pf.que22.mbuf_defrag_failed: 0 > >> > dev.ixl.0.pf.que21.rx_bytes: 0 > >> > dev.ixl.0.pf.que21.rx_packets: 0 > >> > dev.ixl.0.pf.que21.tx_bytes: 2092668 > >> > dev.ixl.0.pf.que21.tx_packets: 6739 > >> > dev.ixl.0.pf.que21.no_desc_avail: 0 > >> > dev.ixl.0.pf.que21.tx_dma_setup: 0 > >> > dev.ixl.0.pf.que21.tso_tx: 0 > >> > dev.ixl.0.pf.que21.irqs: 6728 > >> > dev.ixl.0.pf.que21.dropped: 0 > >> > dev.ixl.0.pf.que21.mbuf_defrag_failed: 0 > >> > dev.ixl.0.pf.que20.rx_bytes: 0 > >> > dev.ixl.0.pf.que20.rx_packets: 0 > >> > dev.ixl.0.pf.que20.tx_bytes: 1742176 > >> > dev.ixl.0.pf.que20.tx_packets: 6246 > >> > dev.ixl.0.pf.que20.no_desc_avail: 0 > >> > dev.ixl.0.pf.que20.tx_dma_setup: 0 > >> > dev.ixl.0.pf.que20.tso_tx: 0 > >> > dev.ixl.0.pf.que20.irqs: 6249 > >> > dev.ixl.0.pf.que20.dropped: 0 > >> > dev.ixl.0.pf.que20.mbuf_defrag_failed: 0 > >> > dev.ixl.0.pf.que19.rx_bytes: 0 > >> > dev.ixl.0.pf.que19.rx_packets: 0 > >> > dev.ixl.0.pf.que19.tx_bytes: 2102284 > >> > dev.ixl.0.pf.que19.tx_packets: 6979 > >> > dev.ixl.0.pf.que19.no_desc_avail: 0 > >> > dev.ixl.0.pf.que19.tx_dma_setup: 0 > >> > dev.ixl.0.pf.que19.tso_tx: 0 > >> > dev.ixl.0.pf.que19.irqs: 6979 > >> > dev.ixl.0.pf.que19.dropped: 0 > >> > dev.ixl.0.pf.que19.mbuf_defrag_failed: 0 > >> > dev.ixl.0.pf.que18.rx_bytes: 0 > >> > dev.ixl.0.pf.que18.rx_packets: 0 > >> > dev.ixl.0.pf.que18.tx_bytes: 1532360 > >> > dev.ixl.0.pf.que18.tx_packets: 5588 > >> > dev.ixl.0.pf.que18.no_desc_avail: 0 > >> > dev.ixl.0.pf.que18.tx_dma_setup: 0 > >> > dev.ixl.0.pf.que18.tso_tx: 0 > >> > dev.ixl.0.pf.que18.irqs: 5588 > >> > dev.ixl.0.pf.que18.dropped: 0 > >> > dev.ixl.0.pf.que18.mbuf_defrag_failed: 0 > >> > dev.ixl.0.pf.que17.rx_bytes: 0 > >> > dev.ixl.0.pf.que17.rx_packets: 0 > >> > dev.ixl.0.pf.que17.tx_bytes: 1809684 > >> > dev.ixl.0.pf.que17.tx_packets: 6136 > >> > dev.ixl.0.pf.que17.no_desc_avail: 0 > >> > dev.ixl.0.pf.que17.tx_dma_setup: 0 > >> > dev.ixl.0.pf.que17.tso_tx: 0 > >> > dev.ixl.0.pf.que17.irqs: 6136 > >> > dev.ixl.0.pf.que17.dropped: 0 > >> > dev.ixl.0.pf.que17.mbuf_defrag_failed: 0 > >> > dev.ixl.0.pf.que16.rx_bytes: 0 > >> > dev.ixl.0.pf.que16.rx_packets: 0 > >> > dev.ixl.0.pf.que16.tx_bytes: 286836299105 > >> > dev.ixl.0.pf.que16.tx_packets: 263532601 > >> > dev.ixl.0.pf.que16.no_desc_avail: 0 > >> > dev.ixl.0.pf.que16.tx_dma_setup: 0 > >> > dev.ixl.0.pf.que16.tso_tx: 0 > >> > dev.ixl.0.pf.que16.irqs: 83232941 > >> > dev.ixl.0.pf.que16.dropped: 0 > >> > dev.ixl.0.pf.que16.mbuf_defrag_failed: 0 > >> > dev.ixl.0.pf.que15.rx_bytes: 106345323488 > >> > dev.ixl.0.pf.que15.rx_packets: 208869912 > >> > dev.ixl.0.pf.que15.tx_bytes: 298825179301 > >> > dev.ixl.0.pf.que15.tx_packets: 288517504 > >> > dev.ixl.0.pf.que15.no_desc_avail: 0 > >> > dev.ixl.0.pf.que15.tx_dma_setup: 0 > >> > dev.ixl.0.pf.que15.tso_tx: 0 > >> > dev.ixl.0.pf.que15.irqs: 223322408 > >> > dev.ixl.0.pf.que15.dropped: 0 > >> > dev.ixl.0.pf.que15.mbuf_defrag_failed: 0 > >> > dev.ixl.0.pf.que14.rx_bytes: 106721900547 > >> > dev.ixl.0.pf.que14.rx_packets: 208566121 > >> > dev.ixl.0.pf.que14.tx_bytes: 288657751920 > >> > dev.ixl.0.pf.que14.tx_packets: 263556000 > >> > dev.ixl.0.pf.que14.no_desc_avail: 0 > >> > dev.ixl.0.pf.que14.tx_dma_setup: 0 > >> > dev.ixl.0.pf.que14.tso_tx: 0 > >> > dev.ixl.0.pf.que14.irqs: 220377537 > >> > dev.ixl.0.pf.que14.dropped: 0 > >> > dev.ixl.0.pf.que14.mbuf_defrag_failed: 0 > >> > dev.ixl.0.pf.que13.rx_bytes: 111978971378 > >> > dev.ixl.0.pf.que13.rx_packets: 218447354 > >> > dev.ixl.0.pf.que13.tx_bytes: 298439860675 > >> > dev.ixl.0.pf.que13.tx_packets: 276806617 > >> > dev.ixl.0.pf.que13.no_desc_avail: 0 > >> > dev.ixl.0.pf.que13.tx_dma_setup: 0 > >> > dev.ixl.0.pf.que13.tso_tx: 0 > >> > dev.ixl.0.pf.que13.irqs: 227474625 > >> > dev.ixl.0.pf.que13.dropped: 0 > >> > dev.ixl.0.pf.que13.mbuf_defrag_failed: 0 > >> > dev.ixl.0.pf.que12.rx_bytes: 112969704706 > >> > dev.ixl.0.pf.que12.rx_packets: 220275562 > >> > dev.ixl.0.pf.que12.tx_bytes: 304750620079 > >> > dev.ixl.0.pf.que12.tx_packets: 272244483 > >> > dev.ixl.0.pf.que12.no_desc_avail: 0 > >> > dev.ixl.0.pf.que12.tx_dma_setup: 0 > >> > dev.ixl.0.pf.que12.tso_tx: 183 > >> > dev.ixl.0.pf.que12.irqs: 230111291 > >> > dev.ixl.0.pf.que12.dropped: 0 > >> > dev.ixl.0.pf.que12.mbuf_defrag_failed: 0 > >> > dev.ixl.0.pf.que11.rx_bytes: 96405343036 > >> > dev.ixl.0.pf.que11.rx_packets: 202329448 > >> > dev.ixl.0.pf.que11.tx_bytes: 302481707696 > >> > dev.ixl.0.pf.que11.tx_packets: 271689246 > >> > dev.ixl.0.pf.que11.no_desc_avail: 0 > >> > dev.ixl.0.pf.que11.tx_dma_setup: 0 > >> > dev.ixl.0.pf.que11.tso_tx: 0 > >> > dev.ixl.0.pf.que11.irqs: 220717612 > >> > dev.ixl.0.pf.que11.dropped: 0 > >> > dev.ixl.0.pf.que11.mbuf_defrag_failed: 0 > >> > dev.ixl.0.pf.que10.rx_bytes: 111280008670 > >> > dev.ixl.0.pf.que10.rx_packets: 214900261 > >> > dev.ixl.0.pf.que10.tx_bytes: 318638566198 > >> > dev.ixl.0.pf.que10.tx_packets: 295011389 > >> > dev.ixl.0.pf.que10.no_desc_avail: 0 > >> > dev.ixl.0.pf.que10.tx_dma_setup: 0 > >> > dev.ixl.0.pf.que10.tso_tx: 0 > >> > dev.ixl.0.pf.que10.irqs: 230681709 > >> > dev.ixl.0.pf.que10.dropped: 0 > >> > dev.ixl.0.pf.que10.mbuf_defrag_failed: 0 > >> > dev.ixl.0.pf.que9.rx_bytes: 93566025126 > >> > dev.ixl.0.pf.que9.rx_packets: 198726483 > >> > dev.ixl.0.pf.que9.tx_bytes: 288858818348 > >> > dev.ixl.0.pf.que9.tx_packets: 258926864 > >> > dev.ixl.0.pf.que9.no_desc_avail: 0 > >> > dev.ixl.0.pf.que9.tx_dma_setup: 0 > >> > dev.ixl.0.pf.que9.tso_tx: 0 > >> > dev.ixl.0.pf.que9.irqs: 217918160 > >> > dev.ixl.0.pf.que9.dropped: 0 > >> > dev.ixl.0.pf.que9.mbuf_defrag_failed: 0 > >> > dev.ixl.0.pf.que8.rx_bytes: 117169019041 > >> > dev.ixl.0.pf.que8.rx_packets: 226938172 > >> > dev.ixl.0.pf.que8.tx_bytes: 665794492752 > >> > dev.ixl.0.pf.que8.tx_packets: 593519436 > >> > dev.ixl.0.pf.que8.no_desc_avail: 0 > >> > dev.ixl.0.pf.que8.tx_dma_setup: 0 > >> > dev.ixl.0.pf.que8.tso_tx: 0 > >> > dev.ixl.0.pf.que8.irqs: 244643578 > >> > dev.ixl.0.pf.que8.dropped: 0 > >> > dev.ixl.0.pf.que8.mbuf_defrag_failed: 0 > >> > dev.ixl.0.pf.que7.rx_bytes: 206974266022 > >> > dev.ixl.0.pf.que7.rx_packets: 449899895 > >> > dev.ixl.0.pf.que7.tx_bytes: 638527685820 > >> > dev.ixl.0.pf.que7.tx_packets: 580750916 > >> > dev.ixl.0.pf.que7.no_desc_avail: 0 > >> > dev.ixl.0.pf.que7.tx_dma_setup: 0 > >> > dev.ixl.0.pf.que7.tso_tx: 0 > >> > dev.ixl.0.pf.que7.irqs: 391760959 > >> > dev.ixl.0.pf.que7.dropped: 0 > >> > dev.ixl.0.pf.que7.mbuf_defrag_failed: 0 > >> > dev.ixl.0.pf.que6.rx_bytes: 204373984670 > >> > dev.ixl.0.pf.que6.rx_packets: 449990985 > >> > dev.ixl.0.pf.que6.tx_bytes: 655511068125 > >> > dev.ixl.0.pf.que6.tx_packets: 600735086 > >> > dev.ixl.0.pf.que6.no_desc_avail: 0 > >> > dev.ixl.0.pf.que6.tx_dma_setup: 0 > >> > dev.ixl.0.pf.que6.tso_tx: 0 > >> > dev.ixl.0.pf.que6.irqs: 394961024 > >> > dev.ixl.0.pf.que6.dropped: 0 > >> > dev.ixl.0.pf.que6.mbuf_defrag_failed: 0 > >> > dev.ixl.0.pf.que5.rx_bytes: 222919535872 > >> > dev.ixl.0.pf.que5.rx_packets: 466659705 > >> > dev.ixl.0.pf.que5.tx_bytes: 647689764751 > >> > dev.ixl.0.pf.que5.tx_packets: 582532691 > >> > dev.ixl.0.pf.que5.no_desc_avail: 0 > >> > dev.ixl.0.pf.que5.tx_dma_setup: 0 > >> > dev.ixl.0.pf.que5.tso_tx: 5 > >> > dev.ixl.0.pf.que5.irqs: 404552229 > >> > dev.ixl.0.pf.que5.dropped: 0 > >> > dev.ixl.0.pf.que5.mbuf_defrag_failed: 0 > >> > dev.ixl.0.pf.que4.rx_bytes: 231706806551 > >> > > From owner-freebsd-net@freebsd.org Mon Aug 24 07:20:47 2015 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id CBB169C1502; Mon, 24 Aug 2015 07:20:47 +0000 (UTC) (envelope-from hps@selasky.org) Received: from mail.turbocat.net (mail.turbocat.net [IPv6:2a01:4f8:d16:4514::2]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 8B58B1730; Mon, 24 Aug 2015 07:20:47 +0000 (UTC) (envelope-from hps@selasky.org) Received: from laptop015.home.selasky.org (cm-176.74.213.204.customer.telag.net [176.74.213.204]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mail.turbocat.net (Postfix) with ESMTPSA id 2B5121FE023; Mon, 24 Aug 2015 09:20:44 +0200 (CEST) Subject: Re: ix(intel) vs mlxen(mellanox) 10Gb performance To: Rick Macklem , Daniel Braniss References: <1D52028A-B39F-4F9B-BD38-CB1D73BF5D56@cs.huji.ac.il> <55D43615.1030401@selasky.org> <2013503980.25726607.1439989235806.JavaMail.zimbra@uoguelph.ca> <20150820023024.GB996@michelle.fasterthan.com> <1153838447.28656490.1440193567940.JavaMail.zimbra@uoguelph.ca> <15D19823-08F7-4E55-BBD0-CE230F67D26E@cs.huji.ac.il> <818666007.28930310.1440244756872.JavaMail.zimbra@uoguelph.ca> <49173B1F-7B5E-4D59-8651-63D97B0CB5AC@cs.huji.ac.il> <1815942485.29539597.1440370972998.JavaMail.zimbra@uoguelph.ca> Cc: pyunyh@gmail.com, FreeBSD stable , FreeBSD Net , Slawa Olhovchenkov , Gleb Smirnoff , Christopher Forgeron From: Hans Petter Selasky Message-ID: <55DAC623.60006@selasky.org> Date: Mon, 24 Aug 2015 09:22:11 +0200 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:38.0) Gecko/20100101 Thunderbird/38.1.0 MIME-Version: 1.0 In-Reply-To: <1815942485.29539597.1440370972998.JavaMail.zimbra@uoguelph.ca> Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 24 Aug 2015 07:20:47 -0000 On 08/24/15 01:02, Rick Macklem wrote: > The other thing is the degradation seems to cut the rate by about half each time. > 300-->150-->70 I have no idea if this helps to explain it. Might be a NUMA binding issue for the processes involved. man cpuset --HPS From owner-freebsd-net@freebsd.org Mon Aug 24 08:13:46 2015 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 3AFC59BF3F9; Mon, 24 Aug 2015 08:13:46 +0000 (UTC) (envelope-from danny@cs.huji.ac.il) Received: from kabab.cs.huji.ac.il (kabab.cs.huji.ac.il [132.65.116.210]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id B29AC161B; Mon, 24 Aug 2015 08:13:45 +0000 (UTC) (envelope-from danny@cs.huji.ac.il) Received: from chamsa.cs.huji.ac.il ([132.65.80.19]) by kabab.cs.huji.ac.il with esmtp id 1ZTmsu-000BBi-69; Mon, 24 Aug 2015 11:13:40 +0300 Mime-Version: 1.0 (Mac OS X Mail 8.2 \(2104\)) Subject: Re: ix(intel) vs mlxen(mellanox) 10Gb performance From: Daniel Braniss In-Reply-To: <1815942485.29539597.1440370972998.JavaMail.zimbra@uoguelph.ca> Date: Mon, 24 Aug 2015 11:13:39 +0300 Cc: pyunyh@gmail.com, Hans Petter Selasky , FreeBSD stable , FreeBSD Net , Slawa Olhovchenkov , Gleb Smirnoff , Christopher Forgeron Message-Id: <0495A92D-0A4C-4DDB-901A-8ACC3D49C866@cs.huji.ac.il> References: <1D52028A-B39F-4F9B-BD38-CB1D73BF5D56@cs.huji.ac.il> <55D43615.1030401@selasky.org> <2013503980.25726607.1439989235806.JavaMail.zimbra@uoguelph.ca> <20150820023024.GB996@michelle.fasterthan.com> <1153838447.28656490.1440193567940.JavaMail.zimbra@uoguelph.ca> <15D19823-08F7-4E55-BBD0-CE230F67D26E@cs.huji.ac.il> <818666007.28930310.1440244756872.JavaMail.zimbra@uoguelph.ca> <49173B1F-7B5E-4D59-8651-63D97B0CB5AC@cs.huji.ac.il> <1815942485.29539597.1440370972998.JavaMail.zimbra@uoguelph.ca> To: Rick Macklem X-Mailer: Apple Mail (2.2104) Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 24 Aug 2015 08:13:46 -0000 > On 24 Aug 2015, at 02:02, Rick Macklem wrote: >=20 > Daniel Braniss wrote: >>=20 >>> On 22 Aug 2015, at 14:59, Rick Macklem wrote: >>>=20 >>> Daniel Braniss wrote: >>>>=20 >>>>> On Aug 22, 2015, at 12:46 AM, Rick Macklem = wrote: >>>>>=20 >>>>> Yonghyeon PYUN wrote: >>>>>> On Wed, Aug 19, 2015 at 09:00:35AM -0400, Rick Macklem wrote: >>>>>>> Hans Petter Selasky wrote: >>>>>>>> On 08/19/15 09:42, Yonghyeon PYUN wrote: >>>>>>>>> On Wed, Aug 19, 2015 at 09:00:52AM +0200, Hans Petter Selasky = wrote: >>>>>>>>>> On 08/18/15 23:54, Rick Macklem wrote: >>>>>>>>>>> Ouch! Yes, I now see that the code that counts the # of = mbufs is >>>>>>>>>>> before >>>>>>>>>>> the >>>>>>>>>>> code that adds the tcp/ip header mbuf. >>>>>>>>>>>=20 >>>>>>>>>>> In my opinion, this should be fixed by setting = if_hw_tsomaxsegcount >>>>>>>>>>> to >>>>>>>>>>> whatever >>>>>>>>>>> the driver provides - 1. It is not the driver's = responsibility to >>>>>>>>>>> know if >>>>>>>>>>> a tcp/ip >>>>>>>>>>> header mbuf will be added and is a lot less confusing that >>>>>>>>>>> expecting >>>>>>>>>>> the >>>>>>>>>>> driver >>>>>>>>>>> author to know to subtract one. (I had mistakenly thought = that >>>>>>>>>>> tcp_output() had >>>>>>>>>>> added the tc/ip header mbuf before the loop that counts = mbufs in >>>>>>>>>>> the >>>>>>>>>>> list. >>>>>>>>>>> Btw, >>>>>>>>>>> this tcp/ip header mbuf also has leading space for the MAC = layer >>>>>>>>>>> header.) >>>>>>>>>>>=20 >>>>>>>>>>=20 >>>>>>>>>> Hi Rick, >>>>>>>>>>=20 >>>>>>>>>> Your question is good. With the Mellanox hardware we have = separate >>>>>>>>>> so-called inline data space for the TCP/IP headers, so if the = TCP >>>>>>>>>> stack >>>>>>>>>> subtracts something, then we would need to add something to = the >>>>>>>>>> limit, >>>>>>>>>> because then the scatter gather list is only used for the = data part. >>>>>>>>>>=20 >>>>>>>>>=20 >>>>>>>>> I think all drivers in tree don't subtract 1 for >>>>>>>>> if_hw_tsomaxsegcount. Probably touching Mellanox driver would = be >>>>>>>>> simpler than fixing all other drivers in tree. >>>>>>>>>=20 >>>>>>>>>> Maybe it can be controlled by some kind of flag, if all the = three >>>>>>>>>> TSO >>>>>>>>>> limits should include the TCP/IP/ethernet headers too. I'm = pretty >>>>>>>>>> sure >>>>>>>>>> we want both versions. >>>>>>>>>>=20 >>>>>>>>>=20 >>>>>>>>> Hmm, I'm afraid it's already complex. Drivers have to tell = almost >>>>>>>>> the same information to both bus_dma(9) and network stack. >>>>>>>>=20 >>>>>>>> Don't forget that not all drivers in the tree set the TSO = limits >>>>>>>> before >>>>>>>> if_attach(), so possibly the subtraction of one TSO fragment = needs to >>>>>>>> go >>>>>>>> into ip_output() .... >>>>>>>>=20 >>>>>>> Ok, I realized that some drivers may not know the answers before >>>>>>> ether_ifattach(), >>>>>>> due to the way they are configured/written (I saw the use of >>>>>>> if_hw_tsomax_update() >>>>>>> in the patch). >>>>>>=20 >>>>>> I was not able to find an interface that configures TSO = parameters >>>>>> after if_t conversion. I'm under the impression >>>>>> if_hw_tsomax_update() is not designed to use this way. Probably = we >>>>>> need a better one?(CCed to Gleb). >>>>>>=20 >>>>>>>=20 >>>>>>> If it is subtracted as a part of the assignment to = if_hw_tsomaxsegcount >>>>>>> in >>>>>>> tcp_output() >>>>>>> at line#791 in tcp_output() like the following, I don't think it = should >>>>>>> matter if the >>>>>>> values are set before ether_ifattach()? >>>>>>> /* >>>>>>> * Subtract 1 for the tcp/ip header mbuf = that >>>>>>> * will be prepended to the mbuf chain = in this >>>>>>> * function in the code below this = block. >>>>>>> */ >>>>>>> if_hw_tsomaxsegcount =3D = tp->t_tsomaxsegcount - 1; >>>>>>>=20 >>>>>>> I don't have a good solution for the case where a driver doesn't = plan >>>>>>> on >>>>>>> using the >>>>>>> tcp/ip header provided by tcp_output() except to say the driver = can add >>>>>>> one >>>>>>> to the >>>>>>> setting to compensate for that (and if they fail to do so, it = still >>>>>>> works, >>>>>>> although >>>>>>> somewhat suboptimally). When I now read the comment in = sys/net/if_var.h >>>>>>> it >>>>>>> is clear >>>>>>> what it means, but for some reason I didn't read it that way = before? (I >>>>>>> think it was >>>>>>> the part that said the driver didn't have to subtract for the = headers >>>>>>> that >>>>>>> confused me?) >>>>>>> In any case, we need to try and come up with a clear definition = of what >>>>>>> they need to >>>>>>> be set to. >>>>>>>=20 >>>>>>> I can now think of two ways to deal with this: >>>>>>> 1 - Leave tcp_output() as is, but provide a macro for the device = driver >>>>>>> authors to use >>>>>>> that sets if_hw_tsomaxsegcount with a flag for "driver uses = tcp/ip >>>>>>> header mbuf", >>>>>>> documenting that this flag should normally be true. >>>>>>> OR >>>>>>> 2 - Change tcp_output() as above, noting that this is a = workaround for >>>>>>> confusion w.r.t. >>>>>>> whether or not if_hw_tsomaxsegcount should include the tcp/ip = header >>>>>>> mbuf and >>>>>>> update the comment in if_var.h to reflect this. Then drivers = that >>>>>>> don't >>>>>>> use the >>>>>>> tcp/ip header mbuf can increase their value for = if_hw_tsomaxsegcount >>>>>>> by >>>>>>> 1. >>>>>>> (The comment should also mention that a value of 35 or greater = is >>>>>>> much >>>>>>> preferred to >>>>>>> 32 if the hardware will support that.) >>>>>>>=20 >>>>>>=20 >>>>>> Both works for me. My preference is 2 just because it's very >>>>>> common for most drivers that use tcp/ip header mbuf. >>>>> Thanks for this comment. I tend to agree, both for the reason you = state >>>>> and >>>>> also >>>>> because the patch is simple enough that it might qualify as an = errata for >>>>> 10.2. >>>>>=20 >>>>> I am hoping Daniel Braniss will be able to test the patch and let = us know >>>>> if it >>>>> improves performance with TSO enabled? >>>>=20 >>>> send me the patch and I=E2=80=99ll test it ASAP. >>>> danny >>>>=20 >>> Patch is attached. The one for head will also include an update to = the >>> comment >>> in sys/net/if_var.h, but that isn't needed for testing. >>=20 >>=20 >> well, the plot thickens. >>=20 >> Yesterday, before running the new kernel, I decided to re run my = test, and to >> my surprise >> i was getting good numbers, about 300MGB/s with and without TSO. >>=20 >> this morning, the numbers were again bad, around 70MGB/s,what the = ^%$#@! >>=20 >> so, after some coffee, I run some more tests, and some conclusions: >> using a netapp(*) as the nfs client: >> - doing >> ifconfig ix0 tso or -tso >> does some magic and numbers are back to normal - for a while >>=20 >> using another Fbsd/zfs as client all is nifty, actually a bit faster = than the >> netapp (not a fair >> comparison, since the zfs client is not heavily used) and I can=E2=80=99= t see any >> degradation. >>=20 > I assume you meant "server" and not "client" above. you are correct. >=20 >> btw, this is with the patch applied, but was seeing similar numbers = before >> the patch. >>=20 >> running with tso, initially I get around 300MGB/s, but after a = while(sorry >> can=E2=80=99t be more scientific) >> it drops down to about half, and finally to a pathetic 70MGB/s >>=20 > Ok, so it sounds like tso isn't the issue. (At least it seems the = patch, > which I believe is needed, doesn't cause a regression.) >=20 > All I can suggest is: > - looking at the ix stats (I know nothing about them), but if you post = them > maybe someone conversant with the chip can help? (Before and after = degredation.) > - if you captured packets for a short period of time when degraded and = then > after doing "ifconfig", looking at the packet capture in wireshark = might give > some indication of what changes? > - For this I'd be focused on the TCP layer (window sizes, etc) and = timing of > packets. > --> I don't know if there is a packet capture tool like tcpdump on a = Netapp, but > that might be better than capturing them on the client, in case = tcpdump affects > the outcome. However, tcpdump run on the client would be a = fallback, I think. >=20 > The other thing is the degradation seems to cut the rate by about half = each time. > 300-->150-->70 I have no idea if this helps to explain it. >=20 the halving is an optical illusion, it starts degrading slowly. actually it=E2=80=99s bad after reboot, fiddling with the two flags = shows the above =E2=80=98fetaure=E2=80=99. one conclusion so far: ix0 behaves much better without TSO when the server is a NetAPP BTW, this thread started because next week, our main NetAPP will be = upgraded, and I wanted to see if there will be any improvement. > Have fun with it, rick love your generosity ;-) cheers, and thanks, danny >=20 >> *: while running the tests I monitored the Netapp, and nothing out of = the >> ordinary there. >>=20 >> cheers, >> danny >>=20 >> _______________________________________________ >> freebsd-stable@freebsd.org = mailing list >> https://lists.freebsd.org/mailman/listinfo/freebsd-stable = >> To unsubscribe, send any mail to = "freebsd-stable-unsubscribe@freebsd.org = " From owner-freebsd-net@freebsd.org Mon Aug 24 08:30:01 2015 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id CEA019BFA10; Mon, 24 Aug 2015 08:30:01 +0000 (UTC) (envelope-from danny@cs.huji.ac.il) Received: from kabab.cs.huji.ac.il (kabab.cs.huji.ac.il [132.65.116.210]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 7D3C9AA; Mon, 24 Aug 2015 08:30:01 +0000 (UTC) (envelope-from danny@cs.huji.ac.il) Received: from chamsa.cs.huji.ac.il ([132.65.80.19]) by kabab.cs.huji.ac.il with esmtp id 1ZTn8g-000BKT-NA; Mon, 24 Aug 2015 11:29:58 +0300 Content-Type: text/plain; charset=utf-8 Mime-Version: 1.0 (Mac OS X Mail 8.2 \(2104\)) Subject: Re: ix(intel) vs mlxen(mellanox) 10Gb performance From: Daniel Braniss In-Reply-To: <55DAC623.60006@selasky.org> Date: Mon, 24 Aug 2015 11:29:58 +0300 Cc: Rick Macklem , pyunyh@gmail.com, FreeBSD stable , FreeBSD Net , Gleb Smirnoff Content-Transfer-Encoding: quoted-printable Message-Id: <62C7B1A3-CC6B-41A1-B254-6399F19F8FF7@cs.huji.ac.il> References: <1D52028A-B39F-4F9B-BD38-CB1D73BF5D56@cs.huji.ac.il> <55D43615.1030401@selasky.org> <2013503980.25726607.1439989235806.JavaMail.zimbra@uoguelph.ca> <20150820023024.GB996@michelle.fasterthan.com> <1153838447.28656490.1440193567940.JavaMail.zimbra@uoguelph.ca> <15D19823-08F7-4E55-BBD0-CE230F67D26E@cs.huji.ac.il> <818666007.28930310.1440244756872.JavaMail.zimbra@uoguelph.ca> <49173B1F-7B5E-4D59-8651-63D97B0CB5AC@cs.huji.ac.il> <1815942485.29539597.1440370972998.JavaMail.zimbra@uoguelph.ca> <55DAC623.60006@selasky.org> To: Hans Petter Selasky X-Mailer: Apple Mail (2.2104) X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 24 Aug 2015 08:30:01 -0000 > On 24 Aug 2015, at 10:22, Hans Petter Selasky wrote: >=20 > On 08/24/15 01:02, Rick Macklem wrote: >> The other thing is the degradation seems to cut the rate by about = half each time. >> 300-->150-->70 I have no idea if this helps to explain it. >=20 > Might be a NUMA binding issue for the processes involved. >=20 > man cpuset >=20 > --HPS I can=E2=80=99t see how this is relevant, given that the same host, = using the mellanox/mlxen behave much better. I=E2=80=99m getting different results with the intel/ix depending who is = the nfs server danny From owner-freebsd-net@freebsd.org Mon Aug 24 09:05:57 2015 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 7D6639C1834 for ; Mon, 24 Aug 2015 09:05:57 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 69CFD1BEF for ; Mon, 24 Aug 2015 09:05:57 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id t7O95vq2057539 for ; Mon, 24 Aug 2015 09:05:57 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-net@FreeBSD.org Subject: [Bug 202510] [CARP] advertisements sourced from CARP IP cause double MASTER situations Date: Mon, 24 Aug 2015 09:05:57 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 10.0-STABLE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Many People X-Bugzilla-Who: dam@my.gd X-Bugzilla-Status: New X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-net@FreeBSD.org X-Bugzilla-Target-Milestone: --- X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 7bit X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 24 Aug 2015 09:05:57 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=202510 --- Comment #1 from dam@my.gd --- As a precision, the problem stems more from how IPs are assigned from rc , than from CARP itself. Either the old ipv4_addrs_ syntax should be deprecated, or it should be moved higher up so that it's enacted before ifconfig__aliasN. -- You are receiving this mail because: You are the assignee for the bug. From owner-freebsd-net@freebsd.org Mon Aug 24 12:25:20 2015 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 8CECE9C2C0F; Mon, 24 Aug 2015 12:25:20 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from esa-annu.net.uoguelph.ca (esa-annu.mail.uoguelph.ca [131.104.91.36]) by mx1.freebsd.org (Postfix) with ESMTP id 1B6272ED; Mon, 24 Aug 2015 12:25:19 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) IronPort-PHdr: 9a23:998oWRF0O/4N2d8MBfZYGZ1GYnF86YWxBRYc798ds5kLTJ75oMuwAkXT6L1XgUPTWs2DsrQf27GQ7v2rBTVIyK3CmU5BWaQEbwUCh8QSkl5oK+++Imq/EsTXaTcnFt9JTl5v8iLzG0FUHMHjew+a+SXqvnYsExnyfTB4Ov7yUtaLyZ/njKbvptaPOk1hv3mUX/BbFF2OtwLft80b08NJC50a7V/3mEZOYPlc3mhyJFiezF7W78a0+4N/oWwL46pyv+YJa6jxfrw5QLpEF3xmdjltvIy4/SXEGCeG4HoRVi08iBNOAhPepEX2V5H3owPxrax9xSube8T9C7EwD2eM9aBuHSXpgyRPEjcy82Xaj4QklqdSqxGlqhlX3onbfYyRLPo4daqLLoBSfnZIQssED38JOYi7dYZaSrNZZes= X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: A2BJAgC3DNtV/61jaINdDoNhaQaDH7pLAQmBbQqCQ4JuSgKBZRQBAQEBAQEBAYEJgh2CBgEBAQMBAQEBIAQnIAsFCwIBCA4KAgINGQICJwEJJgIECAcEARwEiAUIDbEVlTIBAQEBAQEBAwEBAQEBARgEgSKKNYQyBgEBBRcBMweCaYFDBZU0hQaFCoQth0yNT4NoAiaDQFoiMwd+AQYCFyOBBAEBAQ X-IronPort-AV: E=Sophos;i="5.15,737,1432612800"; d="scan'208";a="234429859" Received: from nipigon.cs.uoguelph.ca (HELO zcs1.mail.uoguelph.ca) ([131.104.99.173]) by esa-annu.net.uoguelph.ca with ESMTP; 24 Aug 2015 08:25:12 -0400 Received: from localhost (localhost [127.0.0.1]) by zcs1.mail.uoguelph.ca (Postfix) with ESMTP id 94B2915F55D; Mon, 24 Aug 2015 08:25:12 -0400 (EDT) Received: from zcs1.mail.uoguelph.ca ([127.0.0.1]) by localhost (zcs1.mail.uoguelph.ca [127.0.0.1]) (amavisd-new, port 10032) with ESMTP id vR_Dc3DDNiPK; Mon, 24 Aug 2015 08:25:11 -0400 (EDT) Received: from localhost (localhost [127.0.0.1]) by zcs1.mail.uoguelph.ca (Postfix) with ESMTP id DE95315F563; Mon, 24 Aug 2015 08:25:11 -0400 (EDT) X-Virus-Scanned: amavisd-new at zcs1.mail.uoguelph.ca Received: from zcs1.mail.uoguelph.ca ([127.0.0.1]) by localhost (zcs1.mail.uoguelph.ca [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id npe3rZJYJb-L; Mon, 24 Aug 2015 08:25:11 -0400 (EDT) Received: from zcs1.mail.uoguelph.ca (zcs1.mail.uoguelph.ca [172.17.95.18]) by zcs1.mail.uoguelph.ca (Postfix) with ESMTP id BD72615F55D; Mon, 24 Aug 2015 08:25:11 -0400 (EDT) Date: Mon, 24 Aug 2015 08:25:11 -0400 (EDT) From: Rick Macklem To: Daniel Braniss Cc: Hans Petter Selasky , pyunyh@gmail.com, FreeBSD Net , FreeBSD stable , Gleb Smirnoff Message-ID: <2112273205.29795512.1440419111720.JavaMail.zimbra@uoguelph.ca> In-Reply-To: <62C7B1A3-CC6B-41A1-B254-6399F19F8FF7@cs.huji.ac.il> References: <1D52028A-B39F-4F9B-BD38-CB1D73BF5D56@cs.huji.ac.il> <1153838447.28656490.1440193567940.JavaMail.zimbra@uoguelph.ca> <15D19823-08F7-4E55-BBD0-CE230F67D26E@cs.huji.ac.il> <818666007.28930310.1440244756872.JavaMail.zimbra@uoguelph.ca> <49173B1F-7B5E-4D59-8651-63D97B0CB5AC@cs.huji.ac.il> <1815942485.29539597.1440370972998.JavaMail.zimbra@uoguelph.ca> <55DAC623.60006@selasky.org> <62C7B1A3-CC6B-41A1-B254-6399F19F8FF7@cs.huji.ac.il> Subject: Re: ix(intel) vs mlxen(mellanox) 10Gb performance MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Originating-IP: [172.17.95.11] X-Mailer: Zimbra 8.0.9_GA_6191 (ZimbraWebClient - FF34 (Win)/8.0.9_GA_6191) Thread-Topic: ix(intel) vs mlxen(mellanox) 10Gb performance Thread-Index: ++D1Njh8EEvTZp84OstiEvfCeIwEzQ== X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 24 Aug 2015 12:25:20 -0000 Daniel Braniss wrote: >=20 > > On 24 Aug 2015, at 10:22, Hans Petter Selasky wrote: > >=20 > > On 08/24/15 01:02, Rick Macklem wrote: > >> The other thing is the degradation seems to cut the rate by about half > >> each time. > >> 300-->150-->70 I have no idea if this helps to explain it. > >=20 > > Might be a NUMA binding issue for the processes involved. > >=20 > > man cpuset > >=20 > > --HPS >=20 > I can=E2=80=99t see how this is relevant, given that the same host, using= the > mellanox/mlxen > behave much better. Well, the "ix" driver has a bunch of tunables for things like "number of qu= eues" and although I'll admit I don't understand how these queues are used, I thi= nk they are related to CPUs and their caches. There is also something called I= XGBE_FDIR, which others have recommended be disabled. (The code is #ifdef IXGBE_FDIR, = but I don't know if it defined for your kernel?) There are also tunables for interrupt = rate and something called hw.ixgbe_tx_process_limit, which appears to limit the numb= er of packets to send or something like that? (I suspect Hans would understand this stuff much better than I do, since I = don't understand it at all.;-) At a glance, the mellanox driver looks very different. > I=E2=80=99m getting different results with the intel/ix depending who is = the nfs > server >=20 Who knows until you figure out what is actually going on. It could just be = the timing of handling the write RPCs or when the different servers send acks for the TCP= segments or ... that causes this for one server and not another. One of the principals used when investigating airplane accidents is to "nev= er assume anything" and just try to collect the facts until the pieces of the puzzle fall in pl= ace. I think the same principal works for this kind of stuff. I once had a case where a specific read of one NFS file would fail on certa= in machines. I won't bore you with the details, but after weeks we got to the point wher= e we had a lab of identical machines (exactly the same hardware and exactly the same softw= are loaded on them) and we could reproduce this problem on about half the machines and not the = other half. We (myself and the guy I worked with) finally noticed the failing machines wer= e on network ports for a given switch. We moved the net cables to another switch and the probl= em went away. --> This particular network switch was broken in such a way that it would g= arble one specific packet consistently, but worked fine for everything else. My point here is that, if someone had suggested the "network switch might b= e broken" at the beginning of investigating this, I would have probably dismissed it, based = on "the network is working just fine", but in the end, that was the problem. --> I am not suggesting you have a broken network switch, just "don't take = anything off the table until you know what is actually going on". And to be honest, you may never know, but it is fun to try and solve these = puzzles. Beyond what I already suggested, I'd look at the "ix" driver's stats and tu= nables and see if any of the tunables has an effect. (And, yes, it will take time to w= ork through these.) Good luck with it, rick >=20 > danny >=20 > _______________________________________________ > freebsd-stable@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-stable > To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org" From owner-freebsd-net@freebsd.org Mon Aug 24 13:20:47 2015 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 5A8039C14BC for ; Mon, 24 Aug 2015 13:20:47 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 46AB067D for ; Mon, 24 Aug 2015 13:20:47 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id t7ODKllJ014191 for ; Mon, 24 Aug 2015 13:20:47 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-net@FreeBSD.org Subject: [Bug 202351] [ip6] [panic] Kernel panic in ip6_forward (different from 128247, 131038) Date: Mon, 24 Aug 2015 13:20:46 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 10.2-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: markus.gebert@hostpoint.ch X-Bugzilla-Status: New X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-net@FreeBSD.org X-Bugzilla-Target-Milestone: --- X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: cc Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 7bit X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 24 Aug 2015 13:20:47 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=202351 markus.gebert@hostpoint.ch changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |markus.gebert@hostpoint.ch --- Comment #2 from markus.gebert@hostpoint.ch --- I've been seeing the same panic since upgrading from 10.1 to 10.2. The problem seems to be that the pf ipv6 reassembly/refragmentation code cannot properly cope with multicast packets but if_bridge needs to broadcast that sort of traffic while still passing it through the firewall. Here's what I think happens in detail: 1. One of the bridge members gets an ipv6 multicast packet which was reassembled by pf. 2. if_bridge broadcasts it to the other member(s). 3. if_brigde applies outbound filtering which refragments the packet (forwarding the reassembled packet could cause MTU issues on ipv6, so refragmentation is required). 4. Because multiple packets may result from this, pf injects all of them into the ipv6 stack using ip6_forward() instead of passing a single packet back to if_bridge for further processing. 5. ip6_forward() will refuse to handle multicast packets, because it was written for unicast traffic. 6. Because somewhere along the road (my guess is in the pf reassembly/refragment code) the mbuf->m_pkthdr.rcvif was lost, we see this panic when ip6_forward() is trying to log that it will not forward this multicast packet. Even if we fix the call to log() and/or make sure that the rcvif is always set, which will make the panic go away, ipv6 multicast will still not work together with pf scrubbing and if_bridge. My current workaround is to disable scrubbing on bridge members, because i don't really need it there. Another approach might be to disable it (or at least reassembly) just for ipv6 multicast traffic. Short-term solution: I think pf should be fixed to not reassemble ipv6 multicast traffic, as long as it's unable to reinject that kind of traffic properly after refragmentation. While calling ip6_forward() in the pf refragmention code seems ok for normal (forwarding) traffic, in the bridge case it does not make sense to me even for unicast traffic. IMO bridged traffic should never be passed into the ipv6 stack where it could be routed and even end up on interfaces which are not part of the bridge. Not knowing the code very well, I'm not sure if this scenario would really be possible, but I think that using ip6_forward() for bridged traffic may call for trouble. So in my opinion, as long as pf refragmentation for ipv6 works as it works right now, pf should not reassemble packets that might somehow end up on a bridged interface. But since reassembly happens on the inbound interface, it seems hard to know wether a packet will ever end up on a bridged interface, unless of course in the simplest case where the inbound interface itself is part of a bridge. Ultimately it might be easier to just inform the user that in case of ipv6, pf scrubbing is ok for forwarding traffic but might cause trouble (and lead to unintended routing?) on bridges. Of course the best solution seems to be for the firewall code to be able to pass a list of packets back to the caller and let it decide wether to call into the ip stack (forwarding case) or into the interface transmit code (bridging case) and thereby eliminating the need to directly use ip6_forward() in the first place. But the usage of ip6_forward() suggests that this is currently not possible and probably something not easily changed. Again, I spent only a limited amount of time to find out how all these parts interact, but I also suspect that calling ip6_forward() could prevent a subsequent firewall from denying these packets when using multiple firewalls, doesn't it? -- You are receiving this mail because: You are the assignee for the bug. From owner-freebsd-net@freebsd.org Mon Aug 24 13:45:20 2015 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 92F779C1F1A for ; Mon, 24 Aug 2015 13:45:20 +0000 (UTC) (envelope-from gpalmer@freebsd.org) Received: from mail.in-addr.com (mail.in-addr.com [IPv6:2a01:4f8:191:61e8::2525:2525]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 59FA0122D; Mon, 24 Aug 2015 13:45:20 +0000 (UTC) (envelope-from gpalmer@freebsd.org) Received: from gjp by mail.in-addr.com with local (Exim 4.86 (FreeBSD)) (envelope-from ) id 1ZTs3q-000Iul-90; Mon, 24 Aug 2015 14:45:18 +0100 Date: Mon, 24 Aug 2015 14:45:18 +0100 From: Gary Palmer To: Matthew Seaman , freebsd-net@freebsd.org Subject: Re: Routing IPv6 over tun0 (PPPoE) issue Message-ID: <20150824134518.GG13503@in-addr.com> References: <20150823150408.GE13503@in-addr.com> <55D9E8D4.1050700@FreeBSD.org> <20150823164828.GF13503@in-addr.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20150823164828.GF13503@in-addr.com> X-SA-Exim-Connect-IP: X-SA-Exim-Mail-From: gpalmer@freebsd.org X-SA-Exim-Scanned: No (on mail.in-addr.com); SAEximRunCond expanded to false X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 24 Aug 2015 13:45:20 -0000 On Sun, Aug 23, 2015 at 05:48:28PM +0100, Gary Palmer wrote: > On Sun, Aug 23, 2015 at 04:37:56PM +0100, Matthew Seaman wrote: > > On 23/08/2015 16:04, Gary Palmer wrote: > > > However if I configure other IPs on other interfaces from the netblock that > > > has been delegated to me and either source the traffic from those IPs or > > > try the traceroute from another computer using IPs in that netblock, I > > > don't even see the traffic leaving tun0 with tcpdump, let alone get any > > > replies. > > > > I have a similar setup. Looks to me as if there's a problem with your > > routing internally. > > > > My routing table looks like this (excluding the ff01::, ff02:: and > > ff03:: routes and anything that's a host specific route): > > > > % netstat -rn -f inet6 | grep -vE '(UH|ff0)' > > Routing tables > > > > Internet6: > > Destination Gateway Flags Netif Expire > > ::/96 ::1 UGRS lo0 > > default fe80::203:97ff:fe19:8000%tun0 UGS tun0 > > ::ffff:0.0.0.0/96 ::1 UGRS lo0 > > 2001:8b0:151:1::/64 link#1 U em0 <<<---** > > fe80::/10 ::1 UGRS lo0 > > fe80::%em0/64 link#1 U em0 > > fe80::%re0/64 link#2 U re0 > > fe80::%lo0/64 link#3 U lo0 > > fe80::%tun0/64 link#5 U tun0 > > > > Here em0 is the interface onto my internal network, and any addresses > > from my assigned IPv6 netblock are configured on that interface or the > > network directly attached to it. You should have a route equivalent to > > the one marked with the arrow. > > > > Note that tun0 uses link-local addresses for the IPv6 tunnelling, not > > addresses from my assigned range. Depending on how your ISP has > > configured things you may need a "real" IPv6 address on your tun0 > > interface, but this should be from a distinct subnet to the block you're > > using internally. > > Hi Matthew, > > Thanks for the reply. I may have messed up manually masking the > network data so let me do it by sed this time so I don't mess up. > > aaaa:bbbb:cccc:dddd is the /64 prefix used for the connection > xxxx:yyyy:zzzz is the /48 used for internal IPs > > The tunnelbroker IPs are also configured but I've removed them as they > shouldn't be relevant. I've checked gif0 and none of the traffic is > going out that interface either. > > tun0 shows: > > tun0: flags=8051 metric 0 mtu 1492 > options=80000 > inet6 fe80::200:24ff:fec9:5bbc%tun0 prefixlen 64 scopeid 0xa > inet a.b.c.d --> e.f.g.h netmask 0xffffffff > inet6 aaaa:bbbb:cccc:dddd:200:24ff:fec9:5bbc prefixlen 64 autoconf > nd6 options=23 > Opened by PID 1038 > > vr0 shows: > > vr0: flags=8843 metric 0 mtu 1500 > options=8280b > ether 00:00:24:c9:5b:bc > inet i.j.k.l netmask 0xffffff00 broadcast i.j.k.m > inet6 fe80::200:24ff:fec9:5bbc%vr0 prefixlen 64 scopeid 0x1 > inet6 xxxx:yyyy:zzzz:1::1 prefixlen 64 > nd6 options=21 > media: Ethernet autoselect (100baseTX ) > status: active > > IPv6 routing table: > > Routing tables > > Internet6: > Destination Gateway Flags Netif Expire > ::/96 ::1 UGRS lo0 => > default fe80::230:88ff:fe16:ec4f%tun0 UG tun0 > ::1 link#9 UH lo0 > ::ffff:0.0.0.0/96 ::1 UGRS lo0 > xxxx:yyyy:zzzz:1::/64 link#1 U vr0 > xxxx:yyyy:zzzz:1::1 link#1 UHS lo0 > xxxx:yyyy:zzzz:2::/64 link#3 U vr2 > xxxx:yyyy:zzzz:2::1 link#3 UHS lo0 > aaaa:bbbb:cccc:dddd::/64 link#10 U tun0 > aaaa:bbbb:cccc:dddd:200:24ff:fec9:5bbc link#10 UHS lo0 > > traceroute from tun0 IP (first 4 hops only shown) > > traceroute6 to wfe0.ysv.freebsd.org (2001:1900:2254:206a::50:0) from aaaa:bbbb:cccc:dddd:200:24ff:fec9:5bbc, 4 hops max, 12 byte packets > 1 aaaa:bbbb::3:0:0:2 29.318 ms 29.860 ms 28.065 ms > 2 aaaa:bbbb:0:301:: 28.724 ms 29.064 ms 29.421 ms > 3 aaaa:bbbb:0:4::1 29.881 ms 29.189 ms 28.254 ms > 4 aaaa:bbbb:0:3::1 35.764 ms 36.488 ms 36.054 ms > > traceroute from vr0 IP using 'traceroute6 -s' > > traceroute6 to wfe0.ysv.freebsd.org (2001:1900:2254:206a::50:0) from xxxx:yyyy:zzzz:1::1, 4 hops max, 12 byte packets > 1 * * * > 2 * * * > > > > Hmmm.... you do have 'gateway_enable="YES"' and > > 'ipv6_gateway_enable="YES"' in your /etc/rc.conf ? > > gateway_enable="YES" > ipv6_gateway_enable="YES" > > Yes. v4 continues to work fine. OK, I guess I must have missed something in earlier testing. The packet *was* going out tun0, just not getting a reply. Turns out that the ISP doesn't set up the route for the /48 unless you do an IPv6 DHCP reqeust. Only then does traffic work when using IPs other than the ones on the PPP interface Sorry for the noise Thanks, Gary From owner-freebsd-net@freebsd.org Mon Aug 24 14:16:11 2015 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 8E4779C2B7F for ; Mon, 24 Aug 2015 14:16:11 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 7AA85A61 for ; Mon, 24 Aug 2015 14:16:11 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id t7OEGBUZ040266 for ; Mon, 24 Aug 2015 14:16:11 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-net@FreeBSD.org Subject: [Bug 202510] [CARP] advertisements sourced from CARP IP cause double MASTER situations Date: Mon, 24 Aug 2015 14:16:11 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 10.0-STABLE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Many People X-Bugzilla-Who: dam@my.gd X-Bugzilla-Status: New X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-net@FreeBSD.org X-Bugzilla-Target-Milestone: --- X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: attachments.created Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 7bit X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 24 Aug 2015 14:16:11 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=202510 --- Comment #2 from dam@my.gd --- Created attachment 160307 --> https://bugs.freebsd.org/bugzilla/attachment.cgi?id=160307&action=edit network.subr IP alias order patch Ensures "ipv4_addrs_" addresses are set up before "ifconfig__aliasN" addresses. This makes sure if we're going to use CARPs *in addition to physical IPs*, the advertisements will be sent from the first configured IP. -- You are receiving this mail because: You are the assignee for the bug. From owner-freebsd-net@freebsd.org Mon Aug 24 14:18:06 2015 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 775789C2C5B for ; Mon, 24 Aug 2015 14:18:06 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 64240C49 for ; Mon, 24 Aug 2015 14:18:06 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id t7OEI61R042006 for ; Mon, 24 Aug 2015 14:18:06 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-net@FreeBSD.org Subject: [Bug 202510] [CARP] advertisements sourced from CARP IP cause double MASTER situations Date: Mon, 24 Aug 2015 14:18:06 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 10.0-STABLE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Many People X-Bugzilla-Who: dam@my.gd X-Bugzilla-Status: New X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-net@FreeBSD.org X-Bugzilla-Target-Milestone: --- X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 7bit X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 24 Aug 2015 14:18:06 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=202510 --- Comment #3 from dam@my.gd --- Patch attached, changes rc's behaviour so that "ipv4_addrs" addresses are set up on interfaces before "ifconfig_aliasN" addresses. Seeing this changes the system's behaviour, this may require consensus before approval. -- You are receiving this mail because: You are the assignee for the bug. From owner-freebsd-net@freebsd.org Mon Aug 24 15:33:14 2015 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 63AEA9C0C88; Mon, 24 Aug 2015 15:33:14 +0000 (UTC) (envelope-from markus.gebert@hostpoint.ch) Received: from mail.adm.hostpoint.ch (mail.adm.hostpoint.ch [IPv6:2a00:d70:0:a::e0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 2A09A169E; Mon, 24 Aug 2015 15:33:13 +0000 (UTC) (envelope-from markus.gebert@hostpoint.ch) Received: from [2001:1620:2013:1:b1fa:bcf5:a6e2:f2a0] (port=59328) by mail.adm.hostpoint.ch with esmtpsa (TLSv1:DHE-RSA-AES256-SHA:256) (Exim 4.84 (FreeBSD)) (envelope-from ) id 1ZTtkD-0005yj-Tw; Mon, 24 Aug 2015 17:33:09 +0200 Content-Type: text/plain; charset=utf-8 Mime-Version: 1.0 (Mac OS X Mail 8.2 \(2102\)) Subject: Re: Near-term pf plans From: Markus Gebert In-Reply-To: <20150823150957.GK48727@vega.codepro.be> Date: Mon, 24 Aug 2015 17:33:08 +0200 Cc: freebsd-net@freebsd.org, freebsd-pf@freebsd.org Content-Transfer-Encoding: quoted-printable Message-Id: <3121D8E4-A27E-475B-9771-C09347D1D793@hostpoint.ch> References: <20150823150957.GK48727@vega.codepro.be> To: Kristof Provost X-Mailer: Apple Mail (2.2102) X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 24 Aug 2015 15:33:14 -0000 Hi Kristof > On 23.08.2015, at 17:09, Kristof Provost wrote: >=20 > - PR 202351 > This is a panic after ip6 reassembly in pf. We set the rcvif to NULL > when refragmenting. That seems to go OK execpt when we're = refragmenting > broadcast/multicast packets in the forwarding path. It's not at all > clear to me how that could happen. if_bridge wants to forward ipv6 multicasts. pf refragmentation code = tries to send out the resulting packets using ip6_forward() which does = not handle multicasts, drops the packet and tries to log that fact, = which causes the panic. I=E2=80=99ve updated the PR with some more thoughts about this. Markus From owner-freebsd-net@freebsd.org Mon Aug 24 16:16:15 2015 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id E6A189C1DE0; Mon, 24 Aug 2015 16:16:15 +0000 (UTC) (envelope-from kp@FreeBSD.org) Received: from venus.codepro.be (venus.codepro.be [IPv6:2a01:4f8:162:1127::2]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "*.codepro.be", Issuer "Gandi Standard SSL CA" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 9C8FCCFA; Mon, 24 Aug 2015 16:16:15 +0000 (UTC) (envelope-from kp@FreeBSD.org) Received: from [IPv6:2a02:1811:2419:4e02:6d:6d9f:9966:b7fe] (unknown [IPv6:2a02:1811:2419:4e02:6d:6d9f:9966:b7fe]) by venus.codepro.be (Postfix) with ESMTPSA id 628EBB5A9; Mon, 24 Aug 2015 18:16:12 +0200 (CEST) Content-Type: text/plain; charset=utf-8 Mime-Version: 1.0 (Mac OS X Mail 9.0 \(3083\)) Subject: Re: Near-term pf plans From: Kristof Provost X-Mailer: Apple Mail (2.3083) In-Reply-To: <3121D8E4-A27E-475B-9771-C09347D1D793@hostpoint.ch> Date: Mon, 24 Aug 2015 18:16:12 +0200 Cc: freebsd-net@freebsd.org, freebsd-pf@freebsd.org Content-Transfer-Encoding: quoted-printable Message-Id: <1DDBFAD5-9AFB-4A21-8D16-BD85AB30F448@FreeBSD.org> References: <20150823150957.GK48727@vega.codepro.be> <3121D8E4-A27E-475B-9771-C09347D1D793@hostpoint.ch> To: Markus Gebert X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 24 Aug 2015 16:16:16 -0000 > On 24 Aug 2015, at 17:33, Markus Gebert = wrote: >=20 >> On 23.08.2015, at 17:09, Kristof Provost wrote: >>=20 >> - PR 202351 >> This is a panic after ip6 reassembly in pf. We set the rcvif to NULL >> when refragmenting. That seems to go OK execpt when we're = refragmenting >> broadcast/multicast packets in the forwarding path. It's not at all >> clear to me how that could happen. >=20 > if_bridge wants to forward ipv6 multicasts. pf refragmentation code = tries to send out the resulting packets using ip6_forward() which does = not handle multicasts, drops the packet and tries to log that fact, = which causes the panic. >=20 > I=E2=80=99ve updated the PR with some more thoughts about this. >=20 Yes, I saw that pass by earlier. Thanks for that, I think you did a = great analysis. Unfortunately there are other issues with pf on bridges. (See PR 185633 = for example) I wouldn=E2=80=99t expect the fragmentation and reassembly to work at = all in that scenario. I=E2=80=99ll see what I can do about at least fixing the panic in the = short term. Even if the reassembly/refragmentation doesn=E2=80=99t work (on bridges) = we should at least no panic. Regards, Kristof From owner-freebsd-net@freebsd.org Mon Aug 24 20:11:56 2015 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id B10789C016B for ; Mon, 24 Aug 2015 20:11:56 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 9CFEB12C2 for ; Mon, 24 Aug 2015 20:11:56 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id t7OKBua7099355 for ; Mon, 24 Aug 2015 20:11:56 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-net@FreeBSD.org Subject: [Bug 183407] [rc.d] [patch] Routing restart returns non-zero exitcode in case of no extra routing parameter or missing atm/ipx Date: Mon, 24 Aug 2015 20:11:56 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: conf X-Bugzilla-Version: unspecified X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: cpm@fbsd.es X-Bugzilla-Status: In Progress X-Bugzilla-Priority: Normal X-Bugzilla-Assigned-To: freebsd-net@FreeBSD.org X-Bugzilla-Target-Milestone: --- X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: dependson Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 7bit X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 24 Aug 2015 20:11:56 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=183407 Carlos J Puga Medina changed: What |Removed |Added ---------------------------------------------------------------------------- Depends on| |202602 -- You are receiving this mail because: You are the assignee for the bug. From owner-freebsd-net@freebsd.org Mon Aug 24 20:12:38 2015 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 73E9B9C01EC for ; Mon, 24 Aug 2015 20:12:38 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 5FC9813A0 for ; Mon, 24 Aug 2015 20:12:38 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id t7OKCcGX099943 for ; Mon, 24 Aug 2015 20:12:38 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-net@FreeBSD.org Subject: [Bug 183407] [rc.d] [patch] Routing restart returns non-zero exitcode in case of no extra routing parameter or missing atm/ipx Date: Mon, 24 Aug 2015 20:12:38 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: conf X-Bugzilla-Version: unspecified X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: cpm@fbsd.es X-Bugzilla-Status: In Progress X-Bugzilla-Priority: Normal X-Bugzilla-Assigned-To: freebsd-net@FreeBSD.org X-Bugzilla-Target-Milestone: --- X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: blocked dependson Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 7bit X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 24 Aug 2015 20:12:38 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=183407 Carlos J Puga Medina changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |202602 Depends on|202602 | -- You are receiving this mail because: You are the assignee for the bug. From owner-freebsd-net@freebsd.org Mon Aug 24 22:48:32 2015 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 4453D9C22A1; Mon, 24 Aug 2015 22:48:32 +0000 (UTC) (envelope-from zbeeble@gmail.com) Received: from mail-yk0-x234.google.com (mail-yk0-x234.google.com [IPv6:2607:f8b0:4002:c07::234]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 08C081D69; Mon, 24 Aug 2015 22:48:32 +0000 (UTC) (envelope-from zbeeble@gmail.com) Received: by ykdt205 with SMTP id t205so138578677ykd.1; Mon, 24 Aug 2015 15:48:31 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:date:message-id:subject:from:to:content-type; bh=Lv2RVK9bi4UGneUdLDERC6txUijjI1MAePidQ2FWx9s=; b=exUCuhn2wkcVRgC/EX6HjTQl9A5NHUS+ECzFpWMod77OOOA2o1U9GbnnHoDDnxG9Cc gyzvh1dj0GTuH3brPTZqB96D+U15+JEt5eib3V6HobnmfTFXRYo7fzjh5GEo030X4qBS y5yHmhfqS136uX+LEigm6jIbos5cm4hfN1uxANW7vObCuxcM1Fd/WPrkYxkjSbi85rrR Vd2/fbOJvRdKXDhabX159hMxLPLAU5sUQCbGVBccxFOnc/W8QtpgoadXiSlgkJrAq162 wbqg8i3qnMzMNOxbpp3jni/DViShaRmgJHYiZ6cb6W9Z5xgFsHFO+Qqfc3AmZI0Uk68r KhYA== MIME-Version: 1.0 X-Received: by 10.129.131.84 with SMTP id t81mr33163650ywf.97.1440456511111; Mon, 24 Aug 2015 15:48:31 -0700 (PDT) Received: by 10.37.52.22 with HTTP; Mon, 24 Aug 2015 15:48:31 -0700 (PDT) Date: Mon, 24 Aug 2015 18:48:31 -0400 Message-ID: Subject: FreeBSD not properly padding vlan packets on several interface types. From: Zaphod Beeblebrox To: FreeBSD Net , FreeBSD Hackers Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 24 Aug 2015 22:48:32 -0000 So, as background, the minimum packet size for ethernet packets is 64 bytes. According to at least cisco, the minimum size, then, for 802.1q (vlan, etc) packets is 68 bytes. On at least BGE and BCE interfaces, it seems (according to counters on my switch) that FreeBSD doesn't honour this. "show interface counters error" is the cisco command to check. From owner-freebsd-net@freebsd.org Mon Aug 24 23:04:55 2015 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id DEA319C28E1; Mon, 24 Aug 2015 23:04:54 +0000 (UTC) (envelope-from slw@zxy.spb.ru) Received: from zxy.spb.ru (zxy.spb.ru [195.70.199.98]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 9B3361669; Mon, 24 Aug 2015 23:04:54 +0000 (UTC) (envelope-from slw@zxy.spb.ru) Received: from slw by zxy.spb.ru with local (Exim 4.84 (FreeBSD)) (envelope-from ) id 1ZU0nM-0005iZ-9r; Tue, 25 Aug 2015 02:04:52 +0300 Date: Tue, 25 Aug 2015 02:04:52 +0300 From: Slawa Olhovchenkov To: Zaphod Beeblebrox Cc: FreeBSD Net , FreeBSD Hackers Subject: Re: FreeBSD not properly padding vlan packets on several interface types. Message-ID: <20150824230452.GG21849@zxy.spb.ru> References: MIME-Version: 1.0 Content-Type: text/plain; charset=koi8-r Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.23 (2014-03-12) X-SA-Exim-Connect-IP: X-SA-Exim-Mail-From: slw@zxy.spb.ru X-SA-Exim-Scanned: No (on zxy.spb.ru); SAEximRunCond expanded to false X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 24 Aug 2015 23:04:55 -0000 On Mon, Aug 24, 2015 at 06:48:31PM -0400, Zaphod Beeblebrox wrote: > So, as background, the minimum packet size for ethernet packets is 64 > bytes. According to at least cisco, the minimum size, then, for 802.1q > (vlan, etc) packets is 68 bytes. On at least BGE and BCE interfaces, it > seems (according to counters on my switch) that FreeBSD doesn't honour this. > > "show interface counters error" is the cisco command to check. C.4.4 Padding and frame size considerations C.4.4.1 Treatment of PAD fields in IEEE Std 802.3 frames The minimum frame size constraint placed on IEEE 802.3/Ethernet frames requires frames to carry zero or more pad octets following the MAC client data, in order to ensure that no frame of total length less than 64 octets is transmitted on the medium. This requirement means that frames whose overall length would otherwise be less than 64 octets in length have (64-len) octets of padding added after the MAC client data, where len is the size of the frame before padding. When tagged frames are transmitted by a Bridge on an IEEE Std 802.3 MAC, there are two permissible approaches (7.2), as follows: a) Keep the minimum frame size generated by the Bridge equal to 64 octets. This implies that the number of pad octets in a received untagged IEEE Std 802.3 frame would be reduced by up to 4 octets when that frame was tagged; b) Adopt a minimum tagged frame length of 68 octets. This implies that the number of pad octets in a received untagged IEEE Std 802.3 frame would not be adjusted when tagging such frames; equally, if subsequently untagged, no pad adjustment would be necessary before transmission on IEEE 802.3/Ethernet. C.4.4.2 Maximum PDU size VLAN tagging of an untagged frame, or relaying frames in tagged frame format, can result in an increase in the length of the original frame. If transmission of a given frame in tagged frame format through a given destination Port would result in violation of the maximum PDU size for the destination MAC method, the Bridge discards the frame for that destination Port. NOTE-Violation of the maximum PDU sizes for destination MAC methods can produce undefined results in networks that contain devices that adhere strictly to these maxima, or in MAC methods where these maxima are inherently constrained by the operation of the MAC method itself (e.g., constrained by timing considerations in the MAC state machines). IEEE Std 802.3ac-1998 defines an extension to the normal IEEE 802.3 maximum frame size for the specific purpose of accommodating the additional octets of the VLAN tag header. The example frame translations in this annex make use of this extension to the IEEE 802.3 frame size. C.4.4.3 Minimum PDU size VLAN untagging of a tagged frame results in the original frame decreasing in length. Where the destination MAC is CSMA/CD: a) If untagging a given frame would result in violation of the minimum frame length requirements of CSMA/CD, the Bridge is required to adjust the PAD field to ensure that the frame length equals the minimum length of 64 octets (7.2 and C.4.4.1); b) If a frame is transmitted in tagged frame format, the Bridge may adopt a minimum tagged frame length of either 64 or 68 octets, as an implementation option. If the latter is chosen, the Bridge adjusts the size of the PAD field on transmission for any tagged frame that is less than 68 octets in length (7.2, C.4.4.1). From owner-freebsd-net@freebsd.org Mon Aug 24 23:13:58 2015 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 703C09C2BE9; Mon, 24 Aug 2015 23:13:58 +0000 (UTC) (envelope-from adrian.chadd@gmail.com) Received: from mail-ig0-x22f.google.com (mail-ig0-x22f.google.com [IPv6:2607:f8b0:4001:c05::22f]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 3824C1B94; Mon, 24 Aug 2015 23:13:58 +0000 (UTC) (envelope-from adrian.chadd@gmail.com) Received: by igui7 with SMTP id i7so67696272igu.0; Mon, 24 Aug 2015 16:13:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=KGAF2xSeu2wUg7w3O4j+hiI26ZbFRHAN7BHuGt0n28A=; b=m9KreHrgrxFM9sPJ288MOlnrxkwTmNQ0Zc0TDWMSnUOVpPYosuX4T7AGlJx8y7T8Lm L8nvJhi+dgYdXmHmdGQRWAzoe9T0vF2bwnRdWw2paBCtOt+eR4MQLQh8l8HiQJIyU0r3 Mrvv+Omji2L8+JaplJZRinBp6PB9jVGyx1nAdV019xxsN35QR/NuVIm9Jni+YyGsrWYk yDnhG32k2ioIDT0cqnNk99mkkW8BXU2jbalfuQdbbOpJ5YyXqNHEXILfYbGc19XNIqOH +OKQHveBTjs7PkRsaSotadYF766MgFD0MflfiSQA3VO9/3Ddn9AXY4GxqUbW/etX5ehI admA== MIME-Version: 1.0 X-Received: by 10.50.128.169 with SMTP id np9mr17921927igb.37.1440458037693; Mon, 24 Aug 2015 16:13:57 -0700 (PDT) Received: by 10.36.28.208 with HTTP; Mon, 24 Aug 2015 16:13:57 -0700 (PDT) In-Reply-To: <2112273205.29795512.1440419111720.JavaMail.zimbra@uoguelph.ca> References: <1D52028A-B39F-4F9B-BD38-CB1D73BF5D56@cs.huji.ac.il> <1153838447.28656490.1440193567940.JavaMail.zimbra@uoguelph.ca> <15D19823-08F7-4E55-BBD0-CE230F67D26E@cs.huji.ac.il> <818666007.28930310.1440244756872.JavaMail.zimbra@uoguelph.ca> <49173B1F-7B5E-4D59-8651-63D97B0CB5AC@cs.huji.ac.il> <1815942485.29539597.1440370972998.JavaMail.zimbra@uoguelph.ca> <55DAC623.60006@selasky.org> <62C7B1A3-CC6B-41A1-B254-6399F19F8FF7@cs.huji.ac.il> <2112273205.29795512.1440419111720.JavaMail.zimbra@uoguelph.ca> Date: Mon, 24 Aug 2015 16:13:57 -0700 Message-ID: Subject: Re: ix(intel) vs mlxen(mellanox) 10Gb performance From: Adrian Chadd To: Rick Macklem Cc: Daniel Braniss , Hans Petter Selasky , Yong-Hyeon Pyun , FreeBSD stable , FreeBSD Net Content-Type: text/plain; charset=UTF-8 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 24 Aug 2015 23:13:58 -0000 Hi, Some hand-waving suggestions: * if you're running something before 10.2, please disable IXGBE_FDIR in sys/conf/options and sys/modules/ixgbe/Makefile . It's buggy and it caused a lot of issues. * It sounds like some extra latency is happening, so I'd fiddle around with interrupt settings. By default it does something called adaptive interrupt moderation and it may be getting in the way of what you're trying to do. There's a way to disable AIM in /boot/loader.conf and manually set the interrupt rate. * As others have said, TSO has been a bit of a problem - hps has been working on solidifying the TSO configuration side of things so NICs advertise to the stack what their maximum offload capability is so things like NFS and TCP don't exceed the segment count. I don't know if it's tunable without hacking the driver, but maybe hack the driver to reduce the count a little to make sure you're not overflowing things and causing it to fall back to a slower path (where it copies all the mbufs into a single larger one to send to the NIC.) * Disable software LRO and see if it helps. Since you're doing lots of little non-streaming operations, it may actually be hindering. HTH, -adrian From owner-freebsd-net@freebsd.org Tue Aug 25 00:53:55 2015 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id AC57C9C12D3 for ; Tue, 25 Aug 2015 00:53:55 +0000 (UTC) (envelope-from jmg@gold.funkthat.com) Received: from gold.funkthat.com (gate2.funkthat.com [208.87.223.18]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client CN "gold.funkthat.com", Issuer "gold.funkthat.com" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 8A3B81704 for ; Tue, 25 Aug 2015 00:53:55 +0000 (UTC) (envelope-from jmg@gold.funkthat.com) Received: from gold.funkthat.com (localhost [127.0.0.1]) by gold.funkthat.com (8.14.5/8.14.5) with ESMTP id t7P0rs1p005343 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO) for ; Mon, 24 Aug 2015 17:53:54 -0700 (PDT) (envelope-from jmg@gold.funkthat.com) Received: (from jmg@localhost) by gold.funkthat.com (8.14.5/8.14.5/Submit) id t7P0rsWm005342 for freebsd-net@FreeBSD.org; Mon, 24 Aug 2015 17:53:54 -0700 (PDT) (envelope-from jmg) Date: Mon, 24 Aug 2015 17:53:54 -0700 From: John-Mark Gurney To: freebsd-net@FreeBSD.org Subject: CFT: Jumbo and non-Jumbo hosts on same subnet Message-ID: <20150825005354.GL33167@funkthat.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline X-Operating-System: FreeBSD 9.1-PRERELEASE amd64 X-PGP-Fingerprint: 54BA 873B 6515 3F10 9E88 9322 9CB1 8F74 6D3F A396 X-Files: The truth is out there X-URL: http://resnet.uoregon.edu/~gurney_j/ X-Resume: http://resnet.uoregon.edu/~gurney_j/resume.html X-TipJar: bitcoin:13Qmb6AeTgQecazTWph4XasEsP7nGRbAPE X-to-the-FBI-CIA-and-NSA: HI! HOW YA DOIN? can i haz chizburger? User-Agent: Mutt/1.5.21 (2010-09-15) X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.2.7 (gold.funkthat.com [127.0.0.1]); Mon, 24 Aug 2015 17:53:54 -0700 (PDT) X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 25 Aug 2015 00:53:55 -0000 I've had this idea for a long time (I fixed the kernel to support it in r162205[1]) and even used a manual version of it a long time ago in production for NFS servers, but never got around to producing an automatic version of it. Now I have: https://github.com/jmgurney/automtud It's a simple script that when run, will configure an interface to it's largest support MTU, set the network route to the default normal 1500 byte MTU so that communication w/ other machines works as normal, and then monitor and probe other hosts to figure out just how large of an MTU that host will accept. Sample run: # sh automtud.sh -i re0 setting up: re0 Setting MTU on interface re0 to 6122. setting normal mtu on interface re0 for network 192.168.0.0/24 change net 192.168.0.0: gateway re0 machine 192.168.0.2 add on interface re0 adjusting 192.168.0.2 mtu to 6122 machine 192.168.0.14 add on interface re0 adjusting 192.168.0.14 mtu to 1504 The adjustment to 1504 on .14 is because the interface also has VLANs, but .14 is untagged, and so we can sneak an extra 4 bytes in the packet. The bad part of this is that the iface still appears to have the normal 1500 byte MTU: npe1: flags=8843 metric 0 mtu 1500 options=80008 The script should work on all modern releases of FreeBSD, though feedback where it breaks is welcome. I'd like to hear of any issues that you may run into, but I'm pretty sure often it'll be, we need to fix driver X as most drivers do not handle Jumbo frames well. In most cases, it'll be the driver needs to be changed to use either cluster (2k) or page sized clusters instead of 9k or 16k clusters when configured for jumbo frames. On my old 9.2-R box that I tried it on, I ran into issues where I got tons of 9k cluster allocation failures, probably just need to increase the limit, but it'd still be better to use page sized clusters instead: ITEM SIZE LIMIT USED FREE REQ FAIL SLEEP mbuf_jumbo_9k: 9216, 6400, 0, 1110,65208532,814941, 0 I haven't looked at fixed the em driver yet. [1] https://svnweb.freebsd.org/changeset/base/r162205 P.S. Probing time could be made faster if ping -t supported sub-second values as if a host on a local segment hasn't replied in, say, 100ms, it's probably not going to, or you need to fix the network. -- John-Mark Gurney Voice: +1 415 225 5579 "All that I will do, has been done, All that I have, has not." From owner-freebsd-net@freebsd.org Tue Aug 25 07:36:45 2015 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id C2EEA99A354; Tue, 25 Aug 2015 07:36:45 +0000 (UTC) (envelope-from hps@selasky.org) Received: from mail.turbocat.net (heidi.turbocat.net [88.198.202.214]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 828C4C5B; Tue, 25 Aug 2015 07:36:44 +0000 (UTC) (envelope-from hps@selasky.org) Received: from laptop015.home.selasky.org (cm-176.74.213.204.customer.telag.net [176.74.213.204]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mail.turbocat.net (Postfix) with ESMTPSA id 69BE21FE023; Tue, 25 Aug 2015 09:36:35 +0200 (CEST) Subject: Re: ix(intel) vs mlxen(mellanox) 10Gb performance To: Rick Macklem , Daniel Braniss References: <1D52028A-B39F-4F9B-BD38-CB1D73BF5D56@cs.huji.ac.il> <55D429A4.3010407@selasky.org> <20150819074212.GB964@michelle.fasterthan.com> <55D43615.1030401@selasky.org> <2013503980.25726607.1439989235806.JavaMail.zimbra@uoguelph.ca> <20150820023024.GB996@michelle.fasterthan.com> <1153838447.28656490.1440193567940.JavaMail.zimbra@uoguelph.ca> <15D19823-08F7-4E55-BBD0-CE230F67D26E@cs.huji.ac.il> <818666007.28930310.1440244756872.JavaMail.zimbra@uoguelph.ca> Cc: pyunyh@gmail.com, FreeBSD stable , FreeBSD Net , Slawa Olhovchenkov , Christopher Forgeron From: Hans Petter Selasky Message-ID: <55DC1B5A.8010109@selasky.org> Date: Tue, 25 Aug 2015 09:38:02 +0200 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:38.0) Gecko/20100101 Thunderbird/38.1.0 MIME-Version: 1.0 In-Reply-To: <818666007.28930310.1440244756872.JavaMail.zimbra@uoguelph.ca> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 25 Aug 2015 07:36:45 -0000 Hi, I've made some minor modifications to the patch from Rick, and made this review: https://reviews.freebsd.org/D3477 --HPS From owner-freebsd-net@freebsd.org Tue Aug 25 14:07:57 2015 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 4878899A3C5 for ; Tue, 25 Aug 2015 14:07:57 +0000 (UTC) (envelope-from john@maxnet.ru) Received: from basic.maxnet.ru (mx.maxnet.ru [195.112.97.17]) by mx1.freebsd.org (Postfix) with ESMTP id C9920B6E for ; Tue, 25 Aug 2015 14:07:55 +0000 (UTC) (envelope-from john@maxnet.ru) Received: from [217.15.204.72] (John.Office.Obninsk.MAXnet.ru [217.15.204.72] (may be forged)) by basic.maxnet.ru (8.14.6/8.14.6) with ESMTP id t7PE7ldN014201 for ; Tue, 25 Aug 2015 17:07:48 +0300 (MSK) (envelope-from john@maxnet.ru) Message-ID: <55DC76B7.9060606@maxnet.ru> Date: Tue, 25 Aug 2015 17:07:51 +0300 From: Evgeny Khorokhorin User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:31.0) Gecko/20100101 Thunderbird/31.7.0 MIME-Version: 1.0 To: freebsd-net@freebsd.org Subject: FreeBSD 10.2 , ospf vs. aggregated static routes, performance issue Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 25 Aug 2015 14:07:57 -0000 Hi, I have 10.2-STABLE, 2 CPU Intel E5-2643v3, network Intel XL710 with 1.4.0 driver from Intel I know that going through routing table is very fast (rn_match). But I decided to optimize routing table. I'm using 2 interfaces - ixl0 and ixl1. Behind ixl0 I have 304 networks 172.16.. from /28 to /24 all via the same gw 1.1.1.1 (because ip on ixl0 with /30 mask). And behind ixl1 I have default route via 2.2.2.2. That 304 172.16 networks I receive via OSPF (quagga). Now all is OK - on every interface I have up to 500kpps/395kpps, 4.5Gbps/1.57Gbps (rx/tx on ixl1 and tx/rx on ixl0). If I disable OSPF and in zebra add static route 172.16.0.0/12 via 1.1.1.1, the system works good until traffic grow up to 251kpps/181kpps , 2.27Gbps/637Mbps. After that the system is degrading: ixl's queue threads utilizes 100% CPU and I see many many traffic drops (netstat -i) If I turn on ospfd and receive 304 more specific routes the problem disappears. Where is the problem? Or I have misunderstanding about how FreeBSD uses routing table.. P.S. I use this machine as NAT. I checked this on ipfw and pf, all the same. -- Cheers, Evgeny From owner-freebsd-net@freebsd.org Tue Aug 25 14:45:53 2015 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 2F38E99D4AE for ; Tue, 25 Aug 2015 14:45:53 +0000 (UTC) (envelope-from julian@freebsd.org) Received: from vps1.elischer.org (vps1.elischer.org [204.109.63.16]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "vps1.elischer.org", Issuer "CA Cert Signing Authority" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 09E87691 for ; Tue, 25 Aug 2015 14:45:52 +0000 (UTC) (envelope-from julian@freebsd.org) Received: from Julian-MBP3.local (ppp121-45-243-143.lns20.per4.internode.on.net [121.45.243.143]) (authenticated bits=0) by vps1.elischer.org (8.15.2/8.15.2) with ESMTPSA id t7PEjkBB005480 (version=TLSv1.2 cipher=DHE-RSA-AES128-SHA bits=128 verify=NO); Tue, 25 Aug 2015 07:45:49 -0700 (PDT) (envelope-from julian@freebsd.org) Subject: Re: FreeBSD 10.2 , ospf vs. aggregated static routes, performance issue To: Evgeny Khorokhorin , freebsd-net@freebsd.org References: <55DC76B7.9060606@maxnet.ru> From: Julian Elischer Message-ID: <55DC7F94.1080908@freebsd.org> Date: Tue, 25 Aug 2015 22:45:40 +0800 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:38.0) Gecko/20100101 Thunderbird/38.2.0 MIME-Version: 1.0 In-Reply-To: <55DC76B7.9060606@maxnet.ru> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 25 Aug 2015 14:45:53 -0000 On 8/25/15 10:07 PM, Evgeny Khorokhorin wrote: > Hi, > > I have 10.2-STABLE, 2 CPU Intel E5-2643v3, network Intel XL710 with > 1.4.0 driver from Intel > I know that going through routing table is very fast (rn_match). But > I decided to optimize routing table. > I'm using 2 interfaces - ixl0 and ixl1. > Behind ixl0 I have 304 networks 172.16.. from /28 to /24 all via the > same gw 1.1.1.1 (because ip on ixl0 with /30 mask). And behind ixl1 > I have default route via 2.2.2.2. > That 304 172.16 networks I receive via OSPF (quagga). Now all is OK > - on every interface I have up to 500kpps/395kpps, 4.5Gbps/1.57Gbps > (rx/tx on ixl1 and tx/rx on ixl0). > If I disable OSPF and in zebra add static route 172.16.0.0/12 via > 1.1.1.1, the system works good until traffic grow up to > 251kpps/181kpps , 2.27Gbps/637Mbps. After that the system is > degrading: ixl's queue threads utilizes 100% CPU and I see many many > traffic drops (netstat -i) > If I turn on ospfd and receive 304 more specific routes the problem > disappears. > > Where is the problem? Or I have misunderstanding about how FreeBSD > uses routing table.. > P.S. I use this machine as NAT. I checked this on ipfw and pf, all > the same. without knowing anything, it looks like a lock on the route is a bottleneck. lots of routes spreads the pain.. try two manually added static routes 172.16.0.0/13 and 172.24.0.0/13 (I hope I split that correctly) and see if it changes things.. then try 4.. > > -- Cheers, > Evgeny > > _______________________________________________ > freebsd-net@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" > From owner-freebsd-net@freebsd.org Tue Aug 25 14:59:57 2015 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 3BA729C23DC for ; Tue, 25 Aug 2015 14:59:57 +0000 (UTC) (envelope-from s.tyshchenko@identika.pro) Received: from scale212.ru (scale212.ru [51.254.36.76]) (using TLSv1.2 with cipher DHE-RSA-AES128-SHA (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id B9AE56D1 for ; Tue, 25 Aug 2015 14:59:56 +0000 (UTC) (envelope-from s.tyshchenko@identika.pro) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=scale212.ru; s=default; h=Content-Type:List-Unsubscribe:Message-ID:Sender:From:Date:MIME-Version:Subject:To; bh=o1HnfD4DQ0GhqGvM7JO6uc1e9LL+yC3MtTAj8xSEYlo=; b=HZWbS8SlpRciLnZBaWKf4WuuFRzRbAAt05Cp7Kh4m1dLqG4dpEYe3DoWcAXGq2I01ZW3723UlpYYXxZs9OV7udjHcCE8jfPwIXHPk3yYhhpMhZ+14MTZlpXbuLhVr92KCbxRSE3EtzOIWdpM8fyzFEcrNOEXxLZkM9qo0CpKCI8=; Received: from root by scale212.ru with local (Exim 4.80) (envelope-from ) id 1ZUFPx-0001wQ-G8 for freebsd-net@freebsd.org; Tue, 25 Aug 2015 16:41:41 +0200 To: freebsd-net@freebsd.org Subject: For you MIME-Version: 1.0 Date: Tue, 25 Aug 2015 16:41:41 +0200 From: Sergey Tyshchenko Sender: s.tyshchenko@identika.pro Message-ID: <241791305.26531@scale212.ru> X-Priority: 3 X-Mailer: scale212.ru mailer. Ver. 1.1. Precedence: bulk Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: base64 X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 25 Aug 2015 14:59:57 -0000 Zm9yIHlvdQ0KCQkJDQoNCgkJCQ0KCQkJwqANCg0KCQkJwqANCg0KCQkJwqANCg0KCQkJwqANCg0K CQkJDQoJCQ0KCQkNCgkJCQ0KCQkJDQoJCQkNCgkJCUhlbGxvLCBNeSBuYW1lIGlzIFNlcmdleSwg SSBwcm9wb3NlIHlvdSBjb29wZXJhdGlvbi4gV2UgYXJlIGEgY3JlYXRpdmUgY29tcGFueSBpbiB0 aGUgZGV2ZWxvcG1lbnQgYW5kIGNyZWF0aW9uIG9mIHVuaXF1ZSBwcm9kdWN0cyBmb3IgZGVjb3Jh dGlvbiAtIHRvIGRlc2lnbiBidWlsZGluZ3MgSURFTlRJS0EuUFJPLiBXZScncmUgc3BlY2lhbGl6 ZWQgb24gZGVjb3JhdGlvbiBvZiBzaG9wcywgcGV0cm9sIHN0YXRpb24sIGNhZmUsIGZhc3QgZm9v ZCByZXN0YXVyYW50cywgYW5kIHJldGFpbCBmcmFuY2hpc2VzLldlIGFyZSBpbnRlcmVzdGVkIGF0 IGxvbmctdGVybSBjb29wZXJhdGlvbiB3aXRoIHlvdS4gV2UgcHJvcG9zZSBhIGdvb2QgZW52aXJv bm1lbnQgdG8gd29yayB3aXRoIHBhcnRuZXJzLkV4YW1wbGVzIG9mIG91ciB3b3JrIGh0dHAgOmh0 dHA6Ly9pZGVudGlrYS5wcm8vY291bnRlcl9saW5rL2NvdW50ZXIucGhwP2NsaWNrPXByZXNlbnRh dGlvbl9lbkkgcHJvcG9zZSBZb3UgdG8gYmVjb21lIG91ciBwYXJ0bmVyIGluIHlvdXIgYXJlYS5M ZXQgdXMga25vdyBhYm91dCB5b3VyIGRlY2lzaW9uDQoJCQkNCg0KCQkJDQoJCQnCoA0KDQoJCQnC oA0KDQoJCQnCoA0KDQoJCQnCoA0KDQoJCQkNCgkJDQoJCQ0KCQkJDQoJCQkNCgkJCQ0KCQkJDQoJ CQkNCg0KCQkJDQoJCQnCoA0KDQoJCQnCoA0KDQoJCQnCoA0KDQoJCQnCoA0KDQoJCQkNCgkJDQoJ CQ0KCQkJDQoJCQkNCgkJCQ0KCQkJDQoJCQkNCg0KCQkJDQoJCQnCoA0KDQoJCQnCoA0KDQoJCQnC oA0KDQoJCQnCoA0KDQoJCQkNCgkJDQoJCQ0KCQkJDQoJCQkNCgkJCQ0KCQkJIEV4YW1wbGVzIG9m IG91ciB3b3JrIGh0dHAgOmh0dHA6Ly9pZGVudGlrYS5wcm8vY291bnRlcl9saW5rL2NvdW50ZXIu cGhwP2NsaWNrPXByZXNlbnRhdGlvbl9lbg0KCQkJDQoNCgkJCQ0KCQkJwqANCg0KCQkJwqANCg0K CQkJwqANCg0KCQkJwqANCg0KCQkJDQoJCQ0KCQkNCgkJCQ0KCQkJDQoJCQkNCgkJCVNlcmdleSBU eXNoY2hlbmtvQ0VPIHwgSURFTlRJS0EuUFJPVmliZXI6ICszODA1MDU1NjY5NjUgfCBXaGF0c0Fw cDogKzM4MDUwNTU2Njk2NVNreXBlOiB0LnNlcmdleS5tcy50eXNoY2hlbmtvQGlkZW50aWthLnBy byB8IHd3dy5pZGVudGlrYS5wcm8wMzA0MCB8IEdvbG9zaWl2c2t5aSBBdmUuIDcwIHwgb2ZmaWNl IDUwMiB8IEtpZXYgDQoJCQkNCg0KCQkJDQoJCQnCoA0KDQoJCQnCoA0KDQoJCQnCoA0KDQoJCQnC oA== From owner-freebsd-net@freebsd.org Tue Aug 25 15:34:21 2015 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 0E5A199A786 for ; Tue, 25 Aug 2015 15:34:21 +0000 (UTC) (envelope-from melifaro@ipfw.ru) Received: from forward19h.cmail.yandex.net (forward19h.cmail.yandex.net [IPv6:2a02:6b8:0:f35::a4]) by mx1.freebsd.org (Postfix) with ESMTP id BE0231332; Tue, 25 Aug 2015 15:34:20 +0000 (UTC) (envelope-from melifaro@ipfw.ru) Received: from web10h.yandex.ru (web10h.yandex.ru [IPv6:2a02:6b8:0:f05::20]) by forward19h.cmail.yandex.net (Yandex) with ESMTP id BAC0D219D5; Tue, 25 Aug 2015 18:34:17 +0300 (MSK) Received: from 127.0.0.1 (localhost [127.0.0.1]) by web10h.yandex.ru (Yandex) with ESMTP id 04CD2100017B; Tue, 25 Aug 2015 18:34:16 +0300 (MSK) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ipfw.ru; s=mail; t=1440516857; bh=OXtrR02QUI1hMTgs7rr6RWnFtRi34KGox+gZNKI1w1Y=; h=From:To:In-Reply-To:References:Subject:Date; b=obXlb97AuQYQK2YbaYApKNnTgJE1o1UBPWsn/dXmyB/P1Eo87PzAD0brE7qaC8D/h MyLnRx0ssdSUx9BR/PHdnDpU0aZ33yO6Gwz4+yQZ1t0W7QLBvXBvL+mJ8w5z9VYASP eBIpl1CSTMOQ7U16CjtG2oZfuoc8esjCDqn1Bws4= Received: by web10h.yandex.ru with HTTP; Tue, 25 Aug 2015 18:34:16 +0300 From: Alexander V. Chernikov To: Julian Elischer , Evgeny Khorokhorin , "freebsd-net@freebsd.org" In-Reply-To: <55DC7F94.1080908@freebsd.org> References: <55DC76B7.9060606@maxnet.ru> <55DC7F94.1080908@freebsd.org> Subject: Re: FreeBSD 10.2 , ospf vs. aggregated static routes, performance issue MIME-Version: 1.0 Message-Id: <451201440516856@web10h.yandex.ru> X-Mailer: Yamail [ http://yandex.ru ] 5.0 Date: Tue, 25 Aug 2015 18:34:16 +0300 Content-Transfer-Encoding: 8bit Content-Type: text/plain; charset=koi8-r X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 25 Aug 2015 15:34:21 -0000 25.08.2015, 17:46, "Julian Elischer" : > On 8/25/15 10:07 PM, Evgeny Khorokhorin wrote: >> Hi, >> >> I have 10.2-STABLE, 2 CPU Intel E5-2643v3, network Intel XL710 with >> 1.4.0 driver from Intel >> I know that going through routing table is very fast (rn_match). But >> I decided to optimize routing table. >> I'm using 2 interfaces - ixl0 and ixl1. >> Behind ixl0 I have 304 networks 172.16.. from /28 to /24 all via the >> same gw 1.1.1.1 (because ip on ixl0 with /30 mask). And behind ixl1 >> I have default route via 2.2.2.2. >> That 304 172.16 networks I receive via OSPF (quagga). Now all is OK >> - on every interface I have up to 500kpps/395kpps, 4.5Gbps/1.57Gbps >> (rx/tx on ixl1 and tx/rx on ixl0). >> If I disable OSPF and in zebra add static route 172.16.0.0/12 via >> 1.1.1.1, the system works good until traffic grow up to >> 251kpps/181kpps , 2.27Gbps/637Mbps. After that the system is >> degrading: ixl's queue threads utilizes 100% CPU and I see many many >> traffic drops (netstat -i) >> If I turn on ospfd and receive 304 more specific routes the problem >> disappears. >> >> Where is the problem? Or I have misunderstanding about how FreeBSD >> uses routing table.. >> P.S. I use this machine as NAT. I checked this on ipfw and pf, all >> the same. > > without knowing anything, it looks like a lock on the route is a > bottleneck. > lots of routes spreads the pain.. Julian is absolutely right - there is currently a contention on individual route entries, so I'd better leave ospf (and keep static route so speed of ospf convergence won't hurt). Or if you want to totally get rid of dynamic protos, than yes, you will have to split this static route into smaller pieces to reduce contention.. > > try two manually added static routes 172.16.0.0/13 and 172.24.0.0/13 > (I hope I split that correctly) and see if it changes things.. > then try 4.. > >> -- Cheers, >> Evgeny >> >> _______________________________________________ >> freebsd-net@freebsd.org mailing list >> https://lists.freebsd.org/mailman/listinfo/freebsd-net >> To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" > > _______________________________________________ > freebsd-net@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" From owner-freebsd-net@freebsd.org Tue Aug 25 17:57:00 2015 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 860709C3446; Tue, 25 Aug 2015 17:57:00 +0000 (UTC) (envelope-from ermal.luci@gmail.com) Received: from mail-yk0-x22e.google.com (mail-yk0-x22e.google.com [IPv6:2607:f8b0:4002:c07::22e]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4276D2DF; Tue, 25 Aug 2015 17:57:00 +0000 (UTC) (envelope-from ermal.luci@gmail.com) Received: by ykdt205 with SMTP id t205so164624176ykd.1; Tue, 25 Aug 2015 10:56:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=CvjsxtEFacw8stFLx6NKiIJjAKQRFcY6zzFmW5xSbxA=; b=ILZIBFp8BM8woAs6CYESyc2U4T7Gtql+9Nj+KqI0GHNgG3XMNECjEV8lClodpImqF1 jIf7dEFrxx0SI/7s89cxuSO/loIG2eUtue+UQPDF3W6RL+kjKzBMSQtovFHWr/qrhcaP aHAwfe0j3fDMdJ8dzJQqpO8yOGgrGTl6+wpppTscvQS5DJeG7JTlwXOpskNWpPJ7d73C Q4LE1tCdONBzQ0fYwACKFAZKaXAhji+tqSTvYhAO05zKkfx49qPhUkpzCqS736gmDASy WMLputUBiCD6z44EiwTEOyv5SpPDj+0BZbHXXN/mqc7k3PBXT2/uQwjc6zqFDNhg8NN+ pz/w== MIME-Version: 1.0 X-Received: by 10.170.70.213 with SMTP id m204mr39637810ykm.112.1440525419493; Tue, 25 Aug 2015 10:56:59 -0700 (PDT) Received: by 10.129.73.205 with HTTP; Tue, 25 Aug 2015 10:56:59 -0700 (PDT) In-Reply-To: <20150823150957.GK48727@vega.codepro.be> References: <20150823150957.GK48727@vega.codepro.be> Date: Tue, 25 Aug 2015 19:56:59 +0200 Message-ID: Subject: Re: Near-term pf plans From: =?UTF-8?Q?Ermal_Lu=C3=A7i?= To: Kristof Provost Cc: freebsd-net , "freebsd-pf@freebsd.org" Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 25 Aug 2015 17:57:00 -0000 On Sun, Aug 23, 2015 at 5:09 PM, Kristof Provost wrote: > Hi, > > Some of you may have noticed that I fixed a couple of pf issues (or in > some cases broke things. Sorry Allan.) recently. > > Here's a quick list of my current priorities: > > - PR 127042, 202178: > This is a panic when an interface an ifgroup have the same name. > There's a fix for the immediate problem in > https://reviews.freebsd.org/D3435 > > I'm inclined to say that ifgroups and interfaces should share a > namespace. That would certainly help pf, because it uses both > interchangeably, which just doesn't work unless they're in the same > namespace. That affects more in the network stack though, so I'd like > opinions for people with more experience with ifgroups. > The solution is easy the scenario of interface name and group name should not be allowed. I do not see the use case for this to be allowed at all its just confusion in general in pf(4). > > - PR 202351 > This is a panic after ip6 reassembly in pf. We set the rcvif to NULL > when refragmenting. That seems to go OK execpt when we're refragmenting > broadcast/multicast packets in the forwarding path. It's not at all > clear to me how that could happen. > > - Removal of frag drop / frag crop support in pf. > pf has two (or really three, but crop and drop mode are very similar) > ways to handle fragmented packets. I've recently extended the > 'reassemble' mode to also work for ip6. > The crop/drop modes really only verify that fragments make sense > (i.e. don't claim to deliver data beyond the last fragment, or duplicate > data). > I'd like to remove support for the crop/drop modes for two reasons: > * the code is relatively large and complex. I've already run into a > couple of problems related to it. > * It doesn't do what users expect. Enabling scrub fragment crop > doesn't actually mean pf will filter on the entire fragment. You > still end up having to let all fragments pass through the > firewall. > There's a patch to do just that here: > https://reviews.freebsd.org/D3466 > > While those provide more security protection they do not always work as advertised so i agree on their removal. > - PR 198868, 193579 (TSO issues) > pf has issues on certain network interfaces with TSO enabled. The > most important of these are amazon VMs. > I believe the problem is that pf tries to work with packets with full > checksums. Usually output packets have pseudo-header checksums in the IP > and TCP headers. pf doesn't know about those. It always tries to do > updates on checksums assuming there's a full checksum. (Which is the > case, pf always calculates a full checksum in pf_check_out()) > > I believe the fix for this issue is to have pf be aware of the > pseudo-header checksums. The type of checksum can be determined from the > CSUM_TCP and CSUM_TCP_IPV6 flags. Whenever pf changes a packet header it > will have to check for those flags to figure out if the checksum should > be updated or not. > > I would be inclined to say that the real solution is not check packets generated from the host, otherwise you will have to go into complicated checksum handling which might not be worth it. I know Open has done some work on checksum handling altogether which might be a good reason to go and look there first. > - PR 188511 > divert-reply implementation for pf > I need to review the use case and implementation of the work done by > PiBa.NL.dev@gmail.com > > I am aware of the patch just do not think this will be a good candidate for inclusion. First divert sockets are slow and should not be advertised for use unless properly fixed. Second the better implementation in general, which is on my roadmap, is convert the route-to/reply-to keywords to the same implementation as ipfw fwd and that will reduce code complexity and provide proper functionality for the scenario the patch is written. > - PR 172648 > This bug started out with an issue with TCP reassembly, but I've not > been able to reproduce that. There is a problem with rdr targets for > ipv6 though. > > With rdr to ::1 we fail the scope check in ip6_input() (right after the > pfil hook) because we have a packet to localhost with a > m->m_pkthdr.rcvif which is not a loopback interface. > That can be fixed by having pf rewrite the rcvif, but that'd > special-case rdr to ::1. > > We've got a similar problem for the reply. There we've got a packet from > ::1 to something else. This fails the scope lookup too. > In essence the problem is that we've already made the routing decision > before pf gets the chance to rewrite the destination address. > > I'm not quite sure how to fix this though. > I will try to look back at this but as a general rule look first at Open if they have already fixed this. IPv4 code has some security belts on such packets in pf(4) code to avoid doing the wrong thing. > > If anyone has any comments or questions, or disagrees with any of the > above, please let me know. > > If anyone knows of critical problems not on this list please let me > know, and I'll see what I can do. > > Also, pf is currently an (unpaid) side project for me. I'm reasonably > limited in the amount of time I can invest in it. > > Regards, > Kristof > _______________________________________________ > freebsd-pf@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-pf > To unsubscribe, send any mail to "freebsd-pf-unsubscribe@freebsd.org" > -- Ermal From owner-freebsd-net@freebsd.org Tue Aug 25 18:36:48 2015 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 9181E99A359 for ; Tue, 25 Aug 2015 18:36:48 +0000 (UTC) (envelope-from ricera10@gmail.com) Received: from mail-qg0-f50.google.com (mail-qg0-f50.google.com [209.85.192.50]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 23FFF9CE for ; Tue, 25 Aug 2015 18:36:47 +0000 (UTC) (envelope-from ricera10@gmail.com) Received: by qgj62 with SMTP id 62so111836596qgj.2 for ; Tue, 25 Aug 2015 11:36:41 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc:content-type; bh=uVsJw+HqDL64oXdjl09qWtujzG4aqYwbLLcdmHKTxCI=; b=Pxu0cQa18GOIDkrPugw0Q/EiTank6uBKKob3KvW1fuWllmmFwiqrzFPyl77iRbOtTc FhPIlqWDDVqfTzrYLnIBaphpinot6fk1lOxd5koCIZGasbDQex2usNCTEQM4UNN1khS5 515dCWqrmfngZ47kCyGZQSE5DVyhIgHjl+wh3vmCRHv4PI3ckg4FG02MlUXUQRoqiNku +UaeINRDYkK0VLgdqLUvLug4ca/m45XuP4EOYKDukCiaw3s7oYKE/710+tuNslBiw61S HWtQrL9+RiWMXyjGvoOdeolAIqS25uZB36WJSV2AIqTOUR4TzZ+ojpUk310+nISt94PR ke7g== X-Received: by 10.140.237.140 with SMTP id i134mr72186010qhc.81.1440527801063; Tue, 25 Aug 2015 11:36:41 -0700 (PDT) Received: from mail-qk0-f170.google.com (mail-qk0-f170.google.com. [209.85.220.170]) by smtp.gmail.com with ESMTPSA id z128sm14226785qhd.43.2015.08.25.11.36.40 for (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 25 Aug 2015 11:36:40 -0700 (PDT) Received: by qkch123 with SMTP id h123so99803177qkc.0 for ; Tue, 25 Aug 2015 11:36:40 -0700 (PDT) X-Received: by 10.55.19.31 with SMTP id d31mr4484928qkh.37.1440527800195; Tue, 25 Aug 2015 11:36:40 -0700 (PDT) MIME-Version: 1.0 References: <55D49611.40603@maxnet.ru> <20150819180051.GM94440@strugglingcoder.info> <55D4DAB3.1020401@maxnet.ru> <55DAC32F.4010003@maxnet.ru> In-Reply-To: <55DAC32F.4010003@maxnet.ru> From: Eric Joyner Date: Tue, 25 Aug 2015 18:36:30 +0000 Message-ID: Subject: Re: FreeBSD 10.2-STABLE + Intel XL710 - free queues To: Evgeny Khorokhorin , Adrian Chadd Cc: hiren panchasara , FreeBSD Net Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 25 Aug 2015 18:36:48 -0000 Not yet; I think I will be able to get to it soon. - Eric On Mon, Aug 24, 2015 at 12:09 AM Evgeny Khorokhorin wrote: > Hi Eric, > > Did you manage to try this problem? > > - Evgeny > > 19.08.2015 23:16, Eric Joyner =D0=BF=D0=B8=D1=88=D0=B5=D1=82: > > Yeah; it should be able to do up to 64 queues for the PF's. It's possible > for the NVM to limit the RSS table size and entry width, but that seems > unlikely. > > - Eric > > On Wed, Aug 19, 2015 at 12:41 PM Adrian Chadd > wrote: > >> no, it's not the RSS option - it's the RSS configuration in the NIC >> for steering traffic into different queues based on header contents. >> >> The RSS kernel option includes the framework that ties it all together >> into the network stack - if you don't use it (which is the default), >> the NICs are free to do whatever they want and there's no affinity in >> the network stack. >> >> Eric - does the intel driver / hardware here support receive traffic >> distribution into > 16 queues? >> >> >> >> -adrian >> >> >> On 19 August 2015 at 12:36, Evgeny Khorokhorin wrote: >> > Eric, >> > I updated this driver in kernel, not as module. And I removed >> > #include "opt_rss.h" >> > >> > from if_ixl.c and ixl_txrx.c: >> > >> > #ifndef IXL_STANDALONE_BUILD >> > #include "opt_inet.h" >> > #include "opt_inet6.h" >> > #include "opt_rss.h" >> > #endif >> > >> > because RSS for is only in HEAD >> > Could I break smth by doing this? >> > >> > Best regards, >> > Evgeny Khorokhorin >> > >> > 19.08.2015 21:17, Eric Joyner =D0=BF=D0=B8=D1=88=D0=B5=D1=82: >> >> >> >> The IXLV_MAX_QUEUES value is for the VF driver; the standard driver >> should >> >> be able to allocate and properly use up to 64 queues. >> >> >> >> That said, you're only getting rx traffic on the first 16 queues, so >> that >> >> looks like a bug in the driver. I'll take a look at it. >> >> >> >> - Eric >> >> >> >> On Wed, Aug 19, 2015 at 11:00 AM hiren panchasara >> >> > >> wrote: >> >> >> >> On 08/19/15 at 05:43P, Evgeny Khorokhorin wrote: >> >> > Hi All, >> >> > >> >> > FreeBSD 10.2-STABLE >> >> > 2*CPU Intel E5-2643v3 with HyperThreading enabled >> >> > Intel XL710 network adapter >> >> > I updated the ixl driver to version 1.4.0 from >> >> download.intel.com >> >> >> >> > Every ixl interface create 24 queues (6 cores *2 HT *2 CPUs) bu= t >> >> > utilizes only 16-17 of them. Where is the reason of such >> behavior or >> >> > driver bug? >> >> >> >> Not sure what is the h/w limit but this may be a possible cause: >> >> #define IXLV_MAX_QUEUES 16 >> >> in sys/dev/ixl/ixlv.h >> >> >> >> and ixlv_init_msix() doing: >> >> if (queues > IXLV_MAX_QUEUES) >> >> queues =3D IXLV_MAX_QUEUES; >> >> >> >> Adding eric from intel to confirm. >> >> >> >> Cheers, >> >> Hiren >> >> > >> >> > irq284: ixl0:q0 177563088 2054 >> >> > irq285: ixl0:q1 402668179 4659 >> >> > irq286: ixl0:q2 408885088 4731 >> >> > irq287: ixl0:q3 397744300 4602 >> >> > irq288: ixl0:q4 403040766 4663 >> >> > irq289: ixl0:q5 402499314 4657 >> >> > irq290: ixl0:q6 392693663 4543 >> >> > irq291: ixl0:q7 389364966 4505 >> >> > irq292: ixl0:q8 243244346 2814 >> >> > irq293: ixl0:q9 216834450 2509 >> >> > irq294: ixl0:q10 229460056 2655 >> >> > irq295: ixl0:q11 219591953 2540 >> >> > irq296: ixl0:q12 228944960 2649 >> >> > irq297: ixl0:q13 226385454 2619 >> >> > irq298: ixl0:q14 219174953 2536 >> >> > irq299: ixl0:q15 222151378 2570 >> >> > irq300: ixl0:q16 82799713 958 >> >> > irq301: ixl0:q17 6131 0 >> >> > irq302: ixl0:q18 5586 0 >> >> > irq303: ixl0:q19 6975 0 >> >> > irq304: ixl0:q20 6243 0 >> >> > irq305: ixl0:q21 6729 0 >> >> > irq306: ixl0:q22 6623 0 >> >> > irq307: ixl0:q23 7306 0 >> >> > irq309: ixl1:q0 174074462 2014 >> >> > irq310: ixl1:q1 435716449 5041 >> >> > irq311: ixl1:q2 431030443 4987 >> >> > irq312: ixl1:q3 424156413 4907 >> >> > irq313: ixl1:q4 414791657 4799 >> >> > irq314: ixl1:q5 420260382 4862 >> >> > irq315: ixl1:q6 415645708 4809 >> >> > irq316: ixl1:q7 422783859 4892 >> >> > irq317: ixl1:q8 252737383 2924 >> >> > irq318: ixl1:q9 269655708 3120 >> >> > irq319: ixl1:q10 252397826 2920 >> >> > irq320: ixl1:q11 255649144 2958 >> >> > irq321: ixl1:q12 246025621 2846 >> >> > irq322: ixl1:q13 240176554 2779 >> >> > irq323: ixl1:q14 254882418 2949 >> >> > irq324: ixl1:q15 236846536 2740 >> >> > irq325: ixl1:q16 86794467 1004 >> >> > irq326: ixl1:q17 83 0 >> >> > irq327: ixl1:q18 74 0 >> >> > irq328: ixl1:q19 202 0 >> >> > irq329: ixl1:q20 99 0 >> >> > irq330: ixl1:q21 96 0 >> >> > irq331: ixl1:q22 91 0 >> >> > irq332: ixl1:q23 89 0 >> >> > >> >> > last pid: 28710; load averages: 7.16, 6.76, 6.49 up >> >> 1+00:00:41 17:40:46 >> >> > 391 processes: 32 running, 215 sleeping, 144 waiting >> >> > CPU 0: 0.0% user, 0.0% nice, 0.0% system, 49.2% interrupt, >> >> 50.8% idle >> >> > CPU 1: 0.0% user, 0.0% nice, 0.4% system, 41.3% interrupt, >> >> 58.3% idle >> >> > CPU 2: 0.0% user, 0.0% nice, 0.0% system, 39.0% interrupt, >> >> 61.0% idle >> >> > CPU 3: 0.0% user, 0.0% nice, 0.0% system, 46.5% interrupt, >> >> 53.5% idle >> >> > CPU 4: 0.0% user, 0.0% nice, 0.0% system, 37.4% interrupt, >> >> 62.6% idle >> >> > CPU 5: 0.0% user, 0.0% nice, 0.0% system, 40.9% interrupt, >> >> 59.1% idle >> >> > CPU 6: 0.0% user, 0.0% nice, 0.0% system, 40.2% interrupt, >> >> 59.8% idle >> >> > CPU 7: 0.0% user, 0.0% nice, 0.0% system, 45.3% interrupt, >> >> 54.7% idle >> >> > CPU 8: 0.0% user, 0.0% nice, 0.0% system, 20.5% interrupt, >> >> 79.5% idle >> >> > CPU 9: 0.0% user, 0.0% nice, 0.0% system, 25.2% interrupt, >> >> 74.8% idle >> >> > CPU 10: 0.0% user, 0.0% nice, 0.0% system, 23.2% interrupt, >> >> 76.8% idle >> >> > CPU 11: 0.0% user, 0.0% nice, 0.0% system, 19.3% interrupt, >> >> 80.7% idle >> >> > CPU 12: 0.0% user, 0.0% nice, 0.0% system, 28.7% interrupt, >> >> 71.3% idle >> >> > CPU 13: 0.0% user, 0.0% nice, 0.0% system, 20.5% interrupt, >> >> 79.5% idle >> >> > CPU 14: 0.0% user, 0.0% nice, 0.0% system, 35.0% interrupt, >> >> 65.0% idle >> >> > CPU 15: 0.0% user, 0.0% nice, 0.0% system, 23.2% interrupt, >> >> 76.8% idle >> >> > CPU 16: 0.0% user, 0.0% nice, 0.4% system, 1.2% interrupt, >> >> 98.4% idle >> >> > CPU 17: 0.0% user, 0.0% nice, 2.0% system, 0.0% interrupt, >> >> 98.0% idle >> >> > CPU 18: 0.0% user, 0.0% nice, 2.4% system, 0.0% interrupt, >> >> 97.6% idle >> >> > CPU 19: 0.0% user, 0.0% nice, 2.8% system, 0.0% interrupt, >> >> 97.2% idle >> >> > CPU 20: 0.0% user, 0.0% nice, 2.4% system, 0.0% interrupt, >> >> 97.6% idle >> >> > CPU 21: 0.0% user, 0.0% nice, 1.6% system, 0.0% interrupt, >> >> 98.4% idle >> >> > CPU 22: 0.0% user, 0.0% nice, 2.8% system, 0.0% interrupt, >> >> 97.2% idle >> >> > CPU 23: 0.0% user, 0.0% nice, 0.4% system, 0.0% interrupt, >> >> 99.6% idle >> >> > >> >> > # netstat -I ixl0 -w1 -h >> >> > input ixl0 output >> >> > packets errs idrops bytes packets errs bytes coll= s >> >> > 253K 0 0 136M 311K 0 325M 0 >> >> > 251K 0 0 129M 314K 0 334M 0 >> >> > 250K 0 0 135M 313K 0 333M 0 >> >> > >> >> > hw.ixl.tx_itr: 122 >> >> > hw.ixl.rx_itr: 62 >> >> > hw.ixl.dynamic_tx_itr: 0 >> >> > hw.ixl.dynamic_rx_itr: 0 >> >> > hw.ixl.max_queues: 0 >> >> > hw.ixl.ring_size: 4096 >> >> > hw.ixl.enable_msix: 1 >> >> > dev.ixl.3.mac.xoff_recvd: 0 >> >> > dev.ixl.3.mac.xoff_txd: 0 >> >> > dev.ixl.3.mac.xon_recvd: 0 >> >> > dev.ixl.3.mac.xon_txd: 0 >> >> > dev.ixl.3.mac.tx_frames_big: 0 >> >> > dev.ixl.3.mac.tx_frames_1024_1522: 0 >> >> > dev.ixl.3.mac.tx_frames_512_1023: 0 >> >> > dev.ixl.3.mac.tx_frames_256_511: 0 >> >> > dev.ixl.3.mac.tx_frames_128_255: 0 >> >> > dev.ixl.3.mac.tx_frames_65_127: 0 >> >> > dev.ixl.3.mac.tx_frames_64: 0 >> >> > dev.ixl.3.mac.checksum_errors: 0 >> >> > dev.ixl.3.mac.rx_jabber: 0 >> >> > dev.ixl.3.mac.rx_oversized: 0 >> >> > dev.ixl.3.mac.rx_fragmented: 0 >> >> > dev.ixl.3.mac.rx_undersize: 0 >> >> > dev.ixl.3.mac.rx_frames_big: 0 >> >> > dev.ixl.3.mac.rx_frames_1024_1522: 0 >> >> > dev.ixl.3.mac.rx_frames_512_1023: 0 >> >> > dev.ixl.3.mac.rx_frames_256_511: 0 >> >> > dev.ixl.3.mac.rx_frames_128_255: 0 >> >> > dev.ixl.3.mac.rx_frames_65_127: 0 >> >> > dev.ixl.3.mac.rx_frames_64: 0 >> >> > dev.ixl.3.mac.rx_length_errors: 0 >> >> > dev.ixl.3.mac.remote_faults: 0 >> >> > dev.ixl.3.mac.local_faults: 0 >> >> > dev.ixl.3.mac.illegal_bytes: 0 >> >> > dev.ixl.3.mac.crc_errors: 0 >> >> > dev.ixl.3.mac.bcast_pkts_txd: 0 >> >> > dev.ixl.3.mac.mcast_pkts_txd: 0 >> >> > dev.ixl.3.mac.ucast_pkts_txd: 0 >> >> > dev.ixl.3.mac.good_octets_txd: 0 >> >> > dev.ixl.3.mac.rx_discards: 0 >> >> > dev.ixl.3.mac.bcast_pkts_rcvd: 0 >> >> > dev.ixl.3.mac.mcast_pkts_rcvd: 0 >> >> > dev.ixl.3.mac.ucast_pkts_rcvd: 0 >> >> > dev.ixl.3.mac.good_octets_rcvd: 0 >> >> > dev.ixl.3.pf.que23.rx_bytes: 0 >> >> > dev.ixl.3.pf.que23.rx_packets: 0 >> >> > dev.ixl.3.pf.que23.tx_bytes: 0 >> >> > dev.ixl.3.pf.que23.tx_packets: 0 >> >> > dev.ixl.3.pf.que23.no_desc_avail: 0 >> >> > dev.ixl.3.pf.que23.tx_dma_setup: 0 >> >> > dev.ixl.3.pf.que23.tso_tx: 0 >> >> > dev.ixl.3.pf.que23.irqs: 0 >> >> > dev.ixl.3.pf.que23.dropped: 0 >> >> > dev.ixl.3.pf.que23.mbuf_defrag_failed: 0 >> >> > dev.ixl.3.pf.que22.rx_bytes: 0 >> >> > dev.ixl.3.pf.que22.rx_packets: 0 >> >> > dev.ixl.3.pf.que22.tx_bytes: 0 >> >> > dev.ixl.3.pf.que22.tx_packets: 0 >> >> > dev.ixl.3.pf.que22.no_desc_avail: 0 >> >> > dev.ixl.3.pf.que22.tx_dma_setup: 0 >> >> > dev.ixl.3.pf.que22.tso_tx: 0 >> >> > dev.ixl.3.pf.que22.irqs: 0 >> >> > dev.ixl.3.pf.que22.dropped: 0 >> >> > dev.ixl.3.pf.que22.mbuf_defrag_failed: 0 >> >> > dev.ixl.3.pf.que21.rx_bytes: 0 >> >> > dev.ixl.3.pf.que21.rx_packets: 0 >> >> > dev.ixl.3.pf.que21.tx_bytes: 0 >> >> > dev.ixl.3.pf.que21.tx_packets: 0 >> >> > dev.ixl.3.pf.que21.no_desc_avail: 0 >> >> > dev.ixl.3.pf.que21.tx_dma_setup: 0 >> >> > dev.ixl.3.pf.que21.tso_tx: 0 >> >> > dev.ixl.3.pf.que21.irqs: 0 >> >> > dev.ixl.3.pf.que21.dropped: 0 >> >> > dev.ixl.3.pf.que21.mbuf_defrag_failed: 0 >> >> > dev.ixl.3.pf.que20.rx_bytes: 0 >> >> > dev.ixl.3.pf.que20.rx_packets: 0 >> >> > dev.ixl.3.pf.que20.tx_bytes: 0 >> >> > dev.ixl.3.pf.que20.tx_packets: 0 >> >> > dev.ixl.3.pf.que20.no_desc_avail: 0 >> >> > dev.ixl.3.pf.que20.tx_dma_setup: 0 >> >> > dev.ixl.3.pf.que20.tso_tx: 0 >> >> > dev.ixl.3.pf.que20.irqs: 0 >> >> > dev.ixl.3.pf.que20.dropped: 0 >> >> > dev.ixl.3.pf.que20.mbuf_defrag_failed: 0 >> >> > dev.ixl.3.pf.que19.rx_bytes: 0 >> >> > dev.ixl.3.pf.que19.rx_packets: 0 >> >> > dev.ixl.3.pf.que19.tx_bytes: 0 >> >> > dev.ixl.3.pf.que19.tx_packets: 0 >> >> > dev.ixl.3.pf.que19.no_desc_avail: 0 >> >> > dev.ixl.3.pf.que19.tx_dma_setup: 0 >> >> > dev.ixl.3.pf.que19.tso_tx: 0 >> >> > dev.ixl.3.pf.que19.irqs: 0 >> >> > dev.ixl.3.pf.que19.dropped: 0 >> >> > dev.ixl.3.pf.que19.mbuf_defrag_failed: 0 >> >> > dev.ixl.3.pf.que18.rx_bytes: 0 >> >> > dev.ixl.3.pf.que18.rx_packets: 0 >> >> > dev.ixl.3.pf.que18.tx_bytes: 0 >> >> > dev.ixl.3.pf.que18.tx_packets: 0 >> >> > dev.ixl.3.pf.que18.no_desc_avail: 0 >> >> > dev.ixl.3.pf.que18.tx_dma_setup: 0 >> >> > dev.ixl.3.pf.que18.tso_tx: 0 >> >> > dev.ixl.3.pf.que18.irqs: 0 >> >> > dev.ixl.3.pf.que18.dropped: 0 >> >> > dev.ixl.3.pf.que18.mbuf_defrag_failed: 0 >> >> > dev.ixl.3.pf.que17.rx_bytes: 0 >> >> > dev.ixl.3.pf.que17.rx_packets: 0 >> >> > dev.ixl.3.pf.que17.tx_bytes: 0 >> >> > dev.ixl.3.pf.que17.tx_packets: 0 >> >> > dev.ixl.3.pf.que17.no_desc_avail: 0 >> >> > dev.ixl.3.pf.que17.tx_dma_setup: 0 >> >> > dev.ixl.3.pf.que17.tso_tx: 0 >> >> > dev.ixl.3.pf.que17.irqs: 0 >> >> > dev.ixl.3.pf.que17.dropped: 0 >> >> > dev.ixl.3.pf.que17.mbuf_defrag_failed: 0 >> >> > dev.ixl.3.pf.que16.rx_bytes: 0 >> >> > dev.ixl.3.pf.que16.rx_packets: 0 >> >> > dev.ixl.3.pf.que16.tx_bytes: 0 >> >> > dev.ixl.3.pf.que16.tx_packets: 0 >> >> > dev.ixl.3.pf.que16.no_desc_avail: 0 >> >> > dev.ixl.3.pf.que16.tx_dma_setup: 0 >> >> > dev.ixl.3.pf.que16.tso_tx: 0 >> >> > dev.ixl.3.pf.que16.irqs: 0 >> >> > dev.ixl.3.pf.que16.dropped: 0 >> >> > dev.ixl.3.pf.que16.mbuf_defrag_failed: 0 >> >> > dev.ixl.3.pf.que15.rx_bytes: 0 >> >> > dev.ixl.3.pf.que15.rx_packets: 0 >> >> > dev.ixl.3.pf.que15.tx_bytes: 0 >> >> > dev.ixl.3.pf.que15.tx_packets: 0 >> >> > dev.ixl.3.pf.que15.no_desc_avail: 0 >> >> > dev.ixl.3.pf.que15.tx_dma_setup: 0 >> >> > dev.ixl.3.pf.que15.tso_tx: 0 >> >> > dev.ixl.3.pf.que15.irqs: 0 >> >> > dev.ixl.3.pf.que15.dropped: 0 >> >> > dev.ixl.3.pf.que15.mbuf_defrag_failed: 0 >> >> > dev.ixl.3.pf.que14.rx_bytes: 0 >> >> > dev.ixl.3.pf.que14.rx_packets: 0 >> >> > dev.ixl.3.pf.que14.tx_bytes: 0 >> >> > dev.ixl.3.pf.que14.tx_packets: 0 >> >> > dev.ixl.3.pf.que14.no_desc_avail: 0 >> >> > dev.ixl.3.pf.que14.tx_dma_setup: 0 >> >> > dev.ixl.3.pf.que14.tso_tx: 0 >> >> > dev.ixl.3.pf.que14.irqs: 0 >> >> > dev.ixl.3.pf.que14.dropped: 0 >> >> > dev.ixl.3.pf.que14.mbuf_defrag_failed: 0 >> >> > dev.ixl.3.pf.que13.rx_bytes: 0 >> >> > dev.ixl.3.pf.que13.rx_packets: 0 >> >> > dev.ixl.3.pf.que13.tx_bytes: 0 >> >> > dev.ixl.3.pf.que13.tx_packets: 0 >> >> > dev.ixl.3.pf.que13.no_desc_avail: 0 >> >> > dev.ixl.3.pf.que13.tx_dma_setup: 0 >> >> > dev.ixl.3.pf.que13.tso_tx: 0 >> >> > dev.ixl.3.pf.que13.irqs: 0 >> >> > dev.ixl.3.pf.que13.dropped: 0 >> >> > dev.ixl.3.pf.que13.mbuf_defrag_failed: 0 >> >> > dev.ixl.3.pf.que12.rx_bytes: 0 >> >> > dev.ixl.3.pf.que12.rx_packets: 0 >> >> > dev.ixl.3.pf.que12.tx_bytes: 0 >> >> > dev.ixl.3.pf.que12.tx_packets: 0 >> >> > dev.ixl.3.pf.que12.no_desc_avail: 0 >> >> > dev.ixl.3.pf.que12.tx_dma_setup: 0 >> >> > dev.ixl.3.pf.que12.tso_tx: 0 >> >> > dev.ixl.3.pf.que12.irqs: 0 >> >> > dev.ixl.3.pf.que12.dropped: 0 >> >> > dev.ixl.3.pf.que12.mbuf_defrag_failed: 0 >> >> > dev.ixl.3.pf.que11.rx_bytes: 0 >> >> > dev.ixl.3.pf.que11.rx_packets: 0 >> >> > dev.ixl.3.pf.que11.tx_bytes: 0 >> >> > dev.ixl.3.pf.que11.tx_packets: 0 >> >> > dev.ixl.3.pf.que11.no_desc_avail: 0 >> >> > dev.ixl.3.pf.que11.tx_dma_setup: 0 >> >> > dev.ixl.3.pf.que11.tso_tx: 0 >> >> > dev.ixl.3.pf.que11.irqs: 0 >> >> > dev.ixl.3.pf.que11.dropped: 0 >> >> > dev.ixl.3.pf.que11.mbuf_defrag_failed: 0 >> >> > dev.ixl.3.pf.que10.rx_bytes: 0 >> >> > dev.ixl.3.pf.que10.rx_packets: 0 >> >> > dev.ixl.3.pf.que10.tx_bytes: 0 >> >> > dev.ixl.3.pf.que10.tx_packets: 0 >> >> > dev.ixl.3.pf.que10.no_desc_avail: 0 >> >> > dev.ixl.3.pf.que10.tx_dma_setup: 0 >> >> > dev.ixl.3.pf.que10.tso_tx: 0 >> >> > dev.ixl.3.pf.que10.irqs: 0 >> >> > dev.ixl.3.pf.que10.dropped: 0 >> >> > dev.ixl.3.pf.que10.mbuf_defrag_failed: 0 >> >> > dev.ixl.3.pf.que9.rx_bytes: 0 >> >> > dev.ixl.3.pf.que9.rx_packets: 0 >> >> > dev.ixl.3.pf.que9.tx_bytes: 0 >> >> > dev.ixl.3.pf.que9.tx_packets: 0 >> >> > dev.ixl.3.pf.que9.no_desc_avail: 0 >> >> > dev.ixl.3.pf.que9.tx_dma_setup: 0 >> >> > dev.ixl.3.pf.que9.tso_tx: 0 >> >> > dev.ixl.3.pf.que9.irqs: 0 >> >> > dev.ixl.3.pf.que9.dropped: 0 >> >> > dev.ixl.3.pf.que9.mbuf_defrag_failed: 0 >> >> > dev.ixl.3.pf.que8.rx_bytes: 0 >> >> > dev.ixl.3.pf.que8.rx_packets: 0 >> >> > dev.ixl.3.pf.que8.tx_bytes: 0 >> >> > dev.ixl.3.pf.que8.tx_packets: 0 >> >> > dev.ixl.3.pf.que8.no_desc_avail: 0 >> >> > dev.ixl.3.pf.que8.tx_dma_setup: 0 >> >> > dev.ixl.3.pf.que8.tso_tx: 0 >> >> > dev.ixl.3.pf.que8.irqs: 0 >> >> > dev.ixl.3.pf.que8.dropped: 0 >> >> > dev.ixl.3.pf.que8.mbuf_defrag_failed: 0 >> >> > dev.ixl.3.pf.que7.rx_bytes: 0 >> >> > dev.ixl.3.pf.que7.rx_packets: 0 >> >> > dev.ixl.3.pf.que7.tx_bytes: 0 >> >> > dev.ixl.3.pf.que7.tx_packets: 0 >> >> > dev.ixl.3.pf.que7.no_desc_avail: 0 >> >> > dev.ixl.3.pf.que7.tx_dma_setup: 0 >> >> > dev.ixl.3.pf.que7.tso_tx: 0 >> >> > dev.ixl.3.pf.que7.irqs: 0 >> >> > dev.ixl.3.pf.que7.dropped: 0 >> >> > dev.ixl.3.pf.que7.mbuf_defrag_failed: 0 >> >> > dev.ixl.3.pf.que6.rx_bytes: 0 >> >> > dev.ixl.3.pf.que6.rx_packets: 0 >> >> > dev.ixl.3.pf.que6.tx_bytes: 0 >> >> > dev.ixl.3.pf.que6.tx_packets: 0 >> >> > dev.ixl.3.pf.que6.no_desc_avail: 0 >> >> > dev.ixl.3.pf.que6.tx_dma_setup: 0 >> >> > dev.ixl.3.pf.que6.tso_tx: 0 >> >> > dev.ixl.3.pf.que6.irqs: 0 >> >> > dev.ixl.3.pf.que6.dropped: 0 >> >> > dev.ixl.3.pf.que6.mbuf_defrag_failed: 0 >> >> > dev.ixl.3.pf.que5.rx_bytes: 0 >> >> > dev.ixl.3.pf.que5.rx_packets: 0 >> >> > dev.ixl.3.pf.que5.tx_bytes: 0 >> >> > dev.ixl.3.pf.que5.tx_packets: 0 >> >> > dev.ixl.3.pf.que5.no_desc_avail: 0 >> >> > dev.ixl.3.pf.que5.tx_dma_setup: 0 >> >> > dev.ixl.3.pf.que5.tso_tx: 0 >> >> > dev.ixl.3.pf.que5.irqs: 0 >> >> > dev.ixl.3.pf.que5.dropped: 0 >> >> > dev.ixl.3.pf.que5.mbuf_defrag_failed: 0 >> >> > dev.ixl.3.pf.que4.rx_bytes: 0 >> >> > dev.ixl.3.pf.que4.rx_packets: 0 >> >> > dev.ixl.3.pf.que4.tx_bytes: 0 >> >> > dev.ixl.3.pf.que4.tx_packets: 0 >> >> > dev.ixl.3.pf.que4.no_desc_avail: 0 >> >> > dev.ixl.3.pf.que4.tx_dma_setup: 0 >> >> > dev.ixl.3.pf.que4.tso_tx: 0 >> >> > dev.ixl.3.pf.que4.irqs: 0 >> >> > dev.ixl.3.pf.que4.dropped: 0 >> >> > dev.ixl.3.pf.que4.mbuf_defrag_failed: 0 >> >> > dev.ixl.3.pf.que3.rx_bytes: 0 >> >> > dev.ixl.3.pf.que3.rx_packets: 0 >> >> > dev.ixl.3.pf.que3.tx_bytes: 0 >> >> > dev.ixl.3.pf.que3.tx_packets: 0 >> >> > dev.ixl.3.pf.que3.no_desc_avail: 0 >> >> > dev.ixl.3.pf.que3.tx_dma_setup: 0 >> >> > dev.ixl.3.pf.que3.tso_tx: 0 >> >> > dev.ixl.3.pf.que3.irqs: 0 >> >> > dev.ixl.3.pf.que3.dropped: 0 >> >> > dev.ixl.3.pf.que3.mbuf_defrag_failed: 0 >> >> > dev.ixl.3.pf.que2.rx_bytes: 0 >> >> > dev.ixl.3.pf.que2.rx_packets: 0 >> >> > dev.ixl.3.pf.que2.tx_bytes: 0 >> >> > dev.ixl.3.pf.que2.tx_packets: 0 >> >> > dev.ixl.3.pf.que2.no_desc_avail: 0 >> >> > dev.ixl.3.pf.que2.tx_dma_setup: 0 >> >> > dev.ixl.3.pf.que2.tso_tx: 0 >> >> > dev.ixl.3.pf.que2.irqs: 0 >> >> > dev.ixl.3.pf.que2.dropped: 0 >> >> > dev.ixl.3.pf.que2.mbuf_defrag_failed: 0 >> >> > dev.ixl.3.pf.que1.rx_bytes: 0 >> >> > dev.ixl.3.pf.que1.rx_packets: 0 >> >> > dev.ixl.3.pf.que1.tx_bytes: 0 >> >> > dev.ixl.3.pf.que1.tx_packets: 0 >> >> > dev.ixl.3.pf.que1.no_desc_avail: 0 >> >> > dev.ixl.3.pf.que1.tx_dma_setup: 0 >> >> > dev.ixl.3.pf.que1.tso_tx: 0 >> >> > dev.ixl.3.pf.que1.irqs: 0 >> >> > dev.ixl.3.pf.que1.dropped: 0 >> >> > dev.ixl.3.pf.que1.mbuf_defrag_failed: 0 >> >> > dev.ixl.3.pf.que0.rx_bytes: 0 >> >> > dev.ixl.3.pf.que0.rx_packets: 0 >> >> > dev.ixl.3.pf.que0.tx_bytes: 0 >> >> > dev.ixl.3.pf.que0.tx_packets: 0 >> >> > dev.ixl.3.pf.que0.no_desc_avail: 0 >> >> > dev.ixl.3.pf.que0.tx_dma_setup: 0 >> >> > dev.ixl.3.pf.que0.tso_tx: 0 >> >> > dev.ixl.3.pf.que0.irqs: 0 >> >> > dev.ixl.3.pf.que0.dropped: 0 >> >> > dev.ixl.3.pf.que0.mbuf_defrag_failed: 0 >> >> > dev.ixl.3.pf.bcast_pkts_txd: 0 >> >> > dev.ixl.3.pf.mcast_pkts_txd: 0 >> >> > dev.ixl.3.pf.ucast_pkts_txd: 0 >> >> > dev.ixl.3.pf.good_octets_txd: 0 >> >> > dev.ixl.3.pf.rx_discards: 0 >> >> > dev.ixl.3.pf.bcast_pkts_rcvd: 0 >> >> > dev.ixl.3.pf.mcast_pkts_rcvd: 0 >> >> > dev.ixl.3.pf.ucast_pkts_rcvd: 0 >> >> > dev.ixl.3.pf.good_octets_rcvd: 0 >> >> > dev.ixl.3.vc_debug_level: 1 >> >> > dev.ixl.3.admin_irq: 0 >> >> > dev.ixl.3.watchdog_events: 0 >> >> > dev.ixl.3.debug: 0 >> >> > dev.ixl.3.dynamic_tx_itr: 0 >> >> > dev.ixl.3.tx_itr: 122 >> >> > dev.ixl.3.dynamic_rx_itr: 0 >> >> > dev.ixl.3.rx_itr: 62 >> >> > dev.ixl.3.fw_version: f4.33 a1.2 n04.42 e8000191d >> >> > dev.ixl.3.current_speed: Unknown >> >> > dev.ixl.3.advertise_speed: 0 >> >> > dev.ixl.3.fc: 0 >> >> > dev.ixl.3.%parent: pci129 >> >> > dev.ixl.3.%pnpinfo: vendor=3D0x8086 device=3D0x1572 subvendor= =3D0x8086 >> >> > subdevice=3D0x0000 class=3D0x020000 >> >> > dev.ixl.3.%location: slot=3D0 function=3D3 >> handle=3D\_SB_.PCI1.QR3A.H003 >> >> > dev.ixl.3.%driver: ixl >> >> > dev.ixl.3.%desc: Intel(R) Ethernet Connection XL710 Driver, >> >> Version - 1.4.0 >> >> > dev.ixl.2.mac.xoff_recvd: 0 >> >> > dev.ixl.2.mac.xoff_txd: 0 >> >> > dev.ixl.2.mac.xon_recvd: 0 >> >> > dev.ixl.2.mac.xon_txd: 0 >> >> > dev.ixl.2.mac.tx_frames_big: 0 >> >> > dev.ixl.2.mac.tx_frames_1024_1522: 0 >> >> > dev.ixl.2.mac.tx_frames_512_1023: 0 >> >> > dev.ixl.2.mac.tx_frames_256_511: 0 >> >> > dev.ixl.2.mac.tx_frames_128_255: 0 >> >> > dev.ixl.2.mac.tx_frames_65_127: 0 >> >> > dev.ixl.2.mac.tx_frames_64: 0 >> >> > dev.ixl.2.mac.checksum_errors: 0 >> >> > dev.ixl.2.mac.rx_jabber: 0 >> >> > dev.ixl.2.mac.rx_oversized: 0 >> >> > dev.ixl.2.mac.rx_fragmented: 0 >> >> > dev.ixl.2.mac.rx_undersize: 0 >> >> > dev.ixl.2.mac.rx_frames_big: 0 >> >> > dev.ixl.2.mac.rx_frames_1024_1522: 0 >> >> > dev.ixl.2.mac.rx_frames_512_1023: 0 >> >> > dev.ixl.2.mac.rx_frames_256_511: 0 >> >> > dev.ixl.2.mac.rx_frames_128_255: 0 >> >> > dev.ixl.2.mac.rx_frames_65_127: 0 >> >> > dev.ixl.2.mac.rx_frames_64: 0 >> >> > dev.ixl.2.mac.rx_length_errors: 0 >> >> > dev.ixl.2.mac.remote_faults: 0 >> >> > dev.ixl.2.mac.local_faults: 0 >> >> > dev.ixl.2.mac.illegal_bytes: 0 >> >> > dev.ixl.2.mac.crc_errors: 0 >> >> > dev.ixl.2.mac.bcast_pkts_txd: 0 >> >> > dev.ixl.2.mac.mcast_pkts_txd: 0 >> >> > dev.ixl.2.mac.ucast_pkts_txd: 0 >> >> > dev.ixl.2.mac.good_octets_txd: 0 >> >> > dev.ixl.2.mac.rx_discards: 0 >> >> > dev.ixl.2.mac.bcast_pkts_rcvd: 0 >> >> > dev.ixl.2.mac.mcast_pkts_rcvd: 0 >> >> > dev.ixl.2.mac.ucast_pkts_rcvd: 0 >> >> > dev.ixl.2.mac.good_octets_rcvd: 0 >> >> > dev.ixl.2.pf.que23.rx_bytes: 0 >> >> > dev.ixl.2.pf.que23.rx_packets: 0 >> >> > dev.ixl.2.pf.que23.tx_bytes: 0 >> >> > dev.ixl.2.pf.que23.tx_packets: 0 >> >> > dev.ixl.2.pf.que23.no_desc_avail: 0 >> >> > dev.ixl.2.pf.que23.tx_dma_setup: 0 >> >> > dev.ixl.2.pf.que23.tso_tx: 0 >> >> > dev.ixl.2.pf.que23.irqs: 0 >> >> > dev.ixl.2.pf.que23.dropped: 0 >> >> > dev.ixl.2.pf.que23.mbuf_defrag_failed: 0 >> >> > dev.ixl.2.pf.que22.rx_bytes: 0 >> >> > dev.ixl.2.pf.que22.rx_packets: 0 >> >> > dev.ixl.2.pf.que22.tx_bytes: 0 >> >> > dev.ixl.2.pf.que22.tx_packets: 0 >> >> > dev.ixl.2.pf.que22.no_desc_avail: 0 >> >> > dev.ixl.2.pf.que22.tx_dma_setup: 0 >> >> > dev.ixl.2.pf.que22.tso_tx: 0 >> >> > dev.ixl.2.pf.que22.irqs: 0 >> >> > dev.ixl.2.pf.que22.dropped: 0 >> >> > dev.ixl.2.pf.que22.mbuf_defrag_failed: 0 >> >> > dev.ixl.2.pf.que21.rx_bytes: 0 >> >> > dev.ixl.2.pf.que21.rx_packets: 0 >> >> > dev.ixl.2.pf.que21.tx_bytes: 0 >> >> > dev.ixl.2.pf.que21.tx_packets: 0 >> >> > dev.ixl.2.pf.que21.no_desc_avail: 0 >> >> > dev.ixl.2.pf.que21.tx_dma_setup: 0 >> >> > dev.ixl.2.pf.que21.tso_tx: 0 >> >> > dev.ixl.2.pf.que21.irqs: 0 >> >> > dev.ixl.2.pf.que21.dropped: 0 >> >> > dev.ixl.2.pf.que21.mbuf_defrag_failed: 0 >> >> > dev.ixl.2.pf.que20.rx_bytes: 0 >> >> > dev.ixl.2.pf.que20.rx_packets: 0 >> >> > dev.ixl.2.pf.que20.tx_bytes: 0 >> >> > dev.ixl.2.pf.que20.tx_packets: 0 >> >> > dev.ixl.2.pf.que20.no_desc_avail: 0 >> >> > dev.ixl.2.pf.que20.tx_dma_setup: 0 >> >> > dev.ixl.2.pf.que20.tso_tx: 0 >> >> > dev.ixl.2.pf.que20.irqs: 0 >> >> > dev.ixl.2.pf.que20.dropped: 0 >> >> > dev.ixl.2.pf.que20.mbuf_defrag_failed: 0 >> >> > dev.ixl.2.pf.que19.rx_bytes: 0 >> >> > dev.ixl.2.pf.que19.rx_packets: 0 >> >> > dev.ixl.2.pf.que19.tx_bytes: 0 >> >> > dev.ixl.2.pf.que19.tx_packets: 0 >> >> > dev.ixl.2.pf.que19.no_desc_avail: 0 >> >> > dev.ixl.2.pf.que19.tx_dma_setup: 0 >> >> > dev.ixl.2.pf.que19.tso_tx: 0 >> >> > dev.ixl.2.pf.que19.irqs: 0 >> >> > dev.ixl.2.pf.que19.dropped: 0 >> >> > dev.ixl.2.pf.que19.mbuf_defrag_failed: 0 >> >> > dev.ixl.2.pf.que18.rx_bytes: 0 >> >> > dev.ixl.2.pf.que18.rx_packets: 0 >> >> > dev.ixl.2.pf.que18.tx_bytes: 0 >> >> > dev.ixl.2.pf.que18.tx_packets: 0 >> >> > dev.ixl.2.pf.que18.no_desc_avail: 0 >> >> > dev.ixl.2.pf.que18.tx_dma_setup: 0 >> >> > dev.ixl.2.pf.que18.tso_tx: 0 >> >> > dev.ixl.2.pf.que18.irqs: 0 >> >> > dev.ixl.2.pf.que18.dropped: 0 >> >> > dev.ixl.2.pf.que18.mbuf_defrag_failed: 0 >> >> > dev.ixl.2.pf.que17.rx_bytes: 0 >> >> > dev.ixl.2.pf.que17.rx_packets: 0 >> >> > dev.ixl.2.pf.que17.tx_bytes: 0 >> >> > dev.ixl.2.pf.que17.tx_packets: 0 >> >> > dev.ixl.2.pf.que17.no_desc_avail: 0 >> >> > dev.ixl.2.pf.que17.tx_dma_setup: 0 >> >> > dev.ixl.2.pf.que17.tso_tx: 0 >> >> > dev.ixl.2.pf.que17.irqs: 0 >> >> > dev.ixl.2.pf.que17.dropped: 0 >> >> > dev.ixl.2.pf.que17.mbuf_defrag_failed: 0 >> >> > dev.ixl.2.pf.que16.rx_bytes: 0 >> >> > dev.ixl.2.pf.que16.rx_packets: 0 >> >> > dev.ixl.2.pf.que16.tx_bytes: 0 >> >> > dev.ixl.2.pf.que16.tx_packets: 0 >> >> > dev.ixl.2.pf.que16.no_desc_avail: 0 >> >> > dev.ixl.2.pf.que16.tx_dma_setup: 0 >> >> > dev.ixl.2.pf.que16.tso_tx: 0 >> >> > dev.ixl.2.pf.que16.irqs: 0 >> >> > dev.ixl.2.pf.que16.dropped: 0 >> >> > dev.ixl.2.pf.que16.mbuf_defrag_failed: 0 >> >> > dev.ixl.2.pf.que15.rx_bytes: 0 >> >> > dev.ixl.2.pf.que15.rx_packets: 0 >> >> > dev.ixl.2.pf.que15.tx_bytes: 0 >> >> > dev.ixl.2.pf.que15.tx_packets: 0 >> >> > dev.ixl.2.pf.que15.no_desc_avail: 0 >> >> > dev.ixl.2.pf.que15.tx_dma_setup: 0 >> >> > dev.ixl.2.pf.que15.tso_tx: 0 >> >> > dev.ixl.2.pf.que15.irqs: 0 >> >> > dev.ixl.2.pf.que15.dropped: 0 >> >> > dev.ixl.2.pf.que15.mbuf_defrag_failed: 0 >> >> > dev.ixl.2.pf.que14.rx_bytes: 0 >> >> > dev.ixl.2.pf.que14.rx_packets: 0 >> >> > dev.ixl.2.pf.que14.tx_bytes: 0 >> >> > dev.ixl.2.pf.que14.tx_packets: 0 >> >> > dev.ixl.2.pf.que14.no_desc_avail: 0 >> >> > dev.ixl.2.pf.que14.tx_dma_setup: 0 >> >> > dev.ixl.2.pf.que14.tso_tx: 0 >> >> > dev.ixl.2.pf.que14.irqs: 0 >> >> > dev.ixl.2.pf.que14.dropped: 0 >> >> > dev.ixl.2.pf.que14.mbuf_defrag_failed: 0 >> >> > dev.ixl.2.pf.que13.rx_bytes: 0 >> >> > dev.ixl.2.pf.que13.rx_packets: 0 >> >> > dev.ixl.2.pf.que13.tx_bytes: 0 >> >> > dev.ixl.2.pf.que13.tx_packets: 0 >> >> > dev.ixl.2.pf.que13.no_desc_avail: 0 >> >> > dev.ixl.2.pf.que13.tx_dma_setup: 0 >> >> > dev.ixl.2.pf.que13.tso_tx: 0 >> >> > dev.ixl.2.pf.que13.irqs: 0 >> >> > dev.ixl.2.pf.que13.dropped: 0 >> >> > dev.ixl.2.pf.que13.mbuf_defrag_failed: 0 >> >> > dev.ixl.2.pf.que12.rx_bytes: 0 >> >> > dev.ixl.2.pf.que12.rx_packets: 0 >> >> > dev.ixl.2.pf.que12.tx_bytes: 0 >> >> > dev.ixl.2.pf.que12.tx_packets: 0 >> >> > dev.ixl.2.pf.que12.no_desc_avail: 0 >> >> > dev.ixl.2.pf.que12.tx_dma_setup: 0 >> >> > dev.ixl.2.pf.que12.tso_tx: 0 >> >> > dev.ixl.2.pf.que12.irqs: 0 >> >> > dev.ixl.2.pf.que12.dropped: 0 >> >> > dev.ixl.2.pf.que12.mbuf_defrag_failed: 0 >> >> > dev.ixl.2.pf.que11.rx_bytes: 0 >> >> > dev.ixl.2.pf.que11.rx_packets: 0 >> >> > dev.ixl.2.pf.que11.tx_bytes: 0 >> >> > dev.ixl.2.pf.que11.tx_packets: 0 >> >> > dev.ixl.2.pf.que11.no_desc_avail: 0 >> >> > dev.ixl.2.pf.que11.tx_dma_setup: 0 >> >> > dev.ixl.2.pf.que11.tso_tx: 0 >> >> > dev.ixl.2.pf.que11.irqs: 0 >> >> > dev.ixl.2.pf.que11.dropped: 0 >> >> > dev.ixl.2.pf.que11.mbuf_defrag_failed: 0 >> >> > dev.ixl.2.pf.que10.rx_bytes: 0 >> >> > dev.ixl.2.pf.que10.rx_packets: 0 >> >> > dev.ixl.2.pf.que10.tx_bytes: 0 >> >> > dev.ixl.2.pf.que10.tx_packets: 0 >> >> > dev.ixl.2.pf.que10.no_desc_avail: 0 >> >> > dev.ixl.2.pf.que10.tx_dma_setup: 0 >> >> > dev.ixl.2.pf.que10.tso_tx: 0 >> >> > dev.ixl.2.pf.que10.irqs: 0 >> >> > dev.ixl.2.pf.que10.dropped: 0 >> >> > dev.ixl.2.pf.que10.mbuf_defrag_failed: 0 >> >> > dev.ixl.2.pf.que9.rx_bytes: 0 >> >> > dev.ixl.2.pf.que9.rx_packets: 0 >> >> > dev.ixl.2.pf.que9.tx_bytes: 0 >> >> > dev.ixl.2.pf.que9.tx_packets: 0 >> >> > dev.ixl.2.pf.que9.no_desc_avail: 0 >> >> > dev.ixl.2.pf.que9.tx_dma_setup: 0 >> >> > dev.ixl.2.pf.que9.tso_tx: 0 >> >> > dev.ixl.2.pf.que9.irqs: 0 >> >> > dev.ixl.2.pf.que9.dropped: 0 >> >> > dev.ixl.2.pf.que9.mbuf_defrag_failed: 0 >> >> > dev.ixl.2.pf.que8.rx_bytes: 0 >> >> > dev.ixl.2.pf.que8.rx_packets: 0 >> >> > dev.ixl.2.pf.que8.tx_bytes: 0 >> >> > dev.ixl.2.pf.que8.tx_packets: 0 >> >> > dev.ixl.2.pf.que8.no_desc_avail: 0 >> >> > dev.ixl.2.pf.que8.tx_dma_setup: 0 >> >> > dev.ixl.2.pf.que8.tso_tx: 0 >> >> > dev.ixl.2.pf.que8.irqs: 0 >> >> > dev.ixl.2.pf.que8.dropped: 0 >> >> > dev.ixl.2.pf.que8.mbuf_defrag_failed: 0 >> >> > dev.ixl.2.pf.que7.rx_bytes: 0 >> >> > dev.ixl.2.pf.que7.rx_packets: 0 >> >> > dev.ixl.2.pf.que7.tx_bytes: 0 >> >> > dev.ixl.2.pf.que7.tx_packets: 0 >> >> > dev.ixl.2.pf.que7.no_desc_avail: 0 >> >> > dev.ixl.2.pf.que7.tx_dma_setup: 0 >> >> > dev.ixl.2.pf.que7.tso_tx: 0 >> >> > dev.ixl.2.pf.que7.irqs: 0 >> >> > dev.ixl.2.pf.que7.dropped: 0 >> >> > dev.ixl.2.pf.que7.mbuf_defrag_failed: 0 >> >> > dev.ixl.2.pf.que6.rx_bytes: 0 >> >> > dev.ixl.2.pf.que6.rx_packets: 0 >> >> > dev.ixl.2.pf.que6.tx_bytes: 0 >> >> > dev.ixl.2.pf.que6.tx_packets: 0 >> >> > dev.ixl.2.pf.que6.no_desc_avail: 0 >> >> > dev.ixl.2.pf.que6.tx_dma_setup: 0 >> >> > dev.ixl.2.pf.que6.tso_tx: 0 >> >> > dev.ixl.2.pf.que6.irqs: 0 >> >> > dev.ixl.2.pf.que6.dropped: 0 >> >> > dev.ixl.2.pf.que6.mbuf_defrag_failed: 0 >> >> > dev.ixl.2.pf.que5.rx_bytes: 0 >> >> > dev.ixl.2.pf.que5.rx_packets: 0 >> >> > dev.ixl.2.pf.que5.tx_bytes: 0 >> >> > dev.ixl.2.pf.que5.tx_packets: 0 >> >> > dev.ixl.2.pf.que5.no_desc_avail: 0 >> >> > dev.ixl.2.pf.que5.tx_dma_setup: 0 >> >> > dev.ixl.2.pf.que5.tso_tx: 0 >> >> > dev.ixl.2.pf.que5.irqs: 0 >> >> > dev.ixl.2.pf.que5.dropped: 0 >> >> > dev.ixl.2.pf.que5.mbuf_defrag_failed: 0 >> >> > dev.ixl.2.pf.que4.rx_bytes: 0 >> >> > dev.ixl.2.pf.que4.rx_packets: 0 >> >> > dev.ixl.2.pf.que4.tx_bytes: 0 >> >> > dev.ixl.2.pf.que4.tx_packets: 0 >> >> > dev.ixl.2.pf.que4.no_desc_avail: 0 >> >> > dev.ixl.2.pf.que4.tx_dma_setup: 0 >> >> > dev.ixl.2.pf.que4.tso_tx: 0 >> >> > dev.ixl.2.pf.que4.irqs: 0 >> >> > dev.ixl.2.pf.que4.dropped: 0 >> >> > dev.ixl.2.pf.que4.mbuf_defrag_failed: 0 >> >> > dev.ixl.2.pf.que3.rx_bytes: 0 >> >> > dev.ixl.2.pf.que3.rx_packets: 0 >> >> > dev.ixl.2.pf.que3.tx_bytes: 0 >> >> > dev.ixl.2.pf.que3.tx_packets: 0 >> >> > dev.ixl.2.pf.que3.no_desc_avail: 0 >> >> > dev.ixl.2.pf.que3.tx_dma_setup: 0 >> >> > dev.ixl.2.pf.que3.tso_tx: 0 >> >> > dev.ixl.2.pf.que3.irqs: 0 >> >> > dev.ixl.2.pf.que3.dropped: 0 >> >> > dev.ixl.2.pf.que3.mbuf_defrag_failed: 0 >> >> > dev.ixl.2.pf.que2.rx_bytes: 0 >> >> > dev.ixl.2.pf.que2.rx_packets: 0 >> >> > dev.ixl.2.pf.que2.tx_bytes: 0 >> >> > dev.ixl.2.pf.que2.tx_packets: 0 >> >> > dev.ixl.2.pf.que2.no_desc_avail: 0 >> >> > dev.ixl.2.pf.que2.tx_dma_setup: 0 >> >> > dev.ixl.2.pf.que2.tso_tx: 0 >> >> > dev.ixl.2.pf.que2.irqs: 0 >> >> > dev.ixl.2.pf.que2.dropped: 0 >> >> > dev.ixl.2.pf.que2.mbuf_defrag_failed: 0 >> >> > dev.ixl.2.pf.que1.rx_bytes: 0 >> >> > dev.ixl.2.pf.que1.rx_packets: 0 >> >> > dev.ixl.2.pf.que1.tx_bytes: 0 >> >> > dev.ixl.2.pf.que1.tx_packets: 0 >> >> > dev.ixl.2.pf.que1.no_desc_avail: 0 >> >> > dev.ixl.2.pf.que1.tx_dma_setup: 0 >> >> > dev.ixl.2.pf.que1.tso_tx: 0 >> >> > dev.ixl.2.pf.que1.irqs: 0 >> >> > dev.ixl.2.pf.que1.dropped: 0 >> >> > dev.ixl.2.pf.que1.mbuf_defrag_failed: 0 >> >> > dev.ixl.2.pf.que0.rx_bytes: 0 >> >> > dev.ixl.2.pf.que0.rx_packets: 0 >> >> > dev.ixl.2.pf.que0.tx_bytes: 0 >> >> > dev.ixl.2.pf.que0.tx_packets: 0 >> >> > dev.ixl.2.pf.que0.no_desc_avail: 0 >> >> > dev.ixl.2.pf.que0.tx_dma_setup: 0 >> >> > dev.ixl.2.pf.que0.tso_tx: 0 >> >> > dev.ixl.2.pf.que0.irqs: 0 >> >> > dev.ixl.2.pf.que0.dropped: 0 >> >> > dev.ixl.2.pf.que0.mbuf_defrag_failed: 0 >> >> > dev.ixl.2.pf.bcast_pkts_txd: 0 >> >> > dev.ixl.2.pf.mcast_pkts_txd: 0 >> >> > dev.ixl.2.pf.ucast_pkts_txd: 0 >> >> > dev.ixl.2.pf.good_octets_txd: 0 >> >> > dev.ixl.2.pf.rx_discards: 0 >> >> > dev.ixl.2.pf.bcast_pkts_rcvd: 0 >> >> > dev.ixl.2.pf.mcast_pkts_rcvd: 0 >> >> > dev.ixl.2.pf.ucast_pkts_rcvd: 0 >> >> > dev.ixl.2.pf.good_octets_rcvd: 0 >> >> > dev.ixl.2.vc_debug_level: 1 >> >> > dev.ixl.2.admin_irq: 0 >> >> > dev.ixl.2.watchdog_events: 0 >> >> > dev.ixl.2.debug: 0 >> >> > dev.ixl.2.dynamic_tx_itr: 0 >> >> > dev.ixl.2.tx_itr: 122 >> >> > dev.ixl.2.dynamic_rx_itr: 0 >> >> > dev.ixl.2.rx_itr: 62 >> >> > dev.ixl.2.fw_version: f4.33 a1.2 n04.42 e8000191d >> >> > dev.ixl.2.current_speed: Unknown >> >> > dev.ixl.2.advertise_speed: 0 >> >> > dev.ixl.2.fc: 0 >> >> > dev.ixl.2.%parent: pci129 >> >> > dev.ixl.2.%pnpinfo: vendor=3D0x8086 device=3D0x1572 subvendor= =3D0x8086 >> >> > subdevice=3D0x0000 class=3D0x020000 >> >> > dev.ixl.2.%location: slot=3D0 function=3D2 >> handle=3D\_SB_.PCI1.QR3A.H002 >> >> > dev.ixl.2.%driver: ixl >> >> > dev.ixl.2.%desc: Intel(R) Ethernet Connection XL710 Driver, >> >> Version - 1.4.0 >> >> > dev.ixl.1.mac.xoff_recvd: 0 >> >> > dev.ixl.1.mac.xoff_txd: 0 >> >> > dev.ixl.1.mac.xon_recvd: 0 >> >> > dev.ixl.1.mac.xon_txd: 0 >> >> > dev.ixl.1.mac.tx_frames_big: 0 >> >> > dev.ixl.1.mac.tx_frames_1024_1522: 1565670684 >> >> > dev.ixl.1.mac.tx_frames_512_1023: 101286418 >> >> > dev.ixl.1.mac.tx_frames_256_511: 49713129 >> >> > dev.ixl.1.mac.tx_frames_128_255: 231617277 >> >> > dev.ixl.1.mac.tx_frames_65_127: 2052767669 >> >> > dev.ixl.1.mac.tx_frames_64: 1318689044 >> >> > dev.ixl.1.mac.checksum_errors: 0 >> >> > dev.ixl.1.mac.rx_jabber: 0 >> >> > dev.ixl.1.mac.rx_oversized: 0 >> >> > dev.ixl.1.mac.rx_fragmented: 0 >> >> > dev.ixl.1.mac.rx_undersize: 0 >> >> > dev.ixl.1.mac.rx_frames_big: 0 >> >> > dev.ixl.1.mac.rx_frames_1024_1522: 4960403414 >> >> > dev.ixl.1.mac.rx_frames_512_1023: 113675084 >> >> > dev.ixl.1.mac.rx_frames_256_511: 253904920 >> >> > dev.ixl.1.mac.rx_frames_128_255: 196369726 >> >> > dev.ixl.1.mac.rx_frames_65_127: 1436626211 >> >> > dev.ixl.1.mac.rx_frames_64: 242768681 >> >> > dev.ixl.1.mac.rx_length_errors: 0 >> >> > dev.ixl.1.mac.remote_faults: 0 >> >> > dev.ixl.1.mac.local_faults: 0 >> >> > dev.ixl.1.mac.illegal_bytes: 0 >> >> > dev.ixl.1.mac.crc_errors: 0 >> >> > dev.ixl.1.mac.bcast_pkts_txd: 277 >> >> > dev.ixl.1.mac.mcast_pkts_txd: 0 >> >> > dev.ixl.1.mac.ucast_pkts_txd: 5319743942 >> >> > dev.ixl.1.mac.good_octets_txd: 2642351885737 >> >> > dev.ixl.1.mac.rx_discards: 0 >> >> > dev.ixl.1.mac.bcast_pkts_rcvd: 5 >> >> > dev.ixl.1.mac.mcast_pkts_rcvd: 144 >> >> > dev.ixl.1.mac.ucast_pkts_rcvd: 7203747879 >> >> > dev.ixl.1.mac.good_octets_rcvd: 7770230492434 >> >> > dev.ixl.1.pf.que23.rx_bytes: 0 >> >> > dev.ixl.1.pf.que23.rx_packets: 0 >> >> > dev.ixl.1.pf.que23.tx_bytes: 7111 >> >> > dev.ixl.1.pf.que23.tx_packets: 88 >> >> > dev.ixl.1.pf.que23.no_desc_avail: 0 >> >> > dev.ixl.1.pf.que23.tx_dma_setup: 0 >> >> > dev.ixl.1.pf.que23.tso_tx: 0 >> >> > dev.ixl.1.pf.que23.irqs: 88 >> >> > dev.ixl.1.pf.que23.dropped: 0 >> >> > dev.ixl.1.pf.que23.mbuf_defrag_failed: 0 >> >> > dev.ixl.1.pf.que22.rx_bytes: 0 >> >> > dev.ixl.1.pf.que22.rx_packets: 0 >> >> > dev.ixl.1.pf.que22.tx_bytes: 6792 >> >> > dev.ixl.1.pf.que22.tx_packets: 88 >> >> > dev.ixl.1.pf.que22.no_desc_avail: 0 >> >> > dev.ixl.1.pf.que22.tx_dma_setup: 0 >> >> > dev.ixl.1.pf.que22.tso_tx: 0 >> >> > dev.ixl.1.pf.que22.irqs: 89 >> >> > dev.ixl.1.pf.que22.dropped: 0 >> >> > dev.ixl.1.pf.que22.mbuf_defrag_failed: 0 >> >> > dev.ixl.1.pf.que21.rx_bytes: 0 >> >> > dev.ixl.1.pf.que21.rx_packets: 0 >> >> > dev.ixl.1.pf.que21.tx_bytes: 7486 >> >> > dev.ixl.1.pf.que21.tx_packets: 93 >> >> > dev.ixl.1.pf.que21.no_desc_avail: 0 >> >> > dev.ixl.1.pf.que21.tx_dma_setup: 0 >> >> > dev.ixl.1.pf.que21.tso_tx: 0 >> >> > dev.ixl.1.pf.que21.irqs: 95 >> >> > dev.ixl.1.pf.que21.dropped: 0 >> >> > dev.ixl.1.pf.que21.mbuf_defrag_failed: 0 >> >> > dev.ixl.1.pf.que20.rx_bytes: 0 >> >> > dev.ixl.1.pf.que20.rx_packets: 0 >> >> > dev.ixl.1.pf.que20.tx_bytes: 7850 >> >> > dev.ixl.1.pf.que20.tx_packets: 98 >> >> > dev.ixl.1.pf.que20.no_desc_avail: 0 >> >> > dev.ixl.1.pf.que20.tx_dma_setup: 0 >> >> > dev.ixl.1.pf.que20.tso_tx: 0 >> >> > dev.ixl.1.pf.que20.irqs: 99 >> >> > dev.ixl.1.pf.que20.dropped: 0 >> >> > dev.ixl.1.pf.que20.mbuf_defrag_failed: 0 >> >> > dev.ixl.1.pf.que19.rx_bytes: 0 >> >> > dev.ixl.1.pf.que19.rx_packets: 0 >> >> > dev.ixl.1.pf.que19.tx_bytes: 64643 >> >> > dev.ixl.1.pf.que19.tx_packets: 202 >> >> > dev.ixl.1.pf.que19.no_desc_avail: 0 >> >> > dev.ixl.1.pf.que19.tx_dma_setup: 0 >> >> > dev.ixl.1.pf.que19.tso_tx: 0 >> >> > dev.ixl.1.pf.que19.irqs: 202 >> >> > dev.ixl.1.pf.que19.dropped: 0 >> >> > dev.ixl.1.pf.que19.mbuf_defrag_failed: 0 >> >> > dev.ixl.1.pf.que18.rx_bytes: 0 >> >> > dev.ixl.1.pf.que18.rx_packets: 0 >> >> > dev.ixl.1.pf.que18.tx_bytes: 5940 >> >> > dev.ixl.1.pf.que18.tx_packets: 74 >> >> > dev.ixl.1.pf.que18.no_desc_avail: 0 >> >> > dev.ixl.1.pf.que18.tx_dma_setup: 0 >> >> > dev.ixl.1.pf.que18.tso_tx: 0 >> >> > dev.ixl.1.pf.que18.irqs: 74 >> >> > dev.ixl.1.pf.que18.dropped: 0 >> >> > dev.ixl.1.pf.que18.mbuf_defrag_failed: 0 >> >> > dev.ixl.1.pf.que17.rx_bytes: 0 >> >> > dev.ixl.1.pf.que17.rx_packets: 0 >> >> > dev.ixl.1.pf.que17.tx_bytes: 11675 >> >> > dev.ixl.1.pf.que17.tx_packets: 83 >> >> > dev.ixl.1.pf.que17.no_desc_avail: 0 >> >> > dev.ixl.1.pf.que17.tx_dma_setup: 0 >> >> > dev.ixl.1.pf.que17.tso_tx: 0 >> >> > dev.ixl.1.pf.que17.irqs: 83 >> >> > dev.ixl.1.pf.que17.dropped: 0 >> >> > dev.ixl.1.pf.que17.mbuf_defrag_failed: 0 >> >> > dev.ixl.1.pf.que16.rx_bytes: 0 >> >> > dev.ixl.1.pf.que16.rx_packets: 0 >> >> > dev.ixl.1.pf.que16.tx_bytes: 105750457831 >> >> > dev.ixl.1.pf.que16.tx_packets: 205406766 >> >> > dev.ixl.1.pf.que16.no_desc_avail: 0 >> >> > dev.ixl.1.pf.que16.tx_dma_setup: 0 >> >> > dev.ixl.1.pf.que16.tso_tx: 0 >> >> > dev.ixl.1.pf.que16.irqs: 87222978 >> >> > dev.ixl.1.pf.que16.dropped: 0 >> >> > dev.ixl.1.pf.que16.mbuf_defrag_failed: 0 >> >> > dev.ixl.1.pf.que15.rx_bytes: 289558174088 >> >> > dev.ixl.1.pf.que15.rx_packets: 272466190 >> >> > dev.ixl.1.pf.que15.tx_bytes: 106152524681 >> >> > dev.ixl.1.pf.que15.tx_packets: 205379247 >> >> > dev.ixl.1.pf.que15.no_desc_avail: 0 >> >> > dev.ixl.1.pf.que15.tx_dma_setup: 0 >> >> > dev.ixl.1.pf.que15.tso_tx: 0 >> >> > dev.ixl.1.pf.que15.irqs: 238145862 >> >> > dev.ixl.1.pf.que15.dropped: 0 >> >> > dev.ixl.1.pf.que15.mbuf_defrag_failed: 0 >> >> > dev.ixl.1.pf.que14.rx_bytes: 301934533473 >> >> > dev.ixl.1.pf.que14.rx_packets: 298452930 >> >> > dev.ixl.1.pf.que14.tx_bytes: 111420393725 >> >> > dev.ixl.1.pf.que14.tx_packets: 215722532 >> >> > dev.ixl.1.pf.que14.no_desc_avail: 0 >> >> > dev.ixl.1.pf.que14.tx_dma_setup: 0 >> >> > dev.ixl.1.pf.que14.tso_tx: 0 >> >> > dev.ixl.1.pf.que14.irqs: 256291617 >> >> > dev.ixl.1.pf.que14.dropped: 0 >> >> > dev.ixl.1.pf.que14.mbuf_defrag_failed: 0 >> >> > dev.ixl.1.pf.que13.rx_bytes: 291380746253 >> >> > dev.ixl.1.pf.que13.rx_packets: 273037957 >> >> > dev.ixl.1.pf.que13.tx_bytes: 112417776222 >> >> > dev.ixl.1.pf.que13.tx_packets: 217500943 >> >> > dev.ixl.1.pf.que13.no_desc_avail: 0 >> >> > dev.ixl.1.pf.que13.tx_dma_setup: 0 >> >> > dev.ixl.1.pf.que13.tso_tx: 0 >> >> > dev.ixl.1.pf.que13.irqs: 241422331 >> >> > dev.ixl.1.pf.que13.dropped: 0 >> >> > dev.ixl.1.pf.que13.mbuf_defrag_failed: 0 >> >> > dev.ixl.1.pf.que12.rx_bytes: 301105585425 >> >> > dev.ixl.1.pf.que12.rx_packets: 286137817 >> >> > dev.ixl.1.pf.que12.tx_bytes: 95851784579 >> >> > dev.ixl.1.pf.que12.tx_packets: 199715765 >> >> > dev.ixl.1.pf.que12.no_desc_avail: 0 >> >> > dev.ixl.1.pf.que12.tx_dma_setup: 0 >> >> > dev.ixl.1.pf.que12.tso_tx: 0 >> >> > dev.ixl.1.pf.que12.irqs: 247322880 >> >> > dev.ixl.1.pf.que12.dropped: 0 >> >> > dev.ixl.1.pf.que12.mbuf_defrag_failed: 0 >> >> > dev.ixl.1.pf.que11.rx_bytes: 307105398143 >> >> > dev.ixl.1.pf.que11.rx_packets: 281046463 >> >> > dev.ixl.1.pf.que11.tx_bytes: 110710957789 >> >> > dev.ixl.1.pf.que11.tx_packets: 211784031 >> >> > dev.ixl.1.pf.que11.no_desc_avail: 0 >> >> > dev.ixl.1.pf.que11.tx_dma_setup: 0 >> >> > dev.ixl.1.pf.que11.tso_tx: 0 >> >> > dev.ixl.1.pf.que11.irqs: 256987179 >> >> > dev.ixl.1.pf.que11.dropped: 0 >> >> > dev.ixl.1.pf.que11.mbuf_defrag_failed: 0 >> >> > dev.ixl.1.pf.que10.rx_bytes: 304288000453 >> >> > dev.ixl.1.pf.que10.rx_packets: 278987858 >> >> > dev.ixl.1.pf.que10.tx_bytes: 93022244338 >> >> > dev.ixl.1.pf.que10.tx_packets: 195869210 >> >> > dev.ixl.1.pf.que10.no_desc_avail: 0 >> >> > dev.ixl.1.pf.que10.tx_dma_setup: 0 >> >> > dev.ixl.1.pf.que10.tso_tx: 0 >> >> > dev.ixl.1.pf.que10.irqs: 253622192 >> >> > dev.ixl.1.pf.que10.dropped: 0 >> >> > dev.ixl.1.pf.que10.mbuf_defrag_failed: 0 >> >> > dev.ixl.1.pf.que9.rx_bytes: 320340203822 >> >> > dev.ixl.1.pf.que9.rx_packets: 302309010 >> >> > dev.ixl.1.pf.que9.tx_bytes: 116604776460 >> >> > dev.ixl.1.pf.que9.tx_packets: 223949025 >> >> > dev.ixl.1.pf.que9.no_desc_avail: 0 >> >> > dev.ixl.1.pf.que9.tx_dma_setup: 0 >> >> > dev.ixl.1.pf.que9.tso_tx: 0 >> >> > dev.ixl.1.pf.que9.irqs: 271165440 >> >> > dev.ixl.1.pf.que9.dropped: 0 >> >> > dev.ixl.1.pf.que9.mbuf_defrag_failed: 0 >> >> > dev.ixl.1.pf.que8.rx_bytes: 291403725592 >> >> > dev.ixl.1.pf.que8.rx_packets: 267859568 >> >> > dev.ixl.1.pf.que8.tx_bytes: 205745654558 >> >> > dev.ixl.1.pf.que8.tx_packets: 443349835 >> >> > dev.ixl.1.pf.que8.no_desc_avail: 0 >> >> > dev.ixl.1.pf.que8.tx_dma_setup: 0 >> >> > dev.ixl.1.pf.que8.tso_tx: 0 >> >> > dev.ixl.1.pf.que8.irqs: 254116755 >> >> > dev.ixl.1.pf.que8.dropped: 0 >> >> > dev.ixl.1.pf.que8.mbuf_defrag_failed: 0 >> >> > dev.ixl.1.pf.que7.rx_bytes: 673363127346 >> >> > dev.ixl.1.pf.que7.rx_packets: 617269774 >> >> > dev.ixl.1.pf.que7.tx_bytes: 203162891886 >> >> > dev.ixl.1.pf.que7.tx_packets: 443709339 >> >> > dev.ixl.1.pf.que7.no_desc_avail: 0 >> >> > dev.ixl.1.pf.que7.tx_dma_setup: 0 >> >> > dev.ixl.1.pf.que7.tso_tx: 0 >> >> > dev.ixl.1.pf.que7.irqs: 424706771 >> >> > dev.ixl.1.pf.que7.dropped: 0 >> >> > dev.ixl.1.pf.que7.mbuf_defrag_failed: 0 >> >> > dev.ixl.1.pf.que6.rx_bytes: 644709094218 >> >> > dev.ixl.1.pf.que6.rx_packets: 601892919 >> >> > dev.ixl.1.pf.que6.tx_bytes: 221661735032 >> >> > dev.ixl.1.pf.que6.tx_packets: 460127064 >> >> > dev.ixl.1.pf.que6.no_desc_avail: 0 >> >> > dev.ixl.1.pf.que6.tx_dma_setup: 0 >> >> > dev.ixl.1.pf.que6.tso_tx: 0 >> >> > dev.ixl.1.pf.que6.irqs: 417748074 >> >> > dev.ixl.1.pf.que6.dropped: 0 >> >> > dev.ixl.1.pf.que6.mbuf_defrag_failed: 0 >> >> > dev.ixl.1.pf.que5.rx_bytes: 661904432231 >> >> > dev.ixl.1.pf.que5.rx_packets: 622012837 >> >> > dev.ixl.1.pf.que5.tx_bytes: 230514282876 >> >> > dev.ixl.1.pf.que5.tx_packets: 458571100 >> >> > dev.ixl.1.pf.que5.no_desc_avail: 0 >> >> > dev.ixl.1.pf.que5.tx_dma_setup: 0 >> >> > dev.ixl.1.pf.que5.tso_tx: 0 >> >> > dev.ixl.1.pf.que5.irqs: 422305039 >> >> > dev.ixl.1.pf.que5.dropped: 0 >> >> > dev.ixl.1.pf.que5.mbuf_defrag_failed: 0 >> >> > dev.ixl.1.pf.que4.rx_bytes: 653522179234 >> >> > dev.ixl.1.pf.que4.rx_packets: 603345546 >> >> > dev.ixl.1.pf.que4.tx_bytes: 216761219483 >> >> > dev.ixl.1.pf.que4.tx_packets: 450329641 >> >> > dev.ixl.1.pf.que4.no_desc_avail: 0 >> >> > dev.ixl.1.pf.que4.tx_dma_setup: 0 >> >> > dev.ixl.1.pf.que4.tso_tx: 3 >> >> > dev.ixl.1.pf.que4.irqs: 416920533 >> >> > dev.ixl.1.pf.que4.dropped: 0 >> >> > dev.ixl.1.pf.que4.mbuf_defrag_failed: 0 >> >> > dev.ixl.1.pf.que3.rx_bytes: 676494225882 >> >> > dev.ixl.1.pf.que3.rx_packets: 620605168 >> >> > dev.ixl.1.pf.que3.tx_bytes: 233854020454 >> >> > dev.ixl.1.pf.que3.tx_packets: 464425616 >> >> > dev.ixl.1.pf.que3.no_desc_avail: 0 >> >> > dev.ixl.1.pf.que3.tx_dma_setup: 0 >> >> > dev.ixl.1.pf.que3.tso_tx: 0 >> >> > dev.ixl.1.pf.que3.irqs: 426349030 >> >> > dev.ixl.1.pf.que3.dropped: 0 >> >> > dev.ixl.1.pf.que3.mbuf_defrag_failed: 0 >> >> > dev.ixl.1.pf.que2.rx_bytes: 677779337711 >> >> > dev.ixl.1.pf.que2.rx_packets: 620883699 >> >> > dev.ixl.1.pf.que2.tx_bytes: 211297141668 >> >> > dev.ixl.1.pf.que2.tx_packets: 450501525 >> >> > dev.ixl.1.pf.que2.no_desc_avail: 0 >> >> > dev.ixl.1.pf.que2.tx_dma_setup: 0 >> >> > dev.ixl.1.pf.que2.tso_tx: 0 >> >> > dev.ixl.1.pf.que2.irqs: 433146278 >> >> > dev.ixl.1.pf.que2.dropped: 0 >> >> > dev.ixl.1.pf.que2.mbuf_defrag_failed: 0 >> >> > dev.ixl.1.pf.que1.rx_bytes: 661360798018 >> >> > dev.ixl.1.pf.que1.rx_packets: 619700636 >> >> > dev.ixl.1.pf.que1.tx_bytes: 238264220772 >> >> > dev.ixl.1.pf.que1.tx_packets: 473425354 >> >> > dev.ixl.1.pf.que1.no_desc_avail: 0 >> >> > dev.ixl.1.pf.que1.tx_dma_setup: 0 >> >> > dev.ixl.1.pf.que1.tso_tx: 0 >> >> > dev.ixl.1.pf.que1.irqs: 437959829 >> >> > dev.ixl.1.pf.que1.dropped: 0 >> >> > dev.ixl.1.pf.que1.mbuf_defrag_failed: 0 >> >> > dev.ixl.1.pf.que0.rx_bytes: 685201226330 >> >> > dev.ixl.1.pf.que0.rx_packets: 637772348 >> >> > dev.ixl.1.pf.que0.tx_bytes: 124808 >> >> > dev.ixl.1.pf.que0.tx_packets: 1782 >> >> > dev.ixl.1.pf.que0.no_desc_avail: 0 >> >> > dev.ixl.1.pf.que0.tx_dma_setup: 0 >> >> > dev.ixl.1.pf.que0.tso_tx: 0 >> >> > dev.ixl.1.pf.que0.irqs: 174905480 >> >> > dev.ixl.1.pf.que0.dropped: 0 >> >> > dev.ixl.1.pf.que0.mbuf_defrag_failed: 0 >> >> > dev.ixl.1.pf.bcast_pkts_txd: 277 >> >> > dev.ixl.1.pf.mcast_pkts_txd: 0 >> >> > dev.ixl.1.pf.ucast_pkts_txd: 5319743945 >> >> > dev.ixl.1.pf.good_octets_txd: 2613178367282 >> >> > dev.ixl.1.pf.rx_discards: 0 >> >> > dev.ixl.1.pf.bcast_pkts_rcvd: 1 >> >> > dev.ixl.1.pf.mcast_pkts_rcvd: 0 >> >> > dev.ixl.1.pf.ucast_pkts_rcvd: 7203747890 >> >> > dev.ixl.1.pf.good_octets_rcvd: 7770230490224 >> >> > dev.ixl.1.vc_debug_level: 1 >> >> > dev.ixl.1.admin_irq: 0 >> >> > dev.ixl.1.watchdog_events: 0 >> >> > dev.ixl.1.debug: 0 >> >> > dev.ixl.1.dynamic_tx_itr: 0 >> >> > dev.ixl.1.tx_itr: 122 >> >> > dev.ixl.1.dynamic_rx_itr: 0 >> >> > dev.ixl.1.rx_itr: 62 >> >> > dev.ixl.1.fw_version: f4.33 a1.2 n04.42 e8000191d >> >> > dev.ixl.1.current_speed: 10G >> >> > dev.ixl.1.advertise_speed: 0 >> >> > dev.ixl.1.fc: 0 >> >> > dev.ixl.1.%parent: pci129 >> >> > dev.ixl.1.%pnpinfo: vendor=3D0x8086 device=3D0x1572 subvendor= =3D0x8086 >> >> > subdevice=3D0x0000 class=3D0x020000 >> >> > dev.ixl.1.%location: slot=3D0 function=3D1 >> handle=3D\_SB_.PCI1.QR3A.H001 >> >> > dev.ixl.1.%driver: ixl >> >> > dev.ixl.1.%desc: Intel(R) Ethernet Connection XL710 Driver, >> >> Version - 1.4.0 >> >> > dev.ixl.0.mac.xoff_recvd: 0 >> >> > dev.ixl.0.mac.xoff_txd: 0 >> >> > dev.ixl.0.mac.xon_recvd: 0 >> >> > dev.ixl.0.mac.xon_txd: 0 >> >> > dev.ixl.0.mac.tx_frames_big: 0 >> >> > dev.ixl.0.mac.tx_frames_1024_1522: 4961134019 >> >> > dev.ixl.0.mac.tx_frames_512_1023: 113082136 >> >> > dev.ixl.0.mac.tx_frames_256_511: 123538450 >> >> > dev.ixl.0.mac.tx_frames_128_255: 185051082 >> >> > dev.ixl.0.mac.tx_frames_65_127: 1332798493 >> >> > dev.ixl.0.mac.tx_frames_64: 243338964 >> >> > dev.ixl.0.mac.checksum_errors: 0 >> >> > dev.ixl.0.mac.rx_jabber: 0 >> >> > dev.ixl.0.mac.rx_oversized: 0 >> >> > dev.ixl.0.mac.rx_fragmented: 0 >> >> > dev.ixl.0.mac.rx_undersize: 0 >> >> > dev.ixl.0.mac.rx_frames_big: 0 >> >> > dev.ixl.0.mac.rx_frames_1024_1522: 1566499069 >> >> > dev.ixl.0.mac.rx_frames_512_1023: 101390143 >> >> > dev.ixl.0.mac.rx_frames_256_511: 49831970 >> >> > dev.ixl.0.mac.rx_frames_128_255: 231738168 >> >> > dev.ixl.0.mac.rx_frames_65_127: 2123185819 >> >> > dev.ixl.0.mac.rx_frames_64: 1320404300 >> >> > dev.ixl.0.mac.rx_length_errors: 0 >> >> > dev.ixl.0.mac.remote_faults: 0 >> >> > dev.ixl.0.mac.local_faults: 0 >> >> > dev.ixl.0.mac.illegal_bytes: 0 >> >> > dev.ixl.0.mac.crc_errors: 0 >> >> > dev.ixl.0.mac.bcast_pkts_txd: 302 >> >> > dev.ixl.0.mac.mcast_pkts_txd: 33965 >> >> > dev.ixl.0.mac.ucast_pkts_txd: 6958908862 >> >> > dev.ixl.0.mac.good_octets_txd: 7698936138858 >> >> > dev.ixl.0.mac.rx_discards: 0 >> >> > dev.ixl.0.mac.bcast_pkts_rcvd: 1 >> >> > dev.ixl.0.mac.mcast_pkts_rcvd: 49693 >> >> > dev.ixl.0.mac.ucast_pkts_rcvd: 5392999771 >> >> > dev.ixl.0.mac.good_octets_rcvd: 2648906893811 >> >> > dev.ixl.0.pf.que23.rx_bytes: 0 >> >> > dev.ixl.0.pf.que23.rx_packets: 0 >> >> > dev.ixl.0.pf.que23.tx_bytes: 2371273 >> >> > dev.ixl.0.pf.que23.tx_packets: 7313 >> >> > dev.ixl.0.pf.que23.no_desc_avail: 0 >> >> > dev.ixl.0.pf.que23.tx_dma_setup: 0 >> >> > dev.ixl.0.pf.que23.tso_tx: 0 >> >> > dev.ixl.0.pf.que23.irqs: 7313 >> >> > dev.ixl.0.pf.que23.dropped: 0 >> >> > dev.ixl.0.pf.que23.mbuf_defrag_failed: 0 >> >> > dev.ixl.0.pf.que22.rx_bytes: 0 >> >> > dev.ixl.0.pf.que22.rx_packets: 0 >> >> > dev.ixl.0.pf.que22.tx_bytes: 1908468 >> >> > dev.ixl.0.pf.que22.tx_packets: 6626 >> >> > dev.ixl.0.pf.que22.no_desc_avail: 0 >> >> > dev.ixl.0.pf.que22.tx_dma_setup: 0 >> >> > dev.ixl.0.pf.que22.tso_tx: 0 >> >> > dev.ixl.0.pf.que22.irqs: 6627 >> >> > dev.ixl.0.pf.que22.dropped: 0 >> >> > dev.ixl.0.pf.que22.mbuf_defrag_failed: 0 >> >> > dev.ixl.0.pf.que21.rx_bytes: 0 >> >> > dev.ixl.0.pf.que21.rx_packets: 0 >> >> > dev.ixl.0.pf.que21.tx_bytes: 2092668 >> >> > dev.ixl.0.pf.que21.tx_packets: 6739 >> >> > dev.ixl.0.pf.que21.no_desc_avail: 0 >> >> > dev.ixl.0.pf.que21.tx_dma_setup: 0 >> >> > dev.ixl.0.pf.que21.tso_tx: 0 >> >> > dev.ixl.0.pf.que21.irqs: 6728 >> >> > dev.ixl.0.pf.que21.dropped: 0 >> >> > dev.ixl.0.pf.que21.mbuf_defrag_failed: 0 >> >> > dev.ixl.0.pf.que20.rx_bytes: 0 >> >> > dev.ixl.0.pf.que20.rx_packets: 0 >> >> > dev.ixl.0.pf.que20.tx_bytes: 1742176 >> >> > dev.ixl.0.pf.que20.tx_packets: 6246 >> >> > dev.ixl.0.pf.que20.no_desc_avail: 0 >> >> > dev.ixl.0.pf.que20.tx_dma_setup: 0 >> >> > dev.ixl.0.pf.que20.tso_tx: 0 >> >> > dev.ixl.0.pf.que20.irqs: 6249 >> >> > dev.ixl.0.pf.que20.dropped: 0 >> >> > dev.ixl.0.pf.que20.mbuf_defrag_failed: 0 >> >> > dev.ixl.0.pf.que19.rx_bytes: 0 >> >> > dev.ixl.0.pf.que19.rx_packets: 0 >> >> > dev.ixl.0.pf.que19.tx_bytes: 2102284 >> >> > dev.ixl.0.pf.que19.tx_packets: 6979 >> >> > dev.ixl.0.pf.que19.no_desc_avail: 0 >> >> > dev.ixl.0.pf.que19.tx_dma_setup: 0 >> >> > dev.ixl.0.pf.que19.tso_tx: 0 >> >> > dev.ixl.0.pf.que19.irqs: 6979 >> >> > dev.ixl.0.pf.que19.dropped: 0 >> >> > dev.ixl.0.pf.que19.mbuf_defrag_failed: 0 >> >> > dev.ixl.0.pf.que18.rx_bytes: 0 >> >> > dev.ixl.0.pf.que18.rx_packets: 0 >> >> > dev.ixl.0.pf.que18.tx_bytes: 1532360 >> >> > dev.ixl.0.pf.que18.tx_packets: 5588 >> >> > dev.ixl.0.pf.que18.no_desc_avail: 0 >> >> > dev.ixl.0.pf.que18.tx_dma_setup: 0 >> >> > dev.ixl.0.pf.que18.tso_tx: 0 >> >> > dev.ixl.0.pf.que18.irqs: 5588 >> >> > dev.ixl.0.pf.que18.dropped: 0 >> >> > dev.ixl.0.pf.que18.mbuf_defrag_failed: 0 >> >> > dev.ixl.0.pf.que17.rx_bytes: 0 >> >> > dev.ixl.0.pf.que17.rx_packets: 0 >> >> > dev.ixl.0.pf.que17.tx_bytes: 1809684 >> >> > dev.ixl.0.pf.que17.tx_packets: 6136 >> >> > dev.ixl.0.pf.que17.no_desc_avail: 0 >> >> > dev.ixl.0.pf.que17.tx_dma_setup: 0 >> >> > dev.ixl.0.pf.que17.tso_tx: 0 >> >> > dev.ixl.0.pf.que17.irqs: 6136 >> >> > dev.ixl.0.pf.que17.dropped: 0 >> >> > dev.ixl.0.pf.que17.mbuf_defrag_failed: 0 >> >> > dev.ixl.0.pf.que16.rx_bytes: 0 >> >> > dev.ixl.0.pf.que16.rx_packets: 0 >> >> > dev.ixl.0.pf.que16.tx_bytes: 286836299105 >> >> > dev.ixl.0.pf.que16.tx_packets: 263532601 >> >> > dev.ixl.0.pf.que16.no_desc_avail: 0 >> >> > dev.ixl.0.pf.que16.tx_dma_setup: 0 >> >> > dev.ixl.0.pf.que16.tso_tx: 0 >> >> > dev.ixl.0.pf.que16.irqs: 83232941 >> >> > dev.ixl.0.pf.que16.dropped: 0 >> >> > dev.ixl.0.pf.que16.mbuf_defrag_failed: 0 >> >> > dev.ixl.0.pf.que15.rx_bytes: 106345323488 >> >> > dev.ixl.0.pf.que15.rx_packets: 208869912 >> >> > dev.ixl.0.pf.que15.tx_bytes: 298825179301 >> >> > dev.ixl.0.pf.que15.tx_packets: 288517504 >> >> > dev.ixl.0.pf.que15.no_desc_avail: 0 >> >> > dev.ixl.0.pf.que15.tx_dma_setup: 0 >> >> > dev.ixl.0.pf.que15.tso_tx: 0 >> >> > dev.ixl.0.pf.que15.irqs: 223322408 >> >> > dev.ixl.0.pf.que15.dropped: 0 >> >> > dev.ixl.0.pf.que15.mbuf_defrag_failed: 0 >> >> > dev.ixl.0.pf.que14.rx_bytes: 106721900547 >> >> > dev.ixl.0.pf.que14.rx_packets: 208566121 >> >> > dev.ixl.0.pf.que14.tx_bytes: 288657751920 >> >> > dev.ixl.0.pf.que14.tx_packets: 263556000 >> >> > dev.ixl.0.pf.que14.no_desc_avail: 0 >> >> > dev.ixl.0.pf.que14.tx_dma_setup: 0 >> >> > dev.ixl.0.pf.que14.tso_tx: 0 >> >> > dev.ixl.0.pf.que14.irqs: 220377537 >> >> > dev.ixl.0.pf.que14.dropped: 0 >> >> > dev.ixl.0.pf.que14.mbuf_defrag_failed: 0 >> >> > dev.ixl.0.pf.que13.rx_bytes: 111978971378 >> >> > dev.ixl.0.pf.que13.rx_packets: 218447354 >> >> > dev.ixl.0.pf.que13.tx_bytes: 298439860675 >> >> > dev.ixl.0.pf.que13.tx_packets: 276806617 >> >> > dev.ixl.0.pf.que13.no_desc_avail: 0 >> >> > dev.ixl.0.pf.que13.tx_dma_setup: 0 >> >> > dev.ixl.0.pf.que13.tso_tx: 0 >> >> > dev.ixl.0.pf.que13.irqs: 227474625 >> >> > dev.ixl.0.pf.que13.dropped: 0 >> >> > dev.ixl.0.pf.que13.mbuf_defrag_failed: 0 >> >> > dev.ixl.0.pf.que12.rx_bytes: 112969704706 >> >> > dev.ixl.0.pf.que12.rx_packets: 220275562 >> >> > dev.ixl.0.pf.que12.tx_bytes: 304750620079 >> >> > dev.ixl.0.pf.que12.tx_packets: 272244483 >> >> > dev.ixl.0.pf.que12.no_desc_avail: 0 >> >> > dev.ixl.0.pf.que12.tx_dma_setup: 0 >> >> > dev.ixl.0.pf.que12.tso_tx: 183 >> >> > dev.ixl.0.pf.que12.irqs: 230111291 >> >> > dev.ixl.0.pf.que12.dropped: 0 >> >> > dev.ixl.0.pf.que12.mbuf_defrag_failed: 0 >> >> > dev.ixl.0.pf.que11.rx_bytes: 96405343036 >> >> > dev.ixl.0.pf.que11.rx_packets: 202329448 >> >> > dev.ixl.0.pf.que11.tx_bytes: 302481707696 >> >> > dev.ixl.0.pf.que11.tx_packets: 271689246 >> >> > dev.ixl.0.pf.que11.no_desc_avail: 0 >> >> > dev.ixl.0.pf.que11.tx_dma_setup: 0 >> >> > dev.ixl.0.pf.que11.tso_tx: 0 >> >> > dev.ixl.0.pf.que11.irqs: 220717612 >> >> > dev.ixl.0.pf.que11.dropped: 0 >> >> > dev.ixl.0.pf.que11.mbuf_defrag_failed: 0 >> >> > dev.ixl.0.pf.que10.rx_bytes: 111280008670 >> >> > dev.ixl.0.pf.que10.rx_packets: 214900261 >> >> > dev.ixl.0.pf.que10.tx_bytes: 318638566198 >> >> > dev.ixl.0.pf.que10.tx_packets: 295011389 >> >> > dev.ixl.0.pf.que10.no_desc_avail: 0 >> >> > dev.ixl.0.pf.que10.tx_dma_setup: 0 >> >> > dev.ixl.0.pf.que10.tso_tx: 0 >> >> > dev.ixl.0.pf.que10.irqs: 230681709 >> >> > dev.ixl.0.pf.que10.dropped: 0 >> >> > dev.ixl.0.pf.que10.mbuf_defrag_failed: 0 >> >> > dev.ixl.0.pf.que9.rx_bytes: 93566025126 >> >> > dev.ixl.0.pf.que9.rx_packets: 198726483 >> >> > dev.ixl.0.pf.que9.tx_bytes: 288858818348 >> >> > dev.ixl.0.pf.que9.tx_packets: 258926864 >> >> > dev.ixl.0.pf.que9.no_desc_avail: 0 >> >> > dev.ixl.0.pf.que9.tx_dma_setup: 0 >> >> > dev.ixl.0.pf.que9.tso_tx: 0 >> >> > dev.ixl.0.pf.que9.irqs: 217918160 >> >> > dev.ixl.0.pf.que9.dropped: 0 >> >> > dev.ixl.0.pf.que9.mbuf_defrag_failed: 0 >> >> > dev.ixl.0.pf.que8.rx_bytes: 117169019041 >> >> > dev.ixl.0.pf.que8.rx_packets: 226938172 >> >> > dev.ixl.0.pf.que8.tx_bytes: 665794492752 >> >> > dev.ixl.0.pf.que8.tx_packets: 593519436 >> >> > dev.ixl.0.pf.que8.no_desc_avail: 0 >> >> > dev.ixl.0.pf.que8.tx_dma_setup: 0 >> >> > dev.ixl.0.pf.que8.tso_tx: 0 >> >> > dev.ixl.0.pf.que8.irqs: 244643578 >> >> > dev.ixl.0.pf.que8.dropped: 0 >> >> > dev.ixl.0.pf.que8.mbuf_defrag_failed: 0 >> >> > dev.ixl.0.pf.que7.rx_bytes: 206974266022 >> >> > dev.ixl.0.pf.que7.rx_packets: 449899895 >> >> > dev.ixl.0.pf.que7.tx_bytes: 638527685820 >> >> > dev.ixl.0.pf.que7.tx_packets: 580750916 >> >> > dev.ixl.0.pf.que7.no_desc_avail: 0 >> >> > dev.ixl.0.pf.que7.tx_dma_setup: 0 >> >> > dev.ixl.0.pf.que7.tso_tx: 0 >> >> > dev.ixl.0.pf.que7.irqs: 391760959 >> >> > dev.ixl.0.pf.que7.dropped: 0 >> >> > dev.ixl.0.pf.que7.mbuf_defrag_failed: 0 >> >> > dev.ixl.0.pf.que6.rx_bytes: 204373984670 >> >> > dev.ixl.0.pf.que6.rx_packets: 449990985 >> >> > dev.ixl.0.pf.que6.tx_bytes: 655511068125 >> >> > dev.ixl.0.pf.que6.tx_packets: 600735086 >> >> > dev.ixl.0.pf.que6.no_desc_avail: 0 >> >> > dev.ixl.0.pf.que6.tx_dma_setup: 0 >> >> > dev.ixl.0.pf.que6.tso_tx: 0 >> >> > dev.ixl.0.pf.que6.irqs: 394961024 >> >> > dev.ixl.0.pf.que6.dropped: 0 >> >> > dev.ixl.0.pf.que6.mbuf_defrag_failed: 0 >> >> > dev.ixl.0.pf.que5.rx_bytes: 222919535872 >> >> > dev.ixl.0.pf.que5.rx_packets: 466659705 >> >> > dev.ixl.0.pf.que5.tx_bytes: 647689764751 >> >> > dev.ixl.0.pf.que5.tx_packets: 582532691 >> >> > dev.ixl.0.pf.que5.no_desc_avail: 0 >> >> > dev.ixl.0.pf.que5.tx_dma_setup: 0 >> >> > dev.ixl.0.pf.que5.tso_tx: 5 >> >> > dev.ixl.0.pf.que5.irqs: 404552229 >> >> > dev.ixl.0.pf.que5.dropped: 0 >> >> > dev.ixl.0.pf.que5.mbuf_defrag_failed: 0 >> >> > dev.ixl.0.pf.que4.rx_bytes: 231706806551 >> >> > > > > From owner-freebsd-net@freebsd.org Tue Aug 25 22:54:04 2015 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id D99C499A9D9 for ; Tue, 25 Aug 2015 22:54:04 +0000 (UTC) (envelope-from chris@stankevitz.com) Received: from mango.stankevitz.com (mango.stankevitz.com [208.79.93.194]) by mx1.freebsd.org (Postfix) with ESMTP id CA52F69D for ; Tue, 25 Aug 2015 22:54:04 +0000 (UTC) (envelope-from chris@stankevitz.com) Received: from Chriss-MacBook-Pro.local (209-203-101-124.static.twtelecom.net [209.203.101.124]) (using TLSv1.2 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by mango.stankevitz.com (Postfix) with ESMTPSA id 97E911356 for ; Tue, 25 Aug 2015 15:47:29 -0700 (PDT) Message-ID: <55DCF080.7080208@stankevitz.com> Date: Tue, 25 Aug 2015 15:47:28 -0700 From: Chris Stankevitz User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:31.0) Gecko/20100101 Thunderbird/31.7.0 MIME-Version: 1.0 To: freebsd-net@freebsd.org Subject: ssh over WAN: TCP window too small Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 25 Aug 2015 22:54:04 -0000 Hi, # cat /dev/urandom | ssh root@host 'cat > /dev/null' I use the above ssh command over a high-BDP WAN link (80 ms @ 100 Mbps). tcpdump shows I am TCP window limited to 64 KBytes (yielding 5 Mbps). iperf with default options gets the window opened to 500 KBytes (yielding 35 Mbps). Both sides of the connection: FreeBSD 10.1 w/default sshd options (except I permit root login). In particular, HPN is not disabled. Can anyone explain my abysmally small TCP window? Can anyone recommend some tools/tricks to figure out what in FreeBSD and/or base SSH is limiting the send/recv buffer and/or TCP window? Thank you, Chris From owner-freebsd-net@freebsd.org Tue Aug 25 23:11:38 2015 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id A21479C226F for ; Tue, 25 Aug 2015 23:11:38 +0000 (UTC) (envelope-from bzeeb-lists@lists.zabbadoz.net) Received: from mx1.sbone.de (bird.sbone.de [46.4.1.90]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client CN "mx1.sbone.de", Issuer "SBone.DE" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 59EFFBA for ; Tue, 25 Aug 2015 23:11:37 +0000 (UTC) (envelope-from bzeeb-lists@lists.zabbadoz.net) Received: from mail.sbone.de (mail.sbone.de [IPv6:fde9:577b:c1a9:31::2013:587]) (using TLSv1 with cipher ADH-CAMELLIA256-SHA (256/256 bits)) (No client certificate requested) by mx1.sbone.de (Postfix) with ESMTPS id 9B44925D3A9A; Tue, 25 Aug 2015 23:11:28 +0000 (UTC) Received: from content-filter.sbone.de (content-filter.sbone.de [IPv6:fde9:577b:c1a9:31::2013:2742]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mail.sbone.de (Postfix) with ESMTPS id DDD50C7705E; Tue, 25 Aug 2015 23:11:27 +0000 (UTC) X-Virus-Scanned: amavisd-new at sbone.de Received: from mail.sbone.de ([IPv6:fde9:577b:c1a9:31::2013:587]) by content-filter.sbone.de (content-filter.sbone.de [fde9:577b:c1a9:31::2013:2742]) (amavisd-new, port 10024) with ESMTP id ZrnzE8KfvnFM; Tue, 25 Aug 2015 23:11:26 +0000 (UTC) Received: from [IPv6:fde9:577b:c1a9:4410:9194:cf5a:e0dc:a813] (unknown [IPv6:fde9:577b:c1a9:4410:9194:cf5a:e0dc:a813]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mail.sbone.de (Postfix) with ESMTPSA id 46FD5C76FD9; Tue, 25 Aug 2015 23:11:25 +0000 (UTC) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 8.2 \(2104\)) Subject: Re: ssh over WAN: TCP window too small From: "Bjoern A. Zeeb" In-Reply-To: <55DCF080.7080208@stankevitz.com> Date: Tue, 25 Aug 2015 23:11:06 +0000 Cc: freebsd-net@freebsd.org Content-Transfer-Encoding: quoted-printable Message-Id: <27420EDC-5816-4B9E-A834-E4A035B8411C@lists.zabbadoz.net> References: <55DCF080.7080208@stankevitz.com> To: Chris Stankevitz X-Mailer: Apple Mail (2.2104) X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 25 Aug 2015 23:11:38 -0000 > On 25 Aug 2015, at 22:47 , Chris Stankevitz = wrote: >=20 > Hi, >=20 > # cat /dev/urandom | ssh root@host 'cat > /dev/null' >=20 > I use the above ssh command over a high-BDP WAN link (80 ms @ 100 = Mbps). tcpdump shows I am TCP window limited to 64 KBytes (yielding 5 = Mbps). iperf with default options gets the window opened to 500 KBytes = (yielding 35 Mbps). >=20 > Both sides of the connection: FreeBSD 10.1 w/default sshd options = (except I permit root login). In particular, HPN is not disabled. >=20 > Can anyone explain my abysmally small TCP window? >=20 > Can anyone recommend some tools/tricks to figure out what in FreeBSD = and/or base SSH is limiting the send/recv buffer and/or TCP window? if you have the memory, try these sysctls: kern.ipc.maxsockbuf=3D146800640 net.inet.tcp.recvbuf_max=3D67108864 net.inet.tcp.sendbuf_max=3D67108864= From owner-freebsd-net@freebsd.org Wed Aug 26 00:12:05 2015 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id E76799C3C5B for ; Wed, 26 Aug 2015 00:12:05 +0000 (UTC) (envelope-from chris@stankevitz.com) Received: from mango.stankevitz.com (mango.stankevitz.com [208.79.93.194]) by mx1.freebsd.org (Postfix) with ESMTP id D173DDB3 for ; Wed, 26 Aug 2015 00:12:05 +0000 (UTC) (envelope-from chris@stankevitz.com) Received: from Chriss-MacBook-Pro.local (209-203-101-124.static.twtelecom.net [209.203.101.124]) (using TLSv1.2 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by mango.stankevitz.com (Postfix) with ESMTPSA id 6DC2F136A; Tue, 25 Aug 2015 17:12:04 -0700 (PDT) Message-ID: <55DD0453.3040803@stankevitz.com> Date: Tue, 25 Aug 2015 17:12:03 -0700 From: Chris Stankevitz User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:31.0) Gecko/20100101 Thunderbird/31.7.0 MIME-Version: 1.0 To: "Bjoern A. Zeeb" CC: freebsd-net@freebsd.org Subject: Re: ssh over WAN: TCP window too small References: <55DCF080.7080208@stankevitz.com> <27420EDC-5816-4B9E-A834-E4A035B8411C@lists.zabbadoz.net> In-Reply-To: <27420EDC-5816-4B9E-A834-E4A035B8411C@lists.zabbadoz.net> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 26 Aug 2015 00:12:06 -0000 On 8/25/15 4:11 PM, Bjoern A. Zeeb wrote: > >> On 25 Aug 2015, at 22:47 , Chris Stankevitz wrote: >> >> Can anyone recommend some tools/tricks to figure out what in FreeBSD and/or >> base SSH is limiting the send/recv buffer and/or TCP window? > > if you have the memory, try these sysctls: > > kern.ipc.maxsockbuf=146800640 > net.inet.tcp.recvbuf_max=67108864 > net.inet.tcp.sendbuf_max=67108864 Bjoern, Thank you for the reply. Before your suggestion my sysctls are: kern.ipc.maxsockbuf=2097152 net.inet.tcp.recvbuf_max=2097152 net.inet.tcp.sendbuf_max=2097152 Each of these is much larger than the limit I am experiencing (~64,000). So I [naively] expect changing these values will have no effect. Let me try... ... okay... sure enough the sysctrl changes you suggest did not change the 64,000 bytes-in-flight limit I am experiencing. Thanks for the idea (and keep 'em coming!), Chris From owner-freebsd-net@freebsd.org Wed Aug 26 01:03:30 2015 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 6EFE39C220A for ; Wed, 26 Aug 2015 01:03:30 +0000 (UTC) (envelope-from jmg@gold.funkthat.com) Received: from gold.funkthat.com (gate2.funkthat.com [208.87.223.18]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client CN "gold.funkthat.com", Issuer "gold.funkthat.com" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 2B9AA6F1 for ; Wed, 26 Aug 2015 01:03:29 +0000 (UTC) (envelope-from jmg@gold.funkthat.com) Received: from gold.funkthat.com (localhost [127.0.0.1]) by gold.funkthat.com (8.14.5/8.14.5) with ESMTP id t7Q13NpX021967 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Tue, 25 Aug 2015 18:03:23 -0700 (PDT) (envelope-from jmg@gold.funkthat.com) Received: (from jmg@localhost) by gold.funkthat.com (8.14.5/8.14.5/Submit) id t7Q13NNm021966; Tue, 25 Aug 2015 18:03:23 -0700 (PDT) (envelope-from jmg) Date: Tue, 25 Aug 2015 18:03:23 -0700 From: John-Mark Gurney To: Chris Stankevitz Cc: freebsd-net@freebsd.org Subject: Re: ssh over WAN: TCP window too small Message-ID: <20150826010323.GN33167@funkthat.com> References: <55DCF080.7080208@stankevitz.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <55DCF080.7080208@stankevitz.com> X-Operating-System: FreeBSD 9.1-PRERELEASE amd64 X-PGP-Fingerprint: 54BA 873B 6515 3F10 9E88 9322 9CB1 8F74 6D3F A396 X-Files: The truth is out there X-URL: http://resnet.uoregon.edu/~gurney_j/ X-Resume: http://resnet.uoregon.edu/~gurney_j/resume.html X-TipJar: bitcoin:13Qmb6AeTgQecazTWph4XasEsP7nGRbAPE X-to-the-FBI-CIA-and-NSA: HI! HOW YA DOIN? can i haz chizburger? User-Agent: Mutt/1.5.21 (2010-09-15) X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.2.7 (gold.funkthat.com [127.0.0.1]); Tue, 25 Aug 2015 18:03:23 -0700 (PDT) X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 26 Aug 2015 01:03:30 -0000 Chris Stankevitz wrote this message on Tue, Aug 25, 2015 at 15:47 -0700: > # cat /dev/urandom | ssh root@host 'cat > /dev/null' Don't use this for testing... use /dev/zero or some other device that can produce data faster than this... > I use the above ssh command over a high-BDP WAN link (80 ms @ 100 Mbps). > tcpdump shows I am TCP window limited to 64 KBytes (yielding 5 Mbps). > iperf with default options gets the window opened to 500 KBytes > (yielding 35 Mbps). > > Both sides of the connection: FreeBSD 10.1 w/default sshd options > (except I permit root login). In particular, HPN is not disabled. > > Can anyone explain my abysmally small TCP window? Looks like ssh is propbably hard setting the send/recv buffers to values that are too small... So, our SSH does have the HPN patches: https://www.psc.edu/index.php/hpn-ssh and the README says: BUFFER SIZES: - if HPN is disabled the receive buffer size will be set to the OpenSSH default of 64K. You can read more at: https://svnweb.freebsd.org/base/stable/10/crypto/openssh/README.hpn?annotate=256281 Looks like there are undocumented options like TCPRcvBuf that you can use to adjust the recv buffer window... It looks like OpenSSH hard sets the buffer sizes for some reason... On FreeBSD, these should never be set unless the option is provided and you know what you are doing.. We have code that will auto grow buffer sizes properly so that slow connections won't use up too much buffer space... > Can anyone recommend some tools/tricks to figure out what in FreeBSD > and/or base SSH is limiting the send/recv buffer and/or TCP window? Seems like from looking at the code, things should "just work", so not sure why you are seeing the smaller window size... In a quick test of mine, I'm seeing a buffer size of ~520k from my MacOSX box, and ~776k from my 9.2-R box... Server in both cases is a June -CURRENT... netstat -xAanfinet is helpful on this... Hope this helps! -- John-Mark Gurney Voice: +1 415 225 5579 "All that I will do, has been done, All that I have, has not." From owner-freebsd-net@freebsd.org Wed Aug 26 02:55:22 2015 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 5412899AED5 for ; Wed, 26 Aug 2015 02:55:22 +0000 (UTC) (envelope-from chris@stankevitz.com) Received: from mango.stankevitz.com (mango.stankevitz.com [208.79.93.194]) by mx1.freebsd.org (Postfix) with ESMTP id 3530769E for ; Wed, 26 Aug 2015 02:55:21 +0000 (UTC) (envelope-from chris@stankevitz.com) Received: from Chriss-MacBook-Pro.local (209-203-101-124.static.twtelecom.net [209.203.101.124]) (using TLSv1.2 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by mango.stankevitz.com (Postfix) with ESMTPSA id EE43F1388; Tue, 25 Aug 2015 19:55:20 -0700 (PDT) Message-ID: <55DD2A98.2010605@stankevitz.com> Date: Tue, 25 Aug 2015 19:55:20 -0700 From: Chris Stankevitz User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:31.0) Gecko/20100101 Thunderbird/31.7.0 MIME-Version: 1.0 To: John-Mark Gurney CC: freebsd-net@freebsd.org Subject: Re: ssh over WAN: TCP window too small References: <55DCF080.7080208@stankevitz.com> <20150826010323.GN33167@funkthat.com> In-Reply-To: <20150826010323.GN33167@funkthat.com> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 26 Aug 2015 02:55:22 -0000 John-Mark, Thank you for your reply. On 8/25/15 6:03 PM, John-Mark Gurney wrote: > Chris Stankevitz wrote this message on Tue, Aug 25, 2015 at 15:47 -0700: >> # cat /dev/urandom | ssh root@host 'cat > /dev/null' > > Don't use this for testing... use /dev/zero or some other device > that can produce data faster than this... Okay. As I'm sure you can imagine, I used urandom to avoid compression artifacts. My urandom produces data at ~300 Mbps... but I will use /dev/zero from now on. > So, our SSH does have the HPN patches: > https://www.psc.edu/index.php/hpn-ssh > > and the README says: > BUFFER SIZES: > - if HPN is disabled the receive buffer size will be set to the OpenSSH default > of 64K. Yes... I spent some time reading that document and fretting over whether or not HPN was really incorporated in my setup. I "confirmed" that it was available and enabled by setting "HPNDisabled no" and restarting sshd (on both sides) without complaint. I'm half-tempted to build from ports to be certain. > Looks like there are undocumented options like TCPRcvBuf that you can > use to adjust the recv buffer window... According to the HPN README the default (which I am using) is the "system wide TCP receive buffer size". I don't know what value that is or where it comes from (net.inet.tcp.???). I will experiment with TCPRcvBuf. > We have code that will auto grow > buffer sizes properly so that slow connections won't use up too much > buffer space... That is what I expected, although I believe openssh tries thwart/limit this by requesting particular buffer sizes (I'm really unqualified to talk about this). And it is my understanding that HPN undoes these limitations although I'm not sure if it opens the door to FreeBSD having full control or uses its own voodoo. > In a quick test of mine, I'm seeing a buffer size of ~520k from my > MacOSX box, and ~776k from my 9.2-R box... Server in both cases is > a June -CURRENT Thank you for those numbers. Since my system is basically stock, I wonder if my bad behavior is an artifact of something on my network. Did you invoke ssh more or less as "cat /dev/zero | ssh root@host 'cat > /dev/null'"? Are you quoting S-BCNT numbers? > netstat -xAanfinet is helpful on this... That is brilliant! I was using pcap and wireshark to deduce some of that info. I include my sender and receiver netstat's below for the ssh-ing /dev/zero. It differs from iperf (which works well), most notably in S-BCNT (~1MB for iperf, ~64kB for ssh). I think in my case the question is: - who is keeping S-BCNT so low (openssh, HPN, or FreeBSD)? - Is the limitation introduced by the sending or receiving system? - what is the mechanism by which S-BCNT grows when using ssh over long/fat pipes? Thank you again, Chris SSH Sender Recv-Q 0 Send-Q 50132 R-MBUF 0 S-MBUF 16 R-CLUS 0 S-CLUS 14 R-HIWA 66052 S-HIWA 82852 R-LOWA 1 S-LOWA 2048 R-BCNT 0 S-BCNT 57344 R-BMAX 528416 S-BMAX 662816 rexmt 0.29 persist 0 keep 7199.98 2msl 0 delack 0 rcvtime 0.01 SSH Receiver Recv-Q 0 Send-Q 36 R-MBUF 0 S-MBUF 1 R-CLUS 0 S-CLUS 0 R-HIWA 66052 S-HIWA 33700 R-LOWA 1 S-LOWA 2048 R-BCNT 0 S-BCNT 256 R-BMAX 528416 S-BMAX 269600 rexmt 0.24 persist 0 keep 7199.96 2msl 0 delack 0.06 rcvtime 0.03 From owner-freebsd-net@freebsd.org Wed Aug 26 05:56:41 2015 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id ECCED9C0A8E; Wed, 26 Aug 2015 05:56:40 +0000 (UTC) (envelope-from danny@cs.huji.ac.il) Received: from kabab.cs.huji.ac.il (kabab.cs.huji.ac.il [132.65.116.210]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 7841ECB4; Wed, 26 Aug 2015 05:56:40 +0000 (UTC) (envelope-from danny@cs.huji.ac.il) Received: from mbpro-w.cs.huji.ac.il ([132.65.80.91]) by kabab.cs.huji.ac.il with esmtp id 1ZUThD-000NRe-W0; Wed, 26 Aug 2015 08:56:28 +0300 Mime-Version: 1.0 (Mac OS X Mail 8.2 \(2104\)) Subject: Re: ix(intel) vs mlxen(mellanox) 10Gb performance From: Daniel Braniss In-Reply-To: <2112273205.29795512.1440419111720.JavaMail.zimbra@uoguelph.ca> Date: Wed, 26 Aug 2015 08:56:27 +0300 Cc: Hans Petter Selasky , pyunyh@gmail.com, FreeBSD Net , FreeBSD stable , Gleb Smirnoff Message-Id: <1E679659-BA50-42C3-B569-03579E322685@cs.huji.ac.il> References: <1D52028A-B39F-4F9B-BD38-CB1D73BF5D56@cs.huji.ac.il> <1153838447.28656490.1440193567940.JavaMail.zimbra@uoguelph.ca> <15D19823-08F7-4E55-BBD0-CE230F67D26E@cs.huji.ac.il> <818666007.28930310.1440244756872.JavaMail.zimbra@uoguelph.ca> <49173B1F-7B5E-4D59-8651-63D97B0CB5AC@cs.huji.ac.il> <1815942485.29539597.1440370972998.JavaMail.zimbra@uoguelph.ca> <55DAC623.60006@selasky.org> <62C7B1A3-CC6B-41A1-B254-6399F19F8FF7@cs.huji.ac.il> <2112273205.29795512.1440419111720.JavaMail.zimbra@uoguelph.ca> To: Rick Macklem X-Mailer: Apple Mail (2.2104) Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 26 Aug 2015 05:56:41 -0000 > On Aug 24, 2015, at 3:25 PM, Rick Macklem = wrote: >=20 > Daniel Braniss wrote: >>=20 >>> On 24 Aug 2015, at 10:22, Hans Petter Selasky = wrote: >>>=20 >>> On 08/24/15 01:02, Rick Macklem wrote: >>>> The other thing is the degradation seems to cut the rate by about = half >>>> each time. >>>> 300-->150-->70 I have no idea if this helps to explain it. >>>=20 >>> Might be a NUMA binding issue for the processes involved. >>>=20 >>> man cpuset >>>=20 >>> --HPS >>=20 >> I can=E2=80=99t see how this is relevant, given that the same host, = using the >> mellanox/mlxen >> behave much better. > Well, the "ix" driver has a bunch of tunables for things like "number = of queues" > and although I'll admit I don't understand how these queues are used, = I think > they are related to CPUs and their caches. There is also something = called IXGBE_FDIR, > which others have recommended be disabled. (The code is #ifdef = IXGBE_FDIR, but I don't > know if it defined for your kernel?) There are also tunables for = interrupt rate and > something called hw.ixgbe_tx_process_limit, which appears to limit the = number of packets > to send or something like that? > (I suspect Hans would understand this stuff much better than I do, = since I don't understand > it at all.;-) >=20 but how does this explain the fact that, at the same time, the throughput to the NetApp is about 70MG/s while to a FreeBSD it=E2=80=99s above 150MB/s? (window size negotiation?) switching off TSO evens out this diff. > At a glance, the mellanox driver looks very different. >=20 >> I=E2=80=99m getting different results with the intel/ix depending who = is the nfs >> server >>=20 > Who knows until you figure out what is actually going on. It could = just be the timing of > handling the write RPCs or when the different servers send acks for = the TCP segments or ... > that causes this for one server and not another. >=20 > One of the principals used when investigating airplane accidents is to = "never assume anything" > and just try to collect the facts until the pieces of the puzzle fall = in place. I think the > same principal works for this kind of stuff. > I once had a case where a specific read of one NFS file would fail on = certain machines. > I won't bore you with the details, but after weeks we got to the point = where we had a lab > of identical machines (exactly the same hardware and exactly the same = software loaded on them) > and we could reproduce this problem on about half the machines and not = the other half. We > (myself and the guy I worked with) finally noticed the failing = machines were on network ports > for a given switch. We moved the net cables to another switch and the = problem went away. > --> This particular network switch was broken in such a way that it = would garble one specific > packet consistently, but worked fine for everything else. > My point here is that, if someone had suggested the "network switch = might be broken" at the > beginning of investigating this, I would have probably dismissed it, = based on "the network is > working just fine", but in the end, that was the problem. > --> I am not suggesting you have a broken network switch, just "don't = take anything off the > table until you know what is actually going on". >=20 > And to be honest, you may never know, but it is fun to try and solve = these puzzles. one needs to find the clues =E2=80=A6 at the moment: when things go bad, they stay bad ix/nfs/tcp/tso and NetApp when things are ok, the numbers fluctuate, which is probably due = to loads on the system, but they are far above the 70MB/s (100 to 200) > Beyond what I already suggested, I'd look at the "ix" driver's stats = and tunables and > see if any of the tunables has an effect. (And, yes, it will take time = to work through these.) >=20 > Good luck with it, rick >=20 >>=20 >> danny >>=20 >> _______________________________________________ >> freebsd-stable@freebsd.org = mailing list >> https://lists.freebsd.org/mailman/listinfo/freebsd-stable = >> To unsubscribe, send any mail to = "freebsd-stable-unsubscribe@freebsd.org = " From owner-freebsd-net@freebsd.org Wed Aug 26 08:31:44 2015 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id CEB0099A67A for ; Wed, 26 Aug 2015 08:31:44 +0000 (UTC) (envelope-from jmg@gold.funkthat.com) Received: from gold.funkthat.com (gate2.funkthat.com [208.87.223.18]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client CN "gold.funkthat.com", Issuer "gold.funkthat.com" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id B16528AC for ; Wed, 26 Aug 2015 08:31:44 +0000 (UTC) (envelope-from jmg@gold.funkthat.com) Received: from gold.funkthat.com (localhost [127.0.0.1]) by gold.funkthat.com (8.14.5/8.14.5) with ESMTP id t7Q8Ov9S026986 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Wed, 26 Aug 2015 01:24:57 -0700 (PDT) (envelope-from jmg@gold.funkthat.com) Received: (from jmg@localhost) by gold.funkthat.com (8.14.5/8.14.5/Submit) id t7Q8OvXV026985; Wed, 26 Aug 2015 01:24:57 -0700 (PDT) (envelope-from jmg) Date: Wed, 26 Aug 2015 01:24:57 -0700 From: John-Mark Gurney To: Chris Stankevitz Cc: freebsd-net@freebsd.org Subject: Re: ssh over WAN: TCP window too small Message-ID: <20150826082457.GQ33167@funkthat.com> References: <55DCF080.7080208@stankevitz.com> <20150826010323.GN33167@funkthat.com> <55DD2A98.2010605@stankevitz.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <55DD2A98.2010605@stankevitz.com> X-Operating-System: FreeBSD 9.1-PRERELEASE amd64 X-PGP-Fingerprint: 54BA 873B 6515 3F10 9E88 9322 9CB1 8F74 6D3F A396 X-Files: The truth is out there X-URL: http://resnet.uoregon.edu/~gurney_j/ X-Resume: http://resnet.uoregon.edu/~gurney_j/resume.html X-TipJar: bitcoin:13Qmb6AeTgQecazTWph4XasEsP7nGRbAPE X-to-the-FBI-CIA-and-NSA: HI! HOW YA DOIN? can i haz chizburger? User-Agent: Mutt/1.5.21 (2010-09-15) X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.2.7 (gold.funkthat.com [127.0.0.1]); Wed, 26 Aug 2015 01:24:57 -0700 (PDT) X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 26 Aug 2015 08:31:44 -0000 Chris Stankevitz wrote this message on Tue, Aug 25, 2015 at 19:55 -0700: > John-Mark, > > Thank you for your reply. > > On 8/25/15 6:03 PM, John-Mark Gurney wrote: > > Chris Stankevitz wrote this message on Tue, Aug 25, 2015 at 15:47 -0700: > >> # cat /dev/urandom | ssh root@host 'cat > /dev/null' > > > > Don't use this for testing... use /dev/zero or some other device > > that can produce data faster than this... > > Okay. As I'm sure you can imagine, I used urandom to avoid compression > artifacts. My urandom produces data at ~300 Mbps... but I will use > /dev/zero from now on. Yeh, unless you enable compression, ssh doesn't use it, so it won't be an issue... Also, if you want to free up even more cpu, you can test w/ "-c none" which disables encryption... > > So, our SSH does have the HPN patches: > > https://www.psc.edu/index.php/hpn-ssh > > > > and the README says: > > BUFFER SIZES: > > - if HPN is disabled the receive buffer size will be set to the OpenSSH default > > of 64K. > > Yes... I spent some time reading that document and fretting over whether > or not HPN was really incorporated in my setup. I "confirmed" that it > was available and enabled by setting "HPNDisabled no" and restarting > sshd (on both sides) without complaint. I'm half-tempted to build from > ports to be certain. > > > Looks like there are undocumented options like TCPRcvBuf that you can > > use to adjust the recv buffer window... > > According to the HPN README the default (which I am using) is the > "system wide TCP receive buffer size". I don't know what value that is > or where it comes from (net.inet.tcp.???). I will experiment with > TCPRcvBuf. It does look like the values are in KB, as I tried to set it to 30000, and I got this error message: Couldn't set socket receive buffer to 30720000: No buffer space available Also, don't forget that if you set this in .ssh/config, you only set the client size recive buffer, not the server side, so you'd probably need to add this to the server's sshd_config to enable it for server receive side... > > We have code that will auto grow > > buffer sizes properly so that slow connections won't use up too much > > buffer space... > > That is what I expected, although I believe openssh tries thwart/limit > this by requesting particular buffer sizes (I'm really unqualified to > talk about this). And it is my understanding that HPN undoes these > limitations although I'm not sure if it opens the door to FreeBSD having > full control or uses its own voodoo. You can verify this w/ ktrace -i ssh ... Then after that, you do a kdump | grep SO_RCVBUF | grep setsockopt to see if the program set any... If you see something like: $ kdump | grep SO_RCVBUF | grep setsockopt 6641 ssh CALL setsockopt(0x3,SOL_SOCKET,SO_RCVBUF,0x62dd8c,0x4) 6641 ssh CALL setsockopt(0x7,SOL_SOCKET,SO_RCVBUF,0x7fffffffcaa4,0x4) Then the buffer size is being set... I don't see this w/o tcprcvbuf in my config file, but I do when I add it... > > In a quick test of mine, I'm seeing a buffer size of ~520k from my > > MacOSX box, and ~776k from my 9.2-R box... Server in both cases is > > a June -CURRENT > > Thank you for those numbers. Since my system is basically stock, I > wonder if my bad behavior is an artifact of something on my network. > Did you invoke ssh more or less as "cat /dev/zero | ssh root@host 'cat > > /dev/null'"? Are you quoting S-BCNT numbers? The exact command is: dd if=/dev/zero bs=1m | ssh carbon dd of=/dev/null bs=1m And the numbers I was quoting was the R-BMAX numbers... As that is: R-BMAX Maximum bytes that can be used in the receive buffer. S-BCNT is just the number of bytes waiting to be sent, not the largest possible number that could be buffered... You really can't depend upon that number as only on high latency links will that have an appreciable number, otherwise you'll likely catch it between the ack draining it, and the program running to refill it... > > netstat -xAanfinet is helpful on this... > > That is brilliant! I was using pcap and wireshark to deduce some of > that info. Yep, there are lots of great debugging ways.. You could have also used dtrace for some of this too.. :) > I include my sender and receiver netstat's below for the ssh-ing > /dev/zero. It differs from iperf (which works well), most notably in > S-BCNT (~1MB for iperf, ~64kB for ssh). I think in my case the question is: > > - who is keeping S-BCNT so low (openssh, HPN, or FreeBSD)? As it's easier to change the client's recv buffer, you might want to try the command: ssh dd if=/dev/zero bs=1m > /dev/null And then you can play around w/ tcprcvbuf, though you should verify that SO_RCVBUF is being set before this though... > - Is the limitation introduced by the sending or receiving system? > > - what is the mechanism by which S-BCNT grows when using ssh over > long/fat pipes? Oh, I forgot to ask to make sure that net.inet.tcp.{send,recv}buf_auto is enabled: $ sysctl net.inet.tcp.{send,recv}buf_auto net.inet.tcp.sendbuf_auto: 1 net.inet.tcp.recvbuf_auto: 1 Maybe a dump of your net.inet.tcp might also be helpful... > Thank you again, > > Chris > > SSH Sender > Recv-Q 0 > Send-Q 50132 > R-MBUF 0 > S-MBUF 16 > R-CLUS 0 > S-CLUS 14 > R-HIWA 66052 > S-HIWA 82852 > R-LOWA 1 > S-LOWA 2048 > R-BCNT 0 > S-BCNT 57344 You were probably unlucky when you sampled this value, and caught it at a bad time... Also, look at how much CPU time ssh uses... ssh can introduce additional latency that isn't apparent from the network... Ahhh, also make sure the TCPRcvBufPoll is enabled... I'm not sure if that is the default, but I think that will tell ssh to check to see what the current recv buffer size is, and buffer up to that amount of data: Conditions: HPNBufferSize NOT Set, TCPRcvBufPoll enabled, TCPRcvBuf NOT Set Result: HPN Buffer Size = up to 64MB This is the default state. The HPN buffer size will grow to a maximum of 64MB as the TCP receive buffer grows. The maximum HPN Buffer size of 64MB is geared towards 10GigE transcontinental connections. > R-BMAX 528416 > S-BMAX 662816 These look correct... > rexmt 0.29 > persist 0 > keep 7199.98 > 2msl 0 > delack 0 > rcvtime 0.01 > > SSH Receiver > Recv-Q 0 > Send-Q 36 > R-MBUF 0 > S-MBUF 1 > R-CLUS 0 > S-CLUS 0 > R-HIWA 66052 > S-HIWA 33700 > R-LOWA 1 > S-LOWA 2048 > R-BCNT 0 > S-BCNT 256 > R-BMAX 528416 > S-BMAX 269600 These also look correct... > rexmt 0.24 > persist 0 > keep 7199.96 > 2msl 0 > delack 0.06 > rcvtime 0.03 It's very possible that we don't set any of these values, so what happens is that ssh reads the value of the receive buffer at startup, which is 64k or so, and only does buffering in that size.. Then you end up w/ a latency not of your network, but of the speed at which your computer can encrypt at... Just a thought, but you could also measure latency between writes using ktrace to help figure this out... It really looks like we should set TCPRcvBufPoll by default on FreeBSD... -- John-Mark Gurney Voice: +1 415 225 5579 "All that I will do, has been done, All that I have, has not." From owner-freebsd-net@freebsd.org Wed Aug 26 11:43:53 2015 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 180509C3A37; Wed, 26 Aug 2015 11:43:53 +0000 (UTC) (envelope-from kp@vega.codepro.be) Received: from venus.codepro.be (venus.codepro.be [IPv6:2a01:4f8:162:1127::2]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "*.codepro.be", Issuer "Gandi Standard SSL CA" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id A2D121469; Wed, 26 Aug 2015 11:43:52 +0000 (UTC) (envelope-from kp@vega.codepro.be) Received: from vega.codepro.be (unknown [172.16.1.3]) by venus.codepro.be (Postfix) with ESMTP id 3FDEBB18F; Wed, 26 Aug 2015 13:43:48 +0200 (CEST) Received: by vega.codepro.be (Postfix, from userid 1001) id 3B6DF54C6; Wed, 26 Aug 2015 13:43:48 +0200 (CEST) Date: Wed, 26 Aug 2015 13:43:48 +0200 From: Kristof Provost To: Ermal =?utf-8?B?THXDp2k=?= Cc: freebsd-net , "freebsd-pf@freebsd.org" Subject: Re: Near-term pf plans Message-ID: <20150826114348.GA3433@vega.codepro.be> References: <20150823150957.GK48727@vega.codepro.be> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: X-Checked-By-NSA: Probably User-Agent: Mutt/1.5.23 (2014-03-12) X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 26 Aug 2015 11:43:53 -0000 On 2015-08-25 19:56:59 (+0200), Ermal Luçi wrote: > On Sun, Aug 23, 2015 at 5:09 PM, Kristof Provost wrote: > > > I'm inclined to say that ifgroups and interfaces should share a > > namespace. That would certainly help pf, because it uses both > > interchangeably, which just doesn't work unless they're in the same > > namespace. That affects more in the network stack though, so I'd like > > opinions for people with more experience with ifgroups. > > > > > The solution is easy the scenario of interface name and group name should > not be allowed. > I do not see the use case for this to be allowed at all its just confusion > in general in pf(4). > Agreed. I'll see about putting together a patch to do that. > > > > - Removal of frag drop / frag crop support in pf. ... > While those provide more security protection they do not always work as > advertised so i agree on their removal. > Okay, I'll get the patch committed soon. > > - PR 198868, 193579 (TSO issues) > > pf has issues on certain network interfaces with TSO enabled. The > > most important of these are amazon VMs. > > I believe the problem is that pf tries to work with packets with full > > checksums. Usually output packets have pseudo-header checksums in the IP > > and TCP headers. pf doesn't know about those. It always tries to do > > updates on checksums assuming there's a full checksum. (Which is the > > case, pf always calculates a full checksum in pf_check_out()) > > > > I believe the fix for this issue is to have pf be aware of the > > pseudo-header checksums. The type of checksum can be determined from the > > CSUM_TCP and CSUM_TCP_IPV6 flags. Whenever pf changes a packet header it > > will have to check for those flags to figure out if the checksum should > > be updated or not. > > > > > I would be inclined to say that the real solution is not check packets > generated from the host, This also applies to forwarded packets. Think of things like NAT or port rewriting. We have to change IP addresses for nat, and thus also update checksums. > otherwise you will have to go into complicated checksum handling which > might not be worth it. We should end up with slightly more complicated checksum code (because sometimes we'll update and sometimes we won't), but I expect it to actually be faster. Right now we always calculate a full checksum on outbound packets. In the new case we'll either not touch the checksum, or only calculate updates (depending on scenario). > I know Open has done some work on checksum handling altogether which might > be a good reason > to go and look there first. > Right, I'll check. > > - PR 172648 > > This bug started out with an issue with TCP reassembly, but I've not > > been able to reproduce that. There is a problem with rdr targets for > > ipv6 though. > ... > I will try to look back at this but as a general rule look first at Open if > they have already fixed this. > IPv4 code has some security belts on such packets in pf(4) code to avoid > doing the wrong thing. > I think their checks in ip6_output() are less strict than ours. Regards, Kristof From owner-freebsd-net@freebsd.org Wed Aug 26 12:45:02 2015 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id BB2AF99ADE4 for ; Wed, 26 Aug 2015 12:45:02 +0000 (UTC) (envelope-from Mark.Martinec+freebsd@ijs.si) Received: from mail.ijs.si (mail.ijs.si [IPv6:2001:1470:ff80::25]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 72104E05 for ; Wed, 26 Aug 2015 12:45:02 +0000 (UTC) (envelope-from Mark.Martinec+freebsd@ijs.si) Received: from amavis-ori.ijs.si (localhost [IPv6:::1]) by mail.ijs.si (Postfix) with ESMTP id 3n1RjR370Kz12R for ; Wed, 26 Aug 2015 14:44:59 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ijs.si; h= user-agent:message-id:references:in-reply-to:organization :subject:subject:from:from:date:date:content-transfer-encoding :content-type:content-type:mime-version:received:received :received:received; s=jakla4; t=1440593097; x=1443185098; bh=YMQ jkSnMWXNUL9S44NRswVeGgQ6FzyHHRqrybbRUYu4=; b=kuKtdDwAkHA+1DMSckY JHCB9q/eYNAUORPCNbkjmA9SsgKMRTvdjJnMG1Y4pCy57TqEkiicwlmFEabeJjyr RMTU38Md+G/0U8r4Ter33A10K/9Bp0eEb0x++b/9sDuYLrUTEYuieevq/dwQMEuX 8r+mWsailLLGnGKq2bXjr77o= X-Virus-Scanned: amavisd-new at ijs.si Received: from mail.ijs.si ([IPv6:::1]) by amavis-ori.ijs.si (mail.ijs.si [IPv6:::1]) (amavisd-new, port 10026) with LMTP id 43Ed80tozssV for ; Wed, 26 Aug 2015 14:44:57 +0200 (CEST) Received: from mildred.ijs.si (mailbox.ijs.si [IPv6:2001:1470:ff80::143:1]) by mail.ijs.si (Postfix) with ESMTP id 3n1RjP03BVz12C for ; Wed, 26 Aug 2015 14:44:57 +0200 (CEST) Received: from nabiralnik.ijs.si (mailbox.ijs.si [IPv6:2001:1470:ff80::143:1]) by mildred.ijs.si (Postfix) with ESMTP id 3n1RjN6HjGz87 for ; Wed, 26 Aug 2015 14:44:56 +0200 (CEST) Received: from neli.ijs.si ([2001:1470:ff80:88:21c:c0ff:feb1:8c91]) by nabiralnik.ijs.si with HTTP (HTTP/1.1 POST); Wed, 26 Aug 2015 14:44:56 +0200 MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII; format=flowed Content-Transfer-Encoding: 7bit Date: Wed, 26 Aug 2015 14:44:56 +0200 From: Mark Martinec To: freebsd-net@freebsd.org Subject: Re: ssh over WAN: TCP window too small Organization: Jozef Stefan Institute In-Reply-To: <20150826082457.GQ33167@funkthat.com> References: <55DCF080.7080208@stankevitz.com> <20150826010323.GN33167@funkthat.com> <55DD2A98.2010605@stankevitz.com> <20150826082457.GQ33167@funkthat.com> Message-ID: <4e28129672e933ab87f1a4cabc9575dc@mailbox.ijs.si> X-Sender: Mark.Martinec+freebsd@ijs.si User-Agent: Roundcube Webmail/1.1.2 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 26 Aug 2015 12:45:02 -0000 Chris Stankevitz wrote: > # cat /dev/urandom | ssh root@host 'cat > /dev/null' > > I use the above ssh command over a high-BDP WAN link (80 ms @ 100 > Mbps). > tcpdump shows I am TCP window limited to 64 KBytes (yielding 5 Mbps). > iperf with default options gets the window opened to 500 KBytes > (yielding 35 Mbps). > > Both sides of the connection: FreeBSD 10.1 w/default sshd options > (except I permit root login). In particular, HPN is not disabled. > > Can anyone explain my abysmally small TCP window? > > Can anyone recommend some tools/tricks to figure out what in FreeBSD > and/or base SSH is limiting the send/recv buffer and/or TCP window? As an alternative to ssh for copying large files across high-BDP WAN links consider sysutils/bbcp, optionally coupled with security/hpenc for encryption. Mark From owner-freebsd-net@freebsd.org Wed Aug 26 12:50:34 2015 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id BCB0B99AF84; Wed, 26 Aug 2015 12:50:34 +0000 (UTC) (envelope-from ermal.luci@gmail.com) Received: from mail-yk0-x22c.google.com (mail-yk0-x22c.google.com [IPv6:2607:f8b0:4002:c07::22c]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 90AB961; Wed, 26 Aug 2015 12:50:34 +0000 (UTC) (envelope-from ermal.luci@gmail.com) Received: by ykbi184 with SMTP id i184so185328299ykb.2; Wed, 26 Aug 2015 05:50:33 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=IoOHX2RFm6A9u1Mg2d+ncn0hSoMLEMF7L8ZY+JBSTlk=; b=pfneAVQa+owfoM+AgSORBji2cQ4g2yXDI4T3HzVsEmmLvhm2i5ftyYicJS3e4CRIrR I8IUdobP8+FnGjve8bAELIQKAT9+LmblzA1R2+IQtNfRCWxJen/kJPrkD5OQ89KEg60S XWfdS9sqQr6sVLVSZxDAYITUj0lP+ldi0Xno8FDR5tR3rKH0k2A0Vfz9ltBiS2LuYVjm vO/EVD9BzDU/4J0MqAY850l2UqC3M9bhUZMTCKtGQlQWaUcJIVZ3lk8CxF0bD4rQNUmE taxwVXqJgiGr63rycApoa42xT3LuWT1iOAK9SX0F2YY+k+lK4Z4dlXpu9CsvDfMTVEX6 DHkg== MIME-Version: 1.0 X-Received: by 10.129.78.85 with SMTP id c82mr41079083ywb.162.1440593433587; Wed, 26 Aug 2015 05:50:33 -0700 (PDT) Received: by 10.129.73.205 with HTTP; Wed, 26 Aug 2015 05:50:33 -0700 (PDT) In-Reply-To: <20150826114348.GA3433@vega.codepro.be> References: <20150823150957.GK48727@vega.codepro.be> <20150826114348.GA3433@vega.codepro.be> Date: Wed, 26 Aug 2015 14:50:33 +0200 Message-ID: Subject: Re: Near-term pf plans From: =?UTF-8?Q?Ermal_Lu=C3=A7i?= To: Kristof Provost Cc: freebsd-net , "freebsd-pf@freebsd.org" Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 26 Aug 2015 12:50:34 -0000 On Wed, Aug 26, 2015 at 1:43 PM, Kristof Provost wrote= : > On 2015-08-25 19:56:59 (+0200), Ermal Lu=C3=A7i wr= ote: > > On Sun, Aug 23, 2015 at 5:09 PM, Kristof Provost wrote= : > > > > > I'm inclined to say that ifgroups and interfaces should share a > > > namespace. That would certainly help pf, because it uses both > > > interchangeably, which just doesn't work unless they're in the sam= e > > > namespace. That affects more in the network stack though, so I'd > like > > > opinions for people with more experience with ifgroups. > > > > > > > > > The solution is easy the scenario of interface name and group name shou= ld > > not be allowed. > > I do not see the use case for this to be allowed at all its just > confusion > > in general in pf(4). > > > Agreed. I'll see about putting together a patch to do that. > Ok, keep me in the loop for the review. > > > > > > > - Removal of frag drop / frag crop support in pf. > ... > > While those provide more security protection they do not always work as > > advertised so i agree on their removal. > > > Okay, I'll get the patch committed soon. > > > > - PR 198868, 193579 (TSO issues) > > > pf has issues on certain network interfaces with TSO enabled. The > > > most important of these are amazon VMs. > > > I believe the problem is that pf tries to work with packets with > full > > > checksums. Usually output packets have pseudo-header checksums in > the IP > > > and TCP headers. pf doesn't know about those. It always tries to d= o > > > updates on checksums assuming there's a full checksum. (Which is t= he > > > case, pf always calculates a full checksum in pf_check_out()) > > > > > > I believe the fix for this issue is to have pf be aware of the > > > pseudo-header checksums. The type of checksum can be determined > from the > > > CSUM_TCP and CSUM_TCP_IPV6 flags. Whenever pf changes a packet > header it > > > will have to check for those flags to figure out if the checksum > should > > > be updated or not. > > > > > > > > I would be inclined to say that the real solution is not check packets > > generated from the host, > This also applies to forwarded packets. Think of things like NAT or port > rewriting. We have to change IP addresses for nat, and thus also update > checksums. Normally you would expect those with correct checksum AFAIK since the input path validates those as well! > > > otherwise you will have to go into complicated checksum handling which > > might not be worth it. > We should end up with slightly more complicated checksum code (because > sometimes we'll update and sometimes we won't), but I expect it to > actually be faster. Right now we always calculate a full checksum on > outbound packets. In the new case we'll either not touch the checksum, > or only calculate updates (depending on scenario). > > > I know Open has done some work on checksum handling altogether which > might > > be a good reason > > to go and look there first. > > > Right, I'll check. > > After looking at Open code, their solution is to not check anymore checksums on pf(4) code but only mark them to be recalculated if pf(NAT/RDR/BINAT) touches the packet and let the OS mechanics deal with them. Certainly they went to length on doing this allover but it seems a better choice, also same effort as making the checksum handling right, rather than duplicating whole portions of code which are difficult to get right and keep in sync. How do you say that we try to do the same in FreeBSD and seems a better path in maintainability rather than a complex code that can get out-of-date soon. Relying on the protocol routines deal with checksum handling, since they have already the code, is the better path and with so many offloading features on the NICs nowadays it does not make much sense to have it on pf(4), taking also in consideration the input path verifies them and output path handles them properly even during forwarding and even during host packet transmission. This also has the benefit to increase the performance! > > > - PR 172648 > > > This bug started out with an issue with TCP reassembly, but I've n= ot > > > been able to reproduce that. There is a problem with rdr targets f= or > > > ipv6 though. > > > ... > > I will try to look back at this but as a general rule look first at Ope= n > if > > they have already fixed this. > > IPv4 code has some security belts on such packets in pf(4) code to avoi= d > > doing the wrong thing. > > > I think their checks in ip6_output() are less strict than ours. > Regards, > Kristof > --=20 Ermal From owner-freebsd-net@freebsd.org Wed Aug 26 21:30:09 2015 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 751D19C2C6A for ; Wed, 26 Aug 2015 21:30:09 +0000 (UTC) (envelope-from chris@stankevitz.com) Received: from mango.stankevitz.com (mango.stankevitz.com [208.79.93.194]) by mx1.freebsd.org (Postfix) with ESMTP id 51B2F155A for ; Wed, 26 Aug 2015 21:30:09 +0000 (UTC) (envelope-from chris@stankevitz.com) Received: from Chriss-MacBook-Pro.local (209-203-101-124.static.twtelecom.net [209.203.101.124]) (using TLSv1.2 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by mango.stankevitz.com (Postfix) with ESMTPSA id D8123145E; Wed, 26 Aug 2015 14:30:07 -0700 (PDT) Message-ID: <55DE2FDF.5030707@stankevitz.com> Date: Wed, 26 Aug 2015 14:30:07 -0700 From: Chris Stankevitz User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:31.0) Gecko/20100101 Thunderbird/31.7.0 MIME-Version: 1.0 To: John-Mark Gurney CC: freebsd-net@freebsd.org Subject: Re: ssh over WAN: TCP window too small References: <55DCF080.7080208@stankevitz.com> <20150826010323.GN33167@funkthat.com> <55DD2A98.2010605@stankevitz.com> <20150826082457.GQ33167@funkthat.com> In-Reply-To: <20150826082457.GQ33167@funkthat.com> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 26 Aug 2015 21:30:09 -0000 John-Mark, Thanks again. I appreciate you teaching me "how to fish". I basically spent all morning reading kdump output. On 8/26/15 1:24 AM, John-Mark Gurney wrote: > It really looks like we should set TCPRcvBufPoll by default on > FreeBSD... According to /etc/ssh/sshd_config, TCPRcvBufPoll defaults to true. ktrace (thank you for showing me this) shows the ssh process continuously polling SO_RCVBUF. In my case with TCPRcvBufPoll on and HPNBufferSize and TCPRcvBuf unset, ssh never sets SO_RCVBUF or SO_SNDBUF so presumably FreeBSD has complete control over those values (you saw the same thing). But perhaps all this is moot in my case because netstat shows that the sender and receiver have sufficiently large SND/RCV buffers: (can you confirm my interpretation is correct?) Sender S-BMAX 662816 Receiver R-BMAX 528416 >> I will experiment with TCPRcvBuf. > > It does look like the values are in KB Yes, README.hpn says they are in KB. However, even though it is described in README.hpn, I cannot set TCPRcvBuf in sshd_config: /etc/ssh/sshd_config: line 141: Bad configuration option: TCPRcvBuf /etc/ssh/sshd_config: line 141: Bad configuration option: TcpRcvBuf However, as I described above, I believe the buffer size is a red herring. ktrace shows pretty clearly what is happening. ssh isn't even bothering to read from /dev/zero until select(2) gives the green light. And in my case select(2) blocks ssh for 0.1 second for every ~30KB of data: 94146 ssh 6.592042 CALL getsockopt(0x3,SOL_SOCKET,SO_RCVBUF 94146 ssh 6.592054 RET getsockopt 0 94146 ssh 6.592065 CALL select(0x7,0x803817168,0x803817178,0,0) 94146 ssh 6.685873 RET select 1 94146 ssh 6.685907 CALL read(0x3,0x7fffffffae40,0x2000) 94146 ssh 6.685921 GIO fd 3 read 36 bytes [ read of fd 3 snipped] 94146 ssh 6.685931 RET read 36/0x24 94146 ssh 6.685950 CALL getpid 94146 ssh 6.685962 RET getpid 94146/0x16fc2 94146 ssh 6.685974 CALL getpid 94146 ssh 6.685984 RET getpid 94146/0x16fc2 94146 ssh 6.685996 CALL getpid 94146 ssh 6.686006 RET getpid 94146/0x16fc2 94146 ssh 6.686017 CALL getpid 94146 ssh 6.686027 RET getpid 94146/0x16fc2 94146 ssh 6.686091 CALL getsockopt(0x3,SOL_SOCKET,SO_RCVBUF 94146 ssh 6.686103 RET getsockopt 0 94146 ssh 6.686116 CALL select(0x7,0x803817168,0x803817178,0,0) 94146 ssh 6.686128 RET select 2 94146 ssh 6.686140 CALL read(0x4,0x7fffffff6c70,0x4000) 94146 ssh 6.686154 GIO fd 4 read 4096 bytes [ read of stdin (/dev/zero) snipped) Note in the above the blocking call to select at time 6.592065 that takes ~0.1 second. This is the reason my connection is slow. The content appears to be encrypted... I presume it's an application-level ssh ack. BTW when I disable HPN (HPNdisabled=yes) that same select happens, but it blocks for less time (~0.05 second) and roughly doubles my throughput. > Also, don't forget that if you set this in .ssh/config, you only set > the client size recive buffer, not the server side, so you'd probably > need to add this to the server's sshd_config to enable it for server > receive side... I understand. > ktrace -i ssh ... Thank you, this is awesome. Is there a way for me to ktrace 'b' in this example: `a | b | c`? I tried `ktrace a | b | c` and `ktrace test.sh` (which included a|b|c) but neither seemed to work. I'm using stream redirection now but it doesn't afford me the bs control of dd. Perhaps named pipes is the solution. > Oh, I forgot to ask to make sure that net.inet.tcp.{send,recv}buf_auto > is enabled: Unfortunately it is enabled :-/ > Maybe a dump of your net.inet.tcp might also be helpful... This should all be standard out-of-the-box: net.inet.tcp.rfc1323: 1 net.inet.tcp.mssdflt: 536 net.inet.tcp.keepidle: 7200000 net.inet.tcp.keepintvl: 75000 net.inet.tcp.sendspace: 32768 net.inet.tcp.recvspace: 65536 net.inet.tcp.keepinit: 75000 net.inet.tcp.delacktime: 100 net.inet.tcp.v6mssdflt: 1220 net.inet.tcp.cc.algorithm: newreno net.inet.tcp.cc.available: newreno net.inet.tcp.hostcache.cachelimit: 15360 net.inet.tcp.hostcache.hashsize: 512 net.inet.tcp.hostcache.bucketlimit: 30 net.inet.tcp.hostcache.count: 6 net.inet.tcp.hostcache.expire: 3600 net.inet.tcp.hostcache.prune: 300 net.inet.tcp.hostcache.purge: 0 net.inet.tcp.log_in_vain: 0 net.inet.tcp.blackhole: 0 net.inet.tcp.delayed_ack: 1 net.inet.tcp.drop_synfin: 0 net.inet.tcp.rfc3042: 1 net.inet.tcp.rfc3390: 1 net.inet.tcp.experimental.initcwnd10: 1 net.inet.tcp.rfc3465: 1 net.inet.tcp.abc_l_var: 2 net.inet.tcp.ecn.enable: 0 net.inet.tcp.ecn.maxretries: 1 net.inet.tcp.insecure_rst: 0 net.inet.tcp.recvbuf_auto: 1 net.inet.tcp.recvbuf_inc: 16384 net.inet.tcp.recvbuf_max: 2097152 net.inet.tcp.path_mtu_discovery: 1 net.inet.tcp.tso: 1 net.inet.tcp.sendbuf_auto: 1 net.inet.tcp.sendbuf_inc: 8192 net.inet.tcp.sendbuf_max: 2097152 net.inet.tcp.reass.maxsegments: 255100 net.inet.tcp.reass.cursegments: 0 net.inet.tcp.reass.overflows: 471 net.inet.tcp.sack.enable: 1 net.inet.tcp.sack.maxholes: 128 net.inet.tcp.sack.globalmaxholes: 65536 net.inet.tcp.sack.globalholes: 0 net.inet.tcp.minmss: 216 net.inet.tcp.log_debug: 0 net.inet.tcp.tcbhashsize: 524288 net.inet.tcp.do_tcpdrain: 1 net.inet.tcp.pcbcount: 43 net.inet.tcp.icmp_may_rst: 1 net.inet.tcp.isn_reseed_interval: 0 net.inet.tcp.soreceive_stream: 0 net.inet.tcp.syncookies: 1 net.inet.tcp.syncookies_only: 0 net.inet.tcp.syncache.bucketlimit: 30 net.inet.tcp.syncache.cachelimit: 15375 net.inet.tcp.syncache.count: 0 net.inet.tcp.syncache.hashsize: 512 net.inet.tcp.syncache.rexmtlimit: 3 net.inet.tcp.syncache.rst_on_sock_fail: 1 net.inet.tcp.msl: 30000 net.inet.tcp.rexmit_min: 30 net.inet.tcp.rexmit_slop: 200 net.inet.tcp.always_keepalive: 1 net.inet.tcp.fast_finwait2_recycle: 0 net.inet.tcp.finwait2_timeout: 60000 net.inet.tcp.keepcnt: 8 net.inet.tcp.rexmit_drop_options: 0 net.inet.tcp.per_cpu_timers: 0 net.inet.tcp.timer_race: 0 net.inet.tcp.maxtcptw: 27767 net.inet.tcp.nolocaltimewait: 0 >> S-BCNT 57344 > > You were probably unlucky when you sampled this value, and caught it at > a bad time... Also, look at how much CPU time ssh uses... ssh can > introduce additional latency that isn't apparent from the network... S-BCNT is always ~60KB when HPN is enabled. This jives with my "ssh is spending all of its time waiting on select(2)" theory. > It's very possible that we don't set any of these values, so what > happens is that ssh reads the value of the receive buffer at startup, > which is 64k or so, and only does buffering in that size.. Then you > end up w/ a latency not of your network, but of the speed at which > your computer can encrypt at... Just a thought, but you could also > measure latency between writes using ktrace to help figure this > out... Yes, I believe something like this is happening now. Thank you again for your help... this thread is proving to me one of those quantum leap moments for me in terms of FreeBSD knowledge. Chris From owner-freebsd-net@freebsd.org Wed Aug 26 22:23:11 2015 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 927669C3F58 for ; Wed, 26 Aug 2015 22:23:11 +0000 (UTC) (envelope-from gpalmer@freebsd.org) Received: from mail.in-addr.com (mail.in-addr.com [IPv6:2a01:4f8:191:61e8::2525:2525]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 55F64FAB for ; Wed, 26 Aug 2015 22:23:11 +0000 (UTC) (envelope-from gpalmer@freebsd.org) Received: from gjp by mail.in-addr.com with local (Exim 4.86 (FreeBSD)) (envelope-from ) id 1ZUj64-000Pxu-75; Wed, 26 Aug 2015 23:23:08 +0100 Date: Wed, 26 Aug 2015 23:23:08 +0100 From: Gary Palmer To: Chris Stankevitz Cc: John-Mark Gurney , freebsd-net@freebsd.org Subject: Re: ssh over WAN: TCP window too small Message-ID: <20150826222308.GH13503@in-addr.com> References: <55DCF080.7080208@stankevitz.com> <20150826010323.GN33167@funkthat.com> <55DD2A98.2010605@stankevitz.com> <20150826082457.GQ33167@funkthat.com> <55DE2FDF.5030707@stankevitz.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <55DE2FDF.5030707@stankevitz.com> X-SA-Exim-Connect-IP: X-SA-Exim-Mail-From: gpalmer@freebsd.org X-SA-Exim-Scanned: No (on mail.in-addr.com); SAEximRunCond expanded to false X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 26 Aug 2015 22:23:11 -0000 On Wed, Aug 26, 2015 at 02:30:07PM -0700, Chris Stankevitz wrote: > > ktrace -i ssh ... > > Thank you, this is awesome. Is there a way for me to ktrace 'b' in this > example: `a | b | c`? I tried `ktrace a | b | c` and `ktrace test.sh` > (which included a|b|c) but neither seemed to work. I'm using stream > redirection now but it doesn't afford me the bs control of dd. Perhaps > named pipes is the solution. a | ktrace b | c or ktrace -di test.sh (I suspect only -i is needed, but I've gotten so used to using both flags) You can also start ktrace on an existing process if you know its PID ktrace -p To stop all ongoing tracing: ktrace -C Regards, Gary From owner-freebsd-net@freebsd.org Wed Aug 26 22:32:16 2015 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id A97C99C32B8 for ; Wed, 26 Aug 2015 22:32:16 +0000 (UTC) (envelope-from jmg@gold.funkthat.com) Received: from gold.funkthat.com (gate2.funkthat.com [208.87.223.18]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client CN "gold.funkthat.com", Issuer "gold.funkthat.com" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 88194682 for ; Wed, 26 Aug 2015 22:32:16 +0000 (UTC) (envelope-from jmg@gold.funkthat.com) Received: from gold.funkthat.com (localhost [127.0.0.1]) by gold.funkthat.com (8.14.5/8.14.5) with ESMTP id t7QMWFdc036248 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Wed, 26 Aug 2015 15:32:15 -0700 (PDT) (envelope-from jmg@gold.funkthat.com) Received: (from jmg@localhost) by gold.funkthat.com (8.14.5/8.14.5/Submit) id t7QMWFCa036247; Wed, 26 Aug 2015 15:32:15 -0700 (PDT) (envelope-from jmg) Date: Wed, 26 Aug 2015 15:32:15 -0700 From: John-Mark Gurney To: Chris Stankevitz Cc: freebsd-net@freebsd.org Subject: Re: ssh over WAN: TCP window too small Message-ID: <20150826223215.GS33167@funkthat.com> References: <55DCF080.7080208@stankevitz.com> <20150826010323.GN33167@funkthat.com> <55DD2A98.2010605@stankevitz.com> <20150826082457.GQ33167@funkthat.com> <55DE2FDF.5030707@stankevitz.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <55DE2FDF.5030707@stankevitz.com> X-Operating-System: FreeBSD 9.1-PRERELEASE amd64 X-PGP-Fingerprint: 54BA 873B 6515 3F10 9E88 9322 9CB1 8F74 6D3F A396 X-Files: The truth is out there X-URL: http://resnet.uoregon.edu/~gurney_j/ X-Resume: http://resnet.uoregon.edu/~gurney_j/resume.html X-TipJar: bitcoin:13Qmb6AeTgQecazTWph4XasEsP7nGRbAPE X-to-the-FBI-CIA-and-NSA: HI! HOW YA DOIN? can i haz chizburger? User-Agent: Mutt/1.5.21 (2010-09-15) X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.2.7 (gold.funkthat.com [127.0.0.1]); Wed, 26 Aug 2015 15:32:15 -0700 (PDT) X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 26 Aug 2015 22:32:16 -0000 Chris Stankevitz wrote this message on Wed, Aug 26, 2015 at 14:30 -0700: > Thanks again. I appreciate you teaching me "how to fish". I basically > spent all morning reading kdump output. No worries, glad you're learning... > On 8/26/15 1:24 AM, John-Mark Gurney wrote: > > It really looks like we should set TCPRcvBufPoll by default on > > FreeBSD... > > According to /etc/ssh/sshd_config, TCPRcvBufPoll defaults to true. > ktrace (thank you for showing me this) shows the ssh process > continuously polling SO_RCVBUF. > > In my case with TCPRcvBufPoll on and HPNBufferSize and TCPRcvBuf unset, > ssh never sets SO_RCVBUF or SO_SNDBUF so presumably FreeBSD has complete > control over those values (you saw the same thing). > > But perhaps all this is moot in my case because netstat shows that the > sender and receiver have sufficiently large SND/RCV buffers: (can you > confirm my interpretation is correct?) > > Sender S-BMAX 662816 > Receiver R-BMAX 528416 Yes, this is correct... So definately not an issue w/ window size of socket buffer size... > >> I will experiment with TCPRcvBuf. > > > > It does look like the values are in KB > > Yes, README.hpn says they are in KB. However, even though it is > described in README.hpn, I cannot set TCPRcvBuf in sshd_config: > > /etc/ssh/sshd_config: line 141: Bad configuration option: TCPRcvBuf > /etc/ssh/sshd_config: line 141: Bad configuration option: TcpRcvBuf Looks like: https://www.psc.edu/index.php/networking/695 Says that TCPRcvBuf is client side only.. Which is odd, but not an issue w/ FreeBSD... > However, as I described above, I believe the buffer size is a red herring. Agreed.... > ktrace shows pretty clearly what is happening. ssh isn't even bothering > to read from /dev/zero until select(2) gives the green light. And in my > case select(2) blocks ssh for 0.1 second for every ~30KB of data: > > > 94146 ssh 6.592042 CALL getsockopt(0x3,SOL_SOCKET,SO_RCVBUF > 94146 ssh 6.592054 RET getsockopt 0 > 94146 ssh 6.592065 CALL select(0x7,0x803817168,0x803817178,0,0) > 94146 ssh 6.685873 RET select 1 > 94146 ssh 6.685907 CALL read(0x3,0x7fffffffae40,0x2000) > 94146 ssh 6.685921 GIO fd 3 read 36 bytes > [ read of fd 3 snipped] > 94146 ssh 6.685931 RET read 36/0x24 > 94146 ssh 6.685950 CALL getpid > 94146 ssh 6.685962 RET getpid 94146/0x16fc2 > 94146 ssh 6.685974 CALL getpid > 94146 ssh 6.685984 RET getpid 94146/0x16fc2 > 94146 ssh 6.685996 CALL getpid > 94146 ssh 6.686006 RET getpid 94146/0x16fc2 > 94146 ssh 6.686017 CALL getpid > 94146 ssh 6.686027 RET getpid 94146/0x16fc2 > 94146 ssh 6.686091 CALL getsockopt(0x3,SOL_SOCKET,SO_RCVBUF > 94146 ssh 6.686103 RET getsockopt 0 > 94146 ssh 6.686116 CALL select(0x7,0x803817168,0x803817178,0,0) > 94146 ssh 6.686128 RET select 2 > 94146 ssh 6.686140 CALL read(0x4,0x7fffffff6c70,0x4000) > 94146 ssh 6.686154 GIO fd 4 read 4096 bytes > [ read of stdin (/dev/zero) snipped) It would be interesting to know how long from the read of stdin (and is it really reading stdin in 4k blocks? If so, that should be fixed) to the write out the socket... Basicly, how long does it take to encrypt the data... > Note in the above the blocking call to select at time 6.592065 that > takes ~0.1 second. This is the reason my connection is slow. The > content appears to be encrypted... I presume it's an application-level > ssh ack. > > BTW when I disable HPN (HPNdisabled=yes) that same select happens, but > it blocks for less time (~0.05 second) and roughly doubles my throughput. > > > Also, don't forget that if you set this in .ssh/config, you only set > > the client size recive buffer, not the server side, so you'd probably > > need to add this to the server's sshd_config to enable it for server > > receive side... > > I understand. > > > ktrace -i ssh ... > > Thank you, this is awesome. Is there a way for me to ktrace 'b' in this > example: `a | b | c`? I tried `ktrace a | b | c` and `ktrace test.sh` When you do this, make sure you ktrace -i to inherit the ktrace flag to the children... In the case of a | b | c, just doing: a | ktrace b | c Should work too... > (which included a|b|c) but neither seemed to work. I'm using stream > redirection now but it doesn't afford me the bs control of dd. Perhaps > named pipes is the solution. > > > Oh, I forgot to ask to make sure that net.inet.tcp.{send,recv}buf_auto > > is enabled: > > Unfortunately it is enabled :-/ > > > Maybe a dump of your net.inet.tcp might also be helpful... > > This should all be standard out-of-the-box: [...] > net.inet.tcp.sendspace: 32768 > net.inet.tcp.recvspace: 65536 Try increasing these and reporting back... the buf_auto should override those, but this may be limiting it... > net.inet.tcp.cc.algorithm: newreno > net.inet.tcp.cc.available: newreno Another option could be to look at other cc algorithms... See mod_cc(4) for more info... [...] > >> S-BCNT 57344 > > > > You were probably unlucky when you sampled this value, and caught it at > > a bad time... Also, look at how much CPU time ssh uses... ssh can > > introduce additional latency that isn't apparent from the network... > > S-BCNT is always ~60KB when HPN is enabled. This jives with my "ssh is > spending all of its time waiting on select(2)" theory. So, I looked at what getsockopt SO_RCVBUF returns, and it returns: case SO_RCVBUF: optval = so->so_rcv.sb_hiwat; Which is NOT S-BMAX, so it looks like OpenSSH only ever gets 66052 bytes for the buffer... If it's decided to base it's waiting for ack on SO_RCVBUF, then this is probably where the issue is... Try setting HPNBufferSize to something resonably large, like 1MB, or above your bandwidth-delay product to see if this will make a difference.. Per: Conditions: HPNBufferSize SET, TCPRcvBufPoll enabled, TCPRcvBuf NOT Set Result: HPN Buffer Size = grows to HPNBufferSize The buffer will grow up to the maximum size specified here. This could be an interaction w/ the HPN buffering and the socket buffer size... As the receive buffer never grows large, HPN can't buffer enough data to meet the bandwidth product delay, even though it should be able to.. > > It's very possible that we don't set any of these values, so what > > happens is that ssh reads the value of the receive buffer at startup, > > which is 64k or so, and only does buffering in that size.. Then you > > end up w/ a latency not of your network, but of the speed at which > > your computer can encrypt at... Just a thought, but you could also > > measure latency between writes using ktrace to help figure this > > out... > > Yes, I believe something like this is happening now. > > Thank you again for your help... this thread is proving to me one of > those quantum leap moments for me in terms of FreeBSD knowledge. Glad I could be of help. -- John-Mark Gurney Voice: +1 415 225 5579 "All that I will do, has been done, All that I have, has not." From owner-freebsd-net@freebsd.org Wed Aug 26 23:34:56 2015 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 594219C38EF for ; Wed, 26 Aug 2015 23:34:56 +0000 (UTC) (envelope-from janm@transactionware.com) Received: from mail3.transactionware.com (mail.transactionware.com [203.14.245.7]) by mx1.freebsd.org (Postfix) with SMTP id A3A1E7F1 for ; Wed, 26 Aug 2015 23:34:54 +0000 (UTC) (envelope-from janm@transactionware.com) Received: (qmail 97811 invoked by uid 907); 26 Aug 2015 23:28:11 -0000 Received: from eth222.nsw.adsl.internode.on.net (HELO [192.168.1.101]) (150.101.196.221) (smtp-auth username janm, mechanism plain) by mail3.transactionware.com (qpsmtpd/0.84) with (ECDHE-RSA-AES256-SHA encrypted) ESMTPSA; Thu, 27 Aug 2015 09:28:11 +1000 Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 8.2 \(2104\)) Subject: Re: ssh over WAN: TCP window too small From: Jan Mikkelsen In-Reply-To: <55DCF080.7080208@stankevitz.com> Date: Thu, 27 Aug 2015 09:28:09 +1000 Cc: freebsd-net@freebsd.org Content-Transfer-Encoding: quoted-printable Message-Id: <735A62B2-EFBC-4A4A-9782-F809EC1069E3@transactionware.com> References: <55DCF080.7080208@stankevitz.com> To: Chris Stankevitz X-Mailer: Apple Mail (2.2104) X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 26 Aug 2015 23:34:56 -0000 Hi, > On 26 Aug 2015, at 08:47, Chris Stankevitz = wrote: >=20 > Hi, >=20 > # cat /dev/urandom | ssh root@host 'cat > /dev/null' >=20 > I use the above ssh command over a high-BDP WAN link (80 ms @ 100 = Mbps). tcpdump shows I am TCP window limited to 64 KBytes (yielding 5 = Mbps). iperf with default options gets the window opened to 500 KBytes = (yielding 35 Mbps). Given that you are TCP window limited, do you have something in the = middle preventing the windows size negotiation from working? A stateful = firewall somewhere, perhaps? > Both sides of the connection: FreeBSD 10.1 w/default sshd options = (except I permit root login). In particular, HPN is not disabled. >=20 > Can anyone explain my abysmally small TCP window? >=20 > Can anyone recommend some tools/tricks to figure out what in FreeBSD = and/or base SSH is limiting the send/recv buffer and/or TCP window? Regards, Jan.= From owner-freebsd-net@freebsd.org Thu Aug 27 00:46:25 2015 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 8346B9C01EC for ; Thu, 27 Aug 2015 00:46:25 +0000 (UTC) (envelope-from ddesimone@verio.net) Received: from na01-bl2-obe.outbound.protection.outlook.com (mail-bl2on0064.outbound.protection.outlook.com [65.55.169.64]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "MSIT Machine Auth CA 2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 3592A1BFD for ; Thu, 27 Aug 2015 00:46:24 +0000 (UTC) (envelope-from ddesimone@verio.net) Received: from BLUPR0801MB674.namprd08.prod.outlook.com (10.141.255.11) by BLUPR0801MB673.namprd08.prod.outlook.com (10.141.254.27) with Microsoft SMTP Server (TLS) id 15.1.243.23; Thu, 27 Aug 2015 00:46:16 +0000 Received: from BLUPR0801MB674.namprd08.prod.outlook.com ([10.141.255.11]) by BLUPR0801MB674.namprd08.prod.outlook.com ([10.141.255.11]) with mapi id 15.01.0243.020; Thu, 27 Aug 2015 00:46:16 +0000 From: David DeSimone To: "freebsd-net@freebsd.org" Subject: RE: ssh over WAN: TCP window too small Thread-Topic: ssh over WAN: TCP window too small Thread-Index: AQHQ34j16xp/8cRnZ0W04Z/ulk58zZ4ddvCAgAAfSACAAFwYgIAA21+AgAARXICAACTZcA== Date: Thu, 27 Aug 2015 00:46:16 +0000 Message-ID: References: <55DCF080.7080208@stankevitz.com> <20150826010323.GN33167@funkthat.com> <55DD2A98.2010605@stankevitz.com> <20150826082457.GQ33167@funkthat.com> <55DE2FDF.5030707@stankevitz.com> <20150826223215.GS33167@funkthat.com> In-Reply-To: <20150826223215.GS33167@funkthat.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: spf=none (sender IP is ) smtp.mailfrom=ddesimone@verio.net; x-originating-ip: [173.71.11.10] x-microsoft-exchange-diagnostics: 1; BLUPR0801MB673; 5:PR2tXQjiUq9UECrbvscyycQa1PCUQ2JSCvIEFEG7SzgdP6f7vpcg44/D47ssA9XKdCuWkzC8GChNLHBeTCOIzFxQtrVBl0yhRrheJ+189It9jMudwTepMZsaodfxZgb3xuB8oe69Dd5s/UI20ERWUA==; 24:qHLXDMqNtzoRqabMudEa5uDaXFcYiMinX6oXkordQRIYrzCZh51fVTcEsxYVGWgj2oteN2YEev6g9xGMMaVjci9xSOrFxMrhQwkN3SJbQTo=; 20:sLkMKEwu+YeuYLhrfgmV4RPFXvaA5Przq18yCmqJD6zcKlj828PZpLhVqur29BB731sMNqELwIJVO1GkDCjmeQ== x-microsoft-antispam: UriScan:;BCL:0;PCL:0;RULEID:;SRVR:BLUPR0801MB673; x-microsoft-antispam-prvs: x-exchange-antispam-report-test: UriScan:; x-exchange-antispam-report-cfa-test: BCL:0; PCL:0; RULEID:(601004)(8121501046)(5005006)(3002001); SRVR:BLUPR0801MB673; BCL:0; PCL:0; RULEID:; SRVR:BLUPR0801MB673; x-forefront-prvs: 06818431B9 x-forefront-antispam-report: SFV:NSPM; SFS:(10009020)(6009001)(377454003)(189002)(199003)(24454002)(479174004)(77156002)(2656002)(64706001)(46102003)(81156007)(4001540100001)(93886004)(106116001)(87936001)(189998001)(107886002)(74316001)(122556002)(97736004)(5001830100001)(10400500002)(5001860100001)(110136002)(40100003)(62966003)(450100001)(86362001)(101416001)(2501003)(102836002)(99286002)(54356999)(77096005)(5007970100001)(76176999)(76576001)(2900100001)(5003600100002)(2351001)(2950100001)(68736005)(105586002)(5001960100002)(33656002)(106356001)(5004730100002)(66066001)(50986999)(5002640100001)(92566002)(5890100001)(160913001)(15963001); DIR:OUT; SFP:1101; SCL:1; SRVR:BLUPR0801MB673; H:BLUPR0801MB674.namprd08.prod.outlook.com; FPR:; SPF:None; PTR:InfoNoRecords; MX:1; A:1; LANG:en; received-spf: None (protection.outlook.com: verio.net does not designate permitted sender hosts) spamdiagnosticoutput: 1:23 spamdiagnosticmetadata: NSPM Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: verio.net X-MS-Exchange-CrossTenant-originalarrivaltime: 27 Aug 2015 00:46:16.3601 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 281c3918-264a-4db4-ab20-2dafa1dca324 X-MS-Exchange-Transport-CrossTenantHeadersStamped: BLUPR0801MB673 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 27 Aug 2015 00:46:25 -0000 On 8/26/15 1:24 AM, John-Mark Gurney wrote: > > 94146 ssh 6.686140 CALL read(0x4,0x7fffffff6c70,0x4000) > > 94146 ssh 6.686154 GIO fd 4 read 4096 bytes > > [ read of stdin (/dev/zero) snipped) > > It would be interesting to know how long from the read of stdin (and is > it really reading stdin in 4k blocks? If so, that should be fixed) The read is making a call with 0x4000 =3D 16k buffer size, but it only rece= ives 4k, probably because that is the max size of the pipe buffer. ________________________________ This email message is intended for the use of the person to whom it has bee= n sent, and may contain information that is confidential or legally protect= ed. If you are not the intended recipient or have received this message in = error, you are not authorized to copy, distribute, or otherwise use this me= ssage or its attachments. Please notify the sender immediately by return e-= mail and permanently delete this message and any attachments. makes no warr= anty that this email is error or virus free. Thank you. ________________________________ This email message is intended for the use of the person to whom it has bee= n sent, and may contain information that is confidential or legally protect= ed. If you are not the intended recipient or have received this message in = error, you are not authorized to copy, distribute, or otherwise use this me= ssage or its attachments. Please notify the sender immediately by return e-= mail and permanently delete this message and any attachments. NTT America m= akes no warranty that this email is error or virus free. Thank you. ________________________________ From owner-freebsd-net@freebsd.org Thu Aug 27 06:22:08 2015 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id E19279C2E1D for ; Thu, 27 Aug 2015 06:22:08 +0000 (UTC) (envelope-from azakharov@uniserve.com) Received: from mbob.nabble.com (mbob.nabble.com [162.253.133.15]) by mx1.freebsd.org (Postfix) with ESMTP id D0736660 for ; Thu, 27 Aug 2015 06:22:08 +0000 (UTC) (envelope-from azakharov@uniserve.com) Received: from msam.nabble.com (unknown [162.253.133.85]) by mbob.nabble.com (Postfix) with ESMTP id 000251418CBE for ; Wed, 26 Aug 2015 23:19:37 -0700 (PDT) Date: Wed, 26 Aug 2015 23:22:02 -0700 (MST) From: azakharov To: freebsd-net@freebsd.org Message-ID: <1440656522718-6035324.post@n5.nabble.com> In-Reply-To: References: Subject: Re: cpsw/atphy network drivers MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 27 Aug 2015 06:22:09 -0000 Hello Matt, We have a similar problem with custom board (AM335X and AR8035 PHY). I am trying to run FreeBSD (current) and everything seems ok, except network interface is not working, and behavior is exactly the same. Did you manage to figure out how to solve it? Cheers, Alex -- View this message in context: http://freebsd.1045724.n5.nabble.com/cpsw-atphy-network-drivers-tp5994248p6035324.html Sent from the freebsd-net mailing list archive at Nabble.com. From owner-freebsd-net@freebsd.org Thu Aug 27 07:56:47 2015 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 7D9259C2069 for ; Thu, 27 Aug 2015 07:56:47 +0000 (UTC) (envelope-from eliezer@ngtech.co.il) Received: from mtaout21.012.net.il (mtaout21.012.net.il [80.179.55.169]) by mx1.freebsd.org (Postfix) with ESMTP id 3089E7BD for ; Thu, 27 Aug 2015 07:56:46 +0000 (UTC) (envelope-from eliezer@ngtech.co.il) Received: from conversion-daemon.a-mtaout21.012.net.il by a-mtaout21.012.net.il (HyperSendmail v2007.08) id <0NTQ00K00DUQ9M00@a-mtaout21.012.net.il> for freebsd-net@freebsd.org; Thu, 27 Aug 2015 10:56:45 +0300 (IDT) Received: from mail.ngtech.co.il ([84.95.212.160]) by a-mtaout21.012.net.il (HyperSendmail v2007.08) with ESMTPSA id <0NTQ00KARE2K2590@a-mtaout21.012.net.il> for freebsd-net@freebsd.org; Thu, 27 Aug 2015 10:56:45 +0300 (IDT) Received: by mail.ngtech.co.il (Postfix, from userid 5001) id 26EA523973; Thu, 27 Aug 2015 10:56:44 +0300 (IDT) Received: from [192.168.10.131] (unknown [192.168.10.131]) by mail.ngtech.co.il (Postfix) with ESMTPA id 38648234AF; Thu, 27 Aug 2015 10:56:43 +0300 (IDT) Date: Thu, 27 Aug 2015 10:56:44 +0300 From: Eliezer Croitoru Subject: Re: Issues with MASQUARDE and FreeBSD router. In-reply-to: <55DDEA51.8010902@ngtech.co.il> X-012-Sender: eliezer-111@012.net.il To: netfilter@vger.kernel.org Cc: freebsd-net@freebsd.org Message-id: <55DEC2BC.8030800@ngtech.co.il> MIME-version: 1.0 Content-type: text/plain; charset=utf-8; format=flowed Content-transfer-encoding: 7bit X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on mail.ngtech.co.il References: <55DDEA51.8010902@ngtech.co.il> User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64; rv:38.0) Gecko/20100101 Thunderbird/38.2.0 X-Spam-Status: No, score=-1.0 required=3.0 tests=ALL_TRUSTED autolearn=ham autolearn_force=no version=3.4.0 X-Spam-Level: X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 27 Aug 2015 07:56:47 -0000 I added a filter rule to iptables with a INVALID reject match and any packet that is being passed throw the FreeBSD router is being marked by itpables as INVALID. An example for an INVALID packet: http://ngtech.co.il/nat_issue/proxy2.pcap Eliezer On 26/08/2015 21:24, Eliezer Croitoru wrote: > Hey lists, > > I had a similar issue in the past but now I have found the combination > which results in the issue. > My topology is between two KVM hosts. > Server is on KVM1 ip address 192.168.10.1/24 > Another whole network on the KVM2. > And the traffic is: > client 192.168.11.2/24 --> R1 - 192.168.11.254/24 > R1 192.168.15.1/24 --> R2(NAT SERVER) 192.168.15.254/24 > R3 eth4 NATed(masquerade) 192.168.10.179/24 --> Server 192.168.10.1/24 > > The Above is what is suppose to happen and the reality us that > 192.168.10.1 receives a packet but from 192.168.11.2. > > I can reproduce the issue successfully replacing the R1 server from a > linux box to a FreeBSD 10.1 box.(freebsd causes the issue) > The routers I have used are: > CentOS 7 > VYOS 1.6 > > It is the same for both and I can reproduce the issue successfully. > > I have also tested the R1 replaced with: > VYOS 1.7 > CENTOS 7 > DEBIAN 8 > vSRX > FreeBSD 4.11 with e1000 card, works fine. > FreeBSD 10.1(amd64) with e1000 card, works fine. > *FreeBSD 10.1(amd64) with virtio card, have an issue.* > > Now I am trying to figure out if it's a netfilter issue or FreeBSD > virtio driver issue and if so what might be the direction to make this > issue fixed. > > Tcpdump captures on the NAT router of different packets and sessions are > here: > http://ngtech.co.il/nat_issue/ > > If the issue is probably with the FreeBSD virtio drivers why would the > MASQUERADE pass the packet to the destination server? > > Thanks, > Eliezer > > > From owner-freebsd-net@freebsd.org Thu Aug 27 13:14:33 2015 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id A9F2E9C48C3 for ; Thu, 27 Aug 2015 13:14:33 +0000 (UTC) (envelope-from lidl@pix.net) Received: from hydra.pix.net (hydra.pix.net [IPv6:2001:470:e254::4]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mail.pix.net", Issuer "Pix.Com Technologies, LLC CA" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 5DA4FFEE for ; Thu, 27 Aug 2015 13:14:33 +0000 (UTC) (envelope-from lidl@pix.net) Received: from torb.pix.net (verizon.pix.net [71.178.232.3]) (authenticated bits=0) by hydra.pix.net (8.15.2/8.15.2) with ESMTPA id t7RDEVXT049108; Thu, 27 Aug 2015 09:14:31 -0400 (EDT) (envelope-from lidl@pix.net) X-Authentication-Warning: hydra.pix.net: Host verizon.pix.net [71.178.232.3] claimed to be torb.pix.net To: freebsd-net@freebsd.org References: <55DCF080.7080208@stankevitz.com> Subject: Re: ssh over WAN: TCP window too small From: Kurt Lidl Message-ID: <55DF0D37.5060003@pix.net> Date: Thu, 27 Aug 2015 09:14:31 -0400 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:38.0) Gecko/20100101 Thunderbird/38.2.0 MIME-Version: 1.0 In-Reply-To: <55DCF080.7080208@stankevitz.com> Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 27 Aug 2015 13:14:33 -0000 Chris Stankevitz wrote: > Hi, > > # cat /dev/urandom | ssh root at host 'cat > /dev/null' > > I use the above ssh command over a high-BDP WAN link (80 ms @ 100 Mbps). > tcpdump shows I am TCP window limited to 64 KBytes (yielding 5 Mbps). > iperf with default options gets the window opened to 500 KBytes > (yielding 35 Mbps). > > Both sides of the connection: FreeBSD 10.1 w/default sshd options > (except I permit root login). In particular, HPN is not disabled. > > Can anyone explain my abysmally small TCP window? > > Can anyone recommend some tools/tricks to figure out what in FreeBSD > and/or base SSH is limiting the send/recv buffer and/or TCP window? I know this response is a little late to the party, but... I spent a bit of time last year tuning my FreeBSD 10.1 host to be able to transfer a bunch of data between the east coast of the US and the west cost. My WAN link was more like 70ms @ 75 Mbps, so not too different than yours. The other end of the connection was also a FreeBSD 10.1 host. I have the following in my /etc/sysctl.conf - and I get pretty much all 75Mbps when I scp or rsync a file: # tcp options for long-haul speedups kern.ipc.maxsockbuf=4194304 # (2 * default 2097152) net.inet.tcp.mssdflt=1448 # (default 576) net.inet.tcp.sendbuf_max=4194304 # (2 * default 2097152) net.inet.tcp.recvbuf_max=4194304 # (2 * default 2097152) net.inet.tcp.syncache.rexmtlimit=1 # (default 3) net.inet.tcp.recvspace=262144 # (4 * default 65,536) net.inet.tcp.sendspace=262144 # (4 * default 65,536) net.inet.tcp.sendbuf_inc=65536 # (8 * default 8192) net.inet.tcp.recvbuf_inc=131072 # (8 * default 16384) One thing that was noticed - it can take a really, really, really long time for the TCP window to open up the whole way with the default net.inet.tcp.sendbuf_inc setting! -Kurt From owner-freebsd-net@freebsd.org Thu Aug 27 16:33:08 2015 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 226349C4521 for ; Thu, 27 Aug 2015 16:33:08 +0000 (UTC) (envelope-from ohartman@zedat.fu-berlin.de) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id F304FAA8 for ; Thu, 27 Aug 2015 16:33:07 +0000 (UTC) (envelope-from ohartman@zedat.fu-berlin.de) Received: by mailman.ysv.freebsd.org (Postfix) id EEE1A9C451E; Thu, 27 Aug 2015 16:33:07 +0000 (UTC) Delivered-To: net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id D5BB69C451B; Thu, 27 Aug 2015 16:33:07 +0000 (UTC) (envelope-from ohartman@zedat.fu-berlin.de) Received: from outpost1.zedat.fu-berlin.de (outpost1.zedat.fu-berlin.de [130.133.4.66]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 8F494AA3; Thu, 27 Aug 2015 16:33:07 +0000 (UTC) (envelope-from ohartman@zedat.fu-berlin.de) Received: from inpost2.zedat.fu-berlin.de ([130.133.4.69]) by outpost.zedat.fu-berlin.de (Exim 4.85) with esmtp (envelope-from ) id <1ZV06l-002JFh-A5>; Thu, 27 Aug 2015 18:32:59 +0200 Received: from x5ce1b959.dyn.telefonica.de ([92.225.185.89] helo=thor.walstatt.dynvpn.de) by inpost2.zedat.fu-berlin.de (Exim 4.85) with esmtpsa (envelope-from ) id <1ZV06l-002L8W-2Z>; Thu, 27 Aug 2015 18:32:59 +0200 Date: Thu, 27 Aug 2015 18:32:53 +0200 From: "O. Hartmann" To: Adrian Chadd Cc: Gleb Smirnoff , "current@freebsd.org" , "net@freebsd.org" Subject: Re: [head up!] WiFi drivers changes Message-ID: <20150827183253.55400c5a.ohartman@zedat.fu-berlin.de> In-Reply-To: References: <20150806151355.GL889@FreeBSD.org> <20150807175226.357b5dce.ohartman@zedat.fu-berlin.de> Organization: FU Berlin X-Mailer: Claws Mail 3.12.0 (GTK+ 2.24.28; amd64-portbld-freebsd11.0) MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha256; boundary="Sig_/RkSl9wsmfGqTdWE5yIb+Mw5"; protocol="application/pgp-signature" X-Originating-IP: 92.225.185.89 X-ZEDAT-Hint: A X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 27 Aug 2015 16:33:08 -0000 --Sig_/RkSl9wsmfGqTdWE5yIb+Mw5 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: quoted-printable Am Fri, 7 Aug 2015 16:50:23 -0700 Adrian Chadd schrieb: > Hi, >=20 > Gleb/others doing this: you have 4 days to figure out what's wrong > with things, or I'm backing all of this work out. >=20 > Thanks, >=20 >=20 > -adrian >=20 >=20 > On 7 August 2015 at 08:52, O. Hartmann wrot= e: > > Am Thu, 6 Aug 2015 18:13:55 +0300 > > Gleb Smirnoff schrieb: > > > >> Hi! > >> > >> As part of the "opaque ifnet project" [1], all 802.11 (WiFi) drivers > >> undergo change of not being an interface anymore. Historically in Free= BSD > >> 802.11 stack, 802.11 devices called if_attach() and created an interfa= ce. > >> Later this was generalized and real functioning interface is created by > >> net80211 stack. However, remnant of parent interface remained. If you > >> are running Intel Centrino wireless, then you got iwn0 interface and > >> wlan0 interface. However, the former doesn't do anything. You can't > >> assign addresses to it or modify any of it parameters. Or you can > >> modify them, but that affects nothing. > >> > >> This superfluous ifnet on the list entangles the net80211 stack and > >> also is on the way of [1]. So, decision was made to remove it. I > >> already did preparatory commits back in May, and now it is time to > >> finish that. > >> > >> The patch is: > >> > >> https://reviews.freebsd.org/D2655 > >> > >> And the Wiki page for it is: > >> > >> https://wiki.freebsd.org/projects/ifnet/net80211 > >> > >> The patch modifies every driver, and diff is bulky. However, changes > >> are mechanical and simple, most drivers appeared to work after first > >> run. Most converted drivers are tested to work. > >> > >> This is list of drivers that are not tested, due to lack of testers: > >> > >> mwl, ipw, bwn, wi, upgt, uath. > >> > >> But, as said, changes are mechanical and probability is 95% that > >> they will work. > >> > >> The only complex one is ndis(4). It could be broken by conversion. > >> Since I already got a tester volunteer, I will fix it quickly if > >> anything happens. > >> > >> Another untrivial one is wtap(4), which is not connected to the > >> build and appeared to be broken even before conversion. Anyway, > >> I made it compilable. > >> > >> Now, for the configuration. The sequence of commands you need > >> to run to configure a WiFi interface doesn't change. As before > >> it is: > >> > >> ifconfig wlan0 create wlandev iwn0 > >> ifconfig wlan0 $foo > >> > >> Your rc.conf doesn't need any changes. As before: > >> > >> wlans_iwn0=3D"wlan0" > >> ifconfig_wlan0=3D"DHCP WPA" > >> > >> However, iwn0 disappeared from the 'ifconfig -l'. It is still > >> in devinfo, or in dmesg. For the sake of installers or other > >> configuration software, a sysctl is provided: > >> > >> net.wlan.devices: iwn0 > >> > >> The /etc subsystem needs to be tweaked. Previously the wlan(4) > >> interfaces were created in childif_create(), and the script > >> did check for presence of parent interface. In my patch I > >> provided wlans_up(), that doesn't check. The code in D2655 > >> now works correctly both on patched and on unpatched kernel. > >> > >> Alternatively, I could tweak childif_create() to use net.wlan.devices > >> instead of 'ifconfig -l'. Or, to use them both, to work on older > >> and on newer kernels? > >> > >> I am not sure which path with /etc is better, so seeking for > >> help with that. > >> > > > > After updating to FreeBSD 11.0-CURRENT #0 r286415: Fri Aug 7 17:22:43 = CEST 2015 > > amd64, several APs won't startup anymore: > > > > [...] > > Starting hostapd. > > Configuration file: /etc/hostapd.conf > > bsd_set_if_media: SIOCSIFMEDIA Device not configured > > bsd_init: failed to set operation mode > > bsd driver initialization failed. > > wlan0: interface state UNINITIALIZED->DISABLED > > wlan0: AP-DISABLED > > hostapd_free_hapd_data: Interface wlan0 wasn't started > > ELOOP: remaining socket: sock=3D5 eloop_data=3D0x801c47100 user_data=3D= 0x0 handler=3D0x41a0e0 > > /etc/rc.d/hostapd: WARNING: failed to start hostapd > _______________________________________________ > freebsd-current@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-current > To unsubscribe, send any mail to "freebsd-current-unsubscribe@freebsd.org" Again, with the most recent changes as of r287211, hostapd doens't start my WiFi A= P anymore (FreeBSD 11.0-CURRENT #7 r287169: Wed Aug 26 20:26:49 CEST 2015 amd64 does!= ). --Sig_/RkSl9wsmfGqTdWE5yIb+Mw5 Content-Type: application/pgp-signature Content-Description: OpenPGP digital signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQEcBAEBCAAGBQJV3zu1AAoJEOgBcD7A/5N8ZfIIAMW1FX+G+cj2NAZ4Tpu9CTbb YvRHrBdq+H3YTpO6oetgTUqE2TQCj12qeUWwiBLYJi0nk9ffBJDCqTQi15yNhMmP f1Zw8ikr/tw6YuOU8odMF5kA8or6A8kxK9IEZnsuSltkzgLi7PYGuchTUsWemlV9 49S5wG8lV5eKANPsgdxJgEwLw2RbVfZsFZEVcrNJlnXBhftxAweWhrfhRylJvNh+ 1k2OyH59dRznl1SoViVwhU2Lf4d0Hi5xHfgDeEwRA3RjTornccUuIobpemqzdvmv WoqrTOP7ie66OrLpaVnPkXIHl+8sHsFwvqj++jS0DtQiMc7kVO/OsJvi+/sSIXY= =FuPL -----END PGP SIGNATURE----- --Sig_/RkSl9wsmfGqTdWE5yIb+Mw5-- From owner-freebsd-net@freebsd.org Thu Aug 27 16:44:07 2015 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 755009C48F6 for ; Thu, 27 Aug 2015 16:44:07 +0000 (UTC) (envelope-from glebius@FreeBSD.org) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 594F510F1 for ; Thu, 27 Aug 2015 16:44:07 +0000 (UTC) (envelope-from glebius@FreeBSD.org) Received: by mailman.ysv.freebsd.org (Postfix) id 568829C48F4; Thu, 27 Aug 2015 16:44:07 +0000 (UTC) Delivered-To: net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 55FDE9C48F3; Thu, 27 Aug 2015 16:44:07 +0000 (UTC) (envelope-from glebius@FreeBSD.org) Received: from cell.glebius.int.ru (glebius.int.ru [81.19.69.10]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "cell.glebius.int.ru", Issuer "cell.glebius.int.ru" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id D076710EF; Thu, 27 Aug 2015 16:44:06 +0000 (UTC) (envelope-from glebius@FreeBSD.org) Received: from cell.glebius.int.ru (localhost [127.0.0.1]) by cell.glebius.int.ru (8.15.2/8.15.2) with ESMTPS id t7RGi3WG085655 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Thu, 27 Aug 2015 19:44:04 +0300 (MSK) (envelope-from glebius@FreeBSD.org) Received: (from glebius@localhost) by cell.glebius.int.ru (8.15.2/8.15.2/Submit) id t7RGi3GU085654; Thu, 27 Aug 2015 19:44:03 +0300 (MSK) (envelope-from glebius@FreeBSD.org) X-Authentication-Warning: cell.glebius.int.ru: glebius set sender to glebius@FreeBSD.org using -f Date: Thu, 27 Aug 2015 19:44:03 +0300 From: Gleb Smirnoff To: "O. Hartmann" Cc: Adrian Chadd , "current@freebsd.org" , "net@freebsd.org" , pluknet@FreeBSD.org Subject: Re: [head up!] WiFi drivers changes Message-ID: <20150827164403.GG56997@glebius.int.ru> References: <20150806151355.GL889@FreeBSD.org> <20150807175226.357b5dce.ohartman@zedat.fu-berlin.de> <20150827183253.55400c5a.ohartman@zedat.fu-berlin.de> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20150827183253.55400c5a.ohartman@zedat.fu-berlin.de> User-Agent: Mutt/1.5.23 (2014-03-12) X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 27 Aug 2015 16:44:07 -0000 Oliver, O> Again, O> with the most recent changes as of r287211, hostapd doens't start my WiFi AP anymore O> (FreeBSD 11.0-CURRENT #7 r287169: Wed Aug 26 20:26:49 CEST 2015 amd64 does!). Let's start investigating from scratch, since the /etc part of the patch have changed significantly. Can you please send me privately your configs and debug log of hostapd? [Adding pluknet@ to Cc, who helped a lot with testing the hostap support during patch development] -- Totus tuus, Glebius. From owner-freebsd-net@freebsd.org Thu Aug 27 16:46:51 2015 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 4932D9C4A0B for ; Thu, 27 Aug 2015 16:46:51 +0000 (UTC) (envelope-from ohartman@zedat.fu-berlin.de) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 28FB3136D for ; Thu, 27 Aug 2015 16:46:51 +0000 (UTC) (envelope-from ohartman@zedat.fu-berlin.de) Received: by mailman.ysv.freebsd.org (Postfix) id 2441F9C4A08; Thu, 27 Aug 2015 16:46:51 +0000 (UTC) Delivered-To: net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 09FD29C4A07; Thu, 27 Aug 2015 16:46:51 +0000 (UTC) (envelope-from ohartman@zedat.fu-berlin.de) Received: from outpost1.zedat.fu-berlin.de (outpost1.zedat.fu-berlin.de [130.133.4.66]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id B5DC0136A; Thu, 27 Aug 2015 16:46:50 +0000 (UTC) (envelope-from ohartman@zedat.fu-berlin.de) Received: from inpost2.zedat.fu-berlin.de ([130.133.4.69]) by outpost.zedat.fu-berlin.de (Exim 4.85) with esmtp (envelope-from ) id <1ZV0K8-002NW9-PG>; Thu, 27 Aug 2015 18:46:48 +0200 Received: from x5ce1b92b.dyn.telefonica.de ([92.225.185.43] helo=thor.walstatt.dynvpn.de) by inpost2.zedat.fu-berlin.de (Exim 4.85) with esmtpsa (envelope-from ) id <1ZV0K8-002MAW-HU>; Thu, 27 Aug 2015 18:46:48 +0200 Date: Thu, 27 Aug 2015 18:45:14 +0200 From: "O. Hartmann" To: Adrian Chadd Cc: Gleb Smirnoff , "current@freebsd.org" , "net@freebsd.org" Subject: Re: [head up!] WiFi drivers changes Message-ID: <20150827184514.7e86e531.ohartman@zedat.fu-berlin.de> In-Reply-To: <20150827183253.55400c5a.ohartman@zedat.fu-berlin.de> References: <20150806151355.GL889@FreeBSD.org> <20150807175226.357b5dce.ohartman@zedat.fu-berlin.de> <20150827183253.55400c5a.ohartman@zedat.fu-berlin.de> Organization: FU Berlin X-Mailer: Claws Mail 3.12.0 (GTK+ 2.24.28; amd64-portbld-freebsd11.0) MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha256; boundary="Sig_/jy+C+SSGd=FpQ_VTbJ3Z_zE"; protocol="application/pgp-signature" X-Originating-IP: 92.225.185.43 X-ZEDAT-Hint: A X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 27 Aug 2015 16:46:51 -0000 --Sig_/jy+C+SSGd=FpQ_VTbJ3Z_zE Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: quoted-printable Am Thu, 27 Aug 2015 18:32:53 +0200 "O. Hartmann" schrieb: > Am Fri, 7 Aug 2015 16:50:23 -0700 > Adrian Chadd schrieb: >=20 > > Hi, > >=20 > > Gleb/others doing this: you have 4 days to figure out what's wrong > > with things, or I'm backing all of this work out. > >=20 > > Thanks, > >=20 > >=20 > > -adrian > >=20 > >=20 > > On 7 August 2015 at 08:52, O. Hartmann wr= ote: > > > Am Thu, 6 Aug 2015 18:13:55 +0300 > > > Gleb Smirnoff schrieb: > > > > > >> Hi! > > >> > > >> As part of the "opaque ifnet project" [1], all 802.11 (WiFi) drive= rs > > >> undergo change of not being an interface anymore. Historically in Fr= eeBSD > > >> 802.11 stack, 802.11 devices called if_attach() and created an inter= face. > > >> Later this was generalized and real functioning interface is created= by > > >> net80211 stack. However, remnant of parent interface remained. If you > > >> are running Intel Centrino wireless, then you got iwn0 interface and > > >> wlan0 interface. However, the former doesn't do anything. You can't > > >> assign addresses to it or modify any of it parameters. Or you can > > >> modify them, but that affects nothing. > > >> > > >> This superfluous ifnet on the list entangles the net80211 stack and > > >> also is on the way of [1]. So, decision was made to remove it. I > > >> already did preparatory commits back in May, and now it is time to > > >> finish that. > > >> > > >> The patch is: > > >> > > >> https://reviews.freebsd.org/D2655 > > >> > > >> And the Wiki page for it is: > > >> > > >> https://wiki.freebsd.org/projects/ifnet/net80211 > > >> > > >> The patch modifies every driver, and diff is bulky. However, changes > > >> are mechanical and simple, most drivers appeared to work after first > > >> run. Most converted drivers are tested to work. > > >> > > >> This is list of drivers that are not tested, due to lack of testers: > > >> > > >> mwl, ipw, bwn, wi, upgt, uath. > > >> > > >> But, as said, changes are mechanical and probability is 95% that > > >> they will work. > > >> > > >> The only complex one is ndis(4). It could be broken by conversion. > > >> Since I already got a tester volunteer, I will fix it quickly if > > >> anything happens. > > >> > > >> Another untrivial one is wtap(4), which is not connected to the > > >> build and appeared to be broken even before conversion. Anyway, > > >> I made it compilable. > > >> > > >> Now, for the configuration. The sequence of commands you need > > >> to run to configure a WiFi interface doesn't change. As before > > >> it is: > > >> > > >> ifconfig wlan0 create wlandev iwn0 > > >> ifconfig wlan0 $foo > > >> > > >> Your rc.conf doesn't need any changes. As before: > > >> > > >> wlans_iwn0=3D"wlan0" > > >> ifconfig_wlan0=3D"DHCP WPA" > > >> > > >> However, iwn0 disappeared from the 'ifconfig -l'. It is still > > >> in devinfo, or in dmesg. For the sake of installers or other > > >> configuration software, a sysctl is provided: > > >> > > >> net.wlan.devices: iwn0 > > >> > > >> The /etc subsystem needs to be tweaked. Previously the wlan(4) > > >> interfaces were created in childif_create(), and the script > > >> did check for presence of parent interface. In my patch I > > >> provided wlans_up(), that doesn't check. The code in D2655 > > >> now works correctly both on patched and on unpatched kernel. > > >> > > >> Alternatively, I could tweak childif_create() to use net.wlan.devices > > >> instead of 'ifconfig -l'. Or, to use them both, to work on older > > >> and on newer kernels? > > >> > > >> I am not sure which path with /etc is better, so seeking for > > >> help with that. > > >> > > > > > > After updating to FreeBSD 11.0-CURRENT #0 r286415: Fri Aug 7 17:22:4= 3 CEST 2015 > > > amd64, several APs won't startup anymore: > > > > > > [...] > > > Starting hostapd. > > > Configuration file: /etc/hostapd.conf > > > bsd_set_if_media: SIOCSIFMEDIA Device not configured > > > bsd_init: failed to set operation mode > > > bsd driver initialization failed. > > > wlan0: interface state UNINITIALIZED->DISABLED > > > wlan0: AP-DISABLED > > > hostapd_free_hapd_data: Interface wlan0 wasn't started > > > ELOOP: remaining socket: sock=3D5 eloop_data=3D0x801c47100 user_data= =3D0x0 > > > handler=3D0x41a0e0 /etc/rc.d/hostapd: WARNING: failed to start hostapd > > _______________________________________________ > > freebsd-current@freebsd.org mailing list > > http://lists.freebsd.org/mailman/listinfo/freebsd-current > > To unsubscribe, send any mail to "freebsd-current-unsubscribe@freebsd.o= rg" >=20 > Again, > with the most recent changes as of r287211, hostapd doens't start my WiFi= AP anymore > (FreeBSD 11.0-CURRENT #7 r287169: Wed Aug 26 20:26:49 CEST 2015 amd64 doe= s!). Ups, sory, sorry - I forgot on that specific machine (1 of 5) to mergemaster :-(= So, after mergemaster, the new rc scripts also got installed on the AP server and the= interface is again up and running! Regards, oh --Sig_/jy+C+SSGd=FpQ_VTbJ3Z_zE Content-Type: application/pgp-signature Content-Description: OpenPGP digital signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQEcBAEBCAAGBQJV3z6aAAoJEOgBcD7A/5N8H+wH/A4YCOu0V7CQPAZyJjdA1C2N 5MtxvdKIhpJbdSGkgtr0mCFkGNZyHejo0Rlg0ikiR46QHb4v+oR3OCjfsuzJojMR kITyd92VqvdQoslSB8/GHnBrZScv/que3IeACOuYpQdVwjnuWmiW6smSEMkZkmz6 i+aSIIpRcQpjbnQHqUskNZ7GJX+LUY8IEnBLxP/gd1QOeHoEV2ZhIjsniOBL61s+ mY/9KmDxcCwDFC+obwAU4/0GbLsb85UrIRJ9v3FYZpAT6IQnaUqx5x26SdGzHT2b goTO9aV2gQEfCaYoyJzYfB9pftSHLoYDNff9IQkbiln67dezHZFDVXGbqNChEnc= =9yxY -----END PGP SIGNATURE----- --Sig_/jy+C+SSGd=FpQ_VTbJ3Z_zE-- From owner-freebsd-net@freebsd.org Thu Aug 27 16:58:11 2015 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 6A1DB9C4DCC for ; Thu, 27 Aug 2015 16:58:11 +0000 (UTC) (envelope-from glebius@FreeBSD.org) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 4DE0D1DF6 for ; Thu, 27 Aug 2015 16:58:11 +0000 (UTC) (envelope-from glebius@FreeBSD.org) Received: by mailman.ysv.freebsd.org (Postfix) id 49C409C4DC9; Thu, 27 Aug 2015 16:58:11 +0000 (UTC) Delivered-To: net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 48FAC9C4DC7; Thu, 27 Aug 2015 16:58:11 +0000 (UTC) (envelope-from glebius@FreeBSD.org) Received: from cell.glebius.int.ru (glebius.int.ru [81.19.69.10]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "cell.glebius.int.ru", Issuer "cell.glebius.int.ru" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id C76131DF3; Thu, 27 Aug 2015 16:58:10 +0000 (UTC) (envelope-from glebius@FreeBSD.org) Received: from cell.glebius.int.ru (localhost [127.0.0.1]) by cell.glebius.int.ru (8.15.2/8.15.2) with ESMTPS id t7RGw8SB085728 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Thu, 27 Aug 2015 19:58:08 +0300 (MSK) (envelope-from glebius@FreeBSD.org) Received: (from glebius@localhost) by cell.glebius.int.ru (8.15.2/8.15.2/Submit) id t7RGw8Z0085727; Thu, 27 Aug 2015 19:58:08 +0300 (MSK) (envelope-from glebius@FreeBSD.org) X-Authentication-Warning: cell.glebius.int.ru: glebius set sender to glebius@FreeBSD.org using -f Date: Thu, 27 Aug 2015 19:58:08 +0300 From: Gleb Smirnoff To: "O. Hartmann" Cc: Adrian Chadd , "current@freebsd.org" , "net@freebsd.org" , pluknet@FreeBSD.org Subject: Re: [head up!] WiFi drivers changes Message-ID: <20150827165808.GH56997@glebius.int.ru> References: <20150806151355.GL889@FreeBSD.org> <20150807175226.357b5dce.ohartman@zedat.fu-berlin.de> <20150827183253.55400c5a.ohartman@zedat.fu-berlin.de> <20150827184514.7e86e531.ohartman@zedat.fu-berlin.de> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20150827184514.7e86e531.ohartman@zedat.fu-berlin.de> User-Agent: Mutt/1.5.23 (2014-03-12) X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 27 Aug 2015 16:58:11 -0000 Good news, thanks! On Thu, Aug 27, 2015 at 06:45:14PM +0200, O. Hartmann wrote: O> sory, sorry - I forgot on that specific machine (1 of 5) to mergemaster :-( So, after O> mergemaster, the new rc scripts also got installed on the AP server and the interface is O> again up and running! O> O> Regards, O> O> oh -- Totus tuus, Glebius. From owner-freebsd-net@freebsd.org Thu Aug 27 23:17:32 2015 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 249DD9C327B for ; Thu, 27 Aug 2015 23:17:32 +0000 (UTC) (envelope-from kob6558@gmail.com) Received: from mail-ob0-x232.google.com (mail-ob0-x232.google.com [IPv6:2607:f8b0:4003:c01::232]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id DDFA1164A for ; Thu, 27 Aug 2015 23:17:31 +0000 (UTC) (envelope-from kob6558@gmail.com) Received: by obkg7 with SMTP id g7so29308552obk.3 for ; Thu, 27 Aug 2015 16:17:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=miTuGhkVBKNKUovcaODTjGReUo1940+k9wTxWtdNW50=; b=rOzhTPl6DdRslDjfRGNe0/dwtUJvgIBmAx223m4Dz5dO6DIoY48jsxggjyxob0tJxb o5680SsgaGvEmKW/h7Ck2FzUWWHED+h+8+Q9n0cRNg9Ye6s7loSPJBrqWfY9QixOExbk mgRcPtdPi8U8xIuoa5JH6gK0rOf62Q+D5HYIPUD6i6DLyZWJ7llQ63SGTj8boCgp4Q/o kMw0tJtAW4XmzRbkOIIpeY90DzoteoOIxFGydJ+oOCi6JSEEZjZG6nIlK9PVPytIUEx+ 5bBklzmXkf2dGVTtkbRv9KESpJSjT9l5xjYR3nXL43ISGU7RiA9nwvHYi4cCm25/znHZ frRQ== MIME-Version: 1.0 X-Received: by 10.182.97.10 with SMTP id dw10mr4345237obb.60.1440717450657; Thu, 27 Aug 2015 16:17:30 -0700 (PDT) Sender: kob6558@gmail.com Received: by 10.202.173.83 with HTTP; Thu, 27 Aug 2015 16:17:30 -0700 (PDT) In-Reply-To: <55DD0453.3040803@stankevitz.com> References: <55DCF080.7080208@stankevitz.com> <27420EDC-5816-4B9E-A834-E4A035B8411C@lists.zabbadoz.net> <55DD0453.3040803@stankevitz.com> Date: Thu, 27 Aug 2015 13:17:30 -1000 X-Google-Sender-Auth: jeQjGc-LGBpBOxq56kw6F2lI_js Message-ID: Subject: Re: ssh over WAN: TCP window too small From: Kevin Oberman To: Chris Stankevitz Cc: "Bjoern A. Zeeb" , "freebsd-net@freebsd.org" Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 27 Aug 2015 23:17:32 -0000 On Tue, Aug 25, 2015 at 2:12 PM, Chris Stankevitz wrote: > On 8/25/15 4:11 PM, Bjoern A. Zeeb wrote: > >> >> On 25 Aug 2015, at 22:47 , Chris Stankevitz wrote: >>> >>> Can anyone recommend some tools/tricks to figure out what in FreeBSD >>> and/or >>> >> >> base SSH is limiting the send/recv buffer and/or TCP window? > >> >> if you have the memory, try these sysctls: >> >> kern.ipc.maxsockbuf=146800640 >> net.inet.tcp.recvbuf_max=67108864 >> net.inet.tcp.sendbuf_max=67108864 >> > > Bjoern, > > Thank you for the reply. Before your suggestion my sysctls are: > > kern.ipc.maxsockbuf=2097152 > net.inet.tcp.recvbuf_max=2097152 > net.inet.tcp.sendbuf_max=2097152 > > Each of these is much larger than the limit I am experiencing (~64,000). > So I [naively] expect changing these values will have no effect. Let me > try... > > ... okay... sure enough the sysctrl changes you suggest did not change the > 64,000 bytes-in-flight limit I am experiencing. > > Thanks for the idea (and keep 'em coming!), > > Chris > My former employer, ESnet, was heavily involved in moving very large amounts of data (petabytes) over very long (intercontinental), very fat (100G) pipes. In an effort to improve customer satisfaction they have done extensive research into the issues involved and have published much of it at http://fasterdata.es.net. In particular, they have documented the issues with ssh over long latency links at http://fasterdata.es.net/data-transfer-tools/say-no-to-scp/. It is VERY hard to get good performance on high latency links in the bast of cases and, unfortunately, ssh/scp makes it not the best of cases. -- Kevin Oberman, Goat herder and Retired Network Engineer From owner-freebsd-net@freebsd.org Fri Aug 28 12:11:29 2015 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 751199C430C for ; Fri, 28 Aug 2015 12:11:29 +0000 (UTC) (envelope-from bounce_Ayw26yaeJnSA8M82_xvx4YcsukLNnIm4c_df8eb5e06b037b98_1@pm1009.com) Received: from b3.mtb-2.com (b3.mtb-2.com [69.63.146.172]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 279F69FB for ; Fri, 28 Aug 2015 12:11:28 +0000 (UTC) (envelope-from bounce_Ayw26yaeJnSA8M82_xvx4YcsukLNnIm4c_df8eb5e06b037b98_1@pm1009.com) DKIM-Signature: v=1; a=rsa-sha1; c=relaxed/relaxed; s=mykey2; d=mtb-2.com; h=Reply-To:List-Unsubscribe:From:To:Subject:MIME-Version:Date:Message-Id:Content-Type; bh=bW2ZP9GjjYFdsuxpg9A3KZGQnec=; b=uaVvrLq9QFteF2bLnTt2wBBVnj9OJsIQw6NWQ48OjrU+NCm4rTqm0LmQu7XWgiLyDcw3J/ws5NJ6 6E2dZJ0N4TWCzYjL0C4KKK7GBXCGXHyzrdyDEIsdy8quvJ0flNTN1UWeZ4S919BrtkuG0p6Bnv0F DoAJSBawMD4Gura27dqN0DrS9iMBk64wEyHK3VAh1W+JO23B7XRsPM8EN+vxD45lVXMHh8jpMDSf mS+gyrAqwdpBIxQ352n6of3gwoDWb8UGxk+/q+5onzwAqXpt4n5KViZkj4qK9h8BTZHPK4lHBSh1 8NK1dpdR/NZ2FuHLKfDpRquPmpTSKURjOAMTtw== Received: from localhost (10.10.204.18) by b3.mtb-2.com id hs17v2157o49 for ; Fri, 28 Aug 2015 14:01:20 +0200 (envelope-from ) Reply-To: digital@cgconsulting.co.za X-Priority: 3 X-Report-Abuse: X-Data: xvx4YcsukLNnIm4c.df8eb5e06b037b98 X-Data-EUID: 38c72f7ae0162144901e855c78e5cf06 X-Data-Rating: -2 From: Philips Healthcare To: freebsd-net@freebsd.org Subject: Accelerating Healthcare Innovation MIME-Version: 1.0 Date: Fri, 28 Aug 2015 13:01:20 +0100 Message-Id: <2015082813132820.2129.272@digitalfire.co.za> Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 28 Aug 2015 12:11:29 -0000 Philips Healthcare Issue: August 2015 Dramatic changes in store for the hospital of tomorrow “Big data” and information and communications technology set to revolutionize healthcare. Find out more ( http://dfire.ensighthq.com/content/philips/P020/media/Philips_WSJ-02_Dramatic-changes-in-store-for-the-hospital-of-tomorrow.pdf?origin=3_me_en_healthcare_media_campaign_digitalfire_eb_jul__kenya ) Precious details See how Philips MRI technology helped a team of MRI clinicians and surgeons remove a boy’s brain tumor. Follow Mustafa into the operating room in this touching story of hope. Find out more ( https://www.youtube.com/watch?v=k5KZsxLw1Bg?origin=3_me_en_healthcare_media_campaign_philips_eb_jul__kenya ) Precious details No baby forgotten No baby forgotten Gertrude’s Children’s Hospital used the Philips IntelliVue X2 technology to monitor an abandoned baby’s vital signs and help her thrive. Find out more ( https://www.youtube.com/watch?v=O2OMV-7xgts?origin=3_me_en_healthcare_media_campaign_philips_eb_jul__kenya ) Also in this issue: Leveraging key intersections to accelerate healthcare innovation Sharing risk, responsibility and reward Technology and process re-engineering open the door to coordinated care Leveraging key intersections to accelerate healthcare innovation Sharing risk, responsibility and reward Technology and process re-engineering open the door to coordinated care Find out more ( http://dfire.ensighthq.com/content/philips/P020/media/Philips_WSJ-05_Leveraging-key-intersections-to-accelerate-healthcare-innovation.pdf?origin=3_me_en_healthcare_media_campaign_digitalfire_eb_jul__kenya ) Find out more ( http://dfire.ensighthq.com/content/philips/P020/media/Philips_WSJ-13_Sharing-risk-responsibility-and-reward.pdf?origin=3_me_en_healthcare_media_campaign_digitalfire_eb_jul__kenya ) Find out more ( http://dfire.ensighthq.com/content/philips/P020/media/Philips_WSJ-16_Technology-and-process-re-engineering-open-the-door-to-coordinated-care.pdf?origin=3_me_en_healthcare_media_campaign_digitalfire_eb_jul__kenya ) Subscribe to the Philips Health newsletter ( http://dfire.ensighthq.com/live/content.php?Item_ID=7911 ) Footer From owner-freebsd-net@freebsd.org Fri Aug 28 16:35:29 2015 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 3F70B9C5BBA for ; Fri, 28 Aug 2015 16:35:29 +0000 (UTC) (envelope-from chris@stankevitz.com) Received: from mango.stankevitz.com (mango.stankevitz.com [208.79.93.194]) by mx1.freebsd.org (Postfix) with ESMTP id 2B926304 for ; Fri, 28 Aug 2015 16:35:29 +0000 (UTC) (envelope-from chris@stankevitz.com) Received: from Chriss-MacBook-Pro.local (209-203-101-124.static.twtelecom.net [209.203.101.124]) (using TLSv1.2 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by mango.stankevitz.com (Postfix) with ESMTPSA id 913FC164F; Fri, 28 Aug 2015 09:35:23 -0700 (PDT) Message-ID: <55E08DCA.6000009@stankevitz.com> Date: Fri, 28 Aug 2015 09:35:22 -0700 From: Chris Stankevitz User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:31.0) Gecko/20100101 Thunderbird/31.7.0 MIME-Version: 1.0 To: David DeSimone , "freebsd-net@freebsd.org" Subject: Re: ssh over WAN: TCP window too small References: <55DCF080.7080208@stankevitz.com> <20150826010323.GN33167@funkthat.com> <55DD2A98.2010605@stankevitz.com> <20150826082457.GQ33167@funkthat.com> <55DE2FDF.5030707@stankevitz.com> <20150826223215.GS33167@funkthat.com> In-Reply-To: Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 28 Aug 2015 16:35:29 -0000 On 8/26/15 5:46 PM, David DeSimone wrote: > On 8/26/15 1:24 AM, John-Mark Gurney wrote: >>> 94146 ssh 6.686140 CALL read(0x4,0x7fffffff6c70,0x4000) >>> 94146 ssh 6.686154 GIO fd 4 read 4096 bytes >>> [ read of stdin (/dev/zero) snipped) >> >> It would be interesting to know how long from the read of stdin (and is >> it really reading stdin in 4k blocks? If so, that should be fixed) > > The read is making a call with 0x4000 = 16k buffer size, but it only receives 4k, probably because that is the max size of the pipe buffer. In that example I used `ssh < /dev/null`. I would have used 'dd if=/dev/zero bs=1m | ssh` but at the time I did not know about `foo | ktrace bar`. Although if there is such a thing named "pipe buffer" I'm not sure it would have made a difference... Chris From owner-freebsd-net@freebsd.org Fri Aug 28 16:35:42 2015 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 86B749C5BEF for ; Fri, 28 Aug 2015 16:35:42 +0000 (UTC) (envelope-from chris@stankevitz.com) Received: from mango.stankevitz.com (mango.stankevitz.com [208.79.93.194]) by mx1.freebsd.org (Postfix) with ESMTP id 740BE3A7 for ; Fri, 28 Aug 2015 16:35:42 +0000 (UTC) (envelope-from chris@stankevitz.com) Received: from Chriss-MacBook-Pro.local (209-203-101-124.static.twtelecom.net [209.203.101.124]) (using TLSv1.2 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by mango.stankevitz.com (Postfix) with ESMTPSA id 1D47E1653; Fri, 28 Aug 2015 09:35:42 -0700 (PDT) Message-ID: <55E08DDD.1040804@stankevitz.com> Date: Fri, 28 Aug 2015 09:35:41 -0700 From: Chris Stankevitz User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:31.0) Gecko/20100101 Thunderbird/31.7.0 MIME-Version: 1.0 To: John-Mark Gurney CC: freebsd-net@freebsd.org Subject: Re: ssh over WAN: TCP window too small References: <55DCF080.7080208@stankevitz.com> <20150826010323.GN33167@funkthat.com> <55DD2A98.2010605@stankevitz.com> <20150826082457.GQ33167@funkthat.com> <55DE2FDF.5030707@stankevitz.com> <20150826223215.GS33167@funkthat.com> In-Reply-To: <20150826223215.GS33167@funkthat.com> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 28 Aug 2015 16:35:42 -0000 John-Mark, I'm going to rearrange the conversation a bit in the quotes below: On 8/26/15 3:32 PM, John-Mark Gurney wrote: > So, I looked at what getsockopt SO_RCVBUF returns, and it returns: > case SO_RCVBUF: > optval = so->so_rcv.sb_hiwat; > > Which is NOT S-BMAX, so it looks like OpenSSH only ever gets 66052 bytes > for the buffer... I believe that explains everything we are seeing. HPN developers thought SO_RCVBUF would return S-BMAX but it instead returns S-HIWA. > If it's decided to base it's waiting for ack on SO_RCVBUF, then this > is probably where the issue is... > > Try setting HPNBufferSize to something resonably large, like 1MB, or > above your bandwidth-delay product to see if this will make a difference.. > Per: > Conditions: HPNBufferSize SET, TCPRcvBufPoll enabled, TCPRcvBuf NOT Set > Result: HPN Buffer Size = grows to HPNBufferSize > The buffer will grow up to the maximum size specified here. Setting HPNBufferSize to 1 MB does not change S-BCNT which remains stuck at ~60KB. Although I might not necessarily expect a change. From README.hpn: HPNBufferSize: This is the default buffer size the HPN functionality uses when interacting with non-HPN SSH installations. > It would be interesting to know how long from the read of stdin (and is > it really reading stdin in 4k blocks? If so, that should be fixed) > to the write out the socket... Basicly, how long does it take to encrypt > the data... > >> Note in the above the blocking call to select at time 6.592065 that >> takes ~0.1 second. This is the reason my connection is slow. The >> content appears to be encrypted... I presume it's an application-level >> ssh ack. I posted the in-context kdump at: https://www.stankevitz.com/ssh-ktrace.txt stdin is read in 4k blocks... but each "cycle" reads 3 of these blocks. I define as "cycle" as the period beginning and ending at a 0.1 second select(2) sleep. Looks like encryption of a 4KB block takes 10 microseconds, which means: for every 0.1 seconds in select(2), .000030 seconds are encrypting. >> net.inet.tcp.sendspace: 32768 >> net.inet.tcp.recvspace: 65536 > > Try increasing these and reporting back... the buf_auto should override > those, but this may be limiting it... Okay. Now we're getting somewhere. Increase sendspace on "sending client": no change Increase recvspace on "sending client": no change Increase sendspace on "receiving server": no change Increase recvspace on "receiving server": sending client's S-BCNT increases proportionally! I'm going to try Kurts parameters now... Chris From owner-freebsd-net@freebsd.org Fri Aug 28 18:42:05 2015 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 8295E9C5B6D for ; Fri, 28 Aug 2015 18:42:05 +0000 (UTC) (envelope-from david@catwhisker.org) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 646B51B92 for ; Fri, 28 Aug 2015 18:42:05 +0000 (UTC) (envelope-from david@catwhisker.org) Received: by mailman.ysv.freebsd.org (Postfix) id 633889C5B6A; Fri, 28 Aug 2015 18:42:05 +0000 (UTC) Delivered-To: net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 628729C5B69; Fri, 28 Aug 2015 18:42:05 +0000 (UTC) (envelope-from david@catwhisker.org) Received: from albert.catwhisker.org (mx.catwhisker.org [198.144.209.73]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id B08881B91; Fri, 28 Aug 2015 18:42:03 +0000 (UTC) (envelope-from david@catwhisker.org) Received: from albert.catwhisker.org (localhost [127.0.0.1]) by albert.catwhisker.org (8.15.2/8.15.2) with ESMTP id t7SIg1Qa072491; Fri, 28 Aug 2015 11:42:01 -0700 (PDT) (envelope-from david@albert.catwhisker.org) Received: (from david@localhost) by albert.catwhisker.org (8.15.2/8.15.2/Submit) id t7SIg0Xb072490; Fri, 28 Aug 2015 11:42:00 -0700 (PDT) (envelope-from david) Date: Fri, 28 Aug 2015 11:42:00 -0700 From: David Wolfskill To: stable@freebsd.org, net@freebsd.org, wireless@freebsd.org Subject: Re: Panic [page fault] in _ieee80211_crypto_delkey(): stable/10/amd64 @r286878 Message-ID: <20150828184200.GM1014@albert.catwhisker.org> Mail-Followup-To: David Wolfskill , stable@freebsd.org, net@freebsd.org, wireless@freebsd.org References: <20150818232007.GN1189@albert.catwhisker.org> <20150819160716.GK63584@albert.catwhisker.org> <20150819190852.GN63584@albert.catwhisker.org> <20150819192315.GO63584@albert.catwhisker.org> <20150819200124.GR63584@albert.catwhisker.org> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="nd30OFrAEJM3lM6x" Content-Disposition: inline In-Reply-To: <20150819200124.GR63584@albert.catwhisker.org> User-Agent: Mutt/1.5.23 (2014-03-12) X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 28 Aug 2015 18:42:05 -0000 --nd30OFrAEJM3lM6x Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Wed, Aug 19, 2015 at 01:01:24PM -0700, David Wolfskill wrote: > On Wed, Aug 19, 2015 at 12:25:38PM -0700, Adrian Chadd wrote: > > ... But we definitely ahe enough to put into a PR.. > > ... >=20 > Bug 202494 - Panic [page fault] in _ieee80211_crypto_delkey()=20 > > .... Caught another one (panic) -- this time, after having run "wlandebug +crypto". Above-cited PR updated to reflect updated status & dubugging info. Peace, david --=20 David H. Wolfskill david@catwhisker.org Those who would murder in the name of God or prophet are blasphemous coward= s. See http://www.catwhisker.org/~david/publickey.gpg for my public key. --nd30OFrAEJM3lM6x Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQJ8BAEBCgBmBQJV4Kt4XxSAAAAAAC4AKGlzc3Vlci1mcHJAbm90YXRpb25zLm9w ZW5wZ3AuZmlmdGhob3JzZW1hbi5uZXQ4RThEMDY4QTIxMjc1MDZFRDIzODYzRTc4 QTY3RjlDOERFRjQxOTNCAAoJEIpn+cje9Bk7BYoP/3y9gR+3528NK5VHdHyBzyOy Cik+C2WD9lIMfcQyC3FxxZBXnASm2Sk39zGLUAOntyexfz7UMyIqUqNLItBpi/R6 65GBj5SQLn+sbjUF6iA7vbCWgKndJtEsdTVPp2gWZIc++qfafklJKT94WJbqN/ng ub1ZpfIpAJfO97TPbhOA8aPN7pWCx/3RlC3njOYLYzTiJfUym9GTnJb2tSosgALs ul/RWScVFF6BdERTtHlibjJObYbj7F5uzAgWD2wBJi1RZMDbjCvzw/GiPG0eyHjs z3CM1GXF1aVs6fERvNJ+YKx452RQQWWCgQZSO9pthmxAZuB3PHHSIr3fkhhzv2H3 KWrYupc+MmVQ/V40PSv2ZqvqYYy23J2ApbdmQ5M4EXJr2sKMANCmcZXFk9G2wTTF eKWaGcYvjsFuUOOrha+EwEXB81Y5CveVfeof5v7dNIRdjt5HyN3jDNa7CaVvFepS xVioiwgiYMRYrAvvcYXR09u/bjLbEj4mDRVtOXvzNFcZsNXi75085r/UMG1V5gAg nmyVN551MRMzhg+OqhCmfCPALUZpBHN0edxQNwE9rnK3mkyfsSpgqe7bVjQLyYbo pFtUt/lVNP5eFlKIhjt0cyZllmo/olMqN7Pl0EoOTm6EKU78XxkM8xZkv3aNHMay SswvxIeF2MOsuNtRTVt9 =txc4 -----END PGP SIGNATURE----- --nd30OFrAEJM3lM6x-- From owner-freebsd-net@freebsd.org Fri Aug 28 19:14:27 2015 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 98C019C47C9 for ; Fri, 28 Aug 2015 19:14:27 +0000 (UTC) (envelope-from chris@stankevitz.com) Received: from mango.stankevitz.com (mango.stankevitz.com [208.79.93.194]) by mx1.freebsd.org (Postfix) with ESMTP id 85FB610EB for ; Fri, 28 Aug 2015 19:14:27 +0000 (UTC) (envelope-from chris@stankevitz.com) Received: from Chriss-MacBook-Pro.local (209-203-101-124.static.twtelecom.net [209.203.101.124]) (using TLSv1.2 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by mango.stankevitz.com (Postfix) with ESMTPSA id 5040B1677 for ; Fri, 28 Aug 2015 12:14:26 -0700 (PDT) Message-ID: <55E0B311.9070501@stankevitz.com> Date: Fri, 28 Aug 2015 12:14:25 -0700 From: Chris Stankevitz User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:31.0) Gecko/20100101 Thunderbird/31.7.0 MIME-Version: 1.0 To: freebsd-net@freebsd.org Subject: Re: ssh over WAN: TCP window too small References: <55DCF080.7080208@stankevitz.com> In-Reply-To: <55DCF080.7080208@stankevitz.com> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 28 Aug 2015 19:14:27 -0000 On 8/25/15 3:47 PM, Chris Stankevitz wrote: > Can anyone explain my abysmally small TCP window? So I believe this is the story: 1. openssh limits the size of some outgoing buffer to 65KB 2. openssh/HPN tries to improve on this by increasing the size of the outgoing buffer to match getsockopt(SO_SNDBUF) 3. When asked for the current SO_SNDBUF, FreeBSD 10.1 reports the high watermark of the outgoing buffer, not its capacity. 4. (2) is essentially a no-op because of (3). 5. openssh/HPN can be tricked into increasing its outgoing buffer by increasing sendspace/recvspace My comments: - (3) is not what I would expect -- perhaps the ssh/HPN folks would agree with me. Shouldn't getsockopt(SO_SNDBUF) return the same value set by setsockopt(SO_SNDBUF)? - I do not understand the mechanism by which (5) works. Chris From owner-freebsd-net@freebsd.org Fri Aug 28 19:14:32 2015 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id EEE4C9C47ED for ; Fri, 28 Aug 2015 19:14:32 +0000 (UTC) (envelope-from chris@stankevitz.com) Received: from mango.stankevitz.com (mango.stankevitz.com [208.79.93.194]) by mx1.freebsd.org (Postfix) with ESMTP id DF1801148 for ; Fri, 28 Aug 2015 19:14:32 +0000 (UTC) (envelope-from chris@stankevitz.com) Received: from Chriss-MacBook-Pro.local (209-203-101-124.static.twtelecom.net [209.203.101.124]) (using TLSv1.2 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by mango.stankevitz.com (Postfix) with ESMTPSA id CB7501678; Fri, 28 Aug 2015 12:14:32 -0700 (PDT) Message-ID: <55E0B318.2090709@stankevitz.com> Date: Fri, 28 Aug 2015 12:14:32 -0700 From: Chris Stankevitz User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:31.0) Gecko/20100101 Thunderbird/31.7.0 MIME-Version: 1.0 To: Kurt Lidl , freebsd-net@freebsd.org Subject: Re: ssh over WAN: TCP window too small References: <55DCF080.7080208@stankevitz.com> <55DF0D37.5060003@pix.net> In-Reply-To: <55DF0D37.5060003@pix.net> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 28 Aug 2015 19:14:33 -0000 On 8/27/15 6:14 AM, Kurt Lidl wrote: > # tcp options for long-haul speedups > kern.ipc.maxsockbuf=4194304 # (2 * default 2097152) > net.inet.tcp.mssdflt=1448 # (default 576) > net.inet.tcp.sendbuf_max=4194304 # (2 * default 2097152) > net.inet.tcp.recvbuf_max=4194304 # (2 * default 2097152) > > net.inet.tcp.syncache.rexmtlimit=1 # (default 3) > net.inet.tcp.recvspace=262144 # (4 * default 65,536) > net.inet.tcp.sendspace=262144 # (4 * default 65,536) > > net.inet.tcp.sendbuf_inc=65536 # (8 * default 8192) > net.inet.tcp.recvbuf_inc=131072 # (8 * default 16384) Kurt, Thank you. FYI the default for sendspace is 32768 (not 65536). With these parameters my S-BCNT increases from ~60K to ~200K when doing `ssh < /dev/zero`. Amusingly, something somewhere limits bytes-in-flight to 172,888 bytes (I believe the ssh sending client is limiting -- although I'm not sure why). This effectively caps my bandwidth to 20 Mbps. With iperf this limitation does not exist -- nor does the need to tune these values (except for buf_inc which can get the ball rolling faster as you pointed out). Chris From owner-freebsd-net@freebsd.org Fri Aug 28 19:58:07 2015 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 329F09C5664 for ; Fri, 28 Aug 2015 19:58:07 +0000 (UTC) (envelope-from bz@freebsd.org) Received: from mx1.sbone.de (mx1.sbone.de [IPv6:2a01:4f8:130:3ffc::401:25]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client CN "mx1.sbone.de", Issuer "SBone.DE" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id DC2BE86A; Fri, 28 Aug 2015 19:58:06 +0000 (UTC) (envelope-from bz@freebsd.org) Received: from mail.sbone.de (mail.sbone.de [IPv6:fde9:577b:c1a9:31::2013:587]) (using TLSv1 with cipher ADH-CAMELLIA256-SHA (256/256 bits)) (No client certificate requested) by mx1.sbone.de (Postfix) with ESMTPS id 9F53925D3A93; Fri, 28 Aug 2015 19:58:03 +0000 (UTC) Received: from content-filter.sbone.de (content-filter.sbone.de [IPv6:fde9:577b:c1a9:31::2013:2742]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mail.sbone.de (Postfix) with ESMTPS id EE5F6C770DC; Fri, 28 Aug 2015 19:58:02 +0000 (UTC) X-Virus-Scanned: amavisd-new at sbone.de Received: from mail.sbone.de ([IPv6:fde9:577b:c1a9:31::2013:587]) by content-filter.sbone.de (content-filter.sbone.de [fde9:577b:c1a9:31::2013:2742]) (amavisd-new, port 10024) with ESMTP id e-CvXkG6JikL; Fri, 28 Aug 2015 19:58:01 +0000 (UTC) Received: from [IPv6:fde9:577b:c1a9:4410:9d74:620c:1e79:eff] (unknown [IPv6:fde9:577b:c1a9:4410:9d74:620c:1e79:eff]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mail.sbone.de (Postfix) with ESMTPSA id A99FCC770DB; Fri, 28 Aug 2015 19:58:00 +0000 (UTC) From: Bjoern A. Zeeb Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Subject: UDP6 locking improvement Date: Fri, 28 Aug 2015 19:57:37 +0000 Message-Id: <2AFBBBDF-D57D-4887-9B5C-1363FFBBD417@freebsd.org> Cc: Adrian Chadd To: FreeBSD Net Mime-Version: 1.0 (Mac OS X Mail 8.2 \(2104\)) X-Mailer: Apple Mail (2.2104) X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 28 Aug 2015 19:58:07 -0000 Hi, based on some work I have done a few years back I have updated an UDP6 = locking change and it=E2=80=99s at the =E2=80=9Cit compiles=E2=80=9D = stage. I have not re-read it yet. I=E2=80=99ll have to split it up = into a couple of changes for PB as it also fixes: - some UDP-Lite bug(s) - some control mbuf leak - something else I forgot already and then re-post the rest into PB as well. The old description in p4 was like: ! MFp4 bz_ipv6_fast: ! ! Migrate udp6_send v4mapped code to udp6_output saving us a re-lock = and ! further simplifying the af handling code by eliminating AF_INET = checks ! at the tail end of the function. ! ! Rework output path locking similar to UDP4 allowing for better ! parallelism (see r222488). ! ! Cleanup early initializations, comments, =E2=80=A6 ! ! Sponsored by: The FreeBSD Foundation ! Sponsored by: iXsystems The new version is here: https://people.freebsd.org/~bz/20150828-01-udp6-locking.diff Anyone who wants to give it a read-through or bashing it through a pps = framework is welcome! I=E2=80=99ll try to do some of that hopefully = over the weekend myself. Bjoern=