From owner-freebsd-net@freebsd.org Sun Aug 9 18:37:30 2015 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id ED15F99D12F for ; Sun, 9 Aug 2015 18:37:30 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id D8ED11D73 for ; Sun, 9 Aug 2015 18:37:30 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id t79IbUUb009484 for ; Sun, 9 Aug 2015 18:37:30 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-net@FreeBSD.org Subject: [Bug 202038] [request] source address based routing without pf or ipfw Date: Sun, 09 Aug 2015 18:37:31 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 10.1-STABLE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: linimon@FreeBSD.org X-Bugzilla-Status: New X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-net@FreeBSD.org X-Bugzilla-Target-Milestone: --- X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: assigned_to short_desc Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 7bit X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 09 Aug 2015 18:37:31 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=202038 Mark Linimon changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|freebsd-bugs@FreeBSD.org |freebsd-net@FreeBSD.org Summary|source address based |[request] source address |routing without pf or ipfw |based routing without pf or | |ipfw -- You are receiving this mail because: You are the assignee for the bug. From owner-freebsd-net@freebsd.org Sun Aug 9 19:32:24 2015 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 66D4B99DF70 for ; Sun, 9 Aug 2015 19:32:24 +0000 (UTC) (envelope-from gavin@FreeBSD.org) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 5297A819 for ; Sun, 9 Aug 2015 19:32:24 +0000 (UTC) (envelope-from gavin@FreeBSD.org) Received: by mailman.ysv.freebsd.org (Postfix) id 4FF7599DF6D; Sun, 9 Aug 2015 19:32:24 +0000 (UTC) Delivered-To: net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 4E3C299DF69; Sun, 9 Aug 2015 19:32:24 +0000 (UTC) (envelope-from gavin@FreeBSD.org) Received: from mail-gw14.york.ac.uk (mail-gw14.york.ac.uk [144.32.129.164]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 16A07816; Sun, 9 Aug 2015 19:32:23 +0000 (UTC) (envelope-from gavin@FreeBSD.org) Received: from ury.york.ac.uk ([144.32.64.162]:18117) by mail-gw14.york.ac.uk with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.76) (envelope-from ) id 1ZOWKN-0000w4-Tz; Sun, 09 Aug 2015 20:32:15 +0100 Date: Sun, 9 Aug 2015 20:32:15 +0100 (BST) From: Gavin Atkinson X-X-Sender: gavin@ury.york.ac.uk To: Gleb Smirnoff cc: current@FreeBSD.org, net@FreeBSD.org Subject: Re: [head up!] WiFi drivers changes In-Reply-To: <20150806151355.GL889@FreeBSD.org> Message-ID: References: <20150806151355.GL889@FreeBSD.org> User-Agent: Alpine 2.20 (BSF 67 2015-01-07) MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 09 Aug 2015 19:32:24 -0000 On Thu, 6 Aug 2015, Gleb Smirnoff wrote: > As part of the "opaque ifnet project" [1], all 802.11 (WiFi) drivers > undergo change of not being an interface anymore. Historically in FreeBSD > 802.11 stack, 802.11 devices called if_attach() and created an interface. > Later this was generalized and real functioning interface is created by > net80211 stack. However, remnant of parent interface remained. If you > are running Intel Centrino wireless, then you got iwn0 interface and > wlan0 interface. However, the former doesn't do anything. You can't > assign addresses to it or modify any of it parameters. Or you can > modify them, but that affects nothing. You could, however, change the Ethernet address of the underlying interface before creating the wlanX interfaces, which woud then be used by the child interfaces. This has traditionally been the only way you could change the Ethernet interface of a wifi device - changing it after creating the wlanX interface does not work. How will this be done in the new world? Thanks, Gavin From owner-freebsd-net@freebsd.org Sun Aug 9 20:09:05 2015 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 3AA1E99D9EF for ; Sun, 9 Aug 2015 20:09:05 +0000 (UTC) (envelope-from glebius@FreeBSD.org) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 2235A949 for ; Sun, 9 Aug 2015 20:09:05 +0000 (UTC) (envelope-from glebius@FreeBSD.org) Received: by mailman.ysv.freebsd.org (Postfix) id 1ECD399D9EC; Sun, 9 Aug 2015 20:09:05 +0000 (UTC) Delivered-To: net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 1E42999D9EB; Sun, 9 Aug 2015 20:09:05 +0000 (UTC) (envelope-from glebius@FreeBSD.org) Received: from cell.glebius.int.ru (glebius.int.ru [81.19.69.10]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "cell.glebius.int.ru", Issuer "cell.glebius.int.ru" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id A106D946; Sun, 9 Aug 2015 20:09:03 +0000 (UTC) (envelope-from glebius@FreeBSD.org) Received: from cell.glebius.int.ru (localhost [127.0.0.1]) by cell.glebius.int.ru (8.15.2/8.15.2) with ESMTPS id t79K913w073050 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Sun, 9 Aug 2015 23:09:01 +0300 (MSK) (envelope-from glebius@FreeBSD.org) Received: (from glebius@localhost) by cell.glebius.int.ru (8.15.2/8.15.2/Submit) id t79K918S073049; Sun, 9 Aug 2015 23:09:01 +0300 (MSK) (envelope-from glebius@FreeBSD.org) X-Authentication-Warning: cell.glebius.int.ru: glebius set sender to glebius@FreeBSD.org using -f Date: Sun, 9 Aug 2015 23:09:01 +0300 From: Gleb Smirnoff To: Gavin Atkinson Cc: current@FreeBSD.org, net@FreeBSD.org Subject: Re: [head up!] WiFi drivers changes Message-ID: <20150809200901.GC889@glebius.int.ru> References: <20150806151355.GL889@FreeBSD.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.23 (2014-03-12) X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 09 Aug 2015 20:09:05 -0000 On Sun, Aug 09, 2015 at 08:32:15PM +0100, Gavin Atkinson wrote: G> On Thu, 6 Aug 2015, Gleb Smirnoff wrote: G> > As part of the "opaque ifnet project" [1], all 802.11 (WiFi) drivers G> > undergo change of not being an interface anymore. Historically in FreeBSD G> > 802.11 stack, 802.11 devices called if_attach() and created an interface. G> > Later this was generalized and real functioning interface is created by G> > net80211 stack. However, remnant of parent interface remained. If you G> > are running Intel Centrino wireless, then you got iwn0 interface and G> > wlan0 interface. However, the former doesn't do anything. You can't G> > assign addresses to it or modify any of it parameters. Or you can G> > modify them, but that affects nothing. G> G> You could, however, change the Ethernet address of the underlying G> interface before creating the wlanX interfaces, which woud then be used by G> the child interfaces. This has traditionally been the only way you could G> change the Ethernet interface of a wifi device - changing it after G> creating the wlanX interface does not work. G> G> How will this be done in the new world? In new world you can change it on wlanX. I've tested on iwn(4), it worked for me, although association went not so quickly as before. -- Totus tuus, Glebius. From owner-freebsd-net@freebsd.org Sun Aug 9 21:00:25 2015 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id D238F99D9EF for ; Sun, 9 Aug 2015 21:00:25 +0000 (UTC) (envelope-from bugzilla-noreply@FreeBSD.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id AE9261F8A for ; Sun, 9 Aug 2015 21:00:25 +0000 (UTC) (envelope-from bugzilla-noreply@FreeBSD.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id t79L0PmN037108 for ; Sun, 9 Aug 2015 21:00:25 GMT (envelope-from bugzilla-noreply@FreeBSD.org) Message-Id: <201508092100.t79L0PmN037108@kenobi.freebsd.org> From: bugzilla-noreply@FreeBSD.org To: freebsd-net@FreeBSD.org Subject: Problem reports for freebsd-net@FreeBSD.org that need special attention X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 Date: Sun, 09 Aug 2015 21:00:25 +0000 Content-Type: text/plain; charset="UTF-8" X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 09 Aug 2015 21:00:25 -0000 To view an individual PR, use: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=(Bug Id). The following is a listing of current problems submitted by FreeBSD users, which need special attention. These represent problem reports covering all versions including experimental development code and obsolete releases. Status | Bug Id | Description ------------+-----------+--------------------------------------------------- Open | 194515 | Fatal Trap 12 Kernel with vimage Open | 199136 | [if_tap] Added down_on_close sysctl variable to t 2 problems total for which you should take action. From owner-freebsd-net@freebsd.org Mon Aug 10 13:33:13 2015 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 8299099DE90 for ; Mon, 10 Aug 2015 13:33:13 +0000 (UTC) (envelope-from barney_cordoba@yahoo.com) Received: from nm37-vm8.bullet.mail.ne1.yahoo.com (nm37-vm8.bullet.mail.ne1.yahoo.com [98.138.229.136]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4AD959A1 for ; Mon, 10 Aug 2015 13:33:13 +0000 (UTC) (envelope-from barney_cordoba@yahoo.com) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1439213263; bh=5EoiWtAzogjuSETwFbf3qQajWVK2Nu5bGZNi0rM4jOE=; h=Date:From:Reply-To:To:Cc:In-Reply-To:References:Subject:From:Subject; b=aTZDq4/eN57T3AGY9Lzg/jSvfd1NiL3wEgBL3TENKJ+XlmjvZuj8DTm+IHIdmO9m8WtJfodRK4SF8IDTR+LNa8BCIn2nF6h8oKPSnlKU1BxBtMBS0K3xyFvB8w1YhSgC8IhIFWoPtk5orGKmhZTs95v4jVqcoYC35rkO8XH0tqKuvrDPqd8h6ChqIBL2EmBvDfkTEZrxxh7G+2JVMyAtUIdeyxwqYSQA/ytM3GKhMcPu9YAZZXDZeeq06R10rjvmcBIKVBvwKnediHbk6g1YOTksjc0xHiANeY0EGJHRfbEljfe3Qm4Cz7Yus8dVjffW0akL2ssVkGo92N2IymfI3Q== Received: from [127.0.0.1] by nm37.bullet.mail.ne1.yahoo.com with NNFMP; 10 Aug 2015 13:27:43 -0000 Received: from [98.138.226.177] by nm37.bullet.mail.ne1.yahoo.com with NNFMP; 10 Aug 2015 13:24:43 -0000 Received: from [98.138.87.7] by tm12.bullet.mail.ne1.yahoo.com with NNFMP; 10 Aug 2015 13:24:43 -0000 Received: from [127.0.0.1] by omp1007.mail.ne1.yahoo.com with NNFMP; 10 Aug 2015 13:24:43 -0000 X-Yahoo-Newman-Property: ymail-4 X-Yahoo-Newman-Id: 538996.83387.bm@omp1007.mail.ne1.yahoo.com X-YMail-OSG: 4nWZcPUVM1m9pVZxG5g3YJSX0w0RqHafGEsMlfyHiLZQYhUn4TkjFlrPhWtCMGa 4IXx.0Z8cbYDlBNChZ_HbjXAHw6b9yGM2wZEYXv5yWqy47JDtbAVpAECfvWOoJWaHoTm.87rMFSE xnV6TgLBT3YEYvLKrgist7OW4kq7kinZD4A3wpjpYXJ6Kk5YeEh0mNVSco9npCeRhn2OeuW8gowP cN9auYl4FEprWzqFrSUg8UNcDAZwpYHOV4rRqjTHnxTqJM4dvcAuCVhcIGt8lm0EspkH74t3yX1k PXAXMPqnGWRGbgIoDwhOFdnJgtXv.GdCFoZQqStlODo3glHoogW95VrYppMa5qjxR1vMuLP6GD4q pusdZTUwgw5ELWzfmm1MqF58_nxjBOLqJvP.IYiClGnKeorSfrVawJAyHzku8rBjLnKrE4xyeBsT ibdoq1lNTxi9Fw4LkJAziu6F_MNXCbleeiyQEZZzFM4E2zAzE8JyxuwkAY6SGBVPqfBg5HYQ.Z7u sl5ZuWLoI5wHQAKrTrFj_FDkF Received: by 98.138.101.183; Mon, 10 Aug 2015 13:24:43 +0000 Date: Mon, 10 Aug 2015 13:24:36 +0000 (UTC) From: Barney Cordoba Reply-To: Barney Cordoba To: Kevin Oberman Cc: =?UTF-8?Q?Olivier_Cochard-Labb=C3=A9?= , FreeBSD Net , Adrian Chadd , Eric Joyner , hiren panchasara Message-ID: <1159228245.2025586.1439213076880.JavaMail.yahoo@mail.yahoo.com> In-Reply-To: References: Subject: Re: Exposing full 32bit RSS hash from card for ixgbe(4) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 10 Aug 2015 13:33:13 -0000 On Wednesday, August 5, 2015 4:28 PM, Kevin Oberman wrote: =20 On Wed, Aug 5, 2015 at 7:10 AM, Barney Cordoba via freebsd-net < freebsd-net@freebsd.org> wrote: > > > >=C2=A0 =C2=A0 =C2=A0 On Wednesday, August 5, 2015 2:19 AM, Olivier Cochard= -Labb=C3=A9 < > olivier@cochard.me> wrote: > > >=C2=A0 On Wed, Aug 5, 2015 at 1:15 AM, Barney Cordoba via freebsd-net < > freebsd-net@freebsd.org> wrote: > > > What's the point of all of this gobbledygook anyway? Seriously, 99% of > the > > world needs a driver that passes packets in the most efficient way, and > > every time I look at igb and ixgbe it has another 2 heads. It's up to 8 > > heads, and none of the things wrong with it have been fixed. This is no= w > > even uglier than Kip Macy's cxgb abortion. > > I'm not trying to be snarky here. I wrote a simple driver 3 years ago > that > > runs and runs and uses little cpu; maybe 8% for a full gig load on an E= 3. > > > > =E2=80=8BHi, > > I will be very happy to bench your simple driver. Where can I download th= e > sources ? > > Thanks, > > Olivier > _______________________________________________ > > Another unproductive dick head on the FreeBSD team? Figures. > A typical Barney thread. First he calls the developers incompetent and says he has done better. Then someone who has experience in real world benchmarking (not a trivial thing) offers to evaluate Barney's code, and gets a quick, rude, obscene dismissal. Is it any wonder that, even though he made some valid arguments (at least for some workloads), almost everyone just dismisses him as too obnoxious to try to deal with. Based on my pre-retirement work with high-performance networking, in some cases it was clear that it would be better to locking down things to a single CPU on with FreeBSD or Linux. I can further state that this was NOT true for all workloads, so it is quite possible that Barney's code works for some cases (perhaps his) and would be bad in others. But without good benchmarking, it's hard to tell. I will say that for large volume data transfers (very large flows), a single CPU solution does work best. But if Barney is going at this with his usual attitude, it's probably=C2=A0 not worth it to continue the discussion= . -- the "give us the source and we'll test it" nonsense is kindergarden stuff. = As if my code is open source and you can just have it, and like you know ho= w to benchmark anything since you can't even benchmark what you have.=C2=A0 Some advice is to ignore guys like Oberman who spent their lives randomly p= ounding networks on slow machines with slow busses and bad NICs on OS's tha= t couldn't do SMP properly. Because he'll just lead you down the road to du= sty death. Multicore design isn't simple math; its about efficiency, lock m= inimization and the understanding that shifting memory between cpus unneces= sarily is costly. Today's=C2=A0CPUs and NICs can't be judged using test met= hods of the past. You'll just end up playing the Microsoft Windows game; ge= t bigger machines and more memory and don't worry about the fact that the c= ode is junk. It's just that the "default" in these drivers is so obviously wrong that it= 's mind-boggling. The argument to use 1, 2 or 4 queues is one worth having;= using "all" of the cpus, including the hyperthreads, is just plain incompe= tent. I will contribute one possibly useful tidbit: disable_queue() only disables receive interrupts. Both tx and rx ints are e= ffectively tied together by moderation so you'll just getan interrupt at th= e next slot anyway. BC From owner-freebsd-net@freebsd.org Tue Aug 11 08:09:31 2015 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id E988599F253 for ; Tue, 11 Aug 2015 08:09:31 +0000 (UTC) (envelope-from lakshmi.n@msystechnologies.com) Received: from mail-pa0-f44.google.com (mail-pa0-f44.google.com [209.85.220.44]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id C0EC9765 for ; Tue, 11 Aug 2015 08:09:31 +0000 (UTC) (envelope-from lakshmi.n@msystechnologies.com) Received: by pacgr6 with SMTP id gr6so45618080pac.2 for ; Tue, 11 Aug 2015 01:09:25 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:message-id:mime-version:from:to:cc:subject :importance:date:in-reply-to:references:content-type; bh=umVmpRHXxcwbHEOMcOR9MKV8RyNrA+j71kEW8TL3yw8=; b=iDkKz1dpHw/Ed/Jpx8k56jsy00DJ5y4UKyBQazXk8rYAUFe/UsEi2pkPRa8rHgWMEm kt0jcloSrCRixiBlaWUegEfBQ+cO+pc9PEGlgt2Kq+VSG1L/62d25jsUtc1+bGErs6zJ qKE+rdFAcFgaXRntJD4LEc6+DvWEQweIGoCq/eG0cCOFz1A5SUImY0yvIRpVwCrUmyhE 2+pgqrUr2H79Vz2zOFifRgg5Js2p4jcFB0zSzJmVGqf1X7hPwCjGqeLUaPj2yzs1JYe1 SrkGRhVUff6tVoTOwxjgg9BWeZ2wsur5JOIHivTy+/ckcDpsFA84YwnIULthmL66f3DJ /G2g== X-Gm-Message-State: ALoCoQlC4QounSjgsEVUGVMliSlBg/X69WUxWP4vqxccR4k04ipWPPxW0/IUYgDxgdu6tEnmfc7g X-Received: by 10.66.122.207 with SMTP id lu15mr53768222pab.146.1439280565079; Tue, 11 Aug 2015 01:09:25 -0700 (PDT) Received: from GandivaPC ([113.193.28.226]) by smtp.gmail.com with ESMTPSA id ov3sm1557739pdb.2.2015.08.11.01.09.22 (version=TLSv1 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Tue, 11 Aug 2015 01:09:24 -0700 (PDT) Message-ID: <55c9adb4.a384460a.8fa38.4179@mx.google.com> MIME-Version: 1.0 From: =?utf-8?Q?Lakshmi_Narasimhan_Sundararajan?= To: "=?utf-8?Q?freebsd-net@freebsd.org?=" CC: "=?utf-8?Q?Lewis,_Fred?=" , "=?utf-8?Q?Pokala,_Ravi?=" , "=?utf-8?Q?panasas-network@msystechnologies.com?=" Subject: =?utf-8?Q?Fw:_D3300:_LAG_LACP_timeout_tunable_through_IOCTL?= Importance: Normal Date: Tue, 11 Aug 2015 08:04:37 +0000 In-Reply-To: References: , Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 11 Aug 2015 08:09:32 -0000 SGkgRnJlZUJTRCBUZWFtLA0KDQpXZSBoYXZlIGJlZW4gd29ya2luZyBvbiBMQUNQIHRpbWVvdXQg dHVuYWJsZSB0byBoZWxwIGV4cGVkaXRlIGxpbmsgZmFpbHVyZSBwcm9wYWdhdGlvbiB0aHJvdWdo IGEgSU9DdGwgaW50ZXJmYWNlLg0KDQoNCg0KVGhlIGNoYW5nZXMgaGF2ZSBiZWVuIHRlc3RlZCBh bmQgcmV2aWV3IGlzIGluIHByb2dyZXNzLiANCg0KVGhlIHBoYWJyaWNhdG9yIGxpbmsgaXMgaHR0 cHM6Ly9yZXZpZXdzLmZyZWVic2Qub3JnL0QzMzAwDQoNCldlIHdvdWxkIGtpbmRseSBhcHByZWNp YXRlIGlmIHRoZSBjaGFuZ2VzIGNhbiBiZSBpbnRlZ3JhdGVkIHRvIENVUlJFTlQgYXQgeW91ciBl YXJsaWVzdC4NCg0KDQpUaGFua3MsDQoNCkxODQoNCg0KDQpNU1lTIFRlY2hub2xvZ2llcw0KDQoN Cg0KDQoNCkZyb206IGxha3NobWkubl9tc3lzdGVjaG5vbG9naWVzLmNvbSAoTE4pDQpTZW50OiDi gI5Nb25kYXnigI4sIOKAjkF1Z3VzdOKAjiDigI4xMOKAjiwg4oCOMjAxNSDigI4y4oCOOuKAjjUy 4oCOIOKAjlBNDQpUbzogbGFrc2htaS5uQG1zeXN0ZWNobm9sb2dpZXMuY29tDQoNCg0KDQoNCg0K bGFrc2htaS5uX21zeXN0ZWNobm9sb2dpZXMuY29tIG1hcmtlZCAyIGlubGluZSBjb21tZW50cyBh cyBkb25lLg0KbGFrc2htaS5uX21zeXN0ZWNobm9sb2dpZXMuY29tIGFkZGVkIGEgY29tbWVudC4N Cg0KV2FycmVuLCBJIGhhdmUgYWRkcmVzc2VkIHlvdXIgY29tbWVudHMuIFBsZWFzZSBsZXQgbWUg a25vdyBob3cgSSBjYW4gc3BlZWQgdXAgdGhlIHByb2Nlc3MgdG8gZ2V0IHRoaXMgaW50ZWdyYXRl ZCB0byBtYWlubGluZS4NCg0KDQpSRVZJU0lPTiBERVRBSUwNCiAgaHR0cHM6Ly9yZXZpZXdzLmZy ZWVic2Qub3JnL0QzMzAwDQoNCkVNQUlMIFBSRUZFUkVOQ0VTDQogIGh0dHBzOi8vcmV2aWV3cy5m cmVlYnNkLm9yZy9zZXR0aW5ncy9wYW5lbC9lbWFpbHByZWZlcmVuY2VzLw0KDQpUbzogbGFrc2ht aS5uX21zeXN0ZWNobm9sb2dpZXMuY29tLCBtYW5wYWdlcywgcnBva2FsYS1wYW5hc2FzLmNvbQ0K Q2M6IHdibG9jaywgaW1w From owner-freebsd-net@freebsd.org Tue Aug 11 16:17:22 2015 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 05EF899FECA for ; Tue, 11 Aug 2015 16:17:22 +0000 (UTC) (envelope-from hiren@strugglingcoder.info) Received: from mail.strugglingcoder.info (strugglingcoder.info [65.19.130.35]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id E5B7A26C for ; Tue, 11 Aug 2015 16:17:21 +0000 (UTC) (envelope-from hiren@strugglingcoder.info) Received: from localhost (unknown [10.1.1.3]) (Authenticated sender: hiren@strugglingcoder.info) by mail.strugglingcoder.info (Postfix) with ESMTPSA id 015FB105A04; Tue, 11 Aug 2015 09:17:14 -0700 (PDT) Date: Tue, 11 Aug 2015 09:17:14 -0700 From: hiren panchasara To: Lakshmi Narasimhan Sundararajan Cc: "freebsd-net@freebsd.org" , "Pokala, Ravi" , "Lewis, Fred" , "panasas-network@msystechnologies.com" Subject: Re: Fw: D3300: LAG LACP timeout tunable through IOCTL Message-ID: <20150811161714.GK34072@strugglingcoder.info> References: <55c9adb4.a384460a.8fa38.4179@mx.google.com> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="FCF9ydJxlAJsfRTu" Content-Disposition: inline In-Reply-To: <55c9adb4.a384460a.8fa38.4179@mx.google.com> User-Agent: Mutt/1.5.23 (2014-03-12) X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 11 Aug 2015 16:17:22 -0000 --FCF9ydJxlAJsfRTu Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On 08/11/15 at 08:04P, Lakshmi Narasimhan Sundararajan wrote: > Hi FreeBSD Team, >=20 > We have been working on LACP timeout tunable to help expedite link failur= e propagation through a IOCtl interface. >=20 >=20 >=20 > The changes have been tested and review is in progress.=20 >=20 > The phabricator link is https://reviews.freebsd.org/D3300 Reviewed/approved on phabric. If none objects, I'll commit tomorrow. Cheers, Hiren --FCF9ydJxlAJsfRTu Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (FreeBSD) iQF8BAEBCgBmBQJVyiAKXxSAAAAAAC4AKGlzc3Vlci1mcHJAbm90YXRpb25zLm9w ZW5wZ3AuZmlmdGhob3JzZW1hbi5uZXRBNEUyMEZBMUQ4Nzg4RjNGMTdFNjZGMDI4 QjkyNTBFMTU2M0VERkU1AAoJEIuSUOFWPt/l+YwH/Ar1uOvTG6NQW/aqjWGjhg7y xFwO6haaM/8hQdJltTaTOFpDmwzgbV2cIx9saQ4o61KGnVqaHAcN71CSbOZnAeDb JqyNCD/Wh3GsL/NfEAtrC6fO9jGoKZBW8SafI7baUiC15kaXRLfPXIhCQHOZR9Di NY5pQdJYm2i19aZMMrrkj69OVoo5+4CS7dcpx4We4ti5tDd4RlRnml1C3bwFPoH5 b+PXWtP8wxUIkAo0lnXLSg/jvjyWflVwBBIGvTFj3H4esaZ4UCEG88olclvYNt7R FxHdv9FOwNYKRB6vEEJvbDhevbizTRxkkOcj3diwdF1chtAD8/WZHt+b4ArUdjA= =FGcn -----END PGP SIGNATURE----- --FCF9ydJxlAJsfRTu-- From owner-freebsd-net@freebsd.org Tue Aug 11 21:18:28 2015 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 296EF99F4EE for ; Tue, 11 Aug 2015 21:18:28 +0000 (UTC) (envelope-from sobomax@sippysoft.com) Received: from mail-wi0-f180.google.com (mail-wi0-f180.google.com [209.85.212.180]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id BEBFC75D for ; Tue, 11 Aug 2015 21:18:27 +0000 (UTC) (envelope-from sobomax@sippysoft.com) Received: by wicja10 with SMTP id ja10so90440321wic.1 for ; Tue, 11 Aug 2015 14:18:20 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:sender:date:message-id:subject:from :to:cc:content-type; bh=kkry+rP7kmNzgmCcCR7YBt9YOOvw520ZWisdlFGvHyA=; b=BKiKKocFQ+4UL5tkdehDYZlpgHL6hrjMOua8XD8HHeWTKFwta+ozwUQCEsnlEqAb/0 HApjSEwLeV93ft73VxpAkyndTUvvvVmCeYJJuQtk0TR7EXvb39oY4XS29y8CllkuQf0k 6NqLNgGUhE9/TkTMq3rjsT+jHVatf2AUjAIqM01ckJJKhDYLqxgv8ZPeENVtZKqU9xCm 4NDyVNZWfI0uRMCVkstw/eWTojlEJMBtsdryqMk4GOQswwvwtp+KpqJBxZ6OSrmTYjuY hroBD1DsLaXtFPwj9YBjqaWYkLFf0y+noIKLxWBz5k51BZbv9nugTYxmQWn3DUelivrK fUVA== X-Gm-Message-State: ALoCoQnWBb60cVobrxlCDaORM2LXsB5gNfnsIBFSiACRPt3xKgmMYDEty5Va5Lqvuk5upMZUY6Pm MIME-Version: 1.0 X-Received: by 10.194.78.164 with SMTP id c4mr57929916wjx.65.1439327899891; Tue, 11 Aug 2015 14:18:19 -0700 (PDT) Sender: sobomax@sippysoft.com Received: by 10.27.143.15 with HTTP; Tue, 11 Aug 2015 14:18:19 -0700 (PDT) Date: Tue, 11 Aug 2015 14:18:19 -0700 X-Google-Sender-Auth: R71as7e1Cojoyot60-kr12bxWwE Message-ID: Subject: Poor high-PPS performance of the 10G ixgbe(9) NIC/driver in FreeBSD 10.1 From: Maxim Sobolev To: FreeBSD Net , freebsd@intel.com Cc: =?UTF-8?Q?Jev_Bj=C3=B6rsell?= Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 11 Aug 2015 21:18:28 -0000 Hi folks, We've trying to migrate some of our high-PPS systems to a new hardware that has four X540-AT2 10G NICs and observed that interrupt time goes through roof after we cross around 200K PPS in and 200K out (two ports in LACP). The previous hardware was stable up to about 350K PPS in and 350K out. I believe the old one was equipped with the I350 and had the identical LACP configuration. The new box also has better CPU with more cores (i.e. 24 cores vs. 16 cores before). CPU itself is 2 x E5-2690 v3. After hitting this limit with the default settings, I've tried to tweak the following settings: hw.ix.rx_process_limit="-1" hw.ix.tx_process_limit="-1" hw.ix.enable_aim="0" hw.ix.max_interrupt_rate="-1" hw.ix.rxd="4096" hw.ix.txd="4096" dev.ix.0.fc=0 dev.ix.1.fc=0 dev.ix.2.fc=0 dev.ix.3.fc=0 hw.intr_storm_threshold=0 But there is little or no effect on the performance. The workload is just lot of small UDP packets being relayed between bunch of hosts. The symptoms are always the same - the box runs nice and cool until it his the said PPS threshold, with kernel spending just few percent in the interrupts and then it jumps straight to 100% interrupt time, thereby scaring some traffic away due to packet loss and such, so that the load drops and the system goes into the "cool" state again. It looks very much like some contention in the driver or in the hardware. Linked are some monitoring screenshots displaying the issue unfolding as well as systat -vm screenshots from the "cool" state. http://sobomax.sippysoft.com/ScreenShot387.png <- CPU utilization right before the "bang event" http://sobomax.sippysoft.com/ScreenShot382.png <- issue itself http://sobomax.sippysoft.com/ScreenShot385.png <- systat -vm few minutes after traffic declined somewhat We are now trying to get customer install 1Gig NIC so that we can run it and compare performance with the rest of the hardware and software being essentially the same. Any ideas on how to improve/resolve this problem are welcome. Thanks! From owner-freebsd-net@freebsd.org Tue Aug 11 22:01:43 2015 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 4CEA899FFE3 for ; Tue, 11 Aug 2015 22:01:43 +0000 (UTC) (envelope-from adrian.chadd@gmail.com) Received: from mail-ig0-x231.google.com (mail-ig0-x231.google.com [IPv6:2607:f8b0:4001:c05::231]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 1A8BEFDB; Tue, 11 Aug 2015 22:01:43 +0000 (UTC) (envelope-from adrian.chadd@gmail.com) Received: by igfj19 with SMTP id j19so1471089igf.1; Tue, 11 Aug 2015 15:01:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=E35GSXuP7eghh1Uf9uGqxo6RxRqin4LqJWzkfzXJBKE=; b=J2fuj6RjqWehFU/8Bnx5R0DXpxDe5TRSk055495XhvNiZ/z2SKJ5vO7ezCHxdDAp86 czbCKOiZMz5K/cAqPJI+Ymafd/b5hKEBGNgQvnsQw8ezT/4tqp7naMyLWL1jPM/x2yt7 TblvnGUOv65nOc+3R2UTJ9ANL1qBtowoZLKalRg+MClI2i8gw125R2BqCBmkwZSpaeiB 7krNH7jIGAmitcsYIfmMZwRVsy4G1Q381aHY/xNJHwPOUVShWtcsE74HrcS/JI9ae99X V9K0ZQdgNzqNN1bF4BtiiwIKzfnTgRAZam0XvfEo3C0GsPuQWx6/aKgVOOclaSjS5ypI ikaw== MIME-Version: 1.0 X-Received: by 10.50.61.144 with SMTP id p16mr19490443igr.22.1439330502634; Tue, 11 Aug 2015 15:01:42 -0700 (PDT) Received: by 10.36.38.133 with HTTP; Tue, 11 Aug 2015 15:01:42 -0700 (PDT) In-Reply-To: References: Date: Tue, 11 Aug 2015 15:01:42 -0700 Message-ID: Subject: Re: Poor high-PPS performance of the 10G ixgbe(9) NIC/driver in FreeBSD 10.1 From: Adrian Chadd To: Maxim Sobolev Cc: FreeBSD Net , freebsd@intel.com, =?UTF-8?Q?Jev_Bj=C3=B6rsell?= Content-Type: text/plain; charset=UTF-8 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 11 Aug 2015 22:01:43 -0000 hi, Are you able to graph per-queue interrupt rates? It looks like the traffic is distributed differently (the first two queues are taking interrupts). Does 10.1 have the flow director code disabled? I remember there was some .. interesting behaviour with ixgbe where it'd look at traffic and set up flow director rules to try and "balance" things. It was buggy and programmed the hardware badly, so we disabled it in at least -HEAD. -adrian On 11 August 2015 at 14:18, Maxim Sobolev wrote: > Hi folks, > > We've trying to migrate some of our high-PPS systems to a new hardware that > has four X540-AT2 10G NICs and observed that interrupt time goes through > roof after we cross around 200K PPS in and 200K out (two ports in LACP). > The previous hardware was stable up to about 350K PPS in and 350K out. I > believe the old one was equipped with the I350 and had the identical LACP > configuration. The new box also has better CPU with more cores (i.e. 24 > cores vs. 16 cores before). CPU itself is 2 x E5-2690 v3. > > After hitting this limit with the default settings, I've tried to tweak the > following settings: > > hw.ix.rx_process_limit="-1" > hw.ix.tx_process_limit="-1" > hw.ix.enable_aim="0" > hw.ix.max_interrupt_rate="-1" > hw.ix.rxd="4096" > hw.ix.txd="4096" > > dev.ix.0.fc=0 > dev.ix.1.fc=0 > dev.ix.2.fc=0 > dev.ix.3.fc=0 > > hw.intr_storm_threshold=0 > > But there is little or no effect on the performance. The workload is just > lot of small UDP packets being relayed between bunch of hosts. The symptoms > are always the same - the box runs nice and cool until it his the said PPS > threshold, with kernel spending just few percent in the interrupts and then > it jumps straight to 100% interrupt time, thereby scaring some traffic away > due to packet loss and such, so that the load drops and the system goes > into the "cool" state again. It looks very much like some contention in the > driver or in the hardware. Linked are some monitoring screenshots > displaying the issue unfolding as well as systat -vm screenshots from the > "cool" state. > > http://sobomax.sippysoft.com/ScreenShot387.png <- CPU utilization right > before the "bang event" > http://sobomax.sippysoft.com/ScreenShot382.png <- issue itself > http://sobomax.sippysoft.com/ScreenShot385.png <- systat -vm few minutes > after traffic declined somewhat > > We are now trying to get customer install 1Gig NIC so that we can run it > and compare performance with the rest of the hardware and software being > essentially the same. > > Any ideas on how to improve/resolve this problem are welcome. Thanks! > _______________________________________________ > freebsd-net@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" From owner-freebsd-net@freebsd.org Tue Aug 11 22:16:28 2015 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 16EAB99F31F for ; Tue, 11 Aug 2015 22:16:28 +0000 (UTC) (envelope-from hiren@strugglingcoder.info) Received: from mail.strugglingcoder.info (strugglingcoder.info [65.19.130.35]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 00D74B7A; Tue, 11 Aug 2015 22:16:27 +0000 (UTC) (envelope-from hiren@strugglingcoder.info) Received: from localhost (unknown [10.1.1.3]) (Authenticated sender: hiren@strugglingcoder.info) by mail.strugglingcoder.info (Postfix) with ESMTPSA id D1C85E688; Tue, 11 Aug 2015 15:16:26 -0700 (PDT) Date: Tue, 11 Aug 2015 15:16:26 -0700 From: hiren panchasara To: Adrian Chadd Cc: Maxim Sobolev , FreeBSD Net , freebsd@intel.com, Jev Bj?rsell Subject: Re: Poor high-PPS performance of the 10G ixgbe(9) NIC/driver in FreeBSD 10.1 Message-ID: <20150811221626.GE96509@strugglingcoder.info> References: MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="k3qmt+ucFURmlhDS" Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.23 (2014-03-12) X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 11 Aug 2015 22:16:28 -0000 --k3qmt+ucFURmlhDS Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On 08/11/15 at 03:01P, Adrian Chadd wrote: > hi, >=20 > Are you able to graph per-queue interrupt rates? >=20 > It looks like the traffic is distributed differently (the first two > queues are taking interrupts). Yeah, also check out "# sysctl dev.ix | grep packets" >=20 > Does 10.1 have the flow director code disabled? I remember there was > some .. interesting behaviour with ixgbe where it'd look at traffic > and set up flow director rules to try and "balance" things. It was > buggy and programmed the hardware badly, so we disabled it in at least > -HEAD. Looks like we don't build with IXGBE_FDIR by default on 10 so I assume it's off. There were some lagg/hashing related changes recently so let us know if that is hurting you. Cheers, Hiren >=20 >=20 >=20 > -adrian >=20 >=20 > On 11 August 2015 at 14:18, Maxim Sobolev wrote: > > Hi folks, > > > > We've trying to migrate some of our high-PPS systems to a new hardware = that > > has four X540-AT2 10G NICs and observed that interrupt time goes through > > roof after we cross around 200K PPS in and 200K out (two ports in LACP). > > The previous hardware was stable up to about 350K PPS in and 350K out. I > > believe the old one was equipped with the I350 and had the identical LA= CP > > configuration. The new box also has better CPU with more cores (i.e. 24 > > cores vs. 16 cores before). CPU itself is 2 x E5-2690 v3. > > > > After hitting this limit with the default settings, I've tried to tweak= the > > following settings: > > > > hw.ix.rx_process_limit=3D"-1" > > hw.ix.tx_process_limit=3D"-1" > > hw.ix.enable_aim=3D"0" > > hw.ix.max_interrupt_rate=3D"-1" > > hw.ix.rxd=3D"4096" > > hw.ix.txd=3D"4096" > > > > dev.ix.0.fc=3D0 > > dev.ix.1.fc=3D0 > > dev.ix.2.fc=3D0 > > dev.ix.3.fc=3D0 > > > > hw.intr_storm_threshold=3D0 > > > > But there is little or no effect on the performance. The workload is ju= st > > lot of small UDP packets being relayed between bunch of hosts. The symp= toms > > are always the same - the box runs nice and cool until it his the said = PPS > > threshold, with kernel spending just few percent in the interrupts and = then > > it jumps straight to 100% interrupt time, thereby scaring some traffic = away > > due to packet loss and such, so that the load drops and the system goes > > into the "cool" state again. It looks very much like some contention in= the > > driver or in the hardware. Linked are some monitoring screenshots > > displaying the issue unfolding as well as systat -vm screenshots from t= he > > "cool" state. > > > > http://sobomax.sippysoft.com/ScreenShot387.png <- CPU utilization right > > before the "bang event" > > http://sobomax.sippysoft.com/ScreenShot382.png <- issue itself > > http://sobomax.sippysoft.com/ScreenShot385.png <- systat -vm few minutes > > after traffic declined somewhat > > > > We are now trying to get customer install 1Gig NIC so that we can run it > > and compare performance with the rest of the hardware and software being > > essentially the same. > > > > Any ideas on how to improve/resolve this problem are welcome. Thanks! > > _______________________________________________ > > freebsd-net@freebsd.org mailing list > > http://lists.freebsd.org/mailman/listinfo/freebsd-net > > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" > _______________________________________________ > freebsd-net@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" --k3qmt+ucFURmlhDS Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (FreeBSD) iQF8BAEBCgBmBQJVynQ6XxSAAAAAAC4AKGlzc3Vlci1mcHJAbm90YXRpb25zLm9w ZW5wZ3AuZmlmdGhob3JzZW1hbi5uZXRBNEUyMEZBMUQ4Nzg4RjNGMTdFNjZGMDI4 QjkyNTBFMTU2M0VERkU1AAoJEIuSUOFWPt/l3/gIAIJscBPgQ3wE0WFGWJ1kC4OW rj1drAXAW72tSX8quAfMcnlw99RVtVU7AUFgPHsPokrZpPEFXPHXAQobM5CJfm5O UWkjpY02N1q+90N1kxfQ1Sg8J9rP7xM8rx0T63tecnlpM51KTpOw9oGyHfCIJzUu F38Ir9PGseZT5Ouc+szpqZiqtpZ9vP1hduI8hdA/Aa9iC5heYmu5W/gI5+7xaLWb XroSa8emwcaOQGViJ0GHzncwXnnEM/RNNrW/BUZQKurYscmQhWt/Oxj9JUNREjE3 cbVs1nQ5KifDSFilUjxIlebXhYMUKEOyQj+SpRIS44+K3HDl3d119xJkLB34zkU= =iddr -----END PGP SIGNATURE----- --k3qmt+ucFURmlhDS-- From owner-freebsd-net@freebsd.org Tue Aug 11 22:22:05 2015 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 4167999F4A4 for ; Tue, 11 Aug 2015 22:22:05 +0000 (UTC) (envelope-from hiren@strugglingcoder.info) Received: from mail.strugglingcoder.info (strugglingcoder.info [65.19.130.35]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 2A95CE85; Tue, 11 Aug 2015 22:22:04 +0000 (UTC) (envelope-from hiren@strugglingcoder.info) Received: from localhost (unknown [10.1.1.3]) (Authenticated sender: hiren@strugglingcoder.info) by mail.strugglingcoder.info (Postfix) with ESMTPSA id 3FD54E71D; Tue, 11 Aug 2015 15:22:04 -0700 (PDT) Date: Tue, 11 Aug 2015 15:22:04 -0700 From: hiren panchasara To: Adrian Chadd Cc: Maxim Sobolev , freebsd@intel.com, Jev Bj?rsell , FreeBSD Net Subject: Re: Poor high-PPS performance of the 10G ixgbe(9) NIC/driver in FreeBSD 10.1 Message-ID: <20150811222204.GF96509@strugglingcoder.info> References: <20150811221626.GE96509@strugglingcoder.info> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="IvGM3kKqwtniy32b" Content-Disposition: inline In-Reply-To: <20150811221626.GE96509@strugglingcoder.info> User-Agent: Mutt/1.5.23 (2014-03-12) X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 11 Aug 2015 22:22:05 -0000 --IvGM3kKqwtniy32b Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On 08/11/15 at 03:16P, hiren panchasara wrote: >=20 > There were some lagg/hashing related changes recently so let us know if > that is hurting you. Ah, my bad. Said changes would not be in 10.1. You may want to give 10.2 a try. (rc3 is out now.) Cheers, Hiren --IvGM3kKqwtniy32b Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (FreeBSD) iQF8BAEBCgBmBQJVynWLXxSAAAAAAC4AKGlzc3Vlci1mcHJAbm90YXRpb25zLm9w ZW5wZ3AuZmlmdGhob3JzZW1hbi5uZXRBNEUyMEZBMUQ4Nzg4RjNGMTdFNjZGMDI4 QjkyNTBFMTU2M0VERkU1AAoJEIuSUOFWPt/lXJwIAJfvjov1L2noO12RPVC3NUuu 2d5StfD3M5lnl0n6IHBEi+j7IEnk8g6ft9UaDeOn7MjNtxISPX2HSk62xlBLsq17 CqFG0IdJpwSEsEi7+41JuFYOh969G8KyJ0Ed42r+f/7ZscwA+CHvqHgYyfRXVyAk yAhuUUYat1/RyGsuWnX27FUTIRHIiMErDp1Z0BSUTMSw5U56f5rbU9HqmlxGZoEg F3w2wsudj9yfU1RaZful46CLUGHWQbCbZ5kmRPutJ896Vvsht4KQYFx9dfXri7P6 nDfWK/IIYjgoTzz0kUopWp0DjLeI4WUE0wnHr9+LuvBC48nMf2dgCNB4I2fi53w= =vI5j -----END PGP SIGNATURE----- --IvGM3kKqwtniy32b-- From owner-freebsd-net@freebsd.org Tue Aug 11 22:57:39 2015 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 9BD8B99FCF1 for ; Tue, 11 Aug 2015 22:57:39 +0000 (UTC) (envelope-from sobomax@sippysoft.com) Received: from mail-wi0-f177.google.com (mail-wi0-f177.google.com [209.85.212.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 3EE253D2 for ; Tue, 11 Aug 2015 22:57:38 +0000 (UTC) (envelope-from sobomax@sippysoft.com) Received: by wicne3 with SMTP id ne3so79239857wic.0 for ; Tue, 11 Aug 2015 15:57:31 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:sender:date:message-id:subject:from :to:cc:content-type; bh=tHBFN1xxLoMbxbNLxBojEq1meDDOJz7ZqxhVqYTQNyc=; b=ZUDxt9fbAVoNFnOweqD2pLcjoluoNruDNlDgTarH/L67seJEvyRKYl/IbhPPYBCIV8 EFP+Aj6SJiXX28WfgXiG9uuvI1wF/293ZEKD4PLUuwFNum3lOZ8XLGEylfqfqnEZpLrM Ea5usu+gMxWK2ik+Hvo+g/v2fLrl31SYXg8Fju15zhOmRd3v4IZyOicDZdHMtBI9ig/z lN/BTFAnBGfhvEKH0x3PQvXWQZlCMBX+Mf6nDPYQPNMI85+O4Pwik8a0Z3SsS0QwuJHi BvMYWDxG7DqkeKdXk0vRM6I38fm+hFPIfpdwkFYzUUV3+MP8QnccOi83vhg9Uoi9GLeN cblA== X-Gm-Message-State: ALoCoQnTlCUbw9yK1pduvSwNB6iY9PCC58iuU+kE3TAJKE0WOFs37/3bewQppC7b+dFitx+tg5lj MIME-Version: 1.0 X-Received: by 10.180.206.8 with SMTP id lk8mr42266023wic.12.1439333851225; Tue, 11 Aug 2015 15:57:31 -0700 (PDT) Sender: sobomax@sippysoft.com Received: by 10.27.143.15 with HTTP; Tue, 11 Aug 2015 15:57:31 -0700 (PDT) Date: Tue, 11 Aug 2015 15:57:31 -0700 X-Google-Sender-Auth: 2Sw-WVzvqCzTPm2HtNFEsCsI8DI Message-ID: Subject: Re: Poor high-PPS performance of the 10G ixgbe(9) NIC/driver in FreeBSD 10.1 From: Maxim Sobolev To: hiren panchasara Cc: Adrian Chadd , freebsd@intel.com, "Jev Bj?rsell" , FreeBSD Net Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 11 Aug 2015 22:57:39 -0000 Thanks, we will try, however I don't think it's going to make a huge difference because we run almost 2x the PPS on the otherwise identical (as far as FreeBSD version/lagg code goes) with I350/igb(9) and inferior CPU hardware. That kinda suggests that whatever the problem is it is below lagg. -Maxim On Tue, Aug 11, 2015 at 3:22 PM, hiren panchasara < hiren@strugglingcoder.info> wrote: > On 08/11/15 at 03:16P, hiren panchasara wrote: > > > > There were some lagg/hashing related changes recently so let us know if > > that is hurting you. > > Ah, my bad. Said changes would not be in 10.1. You may want to give 10.2 > a try. (rc3 is out now.) > > Cheers, > Hiren > From owner-freebsd-net@freebsd.org Tue Aug 11 23:05:00 2015 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 83B1499FF3E for ; Tue, 11 Aug 2015 23:05:00 +0000 (UTC) (envelope-from sobomax@sippysoft.com) Received: from mail-wi0-f182.google.com (mail-wi0-f182.google.com [209.85.212.182]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 1C693A6E for ; Tue, 11 Aug 2015 23:04:59 +0000 (UTC) (envelope-from sobomax@sippysoft.com) Received: by wicne3 with SMTP id ne3so79387275wic.0 for ; Tue, 11 Aug 2015 16:04:58 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:sender:in-reply-to:references:date :message-id:subject:from:to:cc:content-type; bh=vMRF75p09oZGRr1Ib3Cl/32DGFtuRvlL/nRJz3MG1wo=; b=QclTyAWGLlaAXxJKDAbW8SgzrYYz0h1GwbgVBOEgCf4Uvh29UMpsHRHqyMQ5Lnn9lW +p6/aACNtLY6t+U1yA94MZC3QEbEsHqgYjxG8lCCRsnK+ttLU9q8wHn0KBNGquzgxwTv HKLoyT+uHQSHJYDx9ws4g+jv9iU2lvaEG3SMZ9EaY1xROT4iP9+DYd+kP2vOeWCqRkDe Z4DNigynzTsf3kGszYaN0Llym/u3g7L5DVl5mZc2kKersNmAJyo67HDoQnEJ8eeRU++x dklgIgT8RNXAm8F1Em4NWD2HpfCWpr0FTBf79gQpg9Bzk2udLFT15qWz5rUl89Ol4OoF vGKg== X-Gm-Message-State: ALoCoQn4taKotiwSp37RTx7Ph0sLiyOzxJj/IVdv0AvfY+KGymkG5t2iMW8nBzwmsIolcXGwF32H MIME-Version: 1.0 X-Received: by 10.180.81.100 with SMTP id z4mr39617940wix.8.1439334298213; Tue, 11 Aug 2015 16:04:58 -0700 (PDT) Sender: sobomax@sippysoft.com Received: by 10.27.143.15 with HTTP; Tue, 11 Aug 2015 16:04:58 -0700 (PDT) In-Reply-To: <20150811221626.GE96509@strugglingcoder.info> References: <20150811221626.GE96509@strugglingcoder.info> Date: Tue, 11 Aug 2015 16:04:58 -0700 X-Google-Sender-Auth: nhz1L6qc9BDtNJk-2YQPno9y9hw Message-ID: Subject: Re: Poor high-PPS performance of the 10G ixgbe(9) NIC/driver in FreeBSD 10.1 From: Maxim Sobolev To: hiren panchasara Cc: Adrian Chadd , FreeBSD Net , freebsd@intel.com, "Jev Bj?rsell" Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 11 Aug 2015 23:05:00 -0000 Here it is, the distribution looks pretty normal to me. dev.ix.0.queue0.tx_packets: 846233384 dev.ix.0.queue0.rx_packets: 856092418 dev.ix.0.queue1.tx_packets: 980356163 dev.ix.0.queue1.rx_packets: 922935329 dev.ix.0.queue2.tx_packets: 970700307 dev.ix.0.queue2.rx_packets: 907776311 dev.ix.0.queue3.tx_packets: 951911927 dev.ix.0.queue3.rx_packets: 903933007 dev.ix.0.queue4.tx_packets: 960075438 dev.ix.0.queue4.rx_packets: 909830391 dev.ix.0.queue5.tx_packets: 957304026 dev.ix.0.queue5.rx_packets: 889722162 dev.ix.0.queue6.tx_packets: 946175921 dev.ix.0.queue6.rx_packets: 898922310 dev.ix.0.queue7.tx_packets: 936382885 dev.ix.0.queue7.rx_packets: 890026885 dev.ix.1.queue0.tx_packets: 844847347 dev.ix.1.queue0.rx_packets: 840770906 dev.ix.1.queue1.tx_packets: 978807036 dev.ix.1.queue1.rx_packets: 906148213 dev.ix.1.queue2.tx_packets: 969026390 dev.ix.1.queue2.rx_packets: 906644000 dev.ix.1.queue3.tx_packets: 950384414 dev.ix.1.queue3.rx_packets: 890646445 dev.ix.1.queue4.tx_packets: 958536903 dev.ix.1.queue4.rx_packets: 887900309 dev.ix.1.queue5.tx_packets: 955802045 dev.ix.1.queue5.rx_packets: 884884583 dev.ix.1.queue6.tx_packets: 944802927 dev.ix.1.queue6.rx_packets: 883266179 dev.ix.1.queue7.tx_packets: 934953601 dev.ix.1.queue7.rx_packets: 886399283 On Tue, Aug 11, 2015 at 3:16 PM, hiren panchasara < hiren@strugglingcoder.info> wrote: > On 08/11/15 at 03:01P, Adrian Chadd wrote: > > hi, > > > > Are you able to graph per-queue interrupt rates? > > > > It looks like the traffic is distributed differently (the first two > > queues are taking interrupts). > > Yeah, also check out "# sysctl dev.ix | grep packets" > > > > Does 10.1 have the flow director code disabled? I remember there was > > some .. interesting behaviour with ixgbe where it'd look at traffic > > and set up flow director rules to try and "balance" things. It was > > buggy and programmed the hardware badly, so we disabled it in at least > > -HEAD. > > Looks like we don't build with IXGBE_FDIR by default on 10 so I assume > it's off. > > There were some lagg/hashing related changes recently so let us know if > that is hurting you. > > Cheers, > Hiren > > > > > > > > -adrian > > > > > > On 11 August 2015 at 14:18, Maxim Sobolev wrote: > > > Hi folks, > > > > > > We've trying to migrate some of our high-PPS systems to a new hardware > that > > > has four X540-AT2 10G NICs and observed that interrupt time goes > through > > > roof after we cross around 200K PPS in and 200K out (two ports in > LACP). > > > The previous hardware was stable up to about 350K PPS in and 350K out. > I > > > believe the old one was equipped with the I350 and had the identical > LACP > > > configuration. The new box also has better CPU with more cores (i.e. 24 > > > cores vs. 16 cores before). CPU itself is 2 x E5-2690 v3. > > > > > > After hitting this limit with the default settings, I've tried to > tweak the > > > following settings: > > > > > > hw.ix.rx_process_limit="-1" > > > hw.ix.tx_process_limit="-1" > > > hw.ix.enable_aim="0" > > > hw.ix.max_interrupt_rate="-1" > > > hw.ix.rxd="4096" > > > hw.ix.txd="4096" > > > > > > dev.ix.0.fc=0 > > > dev.ix.1.fc=0 > > > dev.ix.2.fc=0 > > > dev.ix.3.fc=0 > > > > > > hw.intr_storm_threshold=0 > > > > > > But there is little or no effect on the performance. The workload is > just > > > lot of small UDP packets being relayed between bunch of hosts. The > symptoms > > > are always the same - the box runs nice and cool until it his the said > PPS > > > threshold, with kernel spending just few percent in the interrupts and > then > > > it jumps straight to 100% interrupt time, thereby scaring some traffic > away > > > due to packet loss and such, so that the load drops and the system goes > > > into the "cool" state again. It looks very much like some contention > in the > > > driver or in the hardware. Linked are some monitoring screenshots > > > displaying the issue unfolding as well as systat -vm screenshots from > the > > > "cool" state. > > > > > > http://sobomax.sippysoft.com/ScreenShot387.png <- CPU utilization > right > > > before the "bang event" > > > http://sobomax.sippysoft.com/ScreenShot382.png <- issue itself > > > http://sobomax.sippysoft.com/ScreenShot385.png <- systat -vm few > minutes > > > after traffic declined somewhat > > > > > > We are now trying to get customer install 1Gig NIC so that we can run > it > > > and compare performance with the rest of the hardware and software > being > > > essentially the same. > > > > > > Any ideas on how to improve/resolve this problem are welcome. Thanks! > > > _______________________________________________ > > > freebsd-net@freebsd.org mailing list > > > http://lists.freebsd.org/mailman/listinfo/freebsd-net > > > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" > > _______________________________________________ > > freebsd-net@freebsd.org mailing list > > http://lists.freebsd.org/mailman/listinfo/freebsd-net > > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" > From owner-freebsd-net@freebsd.org Tue Aug 11 23:15:19 2015 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id B4E5799F1C2 for ; Tue, 11 Aug 2015 23:15:19 +0000 (UTC) (envelope-from cochard@gmail.com) Received: from mail-la0-x234.google.com (mail-la0-x234.google.com [IPv6:2a00:1450:4010:c03::234]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 28BA8F1E; Tue, 11 Aug 2015 23:15:19 +0000 (UTC) (envelope-from cochard@gmail.com) Received: by labd1 with SMTP id d1so216066lab.1; Tue, 11 Aug 2015 16:15:17 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:from:date:message-id :subject:to:cc:content-type; bh=sZUgiWWtC6lyqUEYdBBP3TzTyjMNrQVPovKEdV9/DX0=; b=BgaCp7/Ea5REOQ8HJucy9Jye4elQIAAvnIB+Ay4X8kjXHXvdQGCFkvF3Gynxa/5eGG lY3TEedFUOHlRmRgZOsrtyDagDgN/2ffyiA674WoSyKIgPywIDFrtSejQitRv66w8nHI 9hOakWRHGAaFspg3K5NzmmEAaiN5lziAUQO6s3mi3CV/JbOSX6BukbTsRjpKflfRcx+3 xkwP8s2e+t/fyd2GVFZojbUKBdmFmoBk3uF93ifBiKzWzQVA7FpeEPMAeaIxd3VvUwAc y43gO6oUTaYyNFphzjk6xeuiAqjwkHJvm31F+mza+MNqovVP7Qje4cpPnBW9B2V22hfv Rwiw== X-Received: by 10.112.47.73 with SMTP id b9mr29202857lbn.46.1439334917144; Tue, 11 Aug 2015 16:15:17 -0700 (PDT) MIME-Version: 1.0 Sender: cochard@gmail.com Received: by 10.25.157.146 with HTTP; Tue, 11 Aug 2015 16:14:57 -0700 (PDT) In-Reply-To: References: From: =?UTF-8?Q?Olivier_Cochard=2DLabb=C3=A9?= Date: Wed, 12 Aug 2015 01:14:57 +0200 X-Google-Sender-Auth: yFh-H7CiJhxDVxV0eRVfIavIpVI Message-ID: Subject: Re: Poor high-PPS performance of the 10G ixgbe(9) NIC/driver in FreeBSD 10.1 To: Maxim Sobolev Cc: FreeBSD Net , freebsd@intel.com, =?UTF-8?Q?Jev_Bj=C3=B6rsell?= Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 11 Aug 2015 23:15:19 -0000 On Tue, Aug 11, 2015 at 11:18 PM, Maxim Sobolev wrote= : > Hi folks, > > =E2=80=8BHi, =E2=80=8B > We've trying to migrate some of our high-PPS systems to a new hardware th= at > has four X540-AT2 10G NICs and observed that interrupt time goes through > roof after we cross around 200K PPS in and 200K out (two ports in LACP). > The previous hardware was stable up to about 350K PPS in and 350K out. I > believe the old one was equipped with the I350 and had the identical LACP > configuration. The new box also has better CPU with more cores (i.e. 24 > cores vs. 16 cores before). CPU itself is 2 x E5-2690 v3. > =E2=80=8B200K PPS, and even 350K PPS are very low value indeed. On a Intel Xeon L5630 (4 cores only) with one X540-AT2=E2=80=8B =E2=80=8B(then 2 10Gigabit ports)=E2=80=8B I've reached about 1.8Mpps (fast= forwarding enabled) [1]. But my setup didn't use lagg(4): Can you disable lagg configuration and re-measure your performance without lagg ? Do you let Intel NIC drivers using 8 queues for port too? In my use case (forwarding smallest UDP packet size), I obtain better behaviour by limiting NIC queues to 4 (hw.ix.num_queues or hw.ixgbe.num_queues, don't remember) if my system had 8 cores. And this with Gigabit Intel[2] or Chelsio NIC [3]. Don't forget to disable TSO and LRO too. =E2=80=8BRegards, Olivier [1] http://bsdrp.net/documentation/examples/forwarding_performance_lab_of_an_ib= m_system_x3550_m3_with_10-gigabit_intel_x540-at2#graphs [2] http://bsdrp.net/documentation/examples/forwarding_performance_lab_of_a_sup= erserver_5018a-ftn4#graph1 [3] http://bsdrp.net/documentation/examples/forwarding_performance_lab_of_a_hp_= proliant_dl360p_gen8_with_10-gigabit_with_10-gigabit_chelsio_t540-cr#reduci= ng_nic_queues From owner-freebsd-net@freebsd.org Tue Aug 11 23:28:32 2015 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 1B76799F474 for ; Tue, 11 Aug 2015 23:28:32 +0000 (UTC) (envelope-from sobomax@sippysoft.com) Received: from mail-wi0-f180.google.com (mail-wi0-f180.google.com [209.85.212.180]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id A71878B5 for ; Tue, 11 Aug 2015 23:28:31 +0000 (UTC) (envelope-from sobomax@sippysoft.com) Received: by wicne3 with SMTP id ne3so196585227wic.1 for ; Tue, 11 Aug 2015 16:28:28 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:sender:date:message-id:subject:from :to:cc:content-type; bh=Sj4vR0eg/BXhE+tzgNWQnUVTvHp2KNEeSXIWvOKBfYc=; b=kFQ2dcf1v7aVtLRQni073Qgm5rQATsrrZW4E46Wx1LRuXwpButGhIRs1ETdbZm/ffD XTf1PODjq7XNxCzUALIeitd+ut/vEmv35FJS8FmXhp6MqvyfZzKYE+QhbXWeBc35GztK kzLefjdGRtEv6QEYXvFZ0pr2VNKs2jgZWfptff6ov/XdNZGLLcxBdEh1gTljUyWCmHRP 7y9kmx6b8RqfF5xkkdHyxbmSXu2I1Uvo7HPeCrmOkF+Z3HQd5kshwcqCvWgsivV8LAWM 26yCMQGvN1jJG9rEbhz5Q4Tc5jEK0DYqaiOCJ58gaMHKO12LGJNp6w3pZrn2XhXrKT/D 3RcQ== X-Gm-Message-State: ALoCoQkr/z7gQwUTMP4uPeEDz2IQQZtgofo0tpRhizaskJFGBBUP5IA497ivCdtV8lybkQSbLDnX MIME-Version: 1.0 X-Received: by 10.180.105.66 with SMTP id gk2mr22291681wib.73.1439335708827; Tue, 11 Aug 2015 16:28:28 -0700 (PDT) Sender: sobomax@sippysoft.com Received: by 10.27.143.15 with HTTP; Tue, 11 Aug 2015 16:28:28 -0700 (PDT) Date: Tue, 11 Aug 2015 16:28:28 -0700 X-Google-Sender-Auth: CD4jT_nH51QoV4zHzfli9eFiPU8 Message-ID: Subject: Re: Poor high-PPS performance of the 10G ixgbe(9) NIC/driver in FreeBSD 10.1 From: Maxim Sobolev To: =?UTF-8?Q?Olivier_Cochard=2DLabb=C3=A9?= Cc: FreeBSD Net , freebsd@intel.com, =?UTF-8?Q?Jev_Bj=C3=B6rsell?= Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 11 Aug 2015 23:28:32 -0000 Olivier, keep in mind that we are not "kernel forwarding" packets, but "app forwarding", i.e. the packet goes full way net->kernel->recvfrom->app->sendto->kernel->net, which is why we have much lower PPS limits and which is why I think we are actually benefiting from the extra queues. Single-thread sendto() in a loop is CPU-bound at about 220K PPS, and while running the test I am observing that outbound traffic from one thread is mapped into a specific queue (well, pair of queues on two separate adaptors, due to lagg load balancing action). And the peak performance of that test is at 7 threads, which I believe corresponds to the number of queues. We have plenty of CPU cores in the box (24) with HTT/SMT disabled and one CPU is mapped to a specific queue. This leaves us with at least 8 CPUs fully capable of running our app. If you look at the CPU utilization, we are at about 10% when the issue hits. ix0: port 0x6020-0x603f mem 0xc7c00000-0xc7dfffff,0xc7e04000-0xc7e07fff irq 40 at device 0.0 on pci3 ix0: Using MSIX interrupts with 9 vectors ix0: Bound queue 0 to cpu 0 ix0: Bound queue 1 to cpu 1 ix0: Bound queue 2 to cpu 2 ix0: Bound queue 3 to cpu 3 ix0: Bound queue 4 to cpu 4 ix0: Bound queue 5 to cpu 5 ix0: Bound queue 6 to cpu 6 ix0: Bound queue 7 to cpu 7 ix0: Ethernet address: 0c:c4:7a:5e:be:64 ix0: PCI Express Bus: Speed 5.0GT/s Width x8 001.000008 [2705] netmap_attach success for ix0 tx 8/4096 rx 8/4096 queues/slots ix1: port 0x6000-0x601f mem 0xc7a00000-0xc7bfffff,0xc7e00000-0xc7e03fff irq 44 at device 0.1 on pci3 ix1: Using MSIX interrupts with 9 vectors ix1: Bound queue 0 to cpu 8 ix1: Bound queue 1 to cpu 9 ix1: Bound queue 2 to cpu 10 ix1: Bound queue 3 to cpu 11 ix1: Bound queue 4 to cpu 12 ix1: Bound queue 5 to cpu 13 ix1: Bound queue 6 to cpu 14 ix1: Bound queue 7 to cpu 15 ix1: Ethernet address: 0c:c4:7a:5e:be:65 ix1: PCI Express Bus: Speed 5.0GT/s Width x8 001.000009 [2705] netmap_attach success for ix1 tx 8/4096 rx 8/4096 queues/slots On Tue, Aug 11, 2015 at 4:14 PM, Olivier Cochard-Labb=C3=A9 wrote: > On Tue, Aug 11, 2015 at 11:18 PM, Maxim Sobolev > wrote: > >> Hi folks, >> >> =E2=80=8BHi, > =E2=80=8B > > >> We've trying to migrate some of our high-PPS systems to a new hardware >> that >> has four X540-AT2 10G NICs and observed that interrupt time goes through >> roof after we cross around 200K PPS in and 200K out (two ports in LACP). >> The previous hardware was stable up to about 350K PPS in and 350K out. I >> believe the old one was equipped with the I350 and had the identical LAC= P >> configuration. The new box also has better CPU with more cores (i.e. 24 >> cores vs. 16 cores before). CPU itself is 2 x E5-2690 v3. >> > > =E2=80=8B200K PPS, and even 350K PPS are very low value indeed. > On a Intel Xeon L5630 (4 cores only) with one X540-AT2=E2=80=8B > > =E2=80=8B(then 2 10Gigabit ports)=E2=80=8B I've reached about 1.8Mpps (fa= stforwarding > enabled) [1]. > But my setup didn't use lagg(4): Can you disable lagg configuration and > re-measure your performance without lagg ? > > Do you let Intel NIC drivers using 8 queues for port too? > In my use case (forwarding smallest UDP packet size), I obtain better > behaviour by limiting NIC queues to 4 (hw.ix.num_queues or > hw.ixgbe.num_queues, don't remember) if my system had 8 cores. And this > with Gigabit Intel[2] or Chelsio NIC [3]. > > Don't forget to disable TSO and LRO too. > > =E2=80=8BRegards, > > Olivier > > [1] > http://bsdrp.net/documentation/examples/forwarding_performance_lab_of_an_= ibm_system_x3550_m3_with_10-gigabit_intel_x540-at2#graphs > [2] > http://bsdrp.net/documentation/examples/forwarding_performance_lab_of_a_s= uperserver_5018a-ftn4#graph1 > [3] > http://bsdrp.net/documentation/examples/forwarding_performance_lab_of_a_h= p_proliant_dl360p_gen8_with_10-gigabit_with_10-gigabit_chelsio_t540-cr#redu= cing_nic_queues > From owner-freebsd-net@freebsd.org Wed Aug 12 01:28:53 2015 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id BD54099E38D for ; Wed, 12 Aug 2015 01:28:53 +0000 (UTC) (envelope-from barney_cordoba@yahoo.com) Received: from nm24-vm4.bullet.mail.ne1.yahoo.com (nm24-vm4.bullet.mail.ne1.yahoo.com [98.138.91.184]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 86F981AF8 for ; Wed, 12 Aug 2015 01:28:53 +0000 (UTC) (envelope-from barney_cordoba@yahoo.com) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1439342932; bh=UAzc1Q33n4ml/ULtQLGaVjXxXtPDHNFLGm2oGDLQLQw=; h=Date:From:Reply-To:To:Cc:In-Reply-To:References:Subject:From:Subject; b=Ag841ue/I9BtNIywovJa82aQN2Gm/Ya+RP0N3VcevK85OgWf3EE6HZpDAGrQDZZISp+Z9Ej5XsXPiyaWyjnkc0dmw5beFwLnzNs29PwKNGuD1TVW5a2/C9cRsYTGoCpXN8rXX93FvDNuXIyLmWnc52jO7S+mZLRrhMKoBl7o4cSVyMyGe+ehsfz1LD9S37oSL/syEWhjrCDpIKcLfccRujxjiedVlW4Xfu3JVhLTV2QmqSzXKoU+nGvfNqNZSOn55mYUYJHV9p9MsOjiw0InQZYbqBnfgFwc2dDyUj8vRjRyKNDCVv+D10li5vOoxQvBEZqDQWcAZTiZycKbFkI5Fw== Received: from [98.138.226.178] by nm24.bullet.mail.ne1.yahoo.com with NNFMP; 12 Aug 2015 01:28:52 -0000 Received: from [98.138.226.164] by tm13.bullet.mail.ne1.yahoo.com with NNFMP; 12 Aug 2015 01:28:52 -0000 Received: from [127.0.0.1] by omp1065.mail.ne1.yahoo.com with NNFMP; 12 Aug 2015 01:28:52 -0000 X-Yahoo-Newman-Property: ymail-3 X-Yahoo-Newman-Id: 71491.57023.bm@omp1065.mail.ne1.yahoo.com X-YMail-OSG: KZf6pDsVM1niKIUnqaZTKq9C9ecYoe4uuXomJr._V.x_2dJv22.UlpXkSsjO1er WEg_O9X7dIMo2DgQkn4CDJMGh.L94dsn9pAzgf39GRkxyhr6E.6enpZqF7mzXLgdQbLhTxPEfzWd 7L156FglJOtkDJtjqdRgWPKwtzpMEff0mPbRGkphu0SOpRXCqg3wYXiRJpX_a6q1ExHtw8d8uQXC 0e5EqblQRHD_kLUsOgleqA8sLWJz_5Ze5_OUXO1k3JxXKJMZiUZyLSv85HR5UoPFF0ECERnux0pm QokOs60MJMKlrPuqaQru7Lc5dNE7KHo9GGs2lPlOnrFdzH1XI00VNBjr1KHZ6lXBpfYc5OeFQMSl fQKD1piFR.bLx1D4rsR2ZgtLrtwQZcH_WC.F6CN7sLFvBasAywblZinau_WE7CS1s.BBZwOUVuhT iPaFgGXmr9Stn03J0MKBbZvpS_UzMKltLjQVfAuR7kQnT7QKcZR9DwdHRI6px66zjFP9k7pgVQEe mZ_MEOhtQH_Q5Lg-- Received: by 98.138.105.200; Wed, 12 Aug 2015 01:28:51 +0000 Date: Wed, 12 Aug 2015 01:28:29 +0000 (UTC) From: Barney Cordoba Reply-To: Barney Cordoba To: =?UTF-8?Q?Olivier_Cochard-Labb=C3=A9?= , Maxim Sobolev Cc: FreeBSD Net , "freebsd@intel.com" , =?UTF-8?Q?Jev_Bj=C3=B6rsell?= Message-ID: <1931183673.3234634.1439342909940.JavaMail.yahoo@mail.yahoo.com> In-Reply-To: References: Subject: Re: Poor high-PPS performance of the 10G ixgbe(9) NIC/driver in FreeBSD 10.1 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 12 Aug 2015 01:28:53 -0000 Wow, this is really important! if this is a college project, I give you a D= . Maybe a D- because it's almost useless information. You ignore the most important aspect of "performance". Efficiency is arguab= ly the most important aspect of performance.=C2=A0 1M pps at 20% cpu usage is much better "performance" than 1.2M pps at 85%.= =C2=A0 Why don't any of you understand this simple thing? Why does spreading equal= ity really matter, unless you are hitting a wall with your cpus? I don't ca= re which cpu processes which packet. If you weren't doing moronic things li= ke binding to a cpu, then you'd never have to care about distribution unles= s it was extremely unbalanced. BC=20 On Tuesday, August 11, 2015 7:15 PM, Olivier Cochard-Labb=C3=A9 wrote: =20 On Tue, Aug 11, 2015 at 11:18 PM, Maxim Sobolev wrot= e: > Hi folks, > > =E2=80=8BHi, =E2=80=8B > We've trying to migrate some of our high-PPS systems to a new hardware th= at > has four X540-AT2 10G NICs and observed that interrupt time goes through > roof after we cross around 200K PPS in and 200K out (two ports in LACP). > The previous hardware was stable up to about 350K PPS in and 350K out. I > believe the old one was equipped with the I350 and had the identical LACP > configuration. The new box also has better CPU with more cores (i.e. 24 > cores vs. 16 cores before). CPU itself is 2 x E5-2690 v3. > =E2=80=8B200K PPS, and even 350K PPS are very low value indeed. On a Intel Xeon L5630 (4 cores only) with one X540-AT2=E2=80=8B =E2=80=8B(then 2 10Gigabit ports)=E2=80=8B I've reached about 1.8Mpps (fast= forwarding enabled) [1]. But my setup didn't use lagg(4): Can you disable lagg configuration and re-measure your performance without lagg ? Do you let Intel NIC drivers using 8 queues for port too? In my use case (forwarding smallest UDP packet size), I obtain better behaviour by limiting NIC queues to 4 (hw.ix.num_queues or hw.ixgbe.num_queues, don't remember) if my system had 8 cores. And this with Gigabit Intel[2] or Chelsio NIC [3]. Don't forget to disable TSO and LRO too. =E2=80=8BRegards, Olivier [1] http://bsdrp.net/documentation/examples/forwarding_performance_lab_of_an_ib= m_system_x3550_m3_with_10-gigabit_intel_x540-at2#graphs [2] http://bsdrp.net/documentation/examples/forwarding_performance_lab_of_a_sup= erserver_5018a-ftn4#graph1 [3] http://bsdrp.net/documentation/examples/forwarding_performance_lab_of_a_hp_= proliant_dl360p_gen8_with_10-gigabit_with_10-gigabit_chelsio_t540-cr#reduci= ng_nic_queues _______________________________________________ freebsd-net@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" From owner-freebsd-net@freebsd.org Wed Aug 12 01:40:17 2015 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id A2BA399E7B8 for ; Wed, 12 Aug 2015 01:40:17 +0000 (UTC) (envelope-from barney_cordoba@yahoo.com) Received: from nm1-vm0.bullet.mail.ne1.yahoo.com (nm1-vm0.bullet.mail.ne1.yahoo.com [98.138.91.74]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 6C5531CA for ; Wed, 12 Aug 2015 01:40:17 +0000 (UTC) (envelope-from barney_cordoba@yahoo.com) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1439343609; bh=g0kHJWoBbWWQcX7i17mr+ATL4vVJs831Ef4Jrp2PDGM=; h=Date:From:Reply-To:To:Cc:In-Reply-To:References:Subject:From:Subject; b=KzoNi/BIo21zcbWtM9+t9ifIDGyIngCrf8aviCcyhAHyyS1vHq6Gk5bBW1m4cG7oipuWewJLCDsn3XR8gS5WRgsPCmhtUEtWKtfVdnJgLiQfVB3vx7zxYiWT89rkutAlWtbsDbofu2/TyDZi1rTiAdXkT31OauFieL5VbkupuI/uxZMbLPuLp9amcwrGHkjjUg+E7kd2afWkX3BD7uxHNPdq+1+UVqbHt1/OYlsH0X5aHcll1955WTJEy0UYcl4ZMEQctpFx3v9hEOBZURkWiUnubwqapE3/JWqST/Pw3TrU8ubYHrIhrjpl9h1Qg/bNtdkMR+kwne31oX/C3wxC9g== Received: from [98.138.100.115] by nm1.bullet.mail.ne1.yahoo.com with NNFMP; 12 Aug 2015 01:40:09 -0000 Received: from [98.138.88.238] by tm106.bullet.mail.ne1.yahoo.com with NNFMP; 12 Aug 2015 01:40:09 -0000 Received: from [127.0.0.1] by omp1038.mail.ne1.yahoo.com with NNFMP; 12 Aug 2015 01:40:09 -0000 X-Yahoo-Newman-Property: ymail-3 X-Yahoo-Newman-Id: 856165.77852.bm@omp1038.mail.ne1.yahoo.com X-YMail-OSG: SpJcUVsVM1m_AJRXVKNA2xCb_5ie.ahITTyyihLhZWPsb8eYh1jqQ2f2RYlmQ_H XVPV7CPkYUaZi8xBJoLxb3NqnG_siFNKw8NlobR3ZQjLnNtSPlyUnDxV88btcOaNStVKN80ce29p w.5AWUsLzxKm1D7AbhYeEsN58xwvt7YE.8NR1WUapscewyleo_iMUsYm41YAPG1ye7tF3vFghpoS kOHq4_hNDKXtGrCwsOMsodjwwdn8aBiHKu7M7l6VEsq1XqjW1Lb4bujECLPblAXLa3Zl7z7QDNRj jYXsDM6w2PnBVvOZburgdL6rUto1Soem4xULXAUkEf9IukjC2cWvMBpgNOt0ZunZdFedQjt8IAJo YXRWGlt1cTDMGGSIrDfh1QJ0anDtCofCNa9J2f4jnRIe3Cy_Xyq8r4PKxKqYBJcRjwf4vpvx_WTa c8fQx20NirKd5QkZ6cUiRQtS_2x10dnPrxcHsiQz_Qdhwvo0NLD72sdP2pEbbHClGA1s4ipbgXHy .sau.9H4MIDgqGw-- Received: by 98.138.101.170; Wed, 12 Aug 2015 01:40:09 +0000 Date: Wed, 12 Aug 2015 01:40:09 +0000 (UTC) From: Barney Cordoba Reply-To: Barney Cordoba To: Barney Cordoba , =?UTF-8?Q?Olivier_Cochard-Labb=C3=A9?= , Maxim Sobolev Cc: FreeBSD Net , "freebsd@intel.com" , =?UTF-8?Q?Jev_Bj=C3=B6rsell?= Message-ID: <477444474.1722346.1439343609122.JavaMail.yahoo@mail.yahoo.com> In-Reply-To: <1931183673.3234634.1439342909940.JavaMail.yahoo@mail.yahoo.com> References: <1931183673.3234634.1439342909940.JavaMail.yahoo@mail.yahoo.com> Subject: Re: Poor high-PPS performance of the 10G ixgbe(9) NIC/driver in FreeBSD 10.1 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 12 Aug 2015 01:40:17 -0000 Also, using a slow-ass cpu like the atom is completely absurd; first, no-on= e would ever use them.=C2=A0 You have to test cpu usage under 60% cpu usage, because as you get to highe= r cpu usage levels the lock contention increases exponentially. You're incr= easing lock contention by having more queues; so more queues at higher cpu = % usage will perform increasingly bad as usage increases. You'd never run a system at 95% usage (ie totally hammering it) in real wor= ld usage, so why would you benchmark at such a high usage? Everything chang= es as cpu available become scarce.=C2=A0 "What is the pps at 50% cpu usage" is a better question to ask than the one= you're asking. BC=20 On Tuesday, August 11, 2015 9:29 PM, Barney Cordoba via freebsd-net wrote: =20 Wow, this is really important! if this is a college project, I give you a = D. Maybe a D- because it's almost useless information. You ignore the most important aspect of "performance". Efficiency is arguab= ly the most important aspect of performance.=C2=A0 1M pps at 20% cpu usage is much better "performance" than 1.2M pps at 85%.= =C2=A0 Why don't any of you understand this simple thing? Why does spreading equal= ity really matter, unless you are hitting a wall with your cpus? I don't ca= re which cpu processes which packet. If you weren't doing moronic things li= ke binding to a cpu, then you'd never have to care about distribution unles= s it was extremely unbalanced. BC=20 =C2=A0 =C2=A0 On Tuesday, August 11, 2015 7:15 PM, Olivier Cochard-Labb=C3= =A9 wrote: =C2=A0=20 On Tue, Aug 11, 2015 at 11:18 PM, Maxim Sobolev wrot= e: > Hi folks, > > =E2=80=8BHi, =E2=80=8B > We've trying to migrate some of our high-PPS systems to a new hardware th= at > has four X540-AT2 10G NICs and observed that interrupt time goes through > roof after we cross around 200K PPS in and 200K out (two ports in LACP). > The previous hardware was stable up to about 350K PPS in and 350K out. I > believe the old one was equipped with the I350 and had the identical LACP > configuration. The new box also has better CPU with more cores (i.e. 24 > cores vs. 16 cores before). CPU itself is 2 x E5-2690 v3. > =E2=80=8B200K PPS, and even 350K PPS are very low value indeed. On a Intel Xeon L5630 (4 cores only) with one X540-AT2=E2=80=8B =E2=80=8B(then 2 10Gigabit ports)=E2=80=8B I've reached about 1.8Mpps (fast= forwarding enabled) [1]. But my setup didn't use lagg(4): Can you disable lagg configuration and re-measure your performance without lagg ? Do you let Intel NIC drivers using 8 queues for port too? In my use case (forwarding smallest UDP packet size), I obtain better behaviour by limiting NIC queues to 4 (hw.ix.num_queues or hw.ixgbe.num_queues, don't remember) if my system had 8 cores. And this with Gigabit Intel[2] or Chelsio NIC [3]. Don't forget to disable TSO and LRO too. =E2=80=8BRegards, Olivier [1] http://bsdrp.net/documentation/examples/forwarding_performance_lab_of_an_ib= m_system_x3550_m3_with_10-gigabit_intel_x540-at2#graphs [2] http://bsdrp.net/documentation/examples/forwarding_performance_lab_of_a_sup= erserver_5018a-ftn4#graph1 [3] http://bsdrp.net/documentation/examples/forwarding_performance_lab_of_a_hp_= proliant_dl360p_gen8_with_10-gigabit_with_10-gigabit_chelsio_t540-cr#reduci= ng_nic_queues _______________________________________________ freebsd-net@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" =C2=A0=20 _______________________________________________ freebsd-net@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" From owner-freebsd-net@freebsd.org Wed Aug 12 01:46:45 2015 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id CE06499E92A for ; Wed, 12 Aug 2015 01:46:45 +0000 (UTC) (envelope-from sobomax@sippysoft.com) Received: from mail-wi0-f181.google.com (mail-wi0-f181.google.com [209.85.212.181]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 64E90A07 for ; Wed, 12 Aug 2015 01:46:44 +0000 (UTC) (envelope-from sobomax@sippysoft.com) Received: by wicne3 with SMTP id ne3so82163661wic.0 for ; Tue, 11 Aug 2015 18:46:42 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:sender:in-reply-to:references:date :message-id:subject:from:to:cc:content-type; bh=QnUF8NPF+tTwauLQQi4jIFxu/EM5dzbuE3bv602TNLA=; b=IQ8EqxNP84p69cCmQf0TYp9Eu6SIO+xgXrpcKCvkNFOsC7HGCxFapbES6tsclhUlri xhCXNAcVKm1i2H5Aa4STLwTMoq3qrt0Gbb3i+E3WgsRjKgUBZyEsr1fnjXKzjaQNF6Ws 0ALcvBOyGtDxtZRuIR7oGU6RIfPsCuI8auHhCNUPij/Ddl8if0kdRiIi4xtmUh1CrSgl gLxaEgTIJYrlZ7uvyxv8L+qpcwaaPXjVHVMc3P6Qur0pT7AJ6KzqlZqQfeIP0U3zY+xX fTSk11fZxds49Rozt8wZktf4f9d1kMA+8/CK4kQvXKODTFSWXjft+HRpzjxjTwgr0KRI HVqw== X-Gm-Message-State: ALoCoQn6BzD71Yqfk+Zxv5qXKPmBmIo21oSNzpCcMRTwYIeegAxqx08VRJ5W7JqfyDA5jJ+pYf1l MIME-Version: 1.0 X-Received: by 10.180.78.166 with SMTP id c6mr21150419wix.8.1439344002764; Tue, 11 Aug 2015 18:46:42 -0700 (PDT) Sender: sobomax@sippysoft.com Received: by 10.27.143.15 with HTTP; Tue, 11 Aug 2015 18:46:42 -0700 (PDT) In-Reply-To: <1931183673.3234634.1439342909940.JavaMail.yahoo@mail.yahoo.com> References: <1931183673.3234634.1439342909940.JavaMail.yahoo@mail.yahoo.com> Date: Tue, 11 Aug 2015 18:46:42 -0700 X-Google-Sender-Auth: vYWQutNBf8V6QTWySNnYWFiUeYc Message-ID: Subject: Re: Poor high-PPS performance of the 10G ixgbe(9) NIC/driver in FreeBSD 10.1 From: Maxim Sobolev To: Barney Cordoba Cc: =?UTF-8?Q?Olivier_Cochard=2DLabb=C3=A9?= , FreeBSD Net , "freebsd@intel.com" , =?UTF-8?Q?Jev_Bj=C3=B6rsell?= Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 12 Aug 2015 01:46:46 -0000 Thanks Barney for totally useless response and an attempted insult! And yes, we are hitting CPU limit on 12 core E5-2620 v2 systems running I350, so yes, we do know a little bit how to distribute our application at least with the igb. For some reason this does not work with ixgb and we are trying to understand why is it so. http://sobomax.sippysoft.com/ScreenShot388.png On Tue, Aug 11, 2015 at 6:28 PM, Barney Cordoba wrote: > Wow, this is really important! if this is a college project, I give you a > D. Maybe a D- because it's almost useless information. > > You ignore the most important aspect of "performance". Efficiency is > arguably the most important aspect of performance. > > 1M pps at 20% cpu usage is much better "performance" than 1.2M pps at 85%= . > > Why don't any of you understand this simple thing? Why does spreading > equality really matter, unless you are hitting a wall with your cpus? I > don't care which cpu processes which packet. If you weren't doing moronic > things like binding to a cpu, then you'd never have to care about > distribution unless it was extremely unbalanced. > > BC > > > > On Tuesday, August 11, 2015 7:15 PM, Olivier Cochard-Labb=C3=A9 < > olivier@cochard.me> wrote: > > > On Tue, Aug 11, 2015 at 11:18 PM, Maxim Sobolev > wrote: > > > Hi folks, > > > > =E2=80=8BHi, > =E2=80=8B > > > > We've trying to migrate some of our high-PPS systems to a new hardware > that > > has four X540-AT2 10G NICs and observed that interrupt time goes throug= h > > roof after we cross around 200K PPS in and 200K out (two ports in LACP)= . > > The previous hardware was stable up to about 350K PPS in and 350K out. = I > > believe the old one was equipped with the I350 and had the identical LA= CP > > configuration. The new box also has better CPU with more cores (i.e. 24 > > cores vs. 16 cores before). CPU itself is 2 x E5-2690 v3. > > > > =E2=80=8B200K PPS, and even 350K PPS are very low value indeed. > On a Intel Xeon L5630 (4 cores only) with one X540-AT2=E2=80=8B > > =E2=80=8B(then 2 10Gigabit ports)=E2=80=8B I've reached about 1.8Mpps (fa= stforwarding > enabled) [1]. > But my setup didn't use lagg(4): Can you disable lagg configuration and > re-measure your performance without lagg ? > > Do you let Intel NIC drivers using 8 queues for port too? > In my use case (forwarding smallest UDP packet size), I obtain better > behaviour by limiting NIC queues to 4 (hw.ix.num_queues or > hw.ixgbe.num_queues, don't remember) if my system had 8 cores. And this > with Gigabit Intel[2] or Chelsio NIC [3]. > > Don't forget to disable TSO and LRO too. > > =E2=80=8BRegards, > > Olivier > > [1] > > http://bsdrp.net/documentation/examples/forwarding_performance_lab_of_an_= ibm_system_x3550_m3_with_10-gigabit_intel_x540-at2#graphs > [2] > > http://bsdrp.net/documentation/examples/forwarding_performance_lab_of_a_s= uperserver_5018a-ftn4#graph1 > [3] > > http://bsdrp.net/documentation/examples/forwarding_performance_lab_of_a_h= p_proliant_dl360p_gen8_with_10-gigabit_with_10-gigabit_chelsio_t540-cr#redu= cing_nic_queues > > _______________________________________________ > freebsd-net@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" > > > From owner-freebsd-net@freebsd.org Wed Aug 12 02:05:18 2015 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 014E999EECD for ; Wed, 12 Aug 2015 02:05:18 +0000 (UTC) (envelope-from sobomax@sippysoft.com) Received: from mail-wi0-f174.google.com (mail-wi0-f174.google.com [209.85.212.174]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 946FB6AA for ; Wed, 12 Aug 2015 02:05:17 +0000 (UTC) (envelope-from sobomax@sippysoft.com) Received: by wijp15 with SMTP id p15so198945495wij.0 for ; Tue, 11 Aug 2015 19:05:15 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:sender:in-reply-to:references:date :message-id:subject:from:to:content-type; bh=XfDNmOeptl6UJkEU7WV7YKyjpOcBs2jLYR7prUlCw7Y=; b=CEp2L8Gh2H+G/Ngd8VqnTpK0Of0h9K2wEZeqorj/kzfQF2kqgyCnlb+wMaCTXmJZYg X+SJ1xD8K0jDXl23MpJuPii8N/I1SKOjKTXqAnLdChltRYQOzA6UEPwd2CiFXfg7Qv48 Jb4nR28wSXXTSfQngOYsDF2GtghVp2ctUByxGu7d4k+o2E4LUPcw+VWnUiAIDKwoukW+ rfOO9e/Q4WgizGdq47gQQkCFrnuyNvfDcC4IGzI0/cN85dZ8C6hMo8TOFLGiKJSygpZ2 nmNEr7HmXDJOyXaOSnGbAvplpMYudQ0rqZQa5Lld7oRGhdCw8ng+9jz2k/0gMNbki0xw zxTg== X-Gm-Message-State: ALoCoQmS1WEEHzK0/pCzkYXJUmFmj1ce9MXO5UbogZ+PTyOET4aSpfPS7YbKtdP10kLL2v53vxK6 MIME-Version: 1.0 X-Received: by 10.194.78.164 with SMTP id c4mr59995431wjx.65.1439345115412; Tue, 11 Aug 2015 19:05:15 -0700 (PDT) Sender: sobomax@sippysoft.com Received: by 10.27.143.15 with HTTP; Tue, 11 Aug 2015 19:05:15 -0700 (PDT) In-Reply-To: References: Date: Tue, 11 Aug 2015 19:05:15 -0700 X-Google-Sender-Auth: vBPMlHX7RrNBcy8dEZclZvM-w50 Message-ID: Subject: Re: Poor high-PPS performance of the 10G ixgbe(9) NIC/driver in FreeBSD 10.1 From: Maxim Sobolev To: Luigi Rizzo , FreeBSD Net , =?UTF-8?Q?Jev_Bj=C3=B6rsell?= Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 12 Aug 2015 02:05:18 -0000 OK, so following Luigi's suggestion we've re-enabled AIM, set max_interrupt_rate to 8000 (matching igb), and reduced number of queues to 6. We'll have next peak in about 14 hours, I'll try to capture and record history of the per-queue interrupt rate. It still remains somewhat puzzling why something that worked with old-gen hardware/driver almost out of the box now needs some extra tuning. Thanks everyone for useful suggestions in any case! On Tue, Aug 11, 2015 at 4:04 PM, Luigi Rizzo wrote: > On Wed, Aug 12, 2015 at 12:46 AM, Maxim Sobolev > wrote: > > Hi Luigi, thanks for the input. Are you sure about the number of queues? > > Keep in mind we are not using netmap or anything as advanced, so we need > lot > > of parallel pipes just to push packet through the whole UDP/IP kernel > stack. > > To make things worse, our app cannot use connected sockets at the > moment, so > > we are having worst case scenario wrt kernel overhead. Single thread > sender > > on that hardware is limited to only 225K PPS until it hits CPU limit. We > use > > 8 queues on other older servers with I350 NICs. My send-only testing > shows > > that the hardware peaks at about 5 concurrent threads for connected > sockets > > and 7 concurrent threads for unconnected. This is with default 8 hardware > > queues. The test is somewhat crippled since it only tests outgoing > traffic. > > > > thing is, you are using 8 queues per NIC so the total number of different > interrupts is huge and will likely eat all of your cores. > This is why i'd try to start low and ramp up later. > > > http://bit.ly/1MiLFHg > > > > With regards to the moderation, what effect it has on the traffic? Is NIC > > going to drop packets once it starts moderating interrupts? > > moderation only means you get larger batches on each interrupt, > which overall makes the system more efficient. The NIC will only drop > packets when the CPU is not fast enough to drain them,. > > (this said, it might be a good idea to try netmap, passing packets > that you do not know how to handle to the host stack, and instead > intercepting those that you can deal with in netmap. those > unconnected udp sockets are the ideal case for netmap.) > > cheers > luigi > From owner-freebsd-net@freebsd.org Wed Aug 12 04:33:39 2015 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 6F09F99F274 for ; Wed, 12 Aug 2015 04:33:39 +0000 (UTC) (envelope-from adrian.chadd@gmail.com) Received: from mail-io0-x22f.google.com (mail-io0-x22f.google.com [IPv6:2607:f8b0:4001:c06::22f]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 3A0C68F0; Wed, 12 Aug 2015 04:33:39 +0000 (UTC) (envelope-from adrian.chadd@gmail.com) Received: by iods203 with SMTP id s203so8872368iod.0; Tue, 11 Aug 2015 21:33:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=V83jphAQHKy5csMpZc9BMy4j+zrYDuGmryV4DbHuRYw=; b=N0Fxszv2hzKUU7rg62awgEePw3ADxkbR6IwUYcM6JoqFNiZRMBJ1FDZlyEoUN3i/EZ 04lji9d9YpHf4mwX6Uzoc1k5RZGtfuPNeN/9HTX99CrVJxqVvqavBxhEQ/OEWxQfh75k 3RpUldlHfgRy5hxAXjunxE1xuA3oKYU+Ph42/YDnoOAX9MWeYMCB8cP4rixkFqb5sYPl 56rFnJg9T6LLkMUADwErVG2ebrckNDLh1JI8JV90C41s0/ShQtxN73aNFBQ68LnWjgbR OWLuwlP5laA6XnxLjov9gcuf7mQ0hF4GLORi8CQhT6ShZ5sMMWkIQd1qYapzALTQcUQL eygA== MIME-Version: 1.0 X-Received: by 10.107.136.152 with SMTP id s24mr30225566ioi.165.1439354018660; Tue, 11 Aug 2015 21:33:38 -0700 (PDT) Received: by 10.36.38.133 with HTTP; Tue, 11 Aug 2015 21:33:38 -0700 (PDT) In-Reply-To: References: Date: Tue, 11 Aug 2015 21:33:38 -0700 Message-ID: Subject: Re: Poor high-PPS performance of the 10G ixgbe(9) NIC/driver in FreeBSD 10.1 From: Adrian Chadd To: Maxim Sobolev Cc: Luigi Rizzo , FreeBSD Net , =?UTF-8?Q?Jev_Bj=C3=B6rsell?= Content-Type: text/plain; charset=UTF-8 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 12 Aug 2015 04:33:39 -0000 Hi, Can you please do a pciconf -lv on the previous and current hardware? I wonder if it's something to do with a feature that is chipset dependent. (And please, disable flow-director on ixgbe on 10.1. Pretty please.) -a From owner-freebsd-net@freebsd.org Wed Aug 12 10:59:43 2015 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 6594E9A0C31 for ; Wed, 12 Aug 2015 10:59:43 +0000 (UTC) (envelope-from melifaro@ipfw.ru) Received: from forward12h.cmail.yandex.net (forward12h.cmail.yandex.net [IPv6:2a02:6b8:0:f35::9d]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "forwards.mail.yandex.net", Issuer "Certum Level IV CA" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id F192BD1C; Wed, 12 Aug 2015 10:59:42 +0000 (UTC) (envelope-from melifaro@ipfw.ru) Received: from web21h.yandex.ru (web21h.yandex.ru [IPv6:2a02:6b8:0:f05::31]) by forward12h.cmail.yandex.net (Yandex) with ESMTP id 1FC6D21679; Wed, 12 Aug 2015 13:59:28 +0300 (MSK) Received: from 127.0.0.1 (localhost [127.0.0.1]) by web21h.yandex.ru (Yandex) with ESMTP id DCC8B13611E4; Wed, 12 Aug 2015 13:59:26 +0300 (MSK) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ipfw.ru; s=mail; t=1439377167; bh=vVtTFUd1K26J3lr9zq1oNwwkNYlZ1k2dO+8J9LMD2Xc=; h=From:To:Cc:In-Reply-To:References:Subject:Date; b=OuNhzdJErIgtaypS4SrVIb4dF8c3btyhvcuiQ6g5OkFBurgq4whQU8BpzvizOz653 gs4PQK4KWQ+1kRuB8ET2JE0+06Nw5qCfLKNzoFJIZn1/7vLEl73MhjWgAhFc0lO5xt /2qo03IFxA2bVZ21Wj1mcWjus/sMXRStcBlVqK/4= Received: by web21h.yandex.ru with HTTP; Wed, 12 Aug 2015 13:59:24 +0300 From: Alexander V. Chernikov To: Maxim Sobolev , =?utf-8?B?T2xpdmllciBDb2NoYXJkLUxhYmLDqQ==?= Cc: FreeBSD Net , "freebsd@intel.com" , =?utf-8?B?SmV2IEJqw7Zyc2VsbA==?= In-Reply-To: References: null Subject: Re: Poor high-PPS performance of the 10G ixgbe(9) NIC/driver in FreeBSD 10.1 MIME-Version: 1.0 Message-Id: <77171439377164@web21h.yandex.ru> X-Mailer: Yamail [ http://yandex.ru ] 5.0 Date: Wed, 12 Aug 2015 13:59:24 +0300 Content-Transfer-Encoding: 8bit Content-Type: text/plain; charset=utf-8 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 12 Aug 2015 10:59:43 -0000 12.08.2015, 02:28, "Maxim Sobolev" : > Olivier, keep in mind that we are not "kernel forwarding" packets, but "app > forwarding", i.e. the packet goes full way > net->kernel->recvfrom->app->sendto->kernel->net, which is why we have much > lower PPS limits and which is why I think we are actually benefiting from > the extra queues. Single-thread sendto() in a loop is CPU-bound at about > 220K PPS, and while running the test I am observing that outbound traffic > from one thread is mapped into a specific queue (well, pair of queues on > two separate adaptors, due to lagg load balancing action). And the peak > performance of that test is at 7 threads, which I believe corresponds to > the number of queues. We have plenty of CPU cores in the box (24) with > HTT/SMT disabled and one CPU is mapped to a specific queue. This leaves us > with at least 8 CPUs fully capable of running our app. If you look at the > CPU utilization, we are at about 10% when the issue hits. In any case, it would be great if you could provide some profiling info since there could be plenty of problematic places starting from TX rings contention to some locks inside udp or even (in)famous random entropy harvester.. e.g. something like pmcstat -TS instructions -w1 might be sufficient to determine the reason > > ix0: port > 0x6020-0x603f mem 0xc7c00000-0xc7dfffff,0xc7e04000-0xc7e07fff irq 40 at > device 0.0 on pci3 > ix0: Using MSIX interrupts with 9 vectors > ix0: Bound queue 0 to cpu 0 > ix0: Bound queue 1 to cpu 1 > ix0: Bound queue 2 to cpu 2 > ix0: Bound queue 3 to cpu 3 > ix0: Bound queue 4 to cpu 4 > ix0: Bound queue 5 to cpu 5 > ix0: Bound queue 6 to cpu 6 > ix0: Bound queue 7 to cpu 7 > ix0: Ethernet address: 0c:c4:7a:5e:be:64 > ix0: PCI Express Bus: Speed 5.0GT/s Width x8 > 001.000008 [2705] netmap_attach success for ix0 tx 8/4096 rx > 8/4096 queues/slots > ix1: port > 0x6000-0x601f mem 0xc7a00000-0xc7bfffff,0xc7e00000-0xc7e03fff irq 44 at > device 0.1 on pci3 > ix1: Using MSIX interrupts with 9 vectors > ix1: Bound queue 0 to cpu 8 > ix1: Bound queue 1 to cpu 9 > ix1: Bound queue 2 to cpu 10 > ix1: Bound queue 3 to cpu 11 > ix1: Bound queue 4 to cpu 12 > ix1: Bound queue 5 to cpu 13 > ix1: Bound queue 6 to cpu 14 > ix1: Bound queue 7 to cpu 15 > ix1: Ethernet address: 0c:c4:7a:5e:be:65 > ix1: PCI Express Bus: Speed 5.0GT/s Width x8 > 001.000009 [2705] netmap_attach success for ix1 tx 8/4096 rx > 8/4096 queues/slots > > On Tue, Aug 11, 2015 at 4:14 PM, Olivier Cochard-Labbé > wrote: > >>  On Tue, Aug 11, 2015 at 11:18 PM, Maxim Sobolev >>  wrote: >> >>>  Hi folks, >>> >>>  ​Hi, >>  ​ >> >>>  We've trying to migrate some of our high-PPS systems to a new hardware >>>  that >>>  has four X540-AT2 10G NICs and observed that interrupt time goes through >>>  roof after we cross around 200K PPS in and 200K out (two ports in LACP). >>>  The previous hardware was stable up to about 350K PPS in and 350K out. I >>>  believe the old one was equipped with the I350 and had the identical LACP >>>  configuration. The new box also has better CPU with more cores (i.e. 24 >>>  cores vs. 16 cores before). CPU itself is 2 x E5-2690 v3. >> >>  ​200K PPS, and even 350K PPS are very low value indeed. >>  On a Intel Xeon L5630 (4 cores only) with one X540-AT2​ >> >>  ​(then 2 10Gigabit ports)​ I've reached about 1.8Mpps (fastforwarding >>  enabled) [1]. >>  But my setup didn't use lagg(4): Can you disable lagg configuration and >>  re-measure your performance without lagg ? >> >>  Do you let Intel NIC drivers using 8 queues for port too? >>  In my use case (forwarding smallest UDP packet size), I obtain better >>  behaviour by limiting NIC queues to 4 (hw.ix.num_queues or >>  hw.ixgbe.num_queues, don't remember) if my system had 8 cores. And this >>  with Gigabit Intel[2] or Chelsio NIC [3]. >> >>  Don't forget to disable TSO and LRO too. >> >>  ​Regards, >> >>  Olivier >> >>  [1] >>  http://bsdrp.net/documentation/examples/forwarding_performance_lab_of_an_ibm_system_x3550_m3_with_10-gigabit_intel_x540-at2#graphs >>  [2] >>  http://bsdrp.net/documentation/examples/forwarding_performance_lab_of_a_superserver_5018a-ftn4#graph1 >>  [3] >>  http://bsdrp.net/documentation/examples/forwarding_performance_lab_of_a_hp_proliant_dl360p_gen8_with_10-gigabit_with_10-gigabit_chelsio_t540-cr#reducing_nic_queues > > _______________________________________________ > freebsd-net@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" From owner-freebsd-net@freebsd.org Wed Aug 12 11:42:08 2015 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id D805199F8C9 for ; Wed, 12 Aug 2015 11:42:08 +0000 (UTC) (envelope-from farrokhi@FreeBSD.org) Received: from mail.imenpardis.com (mail.imenpardis.com [188.121.128.150]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 2B0FD1678; Wed, 12 Aug 2015 11:42:07 +0000 (UTC) (envelope-from farrokhi@FreeBSD.org) Received: from shark.local (unknown [79.127.116.130]) (using TLSv1 with cipher ECDHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) (Authenticated sender: freebsd@farrokhi.net) by mail.imenpardis.com (Postfix) with ESMTPSA id 3F90F423046; Wed, 12 Aug 2015 16:03:46 +0430 (IRDT) Message-ID: <55CB2F18.40902@FreeBSD.org> Date: Wed, 12 Aug 2015 16:03:44 +0430 From: Babak Farrokhi User-Agent: Postbox 4.0.3 (Macintosh/20150805) MIME-Version: 1.0 To: "Alexander V. Chernikov" CC: Maxim Sobolev , =?UTF-8?B?T2xpdmllciBDb2NoYXJkLUw=?= =?UTF-8?B?YWJiw6k=?= , FreeBSD Net , "freebsd@intel.com" , =?UTF-8?B?SmV2IEJqw7Zyc2VsbA==?= Subject: Re: Poor high-PPS performance of the 10G ixgbe(9) NIC/driver in FreeBSD 10.1 References: null <77171439377164@web21h.yandex.ru> In-Reply-To: <77171439377164@web21h.yandex.ru> X-Enigmail-Version: 1.2.3 OpenPGP: url=http://farrokhi.net/babak-farrokhi-pgp.asc Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 12 Aug 2015 11:42:09 -0000 I ran into the same problem with almost the same hardware (Intel X520) on 10-STABLE. HT/SMT is disabled and cards are configured with 8 queues, with the same sysctl tunings as sobomax@ did. I am not using lagg, no FLOWTABLE. I experimented with pmcstat (RESOURCE_STALLS) a while ago and here [1] [2] you can see the results, including pmc output, callchain, flamegraph and gprof output. I am experiencing huge number of interrupts with 200kpps load: # sysctl dev.ix | grep interrupt_rate dev.ix.1.queue7.interrupt_rate: 125000 dev.ix.1.queue6.interrupt_rate: 6329 dev.ix.1.queue5.interrupt_rate: 500000 dev.ix.1.queue4.interrupt_rate: 100000 dev.ix.1.queue3.interrupt_rate: 50000 dev.ix.1.queue2.interrupt_rate: 500000 dev.ix.1.queue1.interrupt_rate: 500000 dev.ix.1.queue0.interrupt_rate: 100000 dev.ix.0.queue7.interrupt_rate: 500000 dev.ix.0.queue6.interrupt_rate: 6097 dev.ix.0.queue5.interrupt_rate: 10204 dev.ix.0.queue4.interrupt_rate: 5208 dev.ix.0.queue3.interrupt_rate: 5208 dev.ix.0.queue2.interrupt_rate: 71428 dev.ix.0.queue1.interrupt_rate: 5494 dev.ix.0.queue0.interrupt_rate: 6250 [1] http://farrokhi.net/~farrokhi/pmc/6/ [2] http://farrokhi.net/~farrokhi/pmc/7/ Regards, Babak Alexander V. Chernikov wrote: > 12.08.2015, 02:28, "Maxim Sobolev" : >> Olivier, keep in mind that we are not "kernel forwarding" packets, but "app >> forwarding", i.e. the packet goes full way >> net->kernel->recvfrom->app->sendto->kernel->net, which is why we have much >> lower PPS limits and which is why I think we are actually benefiting from >> the extra queues. Single-thread sendto() in a loop is CPU-bound at about >> 220K PPS, and while running the test I am observing that outbound traffic >> from one thread is mapped into a specific queue (well, pair of queues on >> two separate adaptors, due to lagg load balancing action). And the peak >> performance of that test is at 7 threads, which I believe corresponds to >> the number of queues. We have plenty of CPU cores in the box (24) with >> HTT/SMT disabled and one CPU is mapped to a specific queue. This leaves us >> with at least 8 CPUs fully capable of running our app. If you look at the >> CPU utilization, we are at about 10% when the issue hits. > > In any case, it would be great if you could provide some profiling info since there could be > plenty of problematic places starting from TX rings contention to some locks inside udp or even > (in)famous random entropy harvester.. > e.g. something like pmcstat -TS instructions -w1 might be sufficient to determine the reason >> ix0: port >> 0x6020-0x603f mem 0xc7c00000-0xc7dfffff,0xc7e04000-0xc7e07fff irq 40 at >> device 0.0 on pci3 >> ix0: Using MSIX interrupts with 9 vectors >> ix0: Bound queue 0 to cpu 0 >> ix0: Bound queue 1 to cpu 1 >> ix0: Bound queue 2 to cpu 2 >> ix0: Bound queue 3 to cpu 3 >> ix0: Bound queue 4 to cpu 4 >> ix0: Bound queue 5 to cpu 5 >> ix0: Bound queue 6 to cpu 6 >> ix0: Bound queue 7 to cpu 7 >> ix0: Ethernet address: 0c:c4:7a:5e:be:64 >> ix0: PCI Express Bus: Speed 5.0GT/s Width x8 >> 001.000008 [2705] netmap_attach success for ix0 tx 8/4096 rx >> 8/4096 queues/slots >> ix1: port >> 0x6000-0x601f mem 0xc7a00000-0xc7bfffff,0xc7e00000-0xc7e03fff irq 44 at >> device 0.1 on pci3 >> ix1: Using MSIX interrupts with 9 vectors >> ix1: Bound queue 0 to cpu 8 >> ix1: Bound queue 1 to cpu 9 >> ix1: Bound queue 2 to cpu 10 >> ix1: Bound queue 3 to cpu 11 >> ix1: Bound queue 4 to cpu 12 >> ix1: Bound queue 5 to cpu 13 >> ix1: Bound queue 6 to cpu 14 >> ix1: Bound queue 7 to cpu 15 >> ix1: Ethernet address: 0c:c4:7a:5e:be:65 >> ix1: PCI Express Bus: Speed 5.0GT/s Width x8 >> 001.000009 [2705] netmap_attach success for ix1 tx 8/4096 rx >> 8/4096 queues/slots >> >> On Tue, Aug 11, 2015 at 4:14 PM, Olivier Cochard-Labbé >> wrote: >> >>> On Tue, Aug 11, 2015 at 11:18 PM, Maxim Sobolev >>> wrote: >>> >>>> Hi folks, >>>> >>>> ​Hi, >>> ​ >>> >>>> We've trying to migrate some of our high-PPS systems to a new hardware >>>> that >>>> has four X540-AT2 10G NICs and observed that interrupt time goes through >>>> roof after we cross around 200K PPS in and 200K out (two ports in LACP). >>>> The previous hardware was stable up to about 350K PPS in and 350K out. I >>>> believe the old one was equipped with the I350 and had the identical LACP >>>> configuration. The new box also has better CPU with more cores (i.e. 24 >>>> cores vs. 16 cores before). CPU itself is 2 x E5-2690 v3. >>> ​200K PPS, and even 350K PPS are very low value indeed. >>> On a Intel Xeon L5630 (4 cores only) with one X540-AT2​ >>> >>> ​(then 2 10Gigabit ports)​ I've reached about 1.8Mpps (fastforwarding >>> enabled) [1]. >>> But my setup didn't use lagg(4): Can you disable lagg configuration and >>> re-measure your performance without lagg ? >>> >>> Do you let Intel NIC drivers using 8 queues for port too? >>> In my use case (forwarding smallest UDP packet size), I obtain better >>> behaviour by limiting NIC queues to 4 (hw.ix.num_queues or >>> hw.ixgbe.num_queues, don't remember) if my system had 8 cores. And this >>> with Gigabit Intel[2] or Chelsio NIC [3]. >>> >>> Don't forget to disable TSO and LRO too. >>> >>> ​Regards, >>> >>> Olivier >>> >>> [1] >>> http://bsdrp.net/documentation/examples/forwarding_performance_lab_of_an_ibm_system_x3550_m3_with_10-gigabit_intel_x540-at2#graphs >>> [2] >>> http://bsdrp.net/documentation/examples/forwarding_performance_lab_of_a_superserver_5018a-ftn4#graph1 >>> [3] >>> http://bsdrp.net/documentation/examples/forwarding_performance_lab_of_a_hp_proliant_dl360p_gen8_with_10-gigabit_with_10-gigabit_chelsio_t540-cr#reducing_nic_queues >> _______________________________________________ >> freebsd-net@freebsd.org mailing list >> http://lists.freebsd.org/mailman/listinfo/freebsd-net >> To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" > _______________________________________________ > freebsd-net@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" From owner-freebsd-net@freebsd.org Wed Aug 12 12:23:24 2015 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id EDA4B9A06DC for ; Wed, 12 Aug 2015 12:23:24 +0000 (UTC) (envelope-from rizzo.unipi@gmail.com) Received: from mail-lb0-x236.google.com (mail-lb0-x236.google.com [IPv6:2a00:1450:4010:c04::236]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 58C8EB0A; Wed, 12 Aug 2015 12:23:24 +0000 (UTC) (envelope-from rizzo.unipi@gmail.com) Received: by lbbtg9 with SMTP id tg9so8464487lbb.1; Wed, 12 Aug 2015 05:23:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=l5u8/rcE16jzFU2+wNsAk/Xi9c5OuZkvUbiQio7VxXc=; b=CberHlvGX22JU3rTO4xGPg29FTB7rdAC4GYrsq1jNnl/xefIYgeS9Q6jlljz9it0Y2 gekxhhupClaTaFQWjMKfQLOPL4EMngBoj+rkDJqg1+2ybOYKawFUqrjrA6DzAj0XpdOH dtlNlQySHry2tIOTPska+6peSO2KppOuwev1t3kfWiSlBWNj1YMw7tbKAwpBFQReXCKF VZ4ukSmXEnCO8Y2+PrccByFDL0MdOpWcbLgembP37D5o5zmWfg1J92X0P1TPuF8cDX6O 0gnPj0aFY2SLdLoLWP7FW1HnRWv/o9QBKLxdDzcKRKu9fnGOX9dxRmmHFjomnY5MetzA 6FnQ== MIME-Version: 1.0 X-Received: by 10.152.5.65 with SMTP id q1mr29080605laq.83.1439382202040; Wed, 12 Aug 2015 05:23:22 -0700 (PDT) Sender: rizzo.unipi@gmail.com Received: by 10.114.172.140 with HTTP; Wed, 12 Aug 2015 05:23:21 -0700 (PDT) In-Reply-To: <55CB2F18.40902@FreeBSD.org> References: <77171439377164@web21h.yandex.ru> <55CB2F18.40902@FreeBSD.org> Date: Wed, 12 Aug 2015 14:23:21 +0200 X-Google-Sender-Auth: L56WAbDWp4JIf9iDaDcF2KkwWm8 Message-ID: Subject: Re: Poor high-PPS performance of the 10G ixgbe(9) NIC/driver in FreeBSD 10.1 From: Luigi Rizzo To: Babak Farrokhi Cc: "Alexander V. Chernikov" , =?UTF-8?Q?Olivier_Cochard=2DLabb=C3=A9?= , Maxim Sobolev , "freebsd@intel.com" , =?UTF-8?Q?Jev_Bj=C3=B6rsell?= , FreeBSD Net Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 12 Aug 2015 12:23:25 -0000 As I was telling to maxim, you should disable aim because it only matches the max interrupt rate to the average packet size, which is the last thing you want. Setting the interrupt rate with sysctl (one per queue) gives you precise control on the max rate and (hence, extra latency). 20k interrupts/s give you 50us of latency, and the 2k slots in the queue are still enough to absorb a burst of min-sized frames hitting a single queue (the os will start dropping long before that level, but that's another story). Cheers Luigi On Wednesday, August 12, 2015, Babak Farrokhi wrote: > I ran into the same problem with almost the same hardware (Intel X520) > on 10-STABLE. HT/SMT is disabled and cards are configured with 8 queues, > with the same sysctl tunings as sobomax@ did. I am not using lagg, no > FLOWTABLE. > > I experimented with pmcstat (RESOURCE_STALLS) a while ago and here [1] > [2] you can see the results, including pmc output, callchain, flamegraph > and gprof output. > > I am experiencing huge number of interrupts with 200kpps load: > > # sysctl dev.ix | grep interrupt_rate > dev.ix.1.queue7.interrupt_rate: 125000 > dev.ix.1.queue6.interrupt_rate: 6329 > dev.ix.1.queue5.interrupt_rate: 500000 > dev.ix.1.queue4.interrupt_rate: 100000 > dev.ix.1.queue3.interrupt_rate: 50000 > dev.ix.1.queue2.interrupt_rate: 500000 > dev.ix.1.queue1.interrupt_rate: 500000 > dev.ix.1.queue0.interrupt_rate: 100000 > dev.ix.0.queue7.interrupt_rate: 500000 > dev.ix.0.queue6.interrupt_rate: 6097 > dev.ix.0.queue5.interrupt_rate: 10204 > dev.ix.0.queue4.interrupt_rate: 5208 > dev.ix.0.queue3.interrupt_rate: 5208 > dev.ix.0.queue2.interrupt_rate: 71428 > dev.ix.0.queue1.interrupt_rate: 5494 > dev.ix.0.queue0.interrupt_rate: 6250 > > [1] http://farrokhi.net/~farrokhi/pmc/6/ > [2] http://farrokhi.net/~farrokhi/pmc/7/ > > Regards, > Babak > > > Alexander V. Chernikov wrote: > > 12.08.2015, 02:28, "Maxim Sobolev" : > >> Olivier, keep in mind that we are not "kernel forwarding" packets, but > "app > >> forwarding", i.e. the packet goes full way > >> net->kernel->recvfrom->app->sendto->kernel->net, which is why we have > much > >> lower PPS limits and which is why I think we are actually benefiting > from > >> the extra queues. Single-thread sendto() in a loop is CPU-bound at abo= ut > >> 220K PPS, and while running the test I am observing that outbound > traffic > >> from one thread is mapped into a specific queue (well, pair of queues = on > >> two separate adaptors, due to lagg load balancing action). And the pea= k > >> performance of that test is at 7 threads, which I believe corresponds = to > >> the number of queues. We have plenty of CPU cores in the box (24) with > >> HTT/SMT disabled and one CPU is mapped to a specific queue. This leave= s > us > >> with at least 8 CPUs fully capable of running our app. If you look at > the > >> CPU utilization, we are at about 10% when the issue hits. > > > > In any case, it would be great if you could provide some profiling info > since there could be > > plenty of problematic places starting from TX rings contention to some > locks inside udp or even > > (in)famous random entropy harvester.. > > e.g. something like pmcstat -TS instructions -w1 might be sufficient to > determine the reason > >> ix0: > port > >> 0x6020-0x603f mem 0xc7c00000-0xc7dfffff,0xc7e04000-0xc7e07fff irq 40 a= t > >> device 0.0 on pci3 > >> ix0: Using MSIX interrupts with 9 vectors > >> ix0: Bound queue 0 to cpu 0 > >> ix0: Bound queue 1 to cpu 1 > >> ix0: Bound queue 2 to cpu 2 > >> ix0: Bound queue 3 to cpu 3 > >> ix0: Bound queue 4 to cpu 4 > >> ix0: Bound queue 5 to cpu 5 > >> ix0: Bound queue 6 to cpu 6 > >> ix0: Bound queue 7 to cpu 7 > >> ix0: Ethernet address: 0c:c4:7a:5e:be:64 > >> ix0: PCI Express Bus: Speed 5.0GT/s Width x8 > >> 001.000008 [2705] netmap_attach success for ix0 tx 8/4096 rx > >> 8/4096 queues/slots > >> ix1: > port > >> 0x6000-0x601f mem 0xc7a00000-0xc7bfffff,0xc7e00000-0xc7e03fff irq 44 a= t > >> device 0.1 on pci3 > >> ix1: Using MSIX interrupts with 9 vectors > >> ix1: Bound queue 0 to cpu 8 > >> ix1: Bound queue 1 to cpu 9 > >> ix1: Bound queue 2 to cpu 10 > >> ix1: Bound queue 3 to cpu 11 > >> ix1: Bound queue 4 to cpu 12 > >> ix1: Bound queue 5 to cpu 13 > >> ix1: Bound queue 6 to cpu 14 > >> ix1: Bound queue 7 to cpu 15 > >> ix1: Ethernet address: 0c:c4:7a:5e:be:65 > >> ix1: PCI Express Bus: Speed 5.0GT/s Width x8 > >> 001.000009 [2705] netmap_attach success for ix1 tx 8/4096 rx > >> 8/4096 queues/slots > >> > >> On Tue, Aug 11, 2015 at 4:14 PM, Olivier Cochard-Labb=C3=A9 < > olivier@cochard.me > > >> wrote: > >> > >>> On Tue, Aug 11, 2015 at 11:18 PM, Maxim Sobolev > > >>> wrote: > >>> > >>>> Hi folks, > >>>> > >>>> =E2=80=8BHi, > >>> =E2=80=8B > >>> > >>>> We've trying to migrate some of our high-PPS systems to a new > hardware > >>>> that > >>>> has four X540-AT2 10G NICs and observed that interrupt time goes > through > >>>> roof after we cross around 200K PPS in and 200K out (two ports in > LACP). > >>>> The previous hardware was stable up to about 350K PPS in and 350K > out. I > >>>> believe the old one was equipped with the I350 and had the identica= l > LACP > >>>> configuration. The new box also has better CPU with more cores (i.e= . > 24 > >>>> cores vs. 16 cores before). CPU itself is 2 x E5-2690 v3. > >>> =E2=80=8B200K PPS, and even 350K PPS are very low value indeed. > >>> On a Intel Xeon L5630 (4 cores only) with one X540-AT2=E2=80=8B > >>> > >>> =E2=80=8B(then 2 10Gigabit ports)=E2=80=8B I've reached about 1.8Mpp= s (fastforwarding > >>> enabled) [1]. > >>> But my setup didn't use lagg(4): Can you disable lagg configuration > and > >>> re-measure your performance without lagg ? > >>> > >>> Do you let Intel NIC drivers using 8 queues for port too? > >>> In my use case (forwarding smallest UDP packet size), I obtain bette= r > >>> behaviour by limiting NIC queues to 4 (hw.ix.num_queues or > >>> hw.ixgbe.num_queues, don't remember) if my system had 8 cores. And > this > >>> with Gigabit Intel[2] or Chelsio NIC [3]. > >>> > >>> Don't forget to disable TSO and LRO too. > >>> > >>> =E2=80=8BRegards, > >>> > >>> Olivier > >>> > >>> [1] > >>> > http://bsdrp.net/documentation/examples/forwarding_performance_lab_of_an_= ibm_system_x3550_m3_with_10-gigabit_intel_x540-at2#graphs > >>> [2] > >>> > http://bsdrp.net/documentation/examples/forwarding_performance_lab_of_a_s= uperserver_5018a-ftn4#graph1 > >>> [3] > >>> > http://bsdrp.net/documentation/examples/forwarding_performance_lab_of_a_h= p_proliant_dl360p_gen8_with_10-gigabit_with_10-gigabit_chelsio_t540-cr#redu= cing_nic_queues > >> _______________________________________________ > >> freebsd-net@freebsd.org mailing list > >> http://lists.freebsd.org/mailman/listinfo/freebsd-net > >> To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org > " > > _______________________________________________ > > freebsd-net@freebsd.org mailing list > > http://lists.freebsd.org/mailman/listinfo/freebsd-net > > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org > " > _______________________________________________ > freebsd-net@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org > " --=20 -----------------------------------------+------------------------------- Prof. Luigi RIZZO, rizzo@iet.unipi.it . Dip. di Ing. dell'Informazione http://www.iet.unipi.it/~luigi/ . Universita` di Pisa TEL +39-050-2217533 . via Diotisalvi 2 Mobile +39-338-6809875 . 56122 PISA (Italy) -----------------------------------------+------------------------------- From owner-freebsd-net@freebsd.org Wed Aug 12 15:03:29 2015 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 03C2C99F292 for ; Wed, 12 Aug 2015 15:03:29 +0000 (UTC) (envelope-from sobomax@sippysoft.com) Received: from mail-wi0-f181.google.com (mail-wi0-f181.google.com [209.85.212.181]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 873BF7B3 for ; Wed, 12 Aug 2015 15:03:28 +0000 (UTC) (envelope-from sobomax@sippysoft.com) Received: by wicne3 with SMTP id ne3so222294965wic.1 for ; Wed, 12 Aug 2015 08:03:26 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc:content-type; bh=58rgGNy+zT2P/bfuHTV0JSN6rJh5ab4hocLHPySWM+k=; b=W8NEfRZtNYhAOLx2HGGiD5DnwJ/oOSHREKh8DJ0///hWV5WjrTJQ/OKYSmnoW1110P WOblJ4GJUn6LLpN1sKWuoELTKp2YerbBwCYGghgW8jVrUpwJuTGRafhmKGZirMSZGkRn y6aFj/HMJtXlzzAiYktmpCA0NbRQELqu3+l9XGmlRO8obpYBEL3vUr6VDCsI/JJ+Hvyk gWm3L8b1W16QrcW+wz8iDiBXkIGGgiqGqBaVKImcB03WkCJ1aZdtcdGMFHVjVGYNVVbf Fv0nmgoURb03DL7x1kPam3wg5K6N544MHeb6b4a7q3n9vaGEOo6WyGDmV+3MWw2t72il Hgzw== X-Gm-Message-State: ALoCoQkaNkvyab8WFYCSHz1CKrygP9T60iYQO2DnFXx93JikgsU/SXqsAureVYpZ1XRPcVsE2yDi MIME-Version: 1.0 X-Received: by 10.180.78.166 with SMTP id c6mr26925969wix.8.1439391805929; Wed, 12 Aug 2015 08:03:25 -0700 (PDT) Received: by 10.27.143.15 with HTTP; Wed, 12 Aug 2015 08:03:25 -0700 (PDT) In-Reply-To: References: <77171439377164@web21h.yandex.ru> <55CB2F18.40902@FreeBSD.org> Date: Wed, 12 Aug 2015 08:03:25 -0700 Message-ID: Subject: Re: Poor high-PPS performance of the 10G ixgbe(9) NIC/driver in FreeBSD 10.1 From: Maxim Sobolev To: Luigi Rizzo Cc: Babak Farrokhi , "Alexander V. Chernikov" , =?UTF-8?Q?Olivier_Cochard=2DLabb=C3=A9?= , "freebsd@intel.com" , =?UTF-8?Q?Jev_Bj=C3=B6rsell?= , FreeBSD Net Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 12 Aug 2015 15:03:29 -0000 Ok, so my current settings are: hw.ix.max_interrupt_rate: 20000 dev.ix.0.queue0.interrupt_rate: 20000 dev.ix.0.queue1.interrupt_rate: 20000 dev.ix.0.queue2.interrupt_rate: 20000 dev.ix.0.queue3.interrupt_rate: 20000 dev.ix.0.queue4.interrupt_rate: 20000 dev.ix.0.queue5.interrupt_rate: 20000 dev.ix.1.queue0.interrupt_rate: 20000 dev.ix.1.queue1.interrupt_rate: 20000 dev.ix.1.queue2.interrupt_rate: 20000 dev.ix.1.queue3.interrupt_rate: 20000 dev.ix.1.queue4.interrupt_rate: 20000 dev.ix.1.queue5.interrupt_rate: 20000 dev.ix.0.enable_aim: 0 dev.ix.1.enable_aim: 0 dev.ix.2.enable_aim: 0 dev.ix.3.enable_aim: 0 hw.ix.num_queues:6 We also happen to have I210-based system with only 4 hardware queues, it would be interesting to see how it stacks up. On Wed, Aug 12, 2015 at 5:23 AM, Luigi Rizzo wrote: > As I was telling to maxim, you should disable aim because it only matches > the max interrupt rate to the average packet size, which is the last thin= g > you want. > > Setting the interrupt rate with sysctl (one per queue) gives you precise > control on the max rate and (hence, extra latency). 20k interrupts/s give > you 50us of latency, and the 2k slots in the queue are still enough to > absorb a burst of min-sized frames hitting a single queue (the os will > start dropping long before that level, but that's another story). > > Cheers > Luigi > > On Wednesday, August 12, 2015, Babak Farrokhi > wrote: > >> I ran into the same problem with almost the same hardware (Intel X520) >> on 10-STABLE. HT/SMT is disabled and cards are configured with 8 queues, >> with the same sysctl tunings as sobomax@ did. I am not using lagg, no >> FLOWTABLE. >> >> I experimented with pmcstat (RESOURCE_STALLS) a while ago and here [1] >> [2] you can see the results, including pmc output, callchain, flamegraph >> and gprof output. >> >> I am experiencing huge number of interrupts with 200kpps load: >> >> # sysctl dev.ix | grep interrupt_rate >> dev.ix.1.queue7.interrupt_rate: 125000 >> dev.ix.1.queue6.interrupt_rate: 6329 >> dev.ix.1.queue5.interrupt_rate: 500000 >> dev.ix.1.queue4.interrupt_rate: 100000 >> dev.ix.1.queue3.interrupt_rate: 50000 >> dev.ix.1.queue2.interrupt_rate: 500000 >> dev.ix.1.queue1.interrupt_rate: 500000 >> dev.ix.1.queue0.interrupt_rate: 100000 >> dev.ix.0.queue7.interrupt_rate: 500000 >> dev.ix.0.queue6.interrupt_rate: 6097 >> dev.ix.0.queue5.interrupt_rate: 10204 >> dev.ix.0.queue4.interrupt_rate: 5208 >> dev.ix.0.queue3.interrupt_rate: 5208 >> dev.ix.0.queue2.interrupt_rate: 71428 >> dev.ix.0.queue1.interrupt_rate: 5494 >> dev.ix.0.queue0.interrupt_rate: 6250 >> >> [1] http://farrokhi.net/~farrokhi/pmc/6/ >> [2] http://farrokhi.net/~farrokhi/pmc/7/ >> >> Regards, >> Babak >> >> >> Alexander V. Chernikov wrote: >> > 12.08.2015, 02:28, "Maxim Sobolev" : >> >> Olivier, keep in mind that we are not "kernel forwarding" packets, bu= t >> "app >> >> forwarding", i.e. the packet goes full way >> >> net->kernel->recvfrom->app->sendto->kernel->net, which is why we have >> much >> >> lower PPS limits and which is why I think we are actually benefiting >> from >> >> the extra queues. Single-thread sendto() in a loop is CPU-bound at >> about >> >> 220K PPS, and while running the test I am observing that outbound >> traffic >> >> from one thread is mapped into a specific queue (well, pair of queues >> on >> >> two separate adaptors, due to lagg load balancing action). And the pe= ak >> >> performance of that test is at 7 threads, which I believe corresponds >> to >> >> the number of queues. We have plenty of CPU cores in the box (24) wit= h >> >> HTT/SMT disabled and one CPU is mapped to a specific queue. This >> leaves us >> >> with at least 8 CPUs fully capable of running our app. If you look at >> the >> >> CPU utilization, we are at about 10% when the issue hits. >> > >> > In any case, it would be great if you could provide some profiling inf= o >> since there could be >> > plenty of problematic places starting from TX rings contention to some >> locks inside udp or even >> > (in)famous random entropy harvester.. >> > e.g. something like pmcstat -TS instructions -w1 might be sufficient t= o >> determine the reason >> >> ix0: >> port >> >> 0x6020-0x603f mem 0xc7c00000-0xc7dfffff,0xc7e04000-0xc7e07fff irq 40 = at >> >> device 0.0 on pci3 >> >> ix0: Using MSIX interrupts with 9 vectors >> >> ix0: Bound queue 0 to cpu 0 >> >> ix0: Bound queue 1 to cpu 1 >> >> ix0: Bound queue 2 to cpu 2 >> >> ix0: Bound queue 3 to cpu 3 >> >> ix0: Bound queue 4 to cpu 4 >> >> ix0: Bound queue 5 to cpu 5 >> >> ix0: Bound queue 6 to cpu 6 >> >> ix0: Bound queue 7 to cpu 7 >> >> ix0: Ethernet address: 0c:c4:7a:5e:be:64 >> >> ix0: PCI Express Bus: Speed 5.0GT/s Width x8 >> >> 001.000008 [2705] netmap_attach success for ix0 tx 8/4096 rx >> >> 8/4096 queues/slots >> >> ix1: >> port >> >> 0x6000-0x601f mem 0xc7a00000-0xc7bfffff,0xc7e00000-0xc7e03fff irq 44 = at >> >> device 0.1 on pci3 >> >> ix1: Using MSIX interrupts with 9 vectors >> >> ix1: Bound queue 0 to cpu 8 >> >> ix1: Bound queue 1 to cpu 9 >> >> ix1: Bound queue 2 to cpu 10 >> >> ix1: Bound queue 3 to cpu 11 >> >> ix1: Bound queue 4 to cpu 12 >> >> ix1: Bound queue 5 to cpu 13 >> >> ix1: Bound queue 6 to cpu 14 >> >> ix1: Bound queue 7 to cpu 15 >> >> ix1: Ethernet address: 0c:c4:7a:5e:be:65 >> >> ix1: PCI Express Bus: Speed 5.0GT/s Width x8 >> >> 001.000009 [2705] netmap_attach success for ix1 tx 8/4096 rx >> >> 8/4096 queues/slots >> >> >> >> On Tue, Aug 11, 2015 at 4:14 PM, Olivier Cochard-Labb=C3=A9 < >> olivier@cochard.me> >> >> wrote: >> >> >> >>> On Tue, Aug 11, 2015 at 11:18 PM, Maxim Sobolev > > >> >>> wrote: >> >>> >> >>>> Hi folks, >> >>>> >> >>>> =E2=80=8BHi, >> >>> =E2=80=8B >> >>> >> >>>> We've trying to migrate some of our high-PPS systems to a new >> hardware >> >>>> that >> >>>> has four X540-AT2 10G NICs and observed that interrupt time goes >> through >> >>>> roof after we cross around 200K PPS in and 200K out (two ports in >> LACP). >> >>>> The previous hardware was stable up to about 350K PPS in and 350K >> out. I >> >>>> believe the old one was equipped with the I350 and had the >> identical LACP >> >>>> configuration. The new box also has better CPU with more cores >> (i.e. 24 >> >>>> cores vs. 16 cores before). CPU itself is 2 x E5-2690 v3. >> >>> =E2=80=8B200K PPS, and even 350K PPS are very low value indeed. >> >>> On a Intel Xeon L5630 (4 cores only) with one X540-AT2=E2=80=8B >> >>> >> >>> =E2=80=8B(then 2 10Gigabit ports)=E2=80=8B I've reached about 1.8Mp= ps (fastforwarding >> >>> enabled) [1]. >> >>> But my setup didn't use lagg(4): Can you disable lagg configuration >> and >> >>> re-measure your performance without lagg ? >> >>> >> >>> Do you let Intel NIC drivers using 8 queues for port too? >> >>> In my use case (forwarding smallest UDP packet size), I obtain bett= er >> >>> behaviour by limiting NIC queues to 4 (hw.ix.num_queues or >> >>> hw.ixgbe.num_queues, don't remember) if my system had 8 cores. And >> this >> >>> with Gigabit Intel[2] or Chelsio NIC [3]. >> >>> >> >>> Don't forget to disable TSO and LRO too. >> >>> >> >>> =E2=80=8BRegards, >> >>> >> >>> Olivier >> >>> >> >>> [1] >> >>> >> http://bsdrp.net/documentation/examples/forwarding_performance_lab_of_an= _ibm_system_x3550_m3_with_10-gigabit_intel_x540-at2#graphs >> >>> [2] >> >>> >> http://bsdrp.net/documentation/examples/forwarding_performance_lab_of_a_= superserver_5018a-ftn4#graph1 >> >>> [3] >> >>> >> http://bsdrp.net/documentation/examples/forwarding_performance_lab_of_a_= hp_proliant_dl360p_gen8_with_10-gigabit_with_10-gigabit_chelsio_t540-cr#red= ucing_nic_queues >> >> _______________________________________________ >> >> freebsd-net@freebsd.org mailing list >> >> http://lists.freebsd.org/mailman/listinfo/freebsd-net >> >> To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org= " >> > _______________________________________________ >> > freebsd-net@freebsd.org mailing list >> > http://lists.freebsd.org/mailman/listinfo/freebsd-net >> > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" >> _______________________________________________ >> freebsd-net@freebsd.org mailing list >> http://lists.freebsd.org/mailman/listinfo/freebsd-net >> To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" > > > > -- > -----------------------------------------+------------------------------- > Prof. Luigi RIZZO, rizzo@iet.unipi.it . Dip. di Ing. dell'Informazione > http://www.iet.unipi.it/~luigi/ . Universita` di Pisa > TEL +39-050-2217533 . via Diotisalvi 2 > Mobile +39-338-6809875 . 56122 PISA (Italy) > -----------------------------------------+------------------------------- > > --=20 Maksym Sobolyev Sippy Software, Inc. Internet Telephony (VoIP) Experts Tel (Canada): +1-778-783-0474 Tel (Toll-Free): +1-855-747-7779 Fax: +1-866-857-6942 Web: http://www.sippysoft.com MSN: sales@sippysoft.com Skype: SippySoft From owner-freebsd-net@freebsd.org Wed Aug 12 15:05:37 2015 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 6C8E099F35D for ; Wed, 12 Aug 2015 15:05:37 +0000 (UTC) (envelope-from sobomax@sippysoft.com) Received: from mail-wi0-f182.google.com (mail-wi0-f182.google.com [209.85.212.182]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id EEA5788B for ; Wed, 12 Aug 2015 15:05:36 +0000 (UTC) (envelope-from sobomax@sippysoft.com) Received: by wicja10 with SMTP id ja10so116756791wic.1 for ; Wed, 12 Aug 2015 08:05:35 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:sender:date:message-id:subject:from :to:cc:content-type; bh=3DvJfybpnWk9FQ7cZRFNrgiz6qE4u10UXjlOaJV9CfM=; b=AE2WrU8Wh3nlERvSflcKvINIMER+72EbBDFGdxtDgSJ0zaVmNyn65+Xzc1MpsUdQuo YgknVxhY56IPlj5XQDKIgjnOAVi0yf/k2CGqxshj/f6WEAGdiUl3LLA2Zq5NmZoybqU3 NOgx+NbKzQz8ZYMRylOmM8rS7KY6uoWiRgOX0OLtGUH2t4NEdCpNP/D2Ir2ybgntaKCT h+9nN11N9J795xyFFK7xb+augdHCfHIn9Vj9CHg6irjRFj0B0qWdGx3vbw3g5Mj4PEKT K04o7zzXrbqSgbcznq98cmyxQjJQ3S2nqUBgjP8ZJr4BuYz1S3SFilKjBv/tpPCBGk+s HJzw== X-Gm-Message-State: ALoCoQlxhr2b/lBLg5yM98n/bWDCROkxA+Hp0swKYBGX9gBrCUR3OmKyiPnldAGA4z0ToTO5G202 MIME-Version: 1.0 X-Received: by 10.180.81.100 with SMTP id z4mr46455058wix.8.1439391933253; Wed, 12 Aug 2015 08:05:33 -0700 (PDT) Sender: sobomax@sippysoft.com Received: by 10.27.143.15 with HTTP; Wed, 12 Aug 2015 08:05:33 -0700 (PDT) Date: Wed, 12 Aug 2015 08:05:33 -0700 X-Google-Sender-Auth: JiwFI6aWSfREyd2AegNlkQP74CA Message-ID: Subject: Re: Poor high-PPS performance of the 10G ixgbe(9) NIC/driver in FreeBSD 10.1 From: Maxim Sobolev To: Luigi Rizzo Cc: Babak Farrokhi , "Alexander V. Chernikov" , =?UTF-8?Q?Olivier_Cochard=2DLabb=C3=A9?= , "freebsd@intel.com" , =?UTF-8?Q?Jev_Bj=C3=B6rsell?= , FreeBSD Net Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 12 Aug 2015 15:05:37 -0000 igb0@pci0:7:0:0: class=3D0x020000 card=3D0x153315d9 chip=3D0x1533808= 6 rev=3D0x03 hdr=3D0x00 vendor =3D 'Intel Corporation' device =3D 'I210 Gigabit Network Connection' class =3D network subclass =3D ethernet igb1@pci0:8:0:0: class=3D0x020000 card=3D0x153315d9 chip=3D0x1533808= 6 rev=3D0x03 hdr=3D0x00 vendor =3D 'Intel Corporation' device =3D 'I210 Gigabit Network Connection' class =3D network subclass =3D ethernet On Wed, Aug 12, 2015 at 8:03 AM, Maxim Sobolev wrote: > Ok, so my current settings are: > > hw.ix.max_interrupt_rate: 20000 > dev.ix.0.queue0.interrupt_rate: 20000 > dev.ix.0.queue1.interrupt_rate: 20000 > dev.ix.0.queue2.interrupt_rate: 20000 > dev.ix.0.queue3.interrupt_rate: 20000 > dev.ix.0.queue4.interrupt_rate: 20000 > dev.ix.0.queue5.interrupt_rate: 20000 > dev.ix.1.queue0.interrupt_rate: 20000 > dev.ix.1.queue1.interrupt_rate: 20000 > dev.ix.1.queue2.interrupt_rate: 20000 > dev.ix.1.queue3.interrupt_rate: 20000 > dev.ix.1.queue4.interrupt_rate: 20000 > dev.ix.1.queue5.interrupt_rate: 20000 > dev.ix.0.enable_aim: 0 > dev.ix.1.enable_aim: 0 > dev.ix.2.enable_aim: 0 > dev.ix.3.enable_aim: 0 > hw.ix.num_queues:6 > > We also happen to have I210-based system with only 4 hardware queues, it > would be interesting to see how it stacks up. > > On Wed, Aug 12, 2015 at 5:23 AM, Luigi Rizzo wrote: > >> As I was telling to maxim, you should disable aim because it only matche= s >> the max interrupt rate to the average packet size, which is the last thi= ng >> you want. >> >> Setting the interrupt rate with sysctl (one per queue) gives you precise >> control on the max rate and (hence, extra latency). 20k interrupts/s giv= e >> you 50us of latency, and the 2k slots in the queue are still enough to >> absorb a burst of min-sized frames hitting a single queue (the os will >> start dropping long before that level, but that's another story). >> >> Cheers >> Luigi >> >> On Wednesday, August 12, 2015, Babak Farrokhi >> wrote: >> >>> I ran into the same problem with almost the same hardware (Intel X520) >>> on 10-STABLE. HT/SMT is disabled and cards are configured with 8 queues= , >>> with the same sysctl tunings as sobomax@ did. I am not using lagg, no >>> FLOWTABLE. >>> >>> I experimented with pmcstat (RESOURCE_STALLS) a while ago and here [1] >>> [2] you can see the results, including pmc output, callchain, flamegrap= h >>> and gprof output. >>> >>> I am experiencing huge number of interrupts with 200kpps load: >>> >>> # sysctl dev.ix | grep interrupt_rate >>> dev.ix.1.queue7.interrupt_rate: 125000 >>> dev.ix.1.queue6.interrupt_rate: 6329 >>> dev.ix.1.queue5.interrupt_rate: 500000 >>> dev.ix.1.queue4.interrupt_rate: 100000 >>> dev.ix.1.queue3.interrupt_rate: 50000 >>> dev.ix.1.queue2.interrupt_rate: 500000 >>> dev.ix.1.queue1.interrupt_rate: 500000 >>> dev.ix.1.queue0.interrupt_rate: 100000 >>> dev.ix.0.queue7.interrupt_rate: 500000 >>> dev.ix.0.queue6.interrupt_rate: 6097 >>> dev.ix.0.queue5.interrupt_rate: 10204 >>> dev.ix.0.queue4.interrupt_rate: 5208 >>> dev.ix.0.queue3.interrupt_rate: 5208 >>> dev.ix.0.queue2.interrupt_rate: 71428 >>> dev.ix.0.queue1.interrupt_rate: 5494 >>> dev.ix.0.queue0.interrupt_rate: 6250 >>> >>> [1] http://farrokhi.net/~farrokhi/pmc/6/ >>> [2] http://farrokhi.net/~farrokhi/pmc/7/ >>> >>> Regards, >>> Babak >>> >>> >>> Alexander V. Chernikov wrote: >>> > 12.08.2015, 02:28, "Maxim Sobolev" : >>> >> Olivier, keep in mind that we are not "kernel forwarding" packets, >>> but "app >>> >> forwarding", i.e. the packet goes full way >>> >> net->kernel->recvfrom->app->sendto->kernel->net, which is why we hav= e >>> much >>> >> lower PPS limits and which is why I think we are actually benefiting >>> from >>> >> the extra queues. Single-thread sendto() in a loop is CPU-bound at >>> about >>> >> 220K PPS, and while running the test I am observing that outbound >>> traffic >>> >> from one thread is mapped into a specific queue (well, pair of queue= s >>> on >>> >> two separate adaptors, due to lagg load balancing action). And the >>> peak >>> >> performance of that test is at 7 threads, which I believe correspond= s >>> to >>> >> the number of queues. We have plenty of CPU cores in the box (24) wi= th >>> >> HTT/SMT disabled and one CPU is mapped to a specific queue. This >>> leaves us >>> >> with at least 8 CPUs fully capable of running our app. If you look a= t >>> the >>> >> CPU utilization, we are at about 10% when the issue hits. >>> > >>> > In any case, it would be great if you could provide some profiling >>> info since there could be >>> > plenty of problematic places starting from TX rings contention to som= e >>> locks inside udp or even >>> > (in)famous random entropy harvester.. >>> > e.g. something like pmcstat -TS instructions -w1 might be sufficient >>> to determine the reason >>> >> ix0: >> 2.5.15> port >>> >> 0x6020-0x603f mem 0xc7c00000-0xc7dfffff,0xc7e04000-0xc7e07fff irq 40 >>> at >>> >> device 0.0 on pci3 >>> >> ix0: Using MSIX interrupts with 9 vectors >>> >> ix0: Bound queue 0 to cpu 0 >>> >> ix0: Bound queue 1 to cpu 1 >>> >> ix0: Bound queue 2 to cpu 2 >>> >> ix0: Bound queue 3 to cpu 3 >>> >> ix0: Bound queue 4 to cpu 4 >>> >> ix0: Bound queue 5 to cpu 5 >>> >> ix0: Bound queue 6 to cpu 6 >>> >> ix0: Bound queue 7 to cpu 7 >>> >> ix0: Ethernet address: 0c:c4:7a:5e:be:64 >>> >> ix0: PCI Express Bus: Speed 5.0GT/s Width x8 >>> >> 001.000008 [2705] netmap_attach success for ix0 tx 8/4096 rx >>> >> 8/4096 queues/slots >>> >> ix1: >> 2.5.15> port >>> >> 0x6000-0x601f mem 0xc7a00000-0xc7bfffff,0xc7e00000-0xc7e03fff irq 44 >>> at >>> >> device 0.1 on pci3 >>> >> ix1: Using MSIX interrupts with 9 vectors >>> >> ix1: Bound queue 0 to cpu 8 >>> >> ix1: Bound queue 1 to cpu 9 >>> >> ix1: Bound queue 2 to cpu 10 >>> >> ix1: Bound queue 3 to cpu 11 >>> >> ix1: Bound queue 4 to cpu 12 >>> >> ix1: Bound queue 5 to cpu 13 >>> >> ix1: Bound queue 6 to cpu 14 >>> >> ix1: Bound queue 7 to cpu 15 >>> >> ix1: Ethernet address: 0c:c4:7a:5e:be:65 >>> >> ix1: PCI Express Bus: Speed 5.0GT/s Width x8 >>> >> 001.000009 [2705] netmap_attach success for ix1 tx 8/4096 rx >>> >> 8/4096 queues/slots >>> >> >>> >> On Tue, Aug 11, 2015 at 4:14 PM, Olivier Cochard-Labb=C3=A9 < >>> olivier@cochard.me> >>> >> wrote: >>> >> >>> >>> On Tue, Aug 11, 2015 at 11:18 PM, Maxim Sobolev < >>> sobomax@freebsd.org> >>> >>> wrote: >>> >>> >>> >>>> Hi folks, >>> >>>> >>> >>>> =E2=80=8BHi, >>> >>> =E2=80=8B >>> >>> >>> >>>> We've trying to migrate some of our high-PPS systems to a new >>> hardware >>> >>>> that >>> >>>> has four X540-AT2 10G NICs and observed that interrupt time goes >>> through >>> >>>> roof after we cross around 200K PPS in and 200K out (two ports in >>> LACP). >>> >>>> The previous hardware was stable up to about 350K PPS in and 350K >>> out. I >>> >>>> believe the old one was equipped with the I350 and had the >>> identical LACP >>> >>>> configuration. The new box also has better CPU with more cores >>> (i.e. 24 >>> >>>> cores vs. 16 cores before). CPU itself is 2 x E5-2690 v3. >>> >>> =E2=80=8B200K PPS, and even 350K PPS are very low value indeed. >>> >>> On a Intel Xeon L5630 (4 cores only) with one X540-AT2=E2=80=8B >>> >>> >>> >>> =E2=80=8B(then 2 10Gigabit ports)=E2=80=8B I've reached about 1.8M= pps >>> (fastforwarding >>> >>> enabled) [1]. >>> >>> But my setup didn't use lagg(4): Can you disable lagg configuratio= n >>> and >>> >>> re-measure your performance without lagg ? >>> >>> >>> >>> Do you let Intel NIC drivers using 8 queues for port too? >>> >>> In my use case (forwarding smallest UDP packet size), I obtain >>> better >>> >>> behaviour by limiting NIC queues to 4 (hw.ix.num_queues or >>> >>> hw.ixgbe.num_queues, don't remember) if my system had 8 cores. And >>> this >>> >>> with Gigabit Intel[2] or Chelsio NIC [3]. >>> >>> >>> >>> Don't forget to disable TSO and LRO too. >>> >>> >>> >>> =E2=80=8BRegards, >>> >>> >>> >>> Olivier >>> >>> >>> >>> [1] >>> >>> >>> http://bsdrp.net/documentation/examples/forwarding_performance_lab_of_a= n_ibm_system_x3550_m3_with_10-gigabit_intel_x540-at2#graphs >>> >>> [2] >>> >>> >>> http://bsdrp.net/documentation/examples/forwarding_performance_lab_of_a= _superserver_5018a-ftn4#graph1 >>> >>> [3] >>> >>> >>> http://bsdrp.net/documentation/examples/forwarding_performance_lab_of_a= _hp_proliant_dl360p_gen8_with_10-gigabit_with_10-gigabit_chelsio_t540-cr#re= ducing_nic_queues >>> >> _______________________________________________ >>> >> freebsd-net@freebsd.org mailing list >>> >> http://lists.freebsd.org/mailman/listinfo/freebsd-net >>> >> To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.or= g >>> " >>> > _______________________________________________ >>> > freebsd-net@freebsd.org mailing list >>> > http://lists.freebsd.org/mailman/listinfo/freebsd-net >>> > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org= " >>> _______________________________________________ >>> freebsd-net@freebsd.org mailing list >>> http://lists.freebsd.org/mailman/listinfo/freebsd-net >>> To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" >> >> >> >> -- >> -----------------------------------------+------------------------------= - >> Prof. Luigi RIZZO, rizzo@iet.unipi.it . Dip. di Ing. dell'Informazione >> http://www.iet.unipi.it/~luigi/ . Universita` di Pisa >> TEL +39-050-2217533 . via Diotisalvi 2 >> Mobile +39-338-6809875 . 56122 PISA (Italy) >> -----------------------------------------+------------------------------= - >> >> > > > -- > Maksym Sobolyev > Sippy Software, Inc. > Internet Telephony (VoIP) Experts > Tel (Canada): +1-778-783-0474 > Tel (Toll-Free): +1-855-747-7779 > Fax: +1-866-857-6942 > Web: http://www.sippysoft.com > MSN: sales@sippysoft.com > Skype: SippySoft > From owner-freebsd-net@freebsd.org Wed Aug 12 15:23:21 2015 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id D254B99F80D for ; Wed, 12 Aug 2015 15:23:21 +0000 (UTC) (envelope-from adrian.chadd@gmail.com) Received: from mail-io0-x22c.google.com (mail-io0-x22c.google.com [IPv6:2607:f8b0:4001:c06::22c]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 9B44BF9F; Wed, 12 Aug 2015 15:23:21 +0000 (UTC) (envelope-from adrian.chadd@gmail.com) Received: by iodb91 with SMTP id b91so23166215iod.1; Wed, 12 Aug 2015 08:23:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type:content-transfer-encoding; bh=2ilnLqNrNYDqj1G6OdEubEZQSwTlXHpkfkGlXvJGJ6Q=; b=ZGwXQJxNYiBT2JFG8bULVCnmzCDQ1kDpiP/YB/M9OIYR4FSnzG68NG7/EzMjpjv+Yg nh/oifH4oTj9edxK4HU9J2W1KnEVDvyTFCMEmyEWbz4kmsJQXZIgrOLplZAROnSbUA9O cqEjmtoH7SOkEVmIFYZwgQ4XOeDmx0bcwHvMDWySvA/R+1BLCR2XoHkfuRTuQP69iurR CO0aRRn7A3rwuogI8DY436qeNL74O5Nxs09WfSfsdSQVQKlmpym2HEBEqxnR69RmlvOH B/XCDDLgXYPlPFwaZGGR6a28aCFtCe4kWtnZLJOIaSzqrvocuOOroKO65jCZHKl0ttfR M1wg== MIME-Version: 1.0 X-Received: by 10.107.156.73 with SMTP id f70mr809498ioe.165.1439393000473; Wed, 12 Aug 2015 08:23:20 -0700 (PDT) Received: by 10.36.38.133 with HTTP; Wed, 12 Aug 2015 08:23:20 -0700 (PDT) In-Reply-To: References: Date: Wed, 12 Aug 2015 08:23:20 -0700 Message-ID: Subject: Re: Poor high-PPS performance of the 10G ixgbe(9) NIC/driver in FreeBSD 10.1 From: Adrian Chadd To: Maxim Sobolev Cc: Luigi Rizzo , "Alexander V. Chernikov" , FreeBSD Net , Babak Farrokhi , "freebsd@intel.com" , =?UTF-8?Q?Jev_Bj=C3=B6rsell?= , =?UTF-8?Q?Olivier_Cochard=2DLabb=C3=A9?= Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 12 Aug 2015 15:23:22 -0000 Right, and for the ixgbe hardware? -a On 12 August 2015 at 08:05, Maxim Sobolev wrote: > igb0@pci0:7:0:0: class=3D0x020000 card=3D0x153315d9 chip=3D0x15338= 086 > rev=3D0x03 hdr=3D0x00 > vendor =3D 'Intel Corporation' > device =3D 'I210 Gigabit Network Connection' > class =3D network > subclass =3D ethernet > igb1@pci0:8:0:0: class=3D0x020000 card=3D0x153315d9 chip=3D0x15338= 086 > rev=3D0x03 hdr=3D0x00 > vendor =3D 'Intel Corporation' > device =3D 'I210 Gigabit Network Connection' > class =3D network > subclass =3D ethernet > > > On Wed, Aug 12, 2015 at 8:03 AM, Maxim Sobolev > wrote: > >> Ok, so my current settings are: >> >> hw.ix.max_interrupt_rate: 20000 >> dev.ix.0.queue0.interrupt_rate: 20000 >> dev.ix.0.queue1.interrupt_rate: 20000 >> dev.ix.0.queue2.interrupt_rate: 20000 >> dev.ix.0.queue3.interrupt_rate: 20000 >> dev.ix.0.queue4.interrupt_rate: 20000 >> dev.ix.0.queue5.interrupt_rate: 20000 >> dev.ix.1.queue0.interrupt_rate: 20000 >> dev.ix.1.queue1.interrupt_rate: 20000 >> dev.ix.1.queue2.interrupt_rate: 20000 >> dev.ix.1.queue3.interrupt_rate: 20000 >> dev.ix.1.queue4.interrupt_rate: 20000 >> dev.ix.1.queue5.interrupt_rate: 20000 >> dev.ix.0.enable_aim: 0 >> dev.ix.1.enable_aim: 0 >> dev.ix.2.enable_aim: 0 >> dev.ix.3.enable_aim: 0 >> hw.ix.num_queues:6 >> >> We also happen to have I210-based system with only 4 hardware queues, it >> would be interesting to see how it stacks up. >> >> On Wed, Aug 12, 2015 at 5:23 AM, Luigi Rizzo wrote: >> >>> As I was telling to maxim, you should disable aim because it only match= es >>> the max interrupt rate to the average packet size, which is the last th= ing >>> you want. >>> >>> Setting the interrupt rate with sysctl (one per queue) gives you precis= e >>> control on the max rate and (hence, extra latency). 20k interrupts/s gi= ve >>> you 50us of latency, and the 2k slots in the queue are still enough to >>> absorb a burst of min-sized frames hitting a single queue (the os will >>> start dropping long before that level, but that's another story). >>> >>> Cheers >>> Luigi >>> >>> On Wednesday, August 12, 2015, Babak Farrokhi >>> wrote: >>> >>>> I ran into the same problem with almost the same hardware (Intel X520) >>>> on 10-STABLE. HT/SMT is disabled and cards are configured with 8 queue= s, >>>> with the same sysctl tunings as sobomax@ did. I am not using lagg, no >>>> FLOWTABLE. >>>> >>>> I experimented with pmcstat (RESOURCE_STALLS) a while ago and here [1] >>>> [2] you can see the results, including pmc output, callchain, flamegra= ph >>>> and gprof output. >>>> >>>> I am experiencing huge number of interrupts with 200kpps load: >>>> >>>> # sysctl dev.ix | grep interrupt_rate >>>> dev.ix.1.queue7.interrupt_rate: 125000 >>>> dev.ix.1.queue6.interrupt_rate: 6329 >>>> dev.ix.1.queue5.interrupt_rate: 500000 >>>> dev.ix.1.queue4.interrupt_rate: 100000 >>>> dev.ix.1.queue3.interrupt_rate: 50000 >>>> dev.ix.1.queue2.interrupt_rate: 500000 >>>> dev.ix.1.queue1.interrupt_rate: 500000 >>>> dev.ix.1.queue0.interrupt_rate: 100000 >>>> dev.ix.0.queue7.interrupt_rate: 500000 >>>> dev.ix.0.queue6.interrupt_rate: 6097 >>>> dev.ix.0.queue5.interrupt_rate: 10204 >>>> dev.ix.0.queue4.interrupt_rate: 5208 >>>> dev.ix.0.queue3.interrupt_rate: 5208 >>>> dev.ix.0.queue2.interrupt_rate: 71428 >>>> dev.ix.0.queue1.interrupt_rate: 5494 >>>> dev.ix.0.queue0.interrupt_rate: 6250 >>>> >>>> [1] http://farrokhi.net/~farrokhi/pmc/6/ >>>> [2] http://farrokhi.net/~farrokhi/pmc/7/ >>>> >>>> Regards, >>>> Babak >>>> >>>> >>>> Alexander V. Chernikov wrote: >>>> > 12.08.2015, 02:28, "Maxim Sobolev" : >>>> >> Olivier, keep in mind that we are not "kernel forwarding" packets, >>>> but "app >>>> >> forwarding", i.e. the packet goes full way >>>> >> net->kernel->recvfrom->app->sendto->kernel->net, which is why we ha= ve >>>> much >>>> >> lower PPS limits and which is why I think we are actually benefitin= g >>>> from >>>> >> the extra queues. Single-thread sendto() in a loop is CPU-bound at >>>> about >>>> >> 220K PPS, and while running the test I am observing that outbound >>>> traffic >>>> >> from one thread is mapped into a specific queue (well, pair of queu= es >>>> on >>>> >> two separate adaptors, due to lagg load balancing action). And the >>>> peak >>>> >> performance of that test is at 7 threads, which I believe correspon= ds >>>> to >>>> >> the number of queues. We have plenty of CPU cores in the box (24) w= ith >>>> >> HTT/SMT disabled and one CPU is mapped to a specific queue. This >>>> leaves us >>>> >> with at least 8 CPUs fully capable of running our app. If you look = at >>>> the >>>> >> CPU utilization, we are at about 10% when the issue hits. >>>> > >>>> > In any case, it would be great if you could provide some profiling >>>> info since there could be >>>> > plenty of problematic places starting from TX rings contention to so= me >>>> locks inside udp or even >>>> > (in)famous random entropy harvester.. >>>> > e.g. something like pmcstat -TS instructions -w1 might be sufficient >>>> to determine the reason >>>> >> ix0: >>> 2.5.15> port >>>> >> 0x6020-0x603f mem 0xc7c00000-0xc7dfffff,0xc7e04000-0xc7e07fff irq 4= 0 >>>> at >>>> >> device 0.0 on pci3 >>>> >> ix0: Using MSIX interrupts with 9 vectors >>>> >> ix0: Bound queue 0 to cpu 0 >>>> >> ix0: Bound queue 1 to cpu 1 >>>> >> ix0: Bound queue 2 to cpu 2 >>>> >> ix0: Bound queue 3 to cpu 3 >>>> >> ix0: Bound queue 4 to cpu 4 >>>> >> ix0: Bound queue 5 to cpu 5 >>>> >> ix0: Bound queue 6 to cpu 6 >>>> >> ix0: Bound queue 7 to cpu 7 >>>> >> ix0: Ethernet address: 0c:c4:7a:5e:be:64 >>>> >> ix0: PCI Express Bus: Speed 5.0GT/s Width x8 >>>> >> 001.000008 [2705] netmap_attach success for ix0 tx 8/4096 rx >>>> >> 8/4096 queues/slots >>>> >> ix1: >>> 2.5.15> port >>>> >> 0x6000-0x601f mem 0xc7a00000-0xc7bfffff,0xc7e00000-0xc7e03fff irq 4= 4 >>>> at >>>> >> device 0.1 on pci3 >>>> >> ix1: Using MSIX interrupts with 9 vectors >>>> >> ix1: Bound queue 0 to cpu 8 >>>> >> ix1: Bound queue 1 to cpu 9 >>>> >> ix1: Bound queue 2 to cpu 10 >>>> >> ix1: Bound queue 3 to cpu 11 >>>> >> ix1: Bound queue 4 to cpu 12 >>>> >> ix1: Bound queue 5 to cpu 13 >>>> >> ix1: Bound queue 6 to cpu 14 >>>> >> ix1: Bound queue 7 to cpu 15 >>>> >> ix1: Ethernet address: 0c:c4:7a:5e:be:65 >>>> >> ix1: PCI Express Bus: Speed 5.0GT/s Width x8 >>>> >> 001.000009 [2705] netmap_attach success for ix1 tx 8/4096 rx >>>> >> 8/4096 queues/slots >>>> >> >>>> >> On Tue, Aug 11, 2015 at 4:14 PM, Olivier Cochard-Labb=C3=A9 < >>>> olivier@cochard.me> >>>> >> wrote: >>>> >> >>>> >>> On Tue, Aug 11, 2015 at 11:18 PM, Maxim Sobolev < >>>> sobomax@freebsd.org> >>>> >>> wrote: >>>> >>> >>>> >>>> Hi folks, >>>> >>>> >>>> >>>> Hi, >>>> >>> >>>> >>> >>>> >>>> We've trying to migrate some of our high-PPS systems to a new >>>> hardware >>>> >>>> that >>>> >>>> has four X540-AT2 10G NICs and observed that interrupt time goes >>>> through >>>> >>>> roof after we cross around 200K PPS in and 200K out (two ports i= n >>>> LACP). >>>> >>>> The previous hardware was stable up to about 350K PPS in and 350= K >>>> out. I >>>> >>>> believe the old one was equipped with the I350 and had the >>>> identical LACP >>>> >>>> configuration. The new box also has better CPU with more cores >>>> (i.e. 24 >>>> >>>> cores vs. 16 cores before). CPU itself is 2 x E5-2690 v3. >>>> >>> 200K PPS, and even 350K PPS are very low value indeed. >>>> >>> On a Intel Xeon L5630 (4 cores only) with one X540-AT2 >>>> >>> >>>> >>> (then 2 10Gigabit ports) I've reached about 1.8Mpps >>>> (fastforwarding >>>> >>> enabled) [1]. >>>> >>> But my setup didn't use lagg(4): Can you disable lagg configurati= on >>>> and >>>> >>> re-measure your performance without lagg ? >>>> >>> >>>> >>> Do you let Intel NIC drivers using 8 queues for port too? >>>> >>> In my use case (forwarding smallest UDP packet size), I obtain >>>> better >>>> >>> behaviour by limiting NIC queues to 4 (hw.ix.num_queues or >>>> >>> hw.ixgbe.num_queues, don't remember) if my system had 8 cores. An= d >>>> this >>>> >>> with Gigabit Intel[2] or Chelsio NIC [3]. >>>> >>> >>>> >>> Don't forget to disable TSO and LRO too. >>>> >>> >>>> >>> Regards, >>>> >>> >>>> >>> Olivier >>>> >>> >>>> >>> [1] >>>> >>> >>>> http://bsdrp.net/documentation/examples/forwarding_performance_lab_of_= an_ibm_system_x3550_m3_with_10-gigabit_intel_x540-at2#graphs >>>> >>> [2] >>>> >>> >>>> http://bsdrp.net/documentation/examples/forwarding_performance_lab_of_= a_superserver_5018a-ftn4#graph1 >>>> >>> [3] >>>> >>> >>>> http://bsdrp.net/documentation/examples/forwarding_performance_lab_of_= a_hp_proliant_dl360p_gen8_with_10-gigabit_with_10-gigabit_chelsio_t540-cr#r= educing_nic_queues >>>> >> _______________________________________________ >>>> >> freebsd-net@freebsd.org mailing list >>>> >> http://lists.freebsd.org/mailman/listinfo/freebsd-net >>>> >> To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.o= rg >>>> " >>>> > _______________________________________________ >>>> > freebsd-net@freebsd.org mailing list >>>> > http://lists.freebsd.org/mailman/listinfo/freebsd-net >>>> > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.or= g" >>>> _______________________________________________ >>>> freebsd-net@freebsd.org mailing list >>>> http://lists.freebsd.org/mailman/listinfo/freebsd-net >>>> To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" >>> >>> >>> >>> -- >>> -----------------------------------------+-----------------------------= -- >>> Prof. Luigi RIZZO, rizzo@iet.unipi.it . Dip. di Ing. dell'Informazion= e >>> http://www.iet.unipi.it/~luigi/ . Universita` di Pisa >>> TEL +39-050-2217533 . via Diotisalvi 2 >>> Mobile +39-338-6809875 . 56122 PISA (Italy) >>> -----------------------------------------+-----------------------------= -- >>> >>> >> >> >> -- >> Maksym Sobolyev >> Sippy Software, Inc. >> Internet Telephony (VoIP) Experts >> Tel (Canada): +1-778-783-0474 >> Tel (Toll-Free): +1-855-747-7779 >> Fax: +1-866-857-6942 >> Web: http://www.sippysoft.com >> MSN: sales@sippysoft.com >> Skype: SippySoft >> > _______________________________________________ > freebsd-net@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" From owner-freebsd-net@freebsd.org Wed Aug 12 17:20:24 2015 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 4750E99F820 for ; Wed, 12 Aug 2015 17:20:24 +0000 (UTC) (envelope-from ebastan10@gmail.com) Received: from mail-ig0-x22d.google.com (mail-ig0-x22d.google.com [IPv6:2607:f8b0:4001:c05::22d]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 166AEE68 for ; Wed, 12 Aug 2015 17:20:24 +0000 (UTC) (envelope-from ebastan10@gmail.com) Received: by igxp17 with SMTP id p17so4521007igx.1 for ; Wed, 12 Aug 2015 10:20:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:date:message-id:subject:from:to:content-type; bh=JAvOwBYH+vS7coGqiqAZkpInIJ9CdUU/OMBudXgUNYM=; b=b9SVHroA/lMV0BO58XVISem/w5vFPBfa39rQ7m1Le4qeHUFrj6WbsiaTI5qKA8O9mR i8+pPm6Z82dXqLA9XDNmGP82RJZK0YJCOt8rEPyMk2B7hfQws4fMdIdkPApg9ZxsnfNV pXU2wECZ3bOQ12CsNRE9mGoFQv4QC3aHIy3QVDgmRQ0Ym86IA8YGffRUshWYocn3+Vo/ klsJ7VdRh1l2TVLqzllBp8jfSrBZjqhzwLWlAR8XPCeD7sj1p1x4bfiCJpcv+mi1oz2Z bmmpkEEtiYLRJmEN2wmyG+ZhMBH19xI+GeGDSqKa97RGHkn4UGkS9h1z0jbuXETR30g9 8YFA== MIME-Version: 1.0 X-Received: by 10.50.66.197 with SMTP id h5mr25091527igt.81.1439400023425; Wed, 12 Aug 2015 10:20:23 -0700 (PDT) Received: by 10.107.9.67 with HTTP; Wed, 12 Aug 2015 10:20:23 -0700 (PDT) Date: Wed, 12 Aug 2015 21:50:23 +0430 Message-ID: Subject: multicasts are not forwarded on wifi interface From: Hooshang F To: freebsd-net@freebsd.org Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 12 Aug 2015 17:20:24 -0000 Hi, We have a freebsd 8.3 wifi-AP running as a VirtulaBox guest with 2 ethernet (em0 & em1) and one USB wifi interface (wlan0). Everything works except that multicast packets arriving on wlan0 interface are not forwarded on Ethernet interfaces. Also, multicast packets arriving on Ethernet interfaces are not forwarded on wlan0 interface (however, multicasts are forwarded between Ethernet interfaces). As a result, portable devices are not able to discover DLNA media servers using UPnP. The kernel is compiled with MROUTING option and mrouted is running with default configuration. Any points/hints is appreciated. From owner-freebsd-net@freebsd.org Wed Aug 12 17:34:24 2015 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 094EB99FC63 for ; Wed, 12 Aug 2015 17:34:24 +0000 (UTC) (envelope-from adrian.chadd@gmail.com) Received: from mail-io0-x22f.google.com (mail-io0-x22f.google.com [IPv6:2607:f8b0:4001:c06::22f]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id CBD2CCC4 for ; Wed, 12 Aug 2015 17:34:23 +0000 (UTC) (envelope-from adrian.chadd@gmail.com) Received: by iodt126 with SMTP id t126so27327947iod.2 for ; Wed, 12 Aug 2015 10:34:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=fMaU8DU5NNZOkBin6D1gVQ98DEckStZC4ErPvM0zIJs=; b=Zk+vMWEnUiBOL5i/nkwv+5NjwoGs0mR41tqyxJEqcefFLpNxUkR+Q03IwBjGoT9wRK 09bXWS2ZssBmVLCCLp4gN3gIbXKe2zTbpEhRmp5tnZnnfjzYeYKhJkbKN1q2gVjNaHb7 +nTRH+areF53IMu6YgEZiW/0ZTwECiPKeUVn40UTFq/LOBUcn3eXa97FrJy60uqZzsBo UHBtqIt3vGzrVopmSpDYIIjTnujZQHz93Nakp8IEqpS89Lt8YwDhYOa4K2d7qAk+y6Vl cpDjbzgQbf4IsfLQOT/UUAJrmttsOcLrv6NoyYLlSSr9hClIPSQnx24rori2CxdCltOX zqoQ== MIME-Version: 1.0 X-Received: by 10.107.156.73 with SMTP id f70mr1364367ioe.165.1439400863233; Wed, 12 Aug 2015 10:34:23 -0700 (PDT) Received: by 10.36.38.133 with HTTP; Wed, 12 Aug 2015 10:34:23 -0700 (PDT) In-Reply-To: References: Date: Wed, 12 Aug 2015 10:34:23 -0700 Message-ID: Subject: Re: multicasts are not forwarded on wifi interface From: Adrian Chadd To: Hooshang F Cc: FreeBSD Net Content-Type: text/plain; charset=UTF-8 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 12 Aug 2015 17:34:24 -0000 HI, Would you please try freebsd-head? I think this was fixed a while ago, but I don't think it was ever backported to freebsd-8. Thanks! -a On 12 August 2015 at 10:20, Hooshang F wrote: > Hi, > > We have a freebsd 8.3 wifi-AP running as a VirtulaBox guest > with 2 ethernet (em0 & em1) and one USB wifi interface (wlan0). Everything > works except that multicast packets arriving on wlan0 interface are not > forwarded on Ethernet interfaces. Also, multicast packets arriving on > Ethernet interfaces are not forwarded on wlan0 interface (however, > multicasts are forwarded between Ethernet interfaces). As a result, > portable devices are not able to discover > DLNA media servers using UPnP. > > The kernel is compiled with MROUTING option and mrouted > is running with default configuration. > > Any points/hints is appreciated. > _______________________________________________ > freebsd-net@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" From owner-freebsd-net@freebsd.org Thu Aug 13 00:47:38 2015 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 1F65D9B7B90 for ; Thu, 13 Aug 2015 00:47:38 +0000 (UTC) (envelope-from sobomax@sippysoft.com) Received: from mail-wi0-f169.google.com (mail-wi0-f169.google.com [209.85.212.169]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id A06D8130 for ; Thu, 13 Aug 2015 00:47:37 +0000 (UTC) (envelope-from sobomax@sippysoft.com) Received: by wicne3 with SMTP id ne3so239169184wic.1 for ; Wed, 12 Aug 2015 17:47:29 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:sender:in-reply-to:references:date :message-id:subject:from:to:cc:content-type; bh=GgpcSycWe2u5ZwglAoABd/uYKmgiOfXxE00s0Vl1PWA=; b=YAnyCrMnj4L8uLSlzvRY1a0mwFxo0D7TQn9kiMPOJsiy0+qjFcLmnpjECIE3k2dyVo WztaXkmduQWCijJkA+JccgnvvZ/9INPapsnodS67QKXs2YwaUyrFNWTXeUWp7ruMBKL0 X+7nkJbFmKOVcQCVSyaDFniQhdFkqSCc8Mp/1UEoLoxkgo3kmkADc6I6bm/LB+q4tk+g LNb3sEiWLnOBzw4Rno6qHYa4T5rC7baStWc1CiBeUwhWqDtKB5NRAtwHoq8k1x+HAt0C cHV4mTpI9F5d6rtR1RYp4qrxwvX/MOeoXtNS5UsnAGcRXQrz3Dk1WRU3881pXs8l3mdY BsCw== X-Gm-Message-State: ALoCoQlPcwDa07cV3jn/5qGZ0sU+cJkWlrJIKK6p0kCrAwEJELUN9eJG2x3vSGGfJRJgd737YsCP MIME-Version: 1.0 X-Received: by 10.180.21.169 with SMTP id w9mr53059079wie.73.1439426849793; Wed, 12 Aug 2015 17:47:29 -0700 (PDT) Sender: sobomax@sippysoft.com Received: by 10.27.143.15 with HTTP; Wed, 12 Aug 2015 17:47:29 -0700 (PDT) In-Reply-To: References: Date: Wed, 12 Aug 2015 17:47:29 -0700 X-Google-Sender-Auth: -PCgK_0UPAYInp4Qq0N-LEdQ1WI Message-ID: Subject: Re: Poor high-PPS performance of the 10G ixgbe(9) NIC/driver in FreeBSD 10.1 From: Maxim Sobolev To: Adrian Chadd Cc: Luigi Rizzo , "Alexander V. Chernikov" , FreeBSD Net , Babak Farrokhi , "freebsd@intel.com" , =?UTF-8?Q?Jev_Bj=C3=B6rsell?= , =?UTF-8?Q?Olivier_Cochard=2DLabb=C3=A9?= Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 13 Aug 2015 00:47:38 -0000 Here we go (ix2 and ix3 are not used): ix0@pci0:3:0:0: class=3D0x020000 card=3D0x152815d9 chip=3D0x15288086 rev=3D= 0x01 hdr=3D0x00 vendor =3D 'Intel Corporation' device =3D 'Ethernet Controller 10-Gigabit X540-AT2' class =3D network subclass =3D ethernet ix1@pci0:3:0:1: class=3D0x020000 card=3D0x152815d9 chip=3D0x15288086 rev=3D= 0x01 hdr=3D0x00 vendor =3D 'Intel Corporation' device =3D 'Ethernet Controller 10-Gigabit X540-AT2' class =3D network subclass =3D ethernet ix2@pci0:4:0:0: class=3D0x020000 card=3D0x152815d9 chip=3D0x15288086 rev=3D= 0x01 hdr=3D0x00 vendor =3D 'Intel Corporation' device =3D 'Ethernet Controller 10-Gigabit X540-AT2' class =3D network subclass =3D ethernet ix3@pci0:4:0:1: class=3D0x020000 card=3D0x152815d9 chip=3D0x15288086 rev=3D= 0x01 hdr=3D0x00 vendor =3D 'Intel Corporation' device =3D 'Ethernet Controller 10-Gigabit X540-AT2' class =3D network subclass =3D ethernet On Wed, Aug 12, 2015 at 8:23 AM, Adrian Chadd wrote: > Right, and for the ixgbe hardware? > > > > -a > > > On 12 August 2015 at 08:05, Maxim Sobolev wrote: > > igb0@pci0:7:0:0: class=3D0x020000 card=3D0x153315d9 chip=3D0x153= 38086 > > rev=3D0x03 hdr=3D0x00 > > vendor =3D 'Intel Corporation' > > device =3D 'I210 Gigabit Network Connection' > > class =3D network > > subclass =3D ethernet > > igb1@pci0:8:0:0: class=3D0x020000 card=3D0x153315d9 chip=3D0x153= 38086 > > rev=3D0x03 hdr=3D0x00 > > vendor =3D 'Intel Corporation' > > device =3D 'I210 Gigabit Network Connection' > > class =3D network > > subclass =3D ethernet > > > > > > On Wed, Aug 12, 2015 at 8:03 AM, Maxim Sobolev > > wrote: > > > >> Ok, so my current settings are: > >> > >> hw.ix.max_interrupt_rate: 20000 > >> dev.ix.0.queue0.interrupt_rate: 20000 > >> dev.ix.0.queue1.interrupt_rate: 20000 > >> dev.ix.0.queue2.interrupt_rate: 20000 > >> dev.ix.0.queue3.interrupt_rate: 20000 > >> dev.ix.0.queue4.interrupt_rate: 20000 > >> dev.ix.0.queue5.interrupt_rate: 20000 > >> dev.ix.1.queue0.interrupt_rate: 20000 > >> dev.ix.1.queue1.interrupt_rate: 20000 > >> dev.ix.1.queue2.interrupt_rate: 20000 > >> dev.ix.1.queue3.interrupt_rate: 20000 > >> dev.ix.1.queue4.interrupt_rate: 20000 > >> dev.ix.1.queue5.interrupt_rate: 20000 > >> dev.ix.0.enable_aim: 0 > >> dev.ix.1.enable_aim: 0 > >> dev.ix.2.enable_aim: 0 > >> dev.ix.3.enable_aim: 0 > >> hw.ix.num_queues:6 > >> > >> We also happen to have I210-based system with only 4 hardware queues, = it > >> would be interesting to see how it stacks up. > >> > >> On Wed, Aug 12, 2015 at 5:23 AM, Luigi Rizzo > wrote: > >> > >>> As I was telling to maxim, you should disable aim because it only > matches > >>> the max interrupt rate to the average packet size, which is the last > thing > >>> you want. > >>> > >>> Setting the interrupt rate with sysctl (one per queue) gives you > precise > >>> control on the max rate and (hence, extra latency). 20k interrupts/s > give > >>> you 50us of latency, and the 2k slots in the queue are still enough t= o > >>> absorb a burst of min-sized frames hitting a single queue (the os wil= l > >>> start dropping long before that level, but that's another story). > >>> > >>> Cheers > >>> Luigi > >>> > >>> On Wednesday, August 12, 2015, Babak Farrokhi > >>> wrote: > >>> > >>>> I ran into the same problem with almost the same hardware (Intel X52= 0) > >>>> on 10-STABLE. HT/SMT is disabled and cards are configured with 8 > queues, > >>>> with the same sysctl tunings as sobomax@ did. I am not using lagg, n= o > >>>> FLOWTABLE. > >>>> > >>>> I experimented with pmcstat (RESOURCE_STALLS) a while ago and here [= 1] > >>>> [2] you can see the results, including pmc output, callchain, > flamegraph > >>>> and gprof output. > >>>> > >>>> I am experiencing huge number of interrupts with 200kpps load: > >>>> > >>>> # sysctl dev.ix | grep interrupt_rate > >>>> dev.ix.1.queue7.interrupt_rate: 125000 > >>>> dev.ix.1.queue6.interrupt_rate: 6329 > >>>> dev.ix.1.queue5.interrupt_rate: 500000 > >>>> dev.ix.1.queue4.interrupt_rate: 100000 > >>>> dev.ix.1.queue3.interrupt_rate: 50000 > >>>> dev.ix.1.queue2.interrupt_rate: 500000 > >>>> dev.ix.1.queue1.interrupt_rate: 500000 > >>>> dev.ix.1.queue0.interrupt_rate: 100000 > >>>> dev.ix.0.queue7.interrupt_rate: 500000 > >>>> dev.ix.0.queue6.interrupt_rate: 6097 > >>>> dev.ix.0.queue5.interrupt_rate: 10204 > >>>> dev.ix.0.queue4.interrupt_rate: 5208 > >>>> dev.ix.0.queue3.interrupt_rate: 5208 > >>>> dev.ix.0.queue2.interrupt_rate: 71428 > >>>> dev.ix.0.queue1.interrupt_rate: 5494 > >>>> dev.ix.0.queue0.interrupt_rate: 6250 > >>>> > >>>> [1] http://farrokhi.net/~farrokhi/pmc/6/ > >>>> [2] http://farrokhi.net/~farrokhi/pmc/7/ > >>>> > >>>> Regards, > >>>> Babak > >>>> > >>>> > >>>> Alexander V. Chernikov wrote: > >>>> > 12.08.2015, 02:28, "Maxim Sobolev" : > >>>> >> Olivier, keep in mind that we are not "kernel forwarding" packets= , > >>>> but "app > >>>> >> forwarding", i.e. the packet goes full way > >>>> >> net->kernel->recvfrom->app->sendto->kernel->net, which is why we > have > >>>> much > >>>> >> lower PPS limits and which is why I think we are actually > benefiting > >>>> from > >>>> >> the extra queues. Single-thread sendto() in a loop is CPU-bound a= t > >>>> about > >>>> >> 220K PPS, and while running the test I am observing that outbound > >>>> traffic > >>>> >> from one thread is mapped into a specific queue (well, pair of > queues > >>>> on > >>>> >> two separate adaptors, due to lagg load balancing action). And th= e > >>>> peak > >>>> >> performance of that test is at 7 threads, which I believe > corresponds > >>>> to > >>>> >> the number of queues. We have plenty of CPU cores in the box (24) > with > >>>> >> HTT/SMT disabled and one CPU is mapped to a specific queue. This > >>>> leaves us > >>>> >> with at least 8 CPUs fully capable of running our app. If you loo= k > at > >>>> the > >>>> >> CPU utilization, we are at about 10% when the issue hits. > >>>> > > >>>> > In any case, it would be great if you could provide some profiling > >>>> info since there could be > >>>> > plenty of problematic places starting from TX rings contention to > some > >>>> locks inside udp or even > >>>> > (in)famous random entropy harvester.. > >>>> > e.g. something like pmcstat -TS instructions -w1 might be sufficie= nt > >>>> to determine the reason > >>>> >> ix0: >>>> 2.5.15> port > >>>> >> 0x6020-0x603f mem 0xc7c00000-0xc7dfffff,0xc7e04000-0xc7e07fff irq > 40 > >>>> at > >>>> >> device 0.0 on pci3 > >>>> >> ix0: Using MSIX interrupts with 9 vectors > >>>> >> ix0: Bound queue 0 to cpu 0 > >>>> >> ix0: Bound queue 1 to cpu 1 > >>>> >> ix0: Bound queue 2 to cpu 2 > >>>> >> ix0: Bound queue 3 to cpu 3 > >>>> >> ix0: Bound queue 4 to cpu 4 > >>>> >> ix0: Bound queue 5 to cpu 5 > >>>> >> ix0: Bound queue 6 to cpu 6 > >>>> >> ix0: Bound queue 7 to cpu 7 > >>>> >> ix0: Ethernet address: 0c:c4:7a:5e:be:64 > >>>> >> ix0: PCI Express Bus: Speed 5.0GT/s Width x8 > >>>> >> 001.000008 [2705] netmap_attach success for ix0 tx 8/4096 rx > >>>> >> 8/4096 queues/slots > >>>> >> ix1: >>>> 2.5.15> port > >>>> >> 0x6000-0x601f mem 0xc7a00000-0xc7bfffff,0xc7e00000-0xc7e03fff irq > 44 > >>>> at > >>>> >> device 0.1 on pci3 > >>>> >> ix1: Using MSIX interrupts with 9 vectors > >>>> >> ix1: Bound queue 0 to cpu 8 > >>>> >> ix1: Bound queue 1 to cpu 9 > >>>> >> ix1: Bound queue 2 to cpu 10 > >>>> >> ix1: Bound queue 3 to cpu 11 > >>>> >> ix1: Bound queue 4 to cpu 12 > >>>> >> ix1: Bound queue 5 to cpu 13 > >>>> >> ix1: Bound queue 6 to cpu 14 > >>>> >> ix1: Bound queue 7 to cpu 15 > >>>> >> ix1: Ethernet address: 0c:c4:7a:5e:be:65 > >>>> >> ix1: PCI Express Bus: Speed 5.0GT/s Width x8 > >>>> >> 001.000009 [2705] netmap_attach success for ix1 tx 8/4096 rx > >>>> >> 8/4096 queues/slots > >>>> >> > >>>> >> On Tue, Aug 11, 2015 at 4:14 PM, Olivier Cochard-Labb=C3=A9 < > >>>> olivier@cochard.me> > >>>> >> wrote: > >>>> >> > >>>> >>> On Tue, Aug 11, 2015 at 11:18 PM, Maxim Sobolev < > >>>> sobomax@freebsd.org> > >>>> >>> wrote: > >>>> >>> > >>>> >>>> Hi folks, > >>>> >>>> > >>>> >>>> Hi, > >>>> >>> > >>>> >>> > >>>> >>>> We've trying to migrate some of our high-PPS systems to a new > >>>> hardware > >>>> >>>> that > >>>> >>>> has four X540-AT2 10G NICs and observed that interrupt time go= es > >>>> through > >>>> >>>> roof after we cross around 200K PPS in and 200K out (two ports > in > >>>> LACP). > >>>> >>>> The previous hardware was stable up to about 350K PPS in and > 350K > >>>> out. I > >>>> >>>> believe the old one was equipped with the I350 and had the > >>>> identical LACP > >>>> >>>> configuration. The new box also has better CPU with more cores > >>>> (i.e. 24 > >>>> >>>> cores vs. 16 cores before). CPU itself is 2 x E5-2690 v3. > >>>> >>> 200K PPS, and even 350K PPS are very low value indeed. > >>>> >>> On a Intel Xeon L5630 (4 cores only) with one X540-AT2 > >>>> >>> > >>>> >>> (then 2 10Gigabit ports) I've reached about 1.8Mpps > >>>> (fastforwarding > >>>> >>> enabled) [1]. > >>>> >>> But my setup didn't use lagg(4): Can you disable lagg > configuration > >>>> and > >>>> >>> re-measure your performance without lagg ? > >>>> >>> > >>>> >>> Do you let Intel NIC drivers using 8 queues for port too? > >>>> >>> In my use case (forwarding smallest UDP packet size), I obtain > >>>> better > >>>> >>> behaviour by limiting NIC queues to 4 (hw.ix.num_queues or > >>>> >>> hw.ixgbe.num_queues, don't remember) if my system had 8 cores. > And > >>>> this > >>>> >>> with Gigabit Intel[2] or Chelsio NIC [3]. > >>>> >>> > >>>> >>> Don't forget to disable TSO and LRO too. > >>>> >>> > >>>> >>> Regards, > >>>> >>> > >>>> >>> Olivier > >>>> >>> > >>>> >>> [1] > >>>> >>> > >>>> > http://bsdrp.net/documentation/examples/forwarding_performance_lab_of_an_= ibm_system_x3550_m3_with_10-gigabit_intel_x540-at2#graphs > >>>> >>> [2] > >>>> >>> > >>>> > http://bsdrp.net/documentation/examples/forwarding_performance_lab_of_a_s= uperserver_5018a-ftn4#graph1 > >>>> >>> [3] > >>>> >>> > >>>> > http://bsdrp.net/documentation/examples/forwarding_performance_lab_of_a_h= p_proliant_dl360p_gen8_with_10-gigabit_with_10-gigabit_chelsio_t540-cr#redu= cing_nic_queues > >>>> >> _______________________________________________ > >>>> >> freebsd-net@freebsd.org mailing list > >>>> >> http://lists.freebsd.org/mailman/listinfo/freebsd-net > >>>> >> To unsubscribe, send any mail to " > freebsd-net-unsubscribe@freebsd.org > >>>> " > >>>> > _______________________________________________ > >>>> > freebsd-net@freebsd.org mailing list > >>>> > http://lists.freebsd.org/mailman/listinfo/freebsd-net > >>>> > To unsubscribe, send any mail to " > freebsd-net-unsubscribe@freebsd.org" > >>>> _______________________________________________ > >>>> freebsd-net@freebsd.org mailing list > >>>> http://lists.freebsd.org/mailman/listinfo/freebsd-net > >>>> To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.or= g > " > >>> > >>> > >>> > >>> -- > >>> > -----------------------------------------+------------------------------- > >>> Prof. Luigi RIZZO, rizzo@iet.unipi.it . Dip. di Ing. > dell'Informazione > >>> http://www.iet.unipi.it/~luigi/ . Universita` di Pisa > >>> TEL +39-050-2217533 . via Diotisalvi 2 > >>> Mobile +39-338-6809875 . 56122 PISA (Italy) > >>> > -----------------------------------------+------------------------------- > >>> > >>> > >> > >> > >> -- > >> Maksym Sobolyev > >> Sippy Software, Inc. > >> Internet Telephony (VoIP) Experts > >> Tel (Canada): +1-778-783-0474 > >> Tel (Toll-Free): +1-855-747-7779 > >> Fax: +1-866-857-6942 > >> Web: http://www.sippysoft.com > >> MSN: sales@sippysoft.com > >> Skype: SippySoft > >> > > _______________________________________________ > > freebsd-net@freebsd.org mailing list > > http://lists.freebsd.org/mailman/listinfo/freebsd-net > > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" > > From owner-freebsd-net@freebsd.org Thu Aug 13 06:47:10 2015 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 772E59A0C50 for ; Thu, 13 Aug 2015 06:47:10 +0000 (UTC) (envelope-from adrian.chadd@gmail.com) Received: from mail-ig0-x233.google.com (mail-ig0-x233.google.com [IPv6:2607:f8b0:4001:c05::233]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 3EFE0B62; Thu, 13 Aug 2015 06:47:10 +0000 (UTC) (envelope-from adrian.chadd@gmail.com) Received: by igui7 with SMTP id i7so76697597igu.1; Wed, 12 Aug 2015 23:47:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type:content-transfer-encoding; bh=gtdScgKGMI4bGva/O4v67f8hNsOFsJCcyiumhwfPg/A=; b=N9gB3gAo8gAEyaUUkR9AhLsqnaUglrB5sDfpgD1DRjtiLoPe9EY8vs5b+Sd3JP+LA7 B1BibSgQMvx7eCh4D2dRoi/rC9Rjodx3lWM96W5uFf4m08cGC507iTwFCDCPZwNGVS6Q tNZwGCN3WioYNorNsA5HYZAOd87mkT9zCYEhIJM/WUjo6dTucGCouaENs/w/+MPvIqrs bYcofUNxRgzH4ZqVIg3NdI52SRVB5IrQscukSHAiJrbZlRB/B3nDydm3LoQO0UJy5l27 2Jhgr4xBWtc7HDm/l9bcCeGDod29B+AYoq25w7M5ucxypVYLOQLGzBGT4/TPB41f2a+k eg0g== MIME-Version: 1.0 X-Received: by 10.50.43.227 with SMTP id z3mr26160434igl.22.1439448429483; Wed, 12 Aug 2015 23:47:09 -0700 (PDT) Received: by 10.36.38.133 with HTTP; Wed, 12 Aug 2015 23:47:09 -0700 (PDT) In-Reply-To: References: Date: Wed, 12 Aug 2015 23:47:09 -0700 Message-ID: Subject: Re: Poor high-PPS performance of the 10G ixgbe(9) NIC/driver in FreeBSD 10.1 From: Adrian Chadd To: Maxim Sobolev Cc: Luigi Rizzo , "Alexander V. Chernikov" , FreeBSD Net , Babak Farrokhi , "freebsd@intel.com" , =?UTF-8?Q?Jev_Bj=C3=B6rsell?= , =?UTF-8?Q?Olivier_Cochard=2DLabb=C3=A9?= Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 13 Aug 2015 06:47:10 -0000 Hi, Try this: * I'd disable AIM and hard-set interrupts to something sensible; * I'd edit sys/conf/files and sys/dev/ixgbe/Makefile on 10.1 and remove the '-DIXGBE_FDIR' bit that enabled flow director - the software setup for flow director is buggy, and it causes things to get wildly unhappy. -adrian On 12 August 2015 at 17:47, Maxim Sobolev wrote: > Here we go (ix2 and ix3 are not used): > > ix0@pci0:3:0:0: class=3D0x020000 card=3D0x152815d9 chip=3D0x15288086 rev= =3D0x01 > hdr=3D0x00 > vendor =3D 'Intel Corporation' > device =3D 'Ethernet Controller 10-Gigabit X540-AT2' > class =3D network > subclass =3D ethernet > ix1@pci0:3:0:1: class=3D0x020000 card=3D0x152815d9 chip=3D0x15288086 rev= =3D0x01 > hdr=3D0x00 > vendor =3D 'Intel Corporation' > device =3D 'Ethernet Controller 10-Gigabit X540-AT2' > class =3D network > subclass =3D ethernet > ix2@pci0:4:0:0: class=3D0x020000 card=3D0x152815d9 chip=3D0x15288086 rev= =3D0x01 > hdr=3D0x00 > vendor =3D 'Intel Corporation' > device =3D 'Ethernet Controller 10-Gigabit X540-AT2' > class =3D network > subclass =3D ethernet > ix3@pci0:4:0:1: class=3D0x020000 card=3D0x152815d9 chip=3D0x15288086 rev= =3D0x01 > hdr=3D0x00 > vendor =3D 'Intel Corporation' > device =3D 'Ethernet Controller 10-Gigabit X540-AT2' > class =3D network > subclass =3D ethernet > > > On Wed, Aug 12, 2015 at 8:23 AM, Adrian Chadd > wrote: >> >> Right, and for the ixgbe hardware? >> >> >> >> -a >> >> >> On 12 August 2015 at 08:05, Maxim Sobolev wrote: >> > igb0@pci0:7:0:0: class=3D0x020000 card=3D0x153315d9 chip=3D0x15= 338086 >> > rev=3D0x03 hdr=3D0x00 >> > vendor =3D 'Intel Corporation' >> > device =3D 'I210 Gigabit Network Connection' >> > class =3D network >> > subclass =3D ethernet >> > igb1@pci0:8:0:0: class=3D0x020000 card=3D0x153315d9 chip=3D0x15= 338086 >> > rev=3D0x03 hdr=3D0x00 >> > vendor =3D 'Intel Corporation' >> > device =3D 'I210 Gigabit Network Connection' >> > class =3D network >> > subclass =3D ethernet >> > >> > >> > On Wed, Aug 12, 2015 at 8:03 AM, Maxim Sobolev >> > wrote: >> > >> >> Ok, so my current settings are: >> >> >> >> hw.ix.max_interrupt_rate: 20000 >> >> dev.ix.0.queue0.interrupt_rate: 20000 >> >> dev.ix.0.queue1.interrupt_rate: 20000 >> >> dev.ix.0.queue2.interrupt_rate: 20000 >> >> dev.ix.0.queue3.interrupt_rate: 20000 >> >> dev.ix.0.queue4.interrupt_rate: 20000 >> >> dev.ix.0.queue5.interrupt_rate: 20000 >> >> dev.ix.1.queue0.interrupt_rate: 20000 >> >> dev.ix.1.queue1.interrupt_rate: 20000 >> >> dev.ix.1.queue2.interrupt_rate: 20000 >> >> dev.ix.1.queue3.interrupt_rate: 20000 >> >> dev.ix.1.queue4.interrupt_rate: 20000 >> >> dev.ix.1.queue5.interrupt_rate: 20000 >> >> dev.ix.0.enable_aim: 0 >> >> dev.ix.1.enable_aim: 0 >> >> dev.ix.2.enable_aim: 0 >> >> dev.ix.3.enable_aim: 0 >> >> hw.ix.num_queues:6 >> >> >> >> We also happen to have I210-based system with only 4 hardware queues, >> >> it >> >> would be interesting to see how it stacks up. >> >> >> >> On Wed, Aug 12, 2015 at 5:23 AM, Luigi Rizzo >> >> wrote: >> >> >> >>> As I was telling to maxim, you should disable aim because it only >> >>> matches >> >>> the max interrupt rate to the average packet size, which is the last >> >>> thing >> >>> you want. >> >>> >> >>> Setting the interrupt rate with sysctl (one per queue) gives you >> >>> precise >> >>> control on the max rate and (hence, extra latency). 20k interrupts/s >> >>> give >> >>> you 50us of latency, and the 2k slots in the queue are still enough = to >> >>> absorb a burst of min-sized frames hitting a single queue (the os wi= ll >> >>> start dropping long before that level, but that's another story). >> >>> >> >>> Cheers >> >>> Luigi >> >>> >> >>> On Wednesday, August 12, 2015, Babak Farrokhi >> >>> wrote: >> >>> >> >>>> I ran into the same problem with almost the same hardware (Intel >> >>>> X520) >> >>>> on 10-STABLE. HT/SMT is disabled and cards are configured with 8 >> >>>> queues, >> >>>> with the same sysctl tunings as sobomax@ did. I am not using lagg, = no >> >>>> FLOWTABLE. >> >>>> >> >>>> I experimented with pmcstat (RESOURCE_STALLS) a while ago and here >> >>>> [1] >> >>>> [2] you can see the results, including pmc output, callchain, >> >>>> flamegraph >> >>>> and gprof output. >> >>>> >> >>>> I am experiencing huge number of interrupts with 200kpps load: >> >>>> >> >>>> # sysctl dev.ix | grep interrupt_rate >> >>>> dev.ix.1.queue7.interrupt_rate: 125000 >> >>>> dev.ix.1.queue6.interrupt_rate: 6329 >> >>>> dev.ix.1.queue5.interrupt_rate: 500000 >> >>>> dev.ix.1.queue4.interrupt_rate: 100000 >> >>>> dev.ix.1.queue3.interrupt_rate: 50000 >> >>>> dev.ix.1.queue2.interrupt_rate: 500000 >> >>>> dev.ix.1.queue1.interrupt_rate: 500000 >> >>>> dev.ix.1.queue0.interrupt_rate: 100000 >> >>>> dev.ix.0.queue7.interrupt_rate: 500000 >> >>>> dev.ix.0.queue6.interrupt_rate: 6097 >> >>>> dev.ix.0.queue5.interrupt_rate: 10204 >> >>>> dev.ix.0.queue4.interrupt_rate: 5208 >> >>>> dev.ix.0.queue3.interrupt_rate: 5208 >> >>>> dev.ix.0.queue2.interrupt_rate: 71428 >> >>>> dev.ix.0.queue1.interrupt_rate: 5494 >> >>>> dev.ix.0.queue0.interrupt_rate: 6250 >> >>>> >> >>>> [1] http://farrokhi.net/~farrokhi/pmc/6/ >> >>>> [2] http://farrokhi.net/~farrokhi/pmc/7/ >> >>>> >> >>>> Regards, >> >>>> Babak >> >>>> >> >>>> >> >>>> Alexander V. Chernikov wrote: >> >>>> > 12.08.2015, 02:28, "Maxim Sobolev" : >> >>>> >> Olivier, keep in mind that we are not "kernel forwarding" packet= s, >> >>>> but "app >> >>>> >> forwarding", i.e. the packet goes full way >> >>>> >> net->kernel->recvfrom->app->sendto->kernel->net, which is why we >> >>>> >> have >> >>>> much >> >>>> >> lower PPS limits and which is why I think we are actually >> >>>> >> benefiting >> >>>> from >> >>>> >> the extra queues. Single-thread sendto() in a loop is CPU-bound = at >> >>>> about >> >>>> >> 220K PPS, and while running the test I am observing that outboun= d >> >>>> traffic >> >>>> >> from one thread is mapped into a specific queue (well, pair of >> >>>> >> queues >> >>>> on >> >>>> >> two separate adaptors, due to lagg load balancing action). And t= he >> >>>> peak >> >>>> >> performance of that test is at 7 threads, which I believe >> >>>> >> corresponds >> >>>> to >> >>>> >> the number of queues. We have plenty of CPU cores in the box (24= ) >> >>>> >> with >> >>>> >> HTT/SMT disabled and one CPU is mapped to a specific queue. This >> >>>> leaves us >> >>>> >> with at least 8 CPUs fully capable of running our app. If you lo= ok >> >>>> >> at >> >>>> the >> >>>> >> CPU utilization, we are at about 10% when the issue hits. >> >>>> > >> >>>> > In any case, it would be great if you could provide some profilin= g >> >>>> info since there could be >> >>>> > plenty of problematic places starting from TX rings contention to >> >>>> > some >> >>>> locks inside udp or even >> >>>> > (in)famous random entropy harvester.. >> >>>> > e.g. something like pmcstat -TS instructions -w1 might be >> >>>> > sufficient >> >>>> to determine the reason >> >>>> >> ix0: > >>>> 2.5.15> port >> >>>> >> 0x6020-0x603f mem 0xc7c00000-0xc7dfffff,0xc7e04000-0xc7e07fff ir= q >> >>>> >> 40 >> >>>> at >> >>>> >> device 0.0 on pci3 >> >>>> >> ix0: Using MSIX interrupts with 9 vectors >> >>>> >> ix0: Bound queue 0 to cpu 0 >> >>>> >> ix0: Bound queue 1 to cpu 1 >> >>>> >> ix0: Bound queue 2 to cpu 2 >> >>>> >> ix0: Bound queue 3 to cpu 3 >> >>>> >> ix0: Bound queue 4 to cpu 4 >> >>>> >> ix0: Bound queue 5 to cpu 5 >> >>>> >> ix0: Bound queue 6 to cpu 6 >> >>>> >> ix0: Bound queue 7 to cpu 7 >> >>>> >> ix0: Ethernet address: 0c:c4:7a:5e:be:64 >> >>>> >> ix0: PCI Express Bus: Speed 5.0GT/s Width x8 >> >>>> >> 001.000008 [2705] netmap_attach success for ix0 tx 8/4096 rx >> >>>> >> 8/4096 queues/slots >> >>>> >> ix1: > >>>> 2.5.15> port >> >>>> >> 0x6000-0x601f mem 0xc7a00000-0xc7bfffff,0xc7e00000-0xc7e03fff ir= q >> >>>> >> 44 >> >>>> at >> >>>> >> device 0.1 on pci3 >> >>>> >> ix1: Using MSIX interrupts with 9 vectors >> >>>> >> ix1: Bound queue 0 to cpu 8 >> >>>> >> ix1: Bound queue 1 to cpu 9 >> >>>> >> ix1: Bound queue 2 to cpu 10 >> >>>> >> ix1: Bound queue 3 to cpu 11 >> >>>> >> ix1: Bound queue 4 to cpu 12 >> >>>> >> ix1: Bound queue 5 to cpu 13 >> >>>> >> ix1: Bound queue 6 to cpu 14 >> >>>> >> ix1: Bound queue 7 to cpu 15 >> >>>> >> ix1: Ethernet address: 0c:c4:7a:5e:be:65 >> >>>> >> ix1: PCI Express Bus: Speed 5.0GT/s Width x8 >> >>>> >> 001.000009 [2705] netmap_attach success for ix1 tx 8/4096 rx >> >>>> >> 8/4096 queues/slots >> >>>> >> >> >>>> >> On Tue, Aug 11, 2015 at 4:14 PM, Olivier Cochard-Labb=C3=A9 < >> >>>> olivier@cochard.me> >> >>>> >> wrote: >> >>>> >> >> >>>> >>> On Tue, Aug 11, 2015 at 11:18 PM, Maxim Sobolev < >> >>>> sobomax@freebsd.org> >> >>>> >>> wrote: >> >>>> >>> >> >>>> >>>> Hi folks, >> >>>> >>>> >> >>>> >>>> Hi, >> >>>> >>> >> >>>> >>> >> >>>> >>>> We've trying to migrate some of our high-PPS systems to a new >> >>>> hardware >> >>>> >>>> that >> >>>> >>>> has four X540-AT2 10G NICs and observed that interrupt time >> >>>> >>>> goes >> >>>> through >> >>>> >>>> roof after we cross around 200K PPS in and 200K out (two port= s >> >>>> >>>> in >> >>>> LACP). >> >>>> >>>> The previous hardware was stable up to about 350K PPS in and >> >>>> >>>> 350K >> >>>> out. I >> >>>> >>>> believe the old one was equipped with the I350 and had the >> >>>> identical LACP >> >>>> >>>> configuration. The new box also has better CPU with more core= s >> >>>> (i.e. 24 >> >>>> >>>> cores vs. 16 cores before). CPU itself is 2 x E5-2690 v3. >> >>>> >>> 200K PPS, and even 350K PPS are very low value indeed. >> >>>> >>> On a Intel Xeon L5630 (4 cores only) with one X540-AT2 >> >>>> >>> >> >>>> >>> (then 2 10Gigabit ports) I've reached about 1.8Mpps >> >>>> (fastforwarding >> >>>> >>> enabled) [1]. >> >>>> >>> But my setup didn't use lagg(4): Can you disable lagg >> >>>> >>> configuration >> >>>> and >> >>>> >>> re-measure your performance without lagg ? >> >>>> >>> >> >>>> >>> Do you let Intel NIC drivers using 8 queues for port too? >> >>>> >>> In my use case (forwarding smallest UDP packet size), I obtain >> >>>> better >> >>>> >>> behaviour by limiting NIC queues to 4 (hw.ix.num_queues or >> >>>> >>> hw.ixgbe.num_queues, don't remember) if my system had 8 cores. >> >>>> >>> And >> >>>> this >> >>>> >>> with Gigabit Intel[2] or Chelsio NIC [3]. >> >>>> >>> >> >>>> >>> Don't forget to disable TSO and LRO too. >> >>>> >>> >> >>>> >>> Regards, >> >>>> >>> >> >>>> >>> Olivier >> >>>> >>> >> >>>> >>> [1] >> >>>> >>> >> >>>> >> >>>> http://bsdrp.net/documentation/examples/forwarding_performance_lab_= of_an_ibm_system_x3550_m3_with_10-gigabit_intel_x540-at2#graphs >> >>>> >>> [2] >> >>>> >>> >> >>>> >> >>>> http://bsdrp.net/documentation/examples/forwarding_performance_lab_= of_a_superserver_5018a-ftn4#graph1 >> >>>> >>> [3] >> >>>> >>> >> >>>> >> >>>> http://bsdrp.net/documentation/examples/forwarding_performance_lab_= of_a_hp_proliant_dl360p_gen8_with_10-gigabit_with_10-gigabit_chelsio_t540-c= r#reducing_nic_queues >> >>>> >> _______________________________________________ >> >>>> >> freebsd-net@freebsd.org mailing list >> >>>> >> http://lists.freebsd.org/mailman/listinfo/freebsd-net >> >>>> >> To unsubscribe, send any mail to >> >>>> >> "freebsd-net-unsubscribe@freebsd.org >> >>>> " >> >>>> > _______________________________________________ >> >>>> > freebsd-net@freebsd.org mailing list >> >>>> > http://lists.freebsd.org/mailman/listinfo/freebsd-net >> >>>> > To unsubscribe, send any mail to >> >>>> > "freebsd-net-unsubscribe@freebsd.org" >> >>>> _______________________________________________ >> >>>> freebsd-net@freebsd.org mailing list >> >>>> http://lists.freebsd.org/mailman/listinfo/freebsd-net >> >>>> To unsubscribe, send any mail to >> >>>> "freebsd-net-unsubscribe@freebsd.org" >> >>> >> >>> >> >>> >> >>> -- >> >>> >> >>> -----------------------------------------+--------------------------= ----- >> >>> Prof. Luigi RIZZO, rizzo@iet.unipi.it . Dip. di Ing. >> >>> dell'Informazione >> >>> http://www.iet.unipi.it/~luigi/ . Universita` di Pisa >> >>> TEL +39-050-2217533 . via Diotisalvi 2 >> >>> Mobile +39-338-6809875 . 56122 PISA (Italy) >> >>> >> >>> -----------------------------------------+--------------------------= ----- >> >>> >> >>> >> >> >> >> >> >> -- >> >> Maksym Sobolyev >> >> Sippy Software, Inc. >> >> Internet Telephony (VoIP) Experts >> >> Tel (Canada): +1-778-783-0474 >> >> Tel (Toll-Free): +1-855-747-7779 >> >> Fax: +1-866-857-6942 >> >> Web: http://www.sippysoft.com >> >> MSN: sales@sippysoft.com >> >> Skype: SippySoft >> >> >> > _______________________________________________ >> > freebsd-net@freebsd.org mailing list >> > http://lists.freebsd.org/mailman/listinfo/freebsd-net >> > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" >> > From owner-freebsd-net@freebsd.org Thu Aug 13 06:50:03 2015 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id F3DAA9A0CB2 for ; Thu, 13 Aug 2015 06:50:02 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id E0099C59 for ; Thu, 13 Aug 2015 06:50:02 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id t7D6o2Z5041709 for ; Thu, 13 Aug 2015 06:50:02 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-net@FreeBSD.org Subject: [Bug 178782] [ixgbe] 82599EB SFP does not work with passthrough under KVM. Date: Thu, 13 Aug 2015 06:50:03 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 9.1-PRERELEASE X-Bugzilla-Keywords: IntelNetworking X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: legumen@foxmail.com X-Bugzilla-Status: In Progress X-Bugzilla-Priority: Normal X-Bugzilla-Assigned-To: freebsd-net@FreeBSD.org X-Bugzilla-Target-Milestone: --- X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 7bit X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 13 Aug 2015 06:50:03 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=178782 --- Comment #3 from legumen@foxmail.com --- (In reply to Sean Bruno from comment #2) Sorry Sean. I'm not sure if it still occur now or not. I changed my job for quite a long time! And I believe my former colleagues may solve(or have workaround for) this, because I noticed their product was announced last year. ;> -- You are receiving this mail because: You are the assignee for the bug. From owner-freebsd-net@freebsd.org Thu Aug 13 16:29:31 2015 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 8D67A9B8638 for ; Thu, 13 Aug 2015 16:29:31 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 7208F77F for ; Thu, 13 Aug 2015 16:29:31 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id t7DGTVuP021039 for ; Thu, 13 Aug 2015 16:29:31 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-net@FreeBSD.org Subject: [Bug 178782] [ixgbe] 82599EB SFP does not work with passthrough under KVM. Date: Thu, 13 Aug 2015 16:29:31 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 9.1-PRERELEASE X-Bugzilla-Keywords: IntelNetworking X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: sbruno@FreeBSD.org X-Bugzilla-Status: In Progress X-Bugzilla-Priority: Normal X-Bugzilla-Assigned-To: freebsd-net@FreeBSD.org X-Bugzilla-Target-Milestone: --- X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 7bit X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 13 Aug 2015 16:29:31 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=178782 --- Comment #4 from Sean Bruno --- (In reply to legumen from comment #3) ok, I'm hunting around for some linux folks that can test this and validate. -- You are receiving this mail because: You are the assignee for the bug. From owner-freebsd-net@freebsd.org Thu Aug 13 17:11:59 2015 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id B309E9B8E3F for ; Thu, 13 Aug 2015 17:11:59 +0000 (UTC) (envelope-from ebastan10@gmail.com) Received: from mail-io0-x22c.google.com (mail-io0-x22c.google.com [IPv6:2607:f8b0:4001:c06::22c]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 7E62490 for ; Thu, 13 Aug 2015 17:11:59 +0000 (UTC) (envelope-from ebastan10@gmail.com) Received: by iodb91 with SMTP id b91so58707713iod.1 for ; Thu, 13 Aug 2015 10:11:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=g99V0UXrpRvwI5O/O7bMcDPNv3OKa3hh/Wx0KfvRiGU=; b=r/w+QNQB5jYeQd5V6Sd5CgLvFpUEAGNgM/yAZGObx397vqGLPh13m9AvUkhCc1a2W9 Y5LZHvw894y9l7rh8u/pxnce8Bts7CDsu3Wey+1FzO3OaGMWEzgUON491jV2jhv/NEYV LjJSI/VZaZieMKgkOO6p1N5tGFWP7Yr4dkJ8zNYsZ/aS2l9KjjD6ql1LuKjhp5A+R+ve KSxvvQj+6XftqDIIf8x44One9nYAY6e0NfVIFmjKsWP1MYAloaAulHJzbHAtbtcWOmba a8gLAcyC5S5/h61tLNaJKCw+vJ0UizS0aLg12U21RDb8CpO/IgPvyBt80ZkWlnAYFk3Q y7OQ== MIME-Version: 1.0 X-Received: by 10.107.19.94 with SMTP id b91mr3044014ioj.144.1439485918828; Thu, 13 Aug 2015 10:11:58 -0700 (PDT) Received: by 10.107.9.67 with HTTP; Thu, 13 Aug 2015 10:11:58 -0700 (PDT) In-Reply-To: References: Date: Thu, 13 Aug 2015 21:41:58 +0430 Message-ID: Subject: Re: multicasts are not forwarded on wifi interface From: Hooshang F To: Adrian Chadd Cc: FreeBSD Net Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 13 Aug 2015 17:11:59 -0000 On Wed, Aug 12, 2015 at 10:04 PM, Adrian Chadd wrote: > HI, > > Would you please try freebsd-head? I think this was fixed a while ago, > but I don't think it was ever backported to freebsd-8. > > Thanks! > Unfortunately, it is not possible for us to change the OS right now. I would appreciate if you provide some explanation about the problem and links to revisions where the problem has been fixed to see if we can back-port the fix to 8.3. Thanks > > > -a > > > On 12 August 2015 at 10:20, Hooshang F wrote: > > Hi, > > > > We have a freebsd 8.3 wifi-AP running as a VirtulaBox guest > > with 2 ethernet (em0 & em1) and one USB wifi interface (wlan0). > Everything > > works except that multicast packets arriving on wlan0 interface are not > > forwarded on Ethernet interfaces. Also, multicast packets arriving on > > Ethernet interfaces are not forwarded on wlan0 interface (however, > > multicasts are forwarded between Ethernet interfaces). As a result, > > portable devices are not able to discover > > DLNA media servers using UPnP. > > > > The kernel is compiled with MROUTING option and mrouted > > is running with default configuration. > > > > Any points/hints is appreciated. > > _______________________________________________ > > freebsd-net@freebsd.org mailing list > > http://lists.freebsd.org/mailman/listinfo/freebsd-net > > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" > From owner-freebsd-net@freebsd.org Thu Aug 13 17:35:14 2015 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id ED28C9B8821 for ; Thu, 13 Aug 2015 17:35:14 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id D93531C9E for ; Thu, 13 Aug 2015 17:35:14 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id t7DHZESP059633 for ; Thu, 13 Aug 2015 17:35:14 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-net@FreeBSD.org Subject: [Bug 178782] [ixgbe] 82599EB SFP does not work with passthrough under KVM. Date: Thu, 13 Aug 2015 17:35:15 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 9.1-PRERELEASE X-Bugzilla-Keywords: IntelNetworking X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: jeffrey.e.pieper@intel.com X-Bugzilla-Status: In Progress X-Bugzilla-Priority: Normal X-Bugzilla-Assigned-To: freebsd-net@FreeBSD.org X-Bugzilla-Target-Milestone: --- X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: cc Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 7bit X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 13 Aug 2015 17:35:15 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=178782 Jeff Pieper changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |jeffrey.e.pieper@intel.com --- Comment #5 from Jeff Pieper --- I can test this. I'll be using 10.1 VM on a RHEL 7.1 host. -- You are receiving this mail because: You are the assignee for the bug. From owner-freebsd-net@freebsd.org Thu Aug 13 17:40:18 2015 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id CE4B19B8076 for ; Thu, 13 Aug 2015 17:40:18 +0000 (UTC) (envelope-from adrian.chadd@gmail.com) Received: from mail-io0-x22e.google.com (mail-io0-x22e.google.com [IPv6:2607:f8b0:4001:c06::22e]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 9B3872EF for ; Thu, 13 Aug 2015 17:40:18 +0000 (UTC) (envelope-from adrian.chadd@gmail.com) Received: by iodv127 with SMTP id v127so43933029iod.3 for ; Thu, 13 Aug 2015 10:40:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=oe9wSSWN+KnF5CxUlCJyxXPF+Y5Hj8ctgdGzSbixpVA=; b=qh8GrpY+0y5tc6SV3mICx+Uk+s6DuowvC7B1FLbFJMpkTbdICghBlf6SxjZfd1YQv0 MdODa6pEw0XCyx5K+iM1uxQLCT4CuC4TGRcrbLypjoFp2kxtIJG3OJI6HRD2BirKFmQQ RY524lTFFTr5cv1O1zbCS5o9ugPjYKY39IhrAR8sutXQ7v+ah3UcsTv/BZNnf47RgA1U MKGCt4cgNjkcZIGrsvxQWrZCPTIXrWdjspdI8HF7F7wb+H97qdYOFRfNen3mOqQNAxBh onLpzPB4kynnUAiZVYtfooRogcrpkZrJ1tHCl5yUZ1+pOmqRXUzUe8bS60crqXi6qeIU Maew== MIME-Version: 1.0 X-Received: by 10.107.156.73 with SMTP id f70mr6214033ioe.165.1439487618033; Thu, 13 Aug 2015 10:40:18 -0700 (PDT) Received: by 10.36.38.133 with HTTP; Thu, 13 Aug 2015 10:40:17 -0700 (PDT) In-Reply-To: References: Date: Thu, 13 Aug 2015 10:40:17 -0700 Message-ID: Subject: Re: multicasts are not forwarded on wifi interface From: Adrian Chadd To: Hooshang F Cc: FreeBSD Net Content-Type: text/plain; charset=UTF-8 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 13 Aug 2015 17:40:18 -0000 I forget the details; I'd have to go dig through the commit history. Which wifi NIC is it? -a On 13 August 2015 at 10:11, Hooshang F wrote: > > On Wed, Aug 12, 2015 at 10:04 PM, Adrian Chadd > wrote: >> >> HI, >> >> Would you please try freebsd-head? I think this was fixed a while ago, >> but I don't think it was ever backported to freebsd-8. >> >> Thanks! > > > Unfortunately, it is not possible for us to change the OS right now. > I would appreciate if you provide some explanation about the > problem and links to revisions where the problem has been fixed to > see if we can back-port the fix to 8.3. > > Thanks > > >> >> >> >> -a >> >> >> On 12 August 2015 at 10:20, Hooshang F wrote: >> > Hi, >> > >> > We have a freebsd 8.3 wifi-AP running as a VirtulaBox guest >> > with 2 ethernet (em0 & em1) and one USB wifi interface (wlan0). >> > Everything >> > works except that multicast packets arriving on wlan0 interface are not >> > forwarded on Ethernet interfaces. Also, multicast packets arriving on >> > Ethernet interfaces are not forwarded on wlan0 interface (however, >> > multicasts are forwarded between Ethernet interfaces). As a result, >> > portable devices are not able to discover >> > DLNA media servers using UPnP. >> > >> > The kernel is compiled with MROUTING option and mrouted >> > is running with default configuration. >> > >> > Any points/hints is appreciated. >> > _______________________________________________ >> > freebsd-net@freebsd.org mailing list >> > http://lists.freebsd.org/mailman/listinfo/freebsd-net >> > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" > > From owner-freebsd-net@freebsd.org Thu Aug 13 18:50:51 2015 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 8F2F09B8F2C for ; Thu, 13 Aug 2015 18:50:51 +0000 (UTC) (envelope-from ebastan10@gmail.com) Received: from mail-ig0-x22b.google.com (mail-ig0-x22b.google.com [IPv6:2607:f8b0:4001:c05::22b]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 584CC1CAC for ; Thu, 13 Aug 2015 18:50:51 +0000 (UTC) (envelope-from ebastan10@gmail.com) Received: by igfj19 with SMTP id j19so5829649igf.0 for ; Thu, 13 Aug 2015 11:50:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=ax9zomfmFLmQyHPk94At9Lgbvj8GS+16Z+1UWws3eWY=; b=KJ3fZ0IYPJKfJHpOdJMi0hYoQaryrRa/cyj+/fpz+B029/IhVqoI/B0c49CPCWajgW Ekb//1kXWzckgGhoLC401H5CmoPLXt8NKfMMgNH5v3BfA/dLhIxdcQIjXKXcs1FrRTTW Z34WV7RFoqojwSjb4qUIaHS+ZdtzvFhUkCD2EFcfAkgNDOaslqEAwRYpPKSCtfgtJjE6 WVpoc4AnQLculRWqvLmE+n0uJZP5gobu269LDItaMF4Bvcv7gD3VHwDXrWIYIaEeTh59 ZuG2az9qohNH360qN7tZ6tHiA9DK1oxBVrBmLvgz+BtAoZdqPcJF/8Ox6324JQvcyTrr GyPw== MIME-Version: 1.0 X-Received: by 10.50.66.197 with SMTP id h5mr4539641igt.81.1439491850736; Thu, 13 Aug 2015 11:50:50 -0700 (PDT) Received: by 10.107.9.67 with HTTP; Thu, 13 Aug 2015 11:50:50 -0700 (PDT) In-Reply-To: References: Date: Thu, 13 Aug 2015 23:20:50 +0430 Message-ID: Subject: Re: multicasts are not forwarded on wifi interface From: Hooshang F To: Adrian Chadd Cc: FreeBSD Net Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 13 Aug 2015 18:50:51 -0000 On Thu, Aug 13, 2015 at 10:10 PM, Adrian Chadd wrote: > I forget the details; I'd have to go dig through the commit history. > > Which wifi NIC is it? > > > -a > It is a TP-Link TL-WN7200ND wireless USB adapter # dmesg | grep run run0: <1.0> on usbus0 run0: MAC/BBP RT3070 (rev 0x0201), RF RT3020 (MIMO 1T1R), address c0:4a:00:26:02:58 run0: firmware RT2870 loaded #ifconfig wlan0 wlan0: flags=8843 metric 0 mtu 1500 ether c0:4a:00:26:02:58 inet 192.168.20.10 netmask 0xffffff00 broadcast 192.168.20.255 media: IEEE 802.11 Wireless Ethernet autoselect mode 11g status: running ssid xxxx channel 1 (2412 MHz 11g) bssid c0:4a:00:26:02:58 country US authmode WPA2/802.11i privacy MIXED deftxkey 3 AES-CCM 2:128-bit AES-CCM 3:128-bit txpower 0 scanvalid 60 protmode CTS wme dtimperiod 1 -dfs > > > On 13 August 2015 at 10:11, Hooshang F wrote: > > > > On Wed, Aug 12, 2015 at 10:04 PM, Adrian Chadd > > wrote: > >> > >> HI, > >> > >> Would you please try freebsd-head? I think this was fixed a while ago, > >> but I don't think it was ever backported to freebsd-8. > >> > >> Thanks! > > > > > > Unfortunately, it is not possible for us to change the OS right now. > > I would appreciate if you provide some explanation about the > > problem and links to revisions where the problem has been fixed to > > see if we can back-port the fix to 8.3. > > > > Thanks > > > > > >> > >> > >> > >> -a > >> > >> > >> On 12 August 2015 at 10:20, Hooshang F wrote: > >> > Hi, > >> > > >> > We have a freebsd 8.3 wifi-AP running as a VirtulaBox guest > >> > with 2 ethernet (em0 & em1) and one USB wifi interface (wlan0). > >> > Everything > >> > works except that multicast packets arriving on wlan0 interface are > not > >> > forwarded on Ethernet interfaces. Also, multicast packets arriving on > >> > Ethernet interfaces are not forwarded on wlan0 interface (however, > >> > multicasts are forwarded between Ethernet interfaces). As a result, > >> > portable devices are not able to discover > >> > DLNA media servers using UPnP. > >> > > >> > The kernel is compiled with MROUTING option and mrouted > >> > is running with default configuration. > >> > > >> > Any points/hints is appreciated. > >> > _______________________________________________ > >> > freebsd-net@freebsd.org mailing list > >> > http://lists.freebsd.org/mailman/listinfo/freebsd-net > >> > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org > " > > > > > From owner-freebsd-net@freebsd.org Thu Aug 13 20:43:20 2015 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 59E849B8524 for ; Thu, 13 Aug 2015 20:43:20 +0000 (UTC) (envelope-from sobomax@sippysoft.com) Received: from mail-wi0-f179.google.com (mail-wi0-f179.google.com [209.85.212.179]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id EE63FA19 for ; Thu, 13 Aug 2015 20:43:19 +0000 (UTC) (envelope-from sobomax@sippysoft.com) Received: by wicja10 with SMTP id ja10so163503039wic.1 for ; Thu, 13 Aug 2015 13:43:11 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:sender:date:message-id:subject:from :to:cc:content-type; bh=h6gHhS24Cb5lMQUrJIbSyUVRczzMGwfy1Jc8dtGv3cs=; b=labDtKjhk8DvMd75HR7FyiTyEyCWCWUG6mkChRCqCJ6TW0TNuEQtGcAm6qHpVYJLAC 1BCUh2lY1I4wl0alE90OqZzjLe9xgV4qj6oxqrtD34f0sTtw7TwquDAjKDp0RQuBNzWG FEgOFUz9nkkhDEOcKsmz1u67fmgXyOfrnVWvLMjtBzCW8/n+bTZbrgsJhH6UJ1VlH+ar Y3NGRyH3cv7jecVIj1vwIpkZV9EvrIlR10jUYKMiia1sW7NoE4uAlgr8GNepnE3piaBx j8QCKTQXWG5dVeC9mvOoY0EMc2XFoTkL5VXkvM2j4kLPkpxSPZH54nLMd2fMLUzaf88f XdMg== X-Gm-Message-State: ALoCoQlHi8Oo1d9G4s2Um9U6Ln3TeTkloHLcl2itY7MNVHyIsXjKHiwh/+7Pk9SZLvbP3JiHb98x MIME-Version: 1.0 X-Received: by 10.180.91.14 with SMTP id ca14mr10537731wib.5.1439498591733; Thu, 13 Aug 2015 13:43:11 -0700 (PDT) Sender: sobomax@sippysoft.com Received: by 10.27.143.15 with HTTP; Thu, 13 Aug 2015 13:43:11 -0700 (PDT) Date: Thu, 13 Aug 2015 13:43:11 -0700 X-Google-Sender-Auth: vTa5vjzrqCTxG1b6_ad0_zpolho Message-ID: Subject: Re: Poor high-PPS performance of the 10G ixgbe(9) NIC/driver in FreeBSD 10.1 From: Maxim Sobolev To: Adrian Chadd Cc: Luigi Rizzo , "Alexander V. Chernikov" , FreeBSD Net , Babak Farrokhi , "freebsd@intel.com" , =?UTF-8?Q?Jev_Bj=C3=B6rsell?= , =?UTF-8?Q?Olivier_Cochard=2DLabb=C3=A9?= Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 13 Aug 2015 20:43:20 -0000 Thanks, we'll try that as well. We have not got as much traffic in the past 2 days, so we were running at about 140Kpps, well below the level that used to cause issues before. I'll try to redistribute traffic tomorrow so that we get it tested. -Max On Wed, Aug 12, 2015 at 11:47 PM, Adrian Chadd wrote: > Hi, > > Try this: > > * I'd disable AIM and hard-set interrupts to something sensible; > * I'd edit sys/conf/files and sys/dev/ixgbe/Makefile on 10.1 and > remove the '-DIXGBE_FDIR' bit that enabled flow director - the > software setup for flow director is buggy, and it causes things to get > wildly unhappy. > From owner-freebsd-net@freebsd.org Thu Aug 13 21:38:29 2015 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 1FBBD9B8EFA for ; Thu, 13 Aug 2015 21:38:29 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 0C12AAF7 for ; Thu, 13 Aug 2015 21:38:29 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id t7DLcSSW013695 for ; Thu, 13 Aug 2015 21:38:28 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-net@FreeBSD.org Subject: [Bug 178782] [ixgbe] 82599EB SFP does not work with passthrough under KVM. Date: Thu, 13 Aug 2015 21:38:29 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 9.1-PRERELEASE X-Bugzilla-Keywords: IntelNetworking X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: jeffrey.e.pieper@intel.com X-Bugzilla-Status: In Progress X-Bugzilla-Priority: Normal X-Bugzilla-Assigned-To: freebsd-net@FreeBSD.org X-Bugzilla-Target-Milestone: --- X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 7bit X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 13 Aug 2015 21:38:29 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=178782 --- Comment #6 from Jeff Pieper --- 10.1 (ixgbe-2.5.15) still fails (no link and no interrupts), however I also tested a VM using an 11.0-CURRENT-amd64-20150804-r286285 snapshot, which has ix-3.1.0 and it gets link and passes traffic. Since the driver in HEAD is working, I think this can be closed. -- You are receiving this mail because: You are the assignee for the bug. From owner-freebsd-net@freebsd.org Thu Aug 13 21:42:16 2015 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 0EAA699F084 for ; Thu, 13 Aug 2015 21:42:16 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id EF374E17 for ; Thu, 13 Aug 2015 21:42:15 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id t7DLgFgq023339 for ; Thu, 13 Aug 2015 21:42:15 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-net@FreeBSD.org Subject: [Bug 178782] [ixgbe] 82599EB SFP does not work with passthrough under KVM. Date: Thu, 13 Aug 2015 21:42:16 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 9.1-PRERELEASE X-Bugzilla-Keywords: IntelNetworking X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: sbruno@FreeBSD.org X-Bugzilla-Status: In Progress X-Bugzilla-Priority: Normal X-Bugzilla-Assigned-To: freebsd-net@FreeBSD.org X-Bugzilla-Target-Milestone: --- X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 7bit X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 13 Aug 2015 21:42:16 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=178782 --- Comment #7 from Sean Bruno --- (In reply to Jeff Pieper from comment #6) Thanks for testing this, it really saved me quite a bit of time in setup. Do you have time to test 10.2R in the next week or so? I think its about to be pushed out by release engineering. -- You are receiving this mail because: You are the assignee for the bug. From owner-freebsd-net@freebsd.org Thu Aug 13 21:42:57 2015 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id DB2F099F0C1 for ; Thu, 13 Aug 2015 21:42:57 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id C7D19EB0 for ; Thu, 13 Aug 2015 21:42:57 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id t7DLgvBc024119 for ; Thu, 13 Aug 2015 21:42:57 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-net@FreeBSD.org Subject: [Bug 178782] [ixgbe] 82599EB SFP does not work with passthrough under KVM. Date: Thu, 13 Aug 2015 21:42:57 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 9.1-PRERELEASE X-Bugzilla-Keywords: IntelNetworking X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: sbruno@FreeBSD.org X-Bugzilla-Status: In Progress X-Bugzilla-Priority: Normal X-Bugzilla-Assigned-To: freebsd-net@FreeBSD.org X-Bugzilla-Target-Milestone: --- X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 7bit X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 13 Aug 2015 21:42:57 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=178782 --- Comment #8 from Sean Bruno --- (In reply to Sean Bruno from comment #7) ftp://ftp.freebsd.org/pub/FreeBSD/releases/ISO-IMAGES/10.2/ -- You are receiving this mail because: You are the assignee for the bug. From owner-freebsd-net@freebsd.org Thu Aug 13 22:39:21 2015 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id F00A599F9FE for ; Thu, 13 Aug 2015 22:39:21 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id D2759BD3 for ; Thu, 13 Aug 2015 22:39:21 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id t7DMdLGg044161 for ; Thu, 13 Aug 2015 22:39:21 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-net@FreeBSD.org Subject: [Bug 178782] [ixgbe] 82599EB SFP does not work with passthrough under KVM. Date: Thu, 13 Aug 2015 22:39:21 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 9.1-PRERELEASE X-Bugzilla-Keywords: IntelNetworking X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: jeffrey.e.pieper@intel.com X-Bugzilla-Status: In Progress X-Bugzilla-Priority: Normal X-Bugzilla-Assigned-To: freebsd-net@FreeBSD.org X-Bugzilla-Target-Milestone: --- X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 7bit X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 13 Aug 2015 22:39:22 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=178782 --- Comment #9 from Jeff Pieper --- (In reply to Sean Bruno from comment #8) 10.2-RELEASE works too (ix-2.8.3). -- You are receiving this mail because: You are the assignee for the bug. From owner-freebsd-net@freebsd.org Thu Aug 13 22:50:59 2015 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id B818699FBAC for ; Thu, 13 Aug 2015 22:50:59 +0000 (UTC) (envelope-from james@lottspot.com) Received: from mx0.lottspot.com (sfo.lottspot.com [198.199.98.33]) by mx1.freebsd.org (Postfix) with ESMTP id A25B42F for ; Thu, 13 Aug 2015 22:50:59 +0000 (UTC) (envelope-from james@lottspot.com) Received: from localhost (localhost [127.0.0.1]) by mail.lottspot.com (Postfix) with ESMTP id B9B6F41260 for ; Thu, 13 Aug 2015 15:41:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=lottspot.com; h= content-transfer-encoding:content-type:content-type:mime-version :user-agent:date:date:message-id:subject:subject:from:from :received:received; s=mail; t=1439505641; bh=bggeCP8pF1g8eRAQwzS bFQ40Mrsv7YUUQ+Sn/9capzc=; b=siyE98K7Y8WAHu8m8gbSNUbgA09iM+v7yul JKumTtKKgFHD/+ox9ZQv6wgkxbaiDrYTeohjVju72uiiiUwajQhZuCuYIXIp5d/L +I3oEKcIIzBsjaUm5UW+MHmXB3ZLYA1kBNfAHzkmvZpFuBAxt2WPxR1Ax+N/31bJ 3JLpvKK0= X-Virus-Scanned: amavisd-new at lottspot.com Received: from mx0.lottspot.com ([127.0.0.1]) by localhost (mail.lottspot.com [127.0.0.1]) (amavisd-new, port 10024) with LMTP id p7aL-ZpTBZNB for ; Thu, 13 Aug 2015 15:40:41 -0700 (PDT) Received: from [192.168.0.6] (h69-131-58-73.nrfdvt.dsl.dynamic.tds.net [69.131.58.73]) by mx0.lottspot.com (Postfix) with ESMTPSA id CBEC4410DB for ; Thu, 13 Aug 2015 15:40:40 -0700 (PDT) To: freebsd-net@freebsd.org From: James Lott Subject: Ethernet tunneling options under FreeBSD Message-ID: <55CD1CE6.2010502@lottspot.com> Date: Thu, 13 Aug 2015 15:40:38 -0700 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:38.0) Gecko/20100101 Thunderbird/38.1.0 MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 13 Aug 2015 22:50:59 -0000 Hello list, I am in the process of planning a build out of a L2 VPN, in which I'd like to have my primary "switch" and DHCP server be a FreeBSD system. I would like to join each new host to the VPN by establishing an IP tunnel with the primary "switch" which transports ethernet frames over the tunnel. So far, the only protocol I have found supported by FreeBSD which seems capable of this is EtherIP. As far as I can tell, it doesn't look like there is any support for L2TPv3, and none of the PPP implementations available appear to support BCP. I'm not completely opposed to using EtherIP, but if there is something more modern which will meet my needs, I would probably try that first. So my question becomes: * Does anyone know of a method supported under FreeBSD (other than EtherIP) for tunneling ethernet over IP that they may be able to suggest I check out? Thanks for any suggestions! From owner-freebsd-net@freebsd.org Thu Aug 13 22:58:13 2015 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 5283599FD17 for ; Thu, 13 Aug 2015 22:58:13 +0000 (UTC) (envelope-from kpaasial@gmail.com) Received: from mail-ig0-x22a.google.com (mail-ig0-x22a.google.com [IPv6:2607:f8b0:4001:c05::22a]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 21D6336F for ; Thu, 13 Aug 2015 22:58:13 +0000 (UTC) (envelope-from kpaasial@gmail.com) Received: by igui7 with SMTP id i7so1252814igu.0 for ; Thu, 13 Aug 2015 15:58:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=Gt7pdJG6kafcl4wKItosJs9+6r2nnb0LKSHq/GBLISQ=; b=wdotS1ctZvCowfreUltGGaaNQejMumjdkXpYoBg0yKdXmHOt/KaTylst44UkTtwcgl m58jbjF7srf5IfTmwCgUkR4SStZLLs8UlDMsFIZqwPC49ns9i9ZenHFXl2XN6nSPI2GQ eJBaxQfA0WWrV8xfYANvqXu3SIJK6aw9lapd67IImLTZAUlEJGFyTOPU5nL6vXm6h+4P oBWH0FewEU8Ylm9pKBy0haFAs0D91t4t7nyvvsHHQaFDt1LQROoU3T0i63qKGyAOu5fL 6ue9OlYyLOkS5pko/vYyD1zm1g06vFrdpl0ha1v+ySdycRrBl91UAbwRali8iHCzfVXI HfkA== MIME-Version: 1.0 X-Received: by 10.50.78.133 with SMTP id b5mr5839106igx.32.1439506692198; Thu, 13 Aug 2015 15:58:12 -0700 (PDT) Received: by 10.107.131.163 with HTTP; Thu, 13 Aug 2015 15:58:12 -0700 (PDT) In-Reply-To: <55CD1CE6.2010502@lottspot.com> References: <55CD1CE6.2010502@lottspot.com> Date: Fri, 14 Aug 2015 01:58:12 +0300 Message-ID: Subject: Re: Ethernet tunneling options under FreeBSD From: Kimmo Paasiala To: James Lott Cc: FreeBSD Net Content-Type: text/plain; charset=UTF-8 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 13 Aug 2015 22:58:13 -0000 On Fri, Aug 14, 2015 at 1:40 AM, James Lott wrote: > Hello list, > > I am in the process of planning a build out of a L2 VPN, in which I'd like > to have my primary "switch" and DHCP server be a FreeBSD system. I would > like to join each new host to the VPN by establishing an IP tunnel with the > primary "switch" which transports ethernet frames over the tunnel. > > So far, the only protocol I have found supported by FreeBSD which seems > capable of this is EtherIP. As far as I can tell, it doesn't look like there > is any support for L2TPv3, and none of the PPP implementations available > appear to support BCP. > > I'm not completely opposed to using EtherIP, but if there is something more > modern which will meet my needs, I would probably try that first. So my > question becomes: > > * Does anyone know of a method supported under FreeBSD (other than EtherIP) > for tunneling ethernet over IP that they may be able to suggest I check out? > > Thanks for any suggestions! > First thing that comes to my mind is OpenVPN on a tap(4) interface. -Kimmo From owner-freebsd-net@freebsd.org Thu Aug 13 23:03:14 2015 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 0DDEA99FE6B for ; Thu, 13 Aug 2015 23:03:14 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id EE0028E7 for ; Thu, 13 Aug 2015 23:03:13 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id t7DN3Db8019310 for ; Thu, 13 Aug 2015 23:03:13 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-net@FreeBSD.org Subject: [Bug 178782] [ixgbe] 82599EB SFP does not work with passthrough under KVM. Date: Thu, 13 Aug 2015 23:03:13 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 9.1-PRERELEASE X-Bugzilla-Keywords: IntelNetworking X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: sbruno@FreeBSD.org X-Bugzilla-Status: Closed X-Bugzilla-Priority: Normal X-Bugzilla-Assigned-To: freebsd-net@FreeBSD.org X-Bugzilla-Target-Milestone: --- X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: bug_status resolution Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 7bit X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 13 Aug 2015 23:03:14 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=178782 Sean Bruno changed: What |Removed |Added ---------------------------------------------------------------------------- Status|In Progress |Closed Resolution|--- |FIXED -- You are receiving this mail because: You are the assignee for the bug. From owner-freebsd-net@freebsd.org Thu Aug 13 23:39:45 2015 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id AD8E19B839D for ; Thu, 13 Aug 2015 23:39:45 +0000 (UTC) (envelope-from Mark.Martinec+freebsd@ijs.si) Received: from mail.ijs.si (mail.ijs.si [IPv6:2001:1470:ff80::25]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 656A181B for ; Thu, 13 Aug 2015 23:39:45 +0000 (UTC) (envelope-from Mark.Martinec+freebsd@ijs.si) Received: from amavis-ori.ijs.si (localhost [IPv6:::1]) by mail.ijs.si (Postfix) with ESMTP id 3mskrr4xBQz2h; Fri, 14 Aug 2015 01:39:40 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ijs.si; h= user-agent:message-id:references:in-reply-to:organization :subject:subject:from:from:date:date:content-transfer-encoding :content-type:content-type:mime-version:received:received :received:received; s=jakla4; t=1439509178; x=1442101179; bh=57Y HbSyYpipAP54P6Ki9E1Q5h3fTzieKxCIAk/dWXy4=; b=PyVvCHbtJNi4WmKV21X Ts3RjO6wF8S7pCRPZ9lYQSe3voHYyO+Fb4dBJpz447X4qwS4ltDpMxdlvHeTu1na i4KRGmua/ZmlWQpxIU82O2XquXR8+5vHUTFRhvrvUkgHkTRzE8XTjamuHAqQwyjH gbMMm+OOo0u9GEYKyOMWVqZg= X-Virus-Scanned: amavisd-new at ijs.si Received: from mail.ijs.si ([IPv6:::1]) by amavis-ori.ijs.si (mail.ijs.si [IPv6:::1]) (amavisd-new, port 10026) with LMTP id VW1VhKwVFdba; Fri, 14 Aug 2015 01:39:38 +0200 (CEST) Received: from mildred.ijs.si (mailbox.ijs.si [IPv6:2001:1470:ff80::143:1]) by mail.ijs.si (Postfix) with ESMTP id 3mskrn2796z2d; Fri, 14 Aug 2015 01:39:37 +0200 (CEST) Received: from neli.ijs.si (neli.ijs.si [IPv6:2001:1470:ff80:88:21c:c0ff:feb1:8c91]) by mildred.ijs.si (Postfix) with ESMTP id 3mskrn07TRz1R5; Fri, 14 Aug 2015 01:39:37 +0200 (CEST) Received: from sleepy.ijs.si ([2001:1470:ff80:e001::1:1]) by neli.ijs.si with HTTP (HTTP/1.1 POST); Fri, 14 Aug 2015 01:39:36 +0200 MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII; format=flowed Content-Transfer-Encoding: 7bit Date: Fri, 14 Aug 2015 01:39:36 +0200 From: Mark Martinec To: James Lott Cc: freebsd-net@freebsd.org Subject: Re: Ethernet tunneling options under FreeBSD Organization: J. Stefan Institute In-Reply-To: <55CD1CE6.2010502@lottspot.com> References: <55CD1CE6.2010502@lottspot.com> Message-ID: <3e2b39d46709d7329feb57065ce08478@mailbox.ijs.si> X-Sender: Mark.Martinec+freebsd@ijs.si User-Agent: Roundcube Webmail/1.1.2 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 13 Aug 2015 23:39:45 -0000 2015-08-14 00:40 James Lott wrote: > I am in the process of planning a build out of a L2 VPN, in which I'd > like to have my primary "switch" and DHCP server be a FreeBSD system. > I would like to join each new host to the VPN by establishing an IP > tunnel with the primary "switch" which transports ethernet frames over > the tunnel. > > So far, the only protocol I have found supported by FreeBSD which > seems capable of this is EtherIP. As far as I can tell, it doesn't > look like there is any support for L2TPv3, and none of the PPP > implementations available appear to support BCP. > > I'm not completely opposed to using EtherIP, but if there is something > more modern which will meet my needs, I would probably try that first. > So my question becomes: > > * Does anyone know of a method supported under FreeBSD (other than > EtherIP) for tunneling ethernet over IP that they may be able to > suggest I check out? vxlan ? It is new with FreeBSD 10.2. Mark From owner-freebsd-net@freebsd.org Fri Aug 14 01:12:43 2015 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 74F2299EF14 for ; Fri, 14 Aug 2015 01:12:43 +0000 (UTC) (envelope-from jmg@gold.funkthat.com) Received: from gold.funkthat.com (gate2.funkthat.com [208.87.223.18]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client CN "gold.funkthat.com", Issuer "gold.funkthat.com" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 55B621898 for ; Fri, 14 Aug 2015 01:12:42 +0000 (UTC) (envelope-from jmg@gold.funkthat.com) Received: from gold.funkthat.com (localhost [127.0.0.1]) by gold.funkthat.com (8.14.5/8.14.5) with ESMTP id t7E1CgVR092861 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Thu, 13 Aug 2015 18:12:42 -0700 (PDT) (envelope-from jmg@gold.funkthat.com) Received: (from jmg@localhost) by gold.funkthat.com (8.14.5/8.14.5/Submit) id t7E1CgfG092860; Thu, 13 Aug 2015 18:12:42 -0700 (PDT) (envelope-from jmg) Date: Thu, 13 Aug 2015 18:12:42 -0700 From: John-Mark Gurney To: James Lott Cc: freebsd-net@freebsd.org Subject: Re: Ethernet tunneling options under FreeBSD Message-ID: <20150814011241.GZ68509@funkthat.com> References: <55CD1CE6.2010502@lottspot.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <55CD1CE6.2010502@lottspot.com> X-Operating-System: FreeBSD 9.1-PRERELEASE amd64 X-PGP-Fingerprint: 54BA 873B 6515 3F10 9E88 9322 9CB1 8F74 6D3F A396 X-Files: The truth is out there X-URL: http://resnet.uoregon.edu/~gurney_j/ X-Resume: http://resnet.uoregon.edu/~gurney_j/resume.html X-TipJar: bitcoin:13Qmb6AeTgQecazTWph4XasEsP7nGRbAPE X-to-the-FBI-CIA-and-NSA: HI! HOW YA DOIN? can i haz chizburger? User-Agent: Mutt/1.5.21 (2010-09-15) X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.2.7 (gold.funkthat.com [127.0.0.1]); Thu, 13 Aug 2015 18:12:42 -0700 (PDT) X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 14 Aug 2015 01:12:43 -0000 James Lott wrote this message on Thu, Aug 13, 2015 at 15:40 -0700: > * Does anyone know of a method supported under FreeBSD (other than > EtherIP) for tunneling ethernet over IP that they may be able to suggest > I check out? Another option would be ssh + Tunnel... ssh_config(5) for more info.. I haven't tried this out myself, but I may in the near future... -- John-Mark Gurney Voice: +1 415 225 5579 "All that I will do, has been done, All that I have, has not." From owner-freebsd-net@freebsd.org Fri Aug 14 03:15:40 2015 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 4B82F9A03C4 for ; Fri, 14 Aug 2015 03:15:40 +0000 (UTC) (envelope-from james@lottspot.com) Received: from mx0.lottspot.com (sfo.lottspot.com [198.199.98.33]) by mx1.freebsd.org (Postfix) with ESMTP id 3408618A8 for ; Fri, 14 Aug 2015 03:15:38 +0000 (UTC) (envelope-from james@lottspot.com) Received: from localhost (localhost [127.0.0.1]) by mail.lottspot.com (Postfix) with ESMTP id 96CAF411FB; Thu, 13 Aug 2015 20:15:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=lottspot.com; h= date:date:subject:subject:from:from:x-mailer:message-id :content-transfer-encoding:content-type:content-type:in-reply-to :mime-version:references:received:received; s=mail; t= 1439522087; bh=00kmlrYI7p2VODeQUYgBsjIFc/YYz6ACWmUhwu/xqWY=; b=M 1S23jI3SppZrDaKyyvGgHQeGCgmQVU6rr2IMK+2FkCVBrnp0gMirpIPyeDeirFNG +iYodS7f1hyLAsx4m2BTH7qKaZsfit98x+b9SlWmsYwEx1xNwVA8pW1fNWt5F3C6 piZX6TNfb7e//qyAYharG38iGQc/x4D5P3ZK6Td+mg= X-Virus-Scanned: amavisd-new at lottspot.com Received: from mx0.lottspot.com ([127.0.0.1]) by localhost (mail.lottspot.com [127.0.0.1]) (amavisd-new, port 10024) with LMTP id bCZTrdEKXRap; Thu, 13 Aug 2015 20:14:47 -0700 (PDT) Received: from [192.168.0.7] (h69-131-58-73.nrfdvt.dsl.dynamic.tds.net [69.131.58.73]) by mx0.lottspot.com (Postfix) with ESMTPSA id D7C5C41187; Thu, 13 Aug 2015 20:14:46 -0700 (PDT) References: <55CD1CE6.2010502@lottspot.com> <3e2b39d46709d7329feb57065ce08478@mailbox.ijs.si> Mime-Version: 1.0 (1.0) In-Reply-To: <3e2b39d46709d7329feb57065ce08478@mailbox.ijs.si> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable Message-Id: <2AC28251-EFFD-4B9C-BC54-A11172875E1D@lottspot.com> Cc: "freebsd-net@freebsd.org" X-Mailer: iPhone Mail (12F70) From: James Lott Subject: Re: Ethernet tunneling options under FreeBSD Date: Thu, 13 Aug 2015 23:12:57 -0400 To: Mark Martinec X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 14 Aug 2015 03:15:40 -0000 Wow vxlans look awesome! I am definitely going to experiment with those next= !=20 Thanks for the great suggestions though everyone, this thread gave me a lot m= ore to experiment with > On Aug 13, 2015, at 7:39 PM, Mark Martinec w= rote: >=20 > 2015-08-14 00:40 James Lott wrote: >> I am in the process of planning a build out of a L2 VPN, in which I'd >> like to have my primary "switch" and DHCP server be a FreeBSD system. >> I would like to join each new host to the VPN by establishing an IP >> tunnel with the primary "switch" which transports ethernet frames over >> the tunnel. >> So far, the only protocol I have found supported by FreeBSD which >> seems capable of this is EtherIP. As far as I can tell, it doesn't >> look like there is any support for L2TPv3, and none of the PPP >> implementations available appear to support BCP. >> I'm not completely opposed to using EtherIP, but if there is something >> more modern which will meet my needs, I would probably try that first. >> So my question becomes: >> * Does anyone know of a method supported under FreeBSD (other than >> EtherIP) for tunneling ethernet over IP that they may be able to >> suggest I check out? >=20 > vxlan ? It is new with FreeBSD 10.2. >=20 >=20 > Mark > _______________________________________________ > freebsd-net@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" From owner-freebsd-net@freebsd.org Fri Aug 14 15:16:55 2015 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 1AE159B98CA for ; Fri, 14 Aug 2015 15:16:55 +0000 (UTC) (envelope-from julian@freebsd.org) Received: from vps1.elischer.org (vps1.elischer.org [204.109.63.16]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "vps1.elischer.org", Issuer "CA Cert Signing Authority" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id E94221DB6 for ; Fri, 14 Aug 2015 15:16:54 +0000 (UTC) (envelope-from julian@freebsd.org) Received: from Julian-MBP3.local (ppp121-45-227-250.lns20.per1.internode.on.net [121.45.227.250]) (authenticated bits=0) by vps1.elischer.org (8.15.2/8.15.2) with ESMTPSA id t7EFGn9k008348 (version=TLSv1.2 cipher=DHE-RSA-AES128-SHA bits=128 verify=NO); Fri, 14 Aug 2015 08:16:53 -0700 (PDT) (envelope-from julian@freebsd.org) Subject: Re: Ethernet tunneling options under FreeBSD To: James Lott , freebsd-net@freebsd.org References: <55CD1CE6.2010502@lottspot.com> From: Julian Elischer Message-ID: <55CE0659.6050206@freebsd.org> Date: Fri, 14 Aug 2015 23:16:41 +0800 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:38.0) Gecko/20100101 Thunderbird/38.1.0 MIME-Version: 1.0 In-Reply-To: <55CD1CE6.2010502@lottspot.com> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 14 Aug 2015 15:16:55 -0000 On 8/14/15 6:40 AM, James Lott wrote: > Hello list, > > I am in the process of planning a build out of a L2 VPN, in which > I'd like to have my primary "switch" and DHCP server be a FreeBSD > system. I would like to join each new host to the VPN by > establishing an IP tunnel with the primary "switch" which transports > ethernet frames over the tunnel. > you haven't really described the network well enough.. try an ascii-art diagram (don't forget to set fixed width font :-) a VPN required two ends.. one is FreeBSD... what's the other? > So far, the only protocol I have found supported by FreeBSD which > seems capable of this is EtherIP. As far as I can tell, it doesn't > look like there is any support for L2TPv3, and none of the PPP > implementations available appear to support BCP. > > I'm not completely opposed to using EtherIP, but if there is > something more modern which will meet my needs, I would probably try > that first. So my question becomes: > > * Does anyone know of a method supported under FreeBSD (other than > EtherIP) for tunneling ethernet over IP that they may be able to > suggest I check out? if both ends are FreeBSD there are dozens of possibilities.. for example: ng_eif->netgraph->ppp->ipsec->ppp->netgraph->ng_eif ng_eif->ng_ksock(udp)->IPsec->ng_ksock->ng_eif tap->ppp->ppp->tap > > Thanks for any suggestions! > _______________________________________________ > freebsd-net@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" > From owner-freebsd-net@freebsd.org Fri Aug 14 17:29:56 2015 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 79DA99B852D for ; Fri, 14 Aug 2015 17:29:56 +0000 (UTC) (envelope-from sobomax@sippysoft.com) Received: from mail-wi0-f174.google.com (mail-wi0-f174.google.com [209.85.212.174]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 07BFA1192 for ; Fri, 14 Aug 2015 17:29:55 +0000 (UTC) (envelope-from sobomax@sippysoft.com) Received: by wicne3 with SMTP id ne3so24768097wic.0 for ; Fri, 14 Aug 2015 10:29:48 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:sender:in-reply-to:references:date :message-id:subject:from:to:cc:content-type; bh=ZKw0+VPiG8A5RZ2V7bPXTzRds05haJCxyh5sJUBrKEE=; b=hPi1l1Zgfdj7zPqHRTbLU0ixeRRI2VqkLScF1QTXWQGMklEC1F8d0it5e4jNYDcXa2 h0/igUos1OX/9DlqHcxbxUOoaO6VS6ddqVxIRKQxpG+jWcMOxYxA54DixlLOxlw0E+IF VVmHWdYjdmKgdKv4cPlhITzoQyjkSsq8FgR8S08L4DAaa6wrs1CHYtcI0iuATF9zg0Ve G1hQEqnIrS8DOB5KkSv1joaV9GQIIieSeiDupFt4luJA28voj5k8hPM2v0cYtLgSknaV SrrikKVw4Fyhu6VkZEtIJzYx/iXLg6W/yHXFxw55mFvoVsyDnu9T1Zq4ikGKeW/Jhzd8 Pt3g== X-Gm-Message-State: ALoCoQmFI/mVWKS/y+rymtKSW4hVeCbD2iVZUzU0JvVYVEa6Ccexc+ok9sBJSJsKyWe2HH4fn12a MIME-Version: 1.0 X-Received: by 10.194.78.164 with SMTP id c4mr1794421wjx.65.1439573388238; Fri, 14 Aug 2015 10:29:48 -0700 (PDT) Sender: sobomax@sippysoft.com Received: by 10.27.143.15 with HTTP; Fri, 14 Aug 2015 10:29:48 -0700 (PDT) In-Reply-To: References: <77171439377164@web21h.yandex.ru> <55CB2F18.40902@FreeBSD.org> Date: Fri, 14 Aug 2015 10:29:48 -0700 X-Google-Sender-Auth: 1Fmo08uwAHrfN08im56UTZ4wZRg Message-ID: Subject: Re: Poor high-PPS performance of the 10G ixgbe(9) NIC/driver in FreeBSD 10.1 From: Maxim Sobolev To: Luigi Rizzo Cc: Babak Farrokhi , "Alexander V. Chernikov" , =?UTF-8?Q?Olivier_Cochard=2DLabb=C3=A9?= , "freebsd@intel.com" , =?UTF-8?Q?Jev_Bj=C3=B6rsell?= , FreeBSD Net Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 14 Aug 2015 17:29:56 -0000 Hi guys, unfortunately no, neither reduction of the number of queues from 8 to 6 nor pinning interrupt rate at 20000 per queue have not made any difference. The card still goes kaboom at about 200Kpps no matter what. in fact I've gone bit further, and after the first spike went on an pushed interrupt rate even further down to 10000, but again no difference either, it still blows at the same mark. Although it did have effect on interrupt rate reduction from 190K to some 130K according to the systat -vm, so that the moderation itself seems to be working fine. We will try disabling IXGBE_FDIR tomorrow and see if it helps. http://sobomax.sippysoft.com/ScreenShot391.png <- systat -vm with max_interrupt_rate =3D 20000 right before overload http://sobomax.sippysoft.com/ScreenShot392.png <- systat -vm during issue unfolding (max_interrupt_rate =3D 10000) http://sobomax.sippysoft.com/ScreenShot394.png <- cpu/net monitoring, first two spikes are with max_interrupt_rate =3D 20000, the third one max_interrupt_rate =3D 10000 -Max On Wed, Aug 12, 2015 at 5:23 AM, Luigi Rizzo wrote: > As I was telling to maxim, you should disable aim because it only matches > the max interrupt rate to the average packet size, which is the last thin= g > you want. > > Setting the interrupt rate with sysctl (one per queue) gives you precise > control on the max rate and (hence, extra latency). 20k interrupts/s give > you 50us of latency, and the 2k slots in the queue are still enough to > absorb a burst of min-sized frames hitting a single queue (the os will > start dropping long before that level, but that's another story). > > Cheers > Luigi > > On Wednesday, August 12, 2015, Babak Farrokhi > wrote: > >> I ran into the same problem with almost the same hardware (Intel X520) >> on 10-STABLE. HT/SMT is disabled and cards are configured with 8 queues, >> with the same sysctl tunings as sobomax@ did. I am not using lagg, no >> FLOWTABLE. >> >> I experimented with pmcstat (RESOURCE_STALLS) a while ago and here [1] >> [2] you can see the results, including pmc output, callchain, flamegraph >> and gprof output. >> >> I am experiencing huge number of interrupts with 200kpps load: >> >> # sysctl dev.ix | grep interrupt_rate >> dev.ix.1.queue7.interrupt_rate: 125000 >> dev.ix.1.queue6.interrupt_rate: 6329 >> dev.ix.1.queue5.interrupt_rate: 500000 >> dev.ix.1.queue4.interrupt_rate: 100000 >> dev.ix.1.queue3.interrupt_rate: 50000 >> dev.ix.1.queue2.interrupt_rate: 500000 >> dev.ix.1.queue1.interrupt_rate: 500000 >> dev.ix.1.queue0.interrupt_rate: 100000 >> dev.ix.0.queue7.interrupt_rate: 500000 >> dev.ix.0.queue6.interrupt_rate: 6097 >> dev.ix.0.queue5.interrupt_rate: 10204 >> dev.ix.0.queue4.interrupt_rate: 5208 >> dev.ix.0.queue3.interrupt_rate: 5208 >> dev.ix.0.queue2.interrupt_rate: 71428 >> dev.ix.0.queue1.interrupt_rate: 5494 >> dev.ix.0.queue0.interrupt_rate: 6250 >> >> [1] http://farrokhi.net/~farrokhi/pmc/6/ >> [2] http://farrokhi.net/~farrokhi/pmc/7/ >> >> Regards, >> Babak >> >> >> Alexander V. Chernikov wrote: >> > 12.08.2015, 02:28, "Maxim Sobolev" : >> >> Olivier, keep in mind that we are not "kernel forwarding" packets, bu= t >> "app >> >> forwarding", i.e. the packet goes full way >> >> net->kernel->recvfrom->app->sendto->kernel->net, which is why we have >> much >> >> lower PPS limits and which is why I think we are actually benefiting >> from >> >> the extra queues. Single-thread sendto() in a loop is CPU-bound at >> about >> >> 220K PPS, and while running the test I am observing that outbound >> traffic >> >> from one thread is mapped into a specific queue (well, pair of queues >> on >> >> two separate adaptors, due to lagg load balancing action). And the pe= ak >> >> performance of that test is at 7 threads, which I believe corresponds >> to >> >> the number of queues. We have plenty of CPU cores in the box (24) wit= h >> >> HTT/SMT disabled and one CPU is mapped to a specific queue. This >> leaves us >> >> with at least 8 CPUs fully capable of running our app. If you look at >> the >> >> CPU utilization, we are at about 10% when the issue hits. >> > >> > In any case, it would be great if you could provide some profiling inf= o >> since there could be >> > plenty of problematic places starting from TX rings contention to some >> locks inside udp or even >> > (in)famous random entropy harvester.. >> > e.g. something like pmcstat -TS instructions -w1 might be sufficient t= o >> determine the reason >> >> ix0: >> port >> >> 0x6020-0x603f mem 0xc7c00000-0xc7dfffff,0xc7e04000-0xc7e07fff irq 40 = at >> >> device 0.0 on pci3 >> >> ix0: Using MSIX interrupts with 9 vectors >> >> ix0: Bound queue 0 to cpu 0 >> >> ix0: Bound queue 1 to cpu 1 >> >> ix0: Bound queue 2 to cpu 2 >> >> ix0: Bound queue 3 to cpu 3 >> >> ix0: Bound queue 4 to cpu 4 >> >> ix0: Bound queue 5 to cpu 5 >> >> ix0: Bound queue 6 to cpu 6 >> >> ix0: Bound queue 7 to cpu 7 >> >> ix0: Ethernet address: 0c:c4:7a:5e:be:64 >> >> ix0: PCI Express Bus: Speed 5.0GT/s Width x8 >> >> 001.000008 [2705] netmap_attach success for ix0 tx 8/4096 rx >> >> 8/4096 queues/slots >> >> ix1: >> port >> >> 0x6000-0x601f mem 0xc7a00000-0xc7bfffff,0xc7e00000-0xc7e03fff irq 44 = at >> >> device 0.1 on pci3 >> >> ix1: Using MSIX interrupts with 9 vectors >> >> ix1: Bound queue 0 to cpu 8 >> >> ix1: Bound queue 1 to cpu 9 >> >> ix1: Bound queue 2 to cpu 10 >> >> ix1: Bound queue 3 to cpu 11 >> >> ix1: Bound queue 4 to cpu 12 >> >> ix1: Bound queue 5 to cpu 13 >> >> ix1: Bound queue 6 to cpu 14 >> >> ix1: Bound queue 7 to cpu 15 >> >> ix1: Ethernet address: 0c:c4:7a:5e:be:65 >> >> ix1: PCI Express Bus: Speed 5.0GT/s Width x8 >> >> 001.000009 [2705] netmap_attach success for ix1 tx 8/4096 rx >> >> 8/4096 queues/slots >> >> >> >> On Tue, Aug 11, 2015 at 4:14 PM, Olivier Cochard-Labb=C3=A9 < >> olivier@cochard.me> >> >> wrote: >> >> >> >>> On Tue, Aug 11, 2015 at 11:18 PM, Maxim Sobolev > > >> >>> wrote: >> >>> >> >>>> Hi folks, >> >>>> >> >>>> =E2=80=8BHi, >> >>> =E2=80=8B >> >>> >> >>>> We've trying to migrate some of our high-PPS systems to a new >> hardware >> >>>> that >> >>>> has four X540-AT2 10G NICs and observed that interrupt time goes >> through >> >>>> roof after we cross around 200K PPS in and 200K out (two ports in >> LACP). >> >>>> The previous hardware was stable up to about 350K PPS in and 350K >> out. I >> >>>> believe the old one was equipped with the I350 and had the >> identical LACP >> >>>> configuration. The new box also has better CPU with more cores >> (i.e. 24 >> >>>> cores vs. 16 cores before). CPU itself is 2 x E5-2690 v3. >> >>> =E2=80=8B200K PPS, and even 350K PPS are very low value indeed. >> >>> On a Intel Xeon L5630 (4 cores only) with one X540-AT2=E2=80=8B >> >>> >> >>> =E2=80=8B(then 2 10Gigabit ports)=E2=80=8B I've reached about 1.8Mp= ps (fastforwarding >> >>> enabled) [1]. >> >>> But my setup didn't use lagg(4): Can you disable lagg configuration >> and >> >>> re-measure your performance without lagg ? >> >>> >> >>> Do you let Intel NIC drivers using 8 queues for port too? >> >>> In my use case (forwarding smallest UDP packet size), I obtain bett= er >> >>> behaviour by limiting NIC queues to 4 (hw.ix.num_queues or >> >>> hw.ixgbe.num_queues, don't remember) if my system had 8 cores. And >> this >> >>> with Gigabit Intel[2] or Chelsio NIC [3]. >> >>> >> >>> Don't forget to disable TSO and LRO too. >> >>> >> >>> =E2=80=8BRegards, >> >>> >> >>> Olivier >> >>> >> >>> [1] >> >>> >> http://bsdrp.net/documentation/examples/forwarding_performance_lab_of_an= _ibm_system_x3550_m3_with_10-gigabit_intel_x540-at2#graphs >> >>> [2] >> >>> >> http://bsdrp.net/documentation/examples/forwarding_performance_lab_of_a_= superserver_5018a-ftn4#graph1 >> >>> [3] >> >>> >> http://bsdrp.net/documentation/examples/forwarding_performance_lab_of_a_= hp_proliant_dl360p_gen8_with_10-gigabit_with_10-gigabit_chelsio_t540-cr#red= ucing_nic_queues >> >> _______________________________________________ >> >> freebsd-net@freebsd.org mailing list >> >> http://lists.freebsd.org/mailman/listinfo/freebsd-net >> >> To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org= " >> > _______________________________________________ >> > freebsd-net@freebsd.org mailing list >> > http://lists.freebsd.org/mailman/listinfo/freebsd-net >> > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" >> _______________________________________________ >> freebsd-net@freebsd.org mailing list >> http://lists.freebsd.org/mailman/listinfo/freebsd-net >> To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" > > > > -- > -----------------------------------------+------------------------------- > Prof. Luigi RIZZO, rizzo@iet.unipi.it . Dip. di Ing. dell'Informazione > http://www.iet.unipi.it/~luigi/ . Universita` di Pisa > TEL +39-050-2217533 . via Diotisalvi 2 > Mobile +39-338-6809875 . 56122 PISA (Italy) > -----------------------------------------+------------------------------- > > From owner-freebsd-net@freebsd.org Fri Aug 14 17:38:04 2015 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 3BF449B876B for ; Fri, 14 Aug 2015 17:38:04 +0000 (UTC) (envelope-from sobomax@sippysoft.com) Received: from mail-wi0-f177.google.com (mail-wi0-f177.google.com [209.85.212.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id BA8301859 for ; Fri, 14 Aug 2015 17:38:03 +0000 (UTC) (envelope-from sobomax@sippysoft.com) Received: by wicne3 with SMTP id ne3so24952637wic.0 for ; Fri, 14 Aug 2015 10:37:56 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc:content-type; bh=mOQxM7q3l1An63TUhm+y7FqBdV3Xon7nAv76SjhGGjw=; b=MqwyLNgXb26ArmvnLAjbXPbrj441fF9Xi82BdDufg4HGSyMFglhFt3VPMmo7UWFKkN e+arJhqxs8Ok8gBhJrwjtjQNg64g7MFVF6vSd6IdrFXQRsSe9NX90Kd57g62VaKhQD65 bnKbtBZzXO8/ejohk+zO+YtZZWEbqfVwTiPF8YsvMHA2+U1NBLVA3MvdnKGS2FsXtqD6 ekpzcxJUFNbir7oSKAmLg0lV/ADVpvI2D+R42ORyjOb0Rzs8gny35TZhgoq7WiWK3EIb /1RoXpTpmxR9AWiz4FLfavwJmamnaUW4DbWhh42Em+iBEAywXQVE8HA+b7ddbMl/ljHW N2iA== X-Gm-Message-State: ALoCoQkhie+fTyFKIYYwSRt8IjagyByPdMp+OYP+YdJI1D+o7mNjc5B5QjUIJW8EJ5rMq/6IYPZY MIME-Version: 1.0 X-Received: by 10.180.78.166 with SMTP id c6mr8592993wix.8.1439573875870; Fri, 14 Aug 2015 10:37:55 -0700 (PDT) Received: by 10.27.143.15 with HTTP; Fri, 14 Aug 2015 10:37:55 -0700 (PDT) In-Reply-To: References: <77171439377164@web21h.yandex.ru> <55CB2F18.40902@FreeBSD.org> Date: Fri, 14 Aug 2015 10:37:55 -0700 Message-ID: Subject: Re: Poor high-PPS performance of the 10G ixgbe(9) NIC/driver in FreeBSD 10.1 From: Maxim Sobolev To: Luigi Rizzo Cc: Babak Farrokhi , "Alexander V. Chernikov" , =?UTF-8?Q?Olivier_Cochard=2DLabb=C3=A9?= , "freebsd@intel.com" , =?UTF-8?Q?Jev_Bj=C3=B6rsell?= , FreeBSD Net Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 14 Aug 2015 17:38:04 -0000 P.S. Just for the comparison, here is today's stats from the system mentioned here with the low-end I210 chip (4 hardware queues), running happily at some 240Kpps. The system and software is identical otherwise and the igb(9) settings are the default ones: http://sobomax.sippysoft.com/ScreenShot395.png On Fri, Aug 14, 2015 at 10:29 AM, Maxim Sobolev wrote= : > Hi guys, unfortunately no, neither reduction of the number of queues from > 8 to 6 nor pinning interrupt rate at 20000 per queue have not made any > difference. The card still goes kaboom at about 200Kpps no matter what. i= n > fact I've gone bit further, and after the first spike went on an pushed > interrupt rate even further down to 10000, but again no difference either= , > it still blows at the same mark. Although it did have effect on interrupt > rate reduction from 190K to some 130K according to the systat -vm, so tha= t > the moderation itself seems to be working fine. We will try disabling IXG= BE_FDIR > tomorrow and see if it helps. > > http://sobomax.sippysoft.com/ScreenShot391.png <- systat -vm with > max_interrupt_rate =3D 20000 right before overload > > http://sobomax.sippysoft.com/ScreenShot392.png <- systat -vm during issue > unfolding (max_interrupt_rate =3D 10000) > > http://sobomax.sippysoft.com/ScreenShot394.png <- cpu/net monitoring, > first two spikes are with max_interrupt_rate =3D 20000, the third one max= _interrupt_rate > =3D 10000 > > -Max > > On Wed, Aug 12, 2015 at 5:23 AM, Luigi Rizzo wrote: > >> As I was telling to maxim, you should disable aim because it only matche= s >> the max interrupt rate to the average packet size, which is the last thi= ng >> you want. >> >> Setting the interrupt rate with sysctl (one per queue) gives you precise >> control on the max rate and (hence, extra latency). 20k interrupts/s giv= e >> you 50us of latency, and the 2k slots in the queue are still enough to >> absorb a burst of min-sized frames hitting a single queue (the os will >> start dropping long before that level, but that's another story). >> >> Cheers >> Luigi >> >> On Wednesday, August 12, 2015, Babak Farrokhi >> wrote: >> >>> I ran into the same problem with almost the same hardware (Intel X520) >>> on 10-STABLE. HT/SMT is disabled and cards are configured with 8 queues= , >>> with the same sysctl tunings as sobomax@ did. I am not using lagg, no >>> FLOWTABLE. >>> >>> I experimented with pmcstat (RESOURCE_STALLS) a while ago and here [1] >>> [2] you can see the results, including pmc output, callchain, flamegrap= h >>> and gprof output. >>> >>> I am experiencing huge number of interrupts with 200kpps load: >>> >>> # sysctl dev.ix | grep interrupt_rate >>> dev.ix.1.queue7.interrupt_rate: 125000 >>> dev.ix.1.queue6.interrupt_rate: 6329 >>> dev.ix.1.queue5.interrupt_rate: 500000 >>> dev.ix.1.queue4.interrupt_rate: 100000 >>> dev.ix.1.queue3.interrupt_rate: 50000 >>> dev.ix.1.queue2.interrupt_rate: 500000 >>> dev.ix.1.queue1.interrupt_rate: 500000 >>> dev.ix.1.queue0.interrupt_rate: 100000 >>> dev.ix.0.queue7.interrupt_rate: 500000 >>> dev.ix.0.queue6.interrupt_rate: 6097 >>> dev.ix.0.queue5.interrupt_rate: 10204 >>> dev.ix.0.queue4.interrupt_rate: 5208 >>> dev.ix.0.queue3.interrupt_rate: 5208 >>> dev.ix.0.queue2.interrupt_rate: 71428 >>> dev.ix.0.queue1.interrupt_rate: 5494 >>> dev.ix.0.queue0.interrupt_rate: 6250 >>> >>> [1] http://farrokhi.net/~farrokhi/pmc/6/ >>> [2] http://farrokhi.net/~farrokhi/pmc/7/ >>> >>> Regards, >>> Babak >>> >>> >>> Alexander V. Chernikov wrote: >>> > 12.08.2015, 02:28, "Maxim Sobolev" : >>> >> Olivier, keep in mind that we are not "kernel forwarding" packets, >>> but "app >>> >> forwarding", i.e. the packet goes full way >>> >> net->kernel->recvfrom->app->sendto->kernel->net, which is why we hav= e >>> much >>> >> lower PPS limits and which is why I think we are actually benefiting >>> from >>> >> the extra queues. Single-thread sendto() in a loop is CPU-bound at >>> about >>> >> 220K PPS, and while running the test I am observing that outbound >>> traffic >>> >> from one thread is mapped into a specific queue (well, pair of queue= s >>> on >>> >> two separate adaptors, due to lagg load balancing action). And the >>> peak >>> >> performance of that test is at 7 threads, which I believe correspond= s >>> to >>> >> the number of queues. We have plenty of CPU cores in the box (24) wi= th >>> >> HTT/SMT disabled and one CPU is mapped to a specific queue. This >>> leaves us >>> >> with at least 8 CPUs fully capable of running our app. If you look a= t >>> the >>> >> CPU utilization, we are at about 10% when the issue hits. >>> > >>> > In any case, it would be great if you could provide some profiling >>> info since there could be >>> > plenty of problematic places starting from TX rings contention to som= e >>> locks inside udp or even >>> > (in)famous random entropy harvester.. >>> > e.g. something like pmcstat -TS instructions -w1 might be sufficient >>> to determine the reason >>> >> ix0: >> 2.5.15> port >>> >> 0x6020-0x603f mem 0xc7c00000-0xc7dfffff,0xc7e04000-0xc7e07fff irq 40 >>> at >>> >> device 0.0 on pci3 >>> >> ix0: Using MSIX interrupts with 9 vectors >>> >> ix0: Bound queue 0 to cpu 0 >>> >> ix0: Bound queue 1 to cpu 1 >>> >> ix0: Bound queue 2 to cpu 2 >>> >> ix0: Bound queue 3 to cpu 3 >>> >> ix0: Bound queue 4 to cpu 4 >>> >> ix0: Bound queue 5 to cpu 5 >>> >> ix0: Bound queue 6 to cpu 6 >>> >> ix0: Bound queue 7 to cpu 7 >>> >> ix0: Ethernet address: 0c:c4:7a:5e:be:64 >>> >> ix0: PCI Express Bus: Speed 5.0GT/s Width x8 >>> >> 001.000008 [2705] netmap_attach success for ix0 tx 8/4096 rx >>> >> 8/4096 queues/slots >>> >> ix1: >> 2.5.15> port >>> >> 0x6000-0x601f mem 0xc7a00000-0xc7bfffff,0xc7e00000-0xc7e03fff irq 44 >>> at >>> >> device 0.1 on pci3 >>> >> ix1: Using MSIX interrupts with 9 vectors >>> >> ix1: Bound queue 0 to cpu 8 >>> >> ix1: Bound queue 1 to cpu 9 >>> >> ix1: Bound queue 2 to cpu 10 >>> >> ix1: Bound queue 3 to cpu 11 >>> >> ix1: Bound queue 4 to cpu 12 >>> >> ix1: Bound queue 5 to cpu 13 >>> >> ix1: Bound queue 6 to cpu 14 >>> >> ix1: Bound queue 7 to cpu 15 >>> >> ix1: Ethernet address: 0c:c4:7a:5e:be:65 >>> >> ix1: PCI Express Bus: Speed 5.0GT/s Width x8 >>> >> 001.000009 [2705] netmap_attach success for ix1 tx 8/4096 rx >>> >> 8/4096 queues/slots >>> >> >>> >> On Tue, Aug 11, 2015 at 4:14 PM, Olivier Cochard-Labb=C3=A9 < >>> olivier@cochard.me> >>> >> wrote: >>> >> >>> >>> On Tue, Aug 11, 2015 at 11:18 PM, Maxim Sobolev < >>> sobomax@freebsd.org> >>> >>> wrote: >>> >>> >>> >>>> Hi folks, >>> >>>> >>> >>>> =E2=80=8BHi, >>> >>> =E2=80=8B >>> >>> >>> >>>> We've trying to migrate some of our high-PPS systems to a new >>> hardware >>> >>>> that >>> >>>> has four X540-AT2 10G NICs and observed that interrupt time goes >>> through >>> >>>> roof after we cross around 200K PPS in and 200K out (two ports in >>> LACP). >>> >>>> The previous hardware was stable up to about 350K PPS in and 350K >>> out. I >>> >>>> believe the old one was equipped with the I350 and had the >>> identical LACP >>> >>>> configuration. The new box also has better CPU with more cores >>> (i.e. 24 >>> >>>> cores vs. 16 cores before). CPU itself is 2 x E5-2690 v3. >>> >>> =E2=80=8B200K PPS, and even 350K PPS are very low value indeed. >>> >>> On a Intel Xeon L5630 (4 cores only) with one X540-AT2=E2=80=8B >>> >>> >>> >>> =E2=80=8B(then 2 10Gigabit ports)=E2=80=8B I've reached about 1.8M= pps >>> (fastforwarding >>> >>> enabled) [1]. >>> >>> But my setup didn't use lagg(4): Can you disable lagg configuratio= n >>> and >>> >>> re-measure your performance without lagg ? >>> >>> >>> >>> Do you let Intel NIC drivers using 8 queues for port too? >>> >>> In my use case (forwarding smallest UDP packet size), I obtain >>> better >>> >>> behaviour by limiting NIC queues to 4 (hw.ix.num_queues or >>> >>> hw.ixgbe.num_queues, don't remember) if my system had 8 cores. And >>> this >>> >>> with Gigabit Intel[2] or Chelsio NIC [3]. >>> >>> >>> >>> Don't forget to disable TSO and LRO too. >>> >>> >>> >>> =E2=80=8BRegards, >>> >>> >>> >>> Olivier >>> >>> >>> >>> [1] >>> >>> >>> http://bsdrp.net/documentation/examples/forwarding_performance_lab_of_a= n_ibm_system_x3550_m3_with_10-gigabit_intel_x540-at2#graphs >>> >>> [2] >>> >>> >>> http://bsdrp.net/documentation/examples/forwarding_performance_lab_of_a= _superserver_5018a-ftn4#graph1 >>> >>> [3] >>> >>> >>> http://bsdrp.net/documentation/examples/forwarding_performance_lab_of_a= _hp_proliant_dl360p_gen8_with_10-gigabit_with_10-gigabit_chelsio_t540-cr#re= ducing_nic_queues >>> >> _______________________________________________ >>> >> freebsd-net@freebsd.org mailing list >>> >> http://lists.freebsd.org/mailman/listinfo/freebsd-net >>> >> To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.or= g >>> " >>> > _______________________________________________ >>> > freebsd-net@freebsd.org mailing list >>> > http://lists.freebsd.org/mailman/listinfo/freebsd-net >>> > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org= " >>> _______________________________________________ >>> freebsd-net@freebsd.org mailing list >>> http://lists.freebsd.org/mailman/listinfo/freebsd-net >>> To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" >> >> >> >> -- >> -----------------------------------------+------------------------------= - >> Prof. Luigi RIZZO, rizzo@iet.unipi.it . Dip. di Ing. dell'Informazione >> http://www.iet.unipi.it/~luigi/ . Universita` di Pisa >> TEL +39-050-2217533 . via Diotisalvi 2 >> Mobile +39-338-6809875 . 56122 PISA (Italy) >> -----------------------------------------+------------------------------= - >> >> > --=20 Maksym Sobolyev Sippy Software, Inc. Internet Telephony (VoIP) Experts Tel (Canada): +1-778-783-0474 Tel (Toll-Free): +1-855-747-7779 Fax: +1-866-857-6942 Web: http://www.sippysoft.com MSN: sales@sippysoft.com Skype: SippySoft From owner-freebsd-net@freebsd.org Fri Aug 14 17:57:08 2015 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 564DB9B8D4C for ; Fri, 14 Aug 2015 17:57:08 +0000 (UTC) (envelope-from ebastan10@gmail.com) Received: from mail-ig0-x22c.google.com (mail-ig0-x22c.google.com [IPv6:2607:f8b0:4001:c05::22c]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 253B416EE for ; Fri, 14 Aug 2015 17:57:08 +0000 (UTC) (envelope-from ebastan10@gmail.com) Received: by igfj19 with SMTP id j19so17176456igf.1 for ; Fri, 14 Aug 2015 10:57:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:date:message-id:subject:from:to:content-type; bh=XtBE/ozwXfG6uugnX48IV+dz+cV60AVfI1xn8P/NNVw=; b=nbOjj7+XLOoe8Y62o1+uBzcVHYL7RvhxFt6YC8BquMGEywhadA2LEOYZrxXvWJRhio /aupCUdrxstWKI5fri2YY4YA/hNIE5e+d1ob4kmO3dag7uVsHcxEiNEAT8g1/XHnLsAf ii0uADN+9QzWUXs1jhKT5sLIffsWEddhH8mr1EAKdU7qrCetYpoMse69h2Xx4KN2HQ8z k2gN1P+cS6SxoeaNtkiIfabP7Bt8axNpJwhylWMBb9X3rpcDyHAxhhqKBRIrHtTBejuK CiblJvjW24A3tGuPYPu5PvpXzIVY6fDGlvpNDZiy2F6R0FScn2bSnLZyL7C7FNc+Ht4/ SMOA== MIME-Version: 1.0 X-Received: by 10.50.1.115 with SMTP id 19mr3766406igl.67.1439575027610; Fri, 14 Aug 2015 10:57:07 -0700 (PDT) Received: by 10.107.9.67 with HTTP; Fri, 14 Aug 2015 10:57:07 -0700 (PDT) Date: Fri, 14 Aug 2015 22:27:07 +0430 Message-ID: Subject: vlan+bridge questions From: Hooshang F To: FreeBSD Net Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 14 Aug 2015 17:57:08 -0000 Hi, We need to install a freebsd firewall (pf). The freebsd box needs to be placed in bridge mode in the middle of a VLAN truck link between 2 Cisco switches. The em0 and em1 ports are connected to the trunk ports on the 2 switches. We are going to: 1- Define two vlan interfaces for vlan id X. one with em0 as parent and the other on top of em1. 2- Create a bridge interface. 3- Add the two vlan interfaces as members of the bridge. 4- Repeat 1-3 for every vlan id used in the network. 2 questions: 1- Is not there a simpler method which does not involve creating so many vlans & bridges? For instance, is it possible to have a truck interface which accepts 'all' vlan IDs (like cisco) instead of creating two vlan interface per ID? 2- How the untagged traffic should be bridged? Cisco switches send out packets untagged if vlan ID is equal to the trunk port 'native' vlan id. To bridge this packets, we should create a bridge with em0 and em1 as members, but that will effectively disables bridging on vlan interfaces. Right? Thanks in advance. From owner-freebsd-net@freebsd.org Fri Aug 14 20:34:43 2015 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 383069BABE0 for ; Fri, 14 Aug 2015 20:34:43 +0000 (UTC) (envelope-from w8hdkim@gmail.com) Received: from mail-io0-x230.google.com (mail-io0-x230.google.com [IPv6:2607:f8b0:4001:c06::230]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 06D7112FA for ; Fri, 14 Aug 2015 20:34:43 +0000 (UTC) (envelope-from w8hdkim@gmail.com) Received: by iods203 with SMTP id s203so96856534iod.0 for ; Fri, 14 Aug 2015 13:34:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:date:message-id:subject:from:to:content-type; bh=sEy+b+FDc7Hl+LISLkTDNgyHT1xIbx9P0uoNyOSAurc=; b=GBMn5+2DeZRRRatCgu9otkP9/Fu3+3tHJk5OLBoJjREMCpqyylvTGPB8CcDxvqQBXD DpVSxrR7nMi2U0R9jvcz10IzwulZapWD7OL70gozOq1/t+y31ohfIjQrYEwL0TjVmVpU +zWr3TER8gmsGkCbYLKwmAOeoXt1LD9cC5sObMU0x1rSnQWjWNm4JFSluiKZiQcpyeIj ODFDCfwcGlFGXVLPNE8wEWK29jocuWm6uBuMdKxQr5JPuBs3I/jncmKHSlvj1Ev/+GtV xmmqpwEJc9Nqv8tZGQ7PSPNZB+730yDps7H31NO0pmoIJgcRQaAH4tDzet1pDp5TItX1 ceXQ== MIME-Version: 1.0 X-Received: by 10.107.131.20 with SMTP id f20mr34525784iod.100.1439584482471; Fri, 14 Aug 2015 13:34:42 -0700 (PDT) Received: by 10.107.202.132 with HTTP; Fri, 14 Aug 2015 13:34:42 -0700 (PDT) Date: Fri, 14 Aug 2015 16:34:42 -0400 Message-ID: Subject: RE not working on 10.2-RELEASE #0 r286731M From: Kim Culhan To: freebsd-net@freebsd.org Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 14 Aug 2015 20:34:43 -0000 RE on 10.2-RELEASE #0 r286731M appears to pass only arp traffic. Replaced if_re.c with version from 273757, appears to work normally. The diff: 34c34 < __FBSDID("$FreeBSD: stable/10/sys/dev/re/if_re.c 273757 2014-10-28 00:43:00Z yongari $"); --- > __FBSDID("$FreeBSD: releng/10.2/sys/dev/re/if_re.c 285177 2015-07-05 20:16:38Z marius $"); 3198,3202d3197 < * Enable transmit and receive. < */ < CSR_WRITE_1(sc, RL_COMMAND, RL_CMD_TX_ENB|RL_CMD_RX_ENB); < < /* 3227a3223,3227 > /* > * Enable transmit and receive. > */ > CSR_WRITE_1(sc, RL_COMMAND, RL_CMD_TX_ENB | RL_CMD_RX_ENB); > 3251,3254d3250 < #ifdef notdef < /* Enable receiver and transmitter. */ < CSR_WRITE_1(sc, RL_COMMAND, RL_CMD_TX_ENB|RL_CMD_RX_ENB); < #endif Let me know what additional info I can provide. thanks -kim From owner-freebsd-net@freebsd.org Fri Aug 14 21:44:11 2015 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id E809A9B97EF for ; Fri, 14 Aug 2015 21:44:11 +0000 (UTC) (envelope-from sbruno@ignoranthack.me) Received: from mail.ignoranthack.me (ignoranthack.me [199.102.79.106]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id CC4A712C1 for ; Fri, 14 Aug 2015 21:44:11 +0000 (UTC) (envelope-from sbruno@ignoranthack.me) Received: from [192.168.200.200] (unknown [50.136.155.142]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) (Authenticated sender: sbruno@ignoranthack.me) by mail.ignoranthack.me (Postfix) with ESMTPSA id 47AF21928D6 for ; Fri, 14 Aug 2015 21:44:10 +0000 (UTC) Reply-To: sbruno@freebsd.org Subject: Re: RE not working on 10.2-RELEASE #0 r286731M References: To: freebsd-net@freebsd.org From: Sean Bruno Message-ID: <55CE6129.4020003@ignoranthack.me> Date: Fri, 14 Aug 2015 14:44:09 -0700 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:38.0) Gecko/20100101 Thunderbird/38.1.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 14 Aug 2015 21:44:12 -0000 -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512 On 08/14/15 13:34, Kim Culhan wrote: > RE on 10.2-RELEASE #0 r286731M appears to pass only arp traffic. > > Replaced if_re.c with version from 273757, appears to work > normally. > > The diff: > > 34c34 < __FBSDID("$FreeBSD: stable/10/sys/dev/re/if_re.c 273757 > 2014-10-28 00:43:00Z yongari $"); --- >> __FBSDID("$FreeBSD: releng/10.2/sys/dev/re/if_re.c 285177 >> 2015-07-05 > 20:16:38Z marius $"); 3198,3202d3197 < * Enable transmit and > receive. < */ < CSR_WRITE_1(sc, RL_COMMAND, > RL_CMD_TX_ENB|RL_CMD_RX_ENB); < < /* 3227a3223,3227 >> /* * Enable transmit and receive. */ CSR_WRITE_1(sc, RL_COMMAND, >> RL_CMD_TX_ENB | RL_CMD_RX_ENB); >> > 3251,3254d3250 < #ifdef notdef < /* Enable receiver and > transmitter. */ < CSR_WRITE_1(sc, RL_COMMAND, > RL_CMD_TX_ENB|RL_CMD_RX_ENB); < #endif > > Let me know what additional info I can provide. > > thanks -kim _______________________________________________ > freebsd-net@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-net To > unsubscribe, send any mail to > "freebsd-net-unsubscribe@freebsd.org" > I'm running -current with all changes in place, I'm not seeing the issues noted here with my hardware. Can you post your hardware from pciconf -lv? re0@pci0:3:0:0: class=0x020000 card=0x84321043 chip=0x816810ec rev=0x06 hdr=0x00 vendor = 'Realtek Semiconductor Co., Ltd.' device = 'RTL8111/8168B PCI Express Gigabit Ethernet controller' class = network subclass = ethernet re1@pci0:4:5:0: class=0x020000 card=0x43021186 chip=0x43021186 rev=0x10 hdr=0x00 vendor = 'D-Link System Inc' device = 'DGE-530T Gigabit Ethernet Adapter (rev.C1) [Realtek RTL8169]' class = network subclass = ethernet sean -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQF8BAEBCgBmBQJVzmEnXxSAAAAAAC4AKGlzc3Vlci1mcHJAbm90YXRpb25zLm9w ZW5wZ3AuZmlmdGhob3JzZW1hbi5uZXRCQUFENDYzMkU3MTIxREU4RDIwOTk3REQx MjAxRUZDQTFFNzI3RTY0AAoJEBIB78oecn5kv8gH/3Z7onJhiOmuHRO2ZR7f+ao2 rOlpg/5hbQMs0hdklh7LHZWw4bs9j/kFtJ/IGelzT6qBDeObb6w6e80X885Os7Gq IxYruQmeSrv4R/w38mVutEjLnpGvYnYqdBDusj9qU6w5d75mFsfgy40/ZxgXV0kH 7agEWzelhP6bOUobwszT53SAqIsEbgysFMTourLRDqW1oRmeqLMC8NYkWjO9oXEP swdMxXAqHZWjj9jm//gYU9YrBrPJlqk4vOyLISh9i9wH5wEysm3iJNzCPC8jChF9 F95ExRLzChN2NoEZJFKsVyrzyHvDiHUAyWFLp8hRsHa/7Lqxc8ReOB/6sl2x6iE= =ihwv -----END PGP SIGNATURE----- From owner-freebsd-net@freebsd.org Fri Aug 14 22:29:09 2015 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 365949BA041 for ; Fri, 14 Aug 2015 22:29:09 +0000 (UTC) (envelope-from w8hdkim@gmail.com) Received: from mail-ig0-x233.google.com (mail-ig0-x233.google.com [IPv6:2607:f8b0:4001:c05::233]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 0584618D4; Fri, 14 Aug 2015 22:29:09 +0000 (UTC) (envelope-from w8hdkim@gmail.com) Received: by igfj19 with SMTP id j19so21559586igf.1; Fri, 14 Aug 2015 15:29:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:date:message-id:subject:from:to:cc:content-type; bh=kmhyzSKppligxMxYt1++QbpgyKl09gNHvRW9XulC/78=; b=ub+QdJaBp59kwVBiBNnb/Hs8gKhkl4m6Z6GTo88sXRCd/IowEpaxdVLJrUlyJuR1KG S84h93882/tfTkuZL9HSE+2KCWXMNDHc+3klJJnLzxbdLHBGfDKFcHKb0gfbpBaMeXWE MBZE23qfRgXyzxg7PUZSCNDU33SDe2fxNEM8F3NipFx1RAGnaRCZmH4RsjWUnpOwhyF7 vU+zwTwB0sfmpCx7NmyA51H3o0frtvn5MQVQKkPbJfeXwtCN0tYEgJWg8upQMSeD5qcU P9/1gY5JFfNE5Nn8XV+bLoi16IyReaMDedKu4uL18uGI/rh9FCOYduitflDKcQX461UC JzlA== MIME-Version: 1.0 X-Received: by 10.50.30.226 with SMTP id v2mr5381255igh.11.1439591348258; Fri, 14 Aug 2015 15:29:08 -0700 (PDT) Received: by 10.107.202.132 with HTTP; Fri, 14 Aug 2015 15:29:08 -0700 (PDT) Date: Fri, 14 Aug 2015 18:29:08 -0400 Message-ID: Subject: Re: RE not working on 10.2-RELEASE #0 r286731M From: Kim Culhan To: sbruno@freebsd.org Cc: freebsd-net@freebsd.org Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 14 Aug 2015 22:29:09 -0000 > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA512 > > On 08/14/15 13:34, Kim Culhan wrote: >> RE on 10.2-RELEASE #0 r286731M appears to pass only arp traffic. >> >> Replaced if_re.c with version from 273757, appears to work >> normally. >> >> The diff: >> >> 34c34 < __FBSDID("$FreeBSD: stable/10/sys/dev/re/if_re.c 273757 >> 2014-10-28 00:43:00Z yongari $"); --- >>> __FBSDID("$FreeBSD: releng/10.2/sys/dev/re/if_re.c 285177 >>> 2015-07-05 >> 20:16:38Z marius $"); 3198,3202d3197 < * Enable transmit and >> receive. < */ < CSR_WRITE_1(sc, RL_COMMAND, >> RL_CMD_TX_ENB|RL_CMD_RX_ENB); < < /* 3227a3223,3227 >>> /* * Enable transmit and receive. */ CSR_WRITE_1(sc, RL_COMMAND, >>> RL_CMD_TX_ENB | RL_CMD_RX_ENB); >>> >> 3251,3254d3250 < #ifdef notdef < /* Enable receiver and >> transmitter. */ < CSR_WRITE_1(sc, RL_COMMAND, >> RL_CMD_TX_ENB|RL_CMD_RX_ENB); < #endif >> >> Let me know what additional info I can provide. >> >> thanks -kim _______________________________________________ >> freebsd-net@freebsd.org mailing list >> https://lists.freebsd.org/mailman/listinfo/freebsd-net To >> unsubscribe, send any mail to >> "freebsd-net-unsubscribe@freebsd.org" >> > > > I'm running -current with all changes in place, I'm not seeing the > issues noted here with my hardware. Can you post your hardware from > pciconf -lv? > > re0@pci0:3:0:0: class=0x020000 card=0x84321043 chip=0x816810ec > rev=0x06 hdr=0x00 > vendor = 'Realtek Semiconductor Co., Ltd.' > device = 'RTL8111/8168B PCI Express Gigabit Ethernet controller' > class = network > subclass = ethernet > re1@pci0:4:5:0: class=0x020000 card=0x43021186 chip=0x43021186 > rev=0x10 hdr=0x00 > vendor = 'D-Link System Inc' > device = 'DGE-530T Gigabit Ethernet Adapter (rev.C1) [Realtek > RTL8169]' > class = network > subclass = ethernet > > > sean pciconf -lv re0@pci0:2:0:0: class=0x020000 card=0x83671043 chip=0x816810ec rev=0x02 hdr=0x00 vendor = 'Realtek Semiconductor Co., Ltd.' device = 'RTL8111/8168B PCI Express Gigabit Ethernet controller' class = network subclass = ethernet re1@pci0:6:0:0: class=0x020000 card=0x816910ec chip=0x816910ec rev=0x10 hdr=0x00 vendor = 'Realtek Semiconductor Co., Ltd.' device = 'RTL8169 PCI Gigabit Ethernet Controller' class = network subclass = ethernet re2@pci0:6:1:0: class=0x020000 card=0x4c001186 chip=0x43001186 rev=0x10 hdr=0x00 vendor = 'D-Link System Inc' device = 'DGE-528T Gigabit Ethernet Adapter' class = network subclass = ethernet The problem was noted on re2, re0 and re1 appeared to be working normally. thanks -kim From owner-freebsd-net@freebsd.org Sat Aug 15 02:42:12 2015 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 914739B954A for ; Sat, 15 Aug 2015 02:42:12 +0000 (UTC) (envelope-from james@lottspot.com) Received: from mx0.lottspot.com (sfo.lottspot.com [198.199.98.33]) by mx1.freebsd.org (Postfix) with ESMTP id 6A8CA170B for ; Sat, 15 Aug 2015 02:42:11 +0000 (UTC) (envelope-from james@lottspot.com) Received: from localhost (localhost [127.0.0.1]) by mail.lottspot.com (Postfix) with ESMTP id 9BE2741298 for ; Fri, 14 Aug 2015 19:42:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=lottspot.com; h= content-type:content-type:content-transfer-encoding:mime-version :references:in-reply-to:user-agent:organization:message-id:date :date:subject:subject:from:from:received:received; s=mail; t= 1439606470; bh=iRWs93L7MdvbY73GoTGCNuOI7Hb6tJnYTWJQOBe/J2I=; b=U W0yzIkahsI/ZGppu6dVcYalcBpnVtV7UlqVvoxOOMrYZk6Ye3hHaxQNdmevi531p QkvfzHzFxMBBprYuxbWuOEKh6AbBz8CmBTam0/yr+E6YPTdPEc3p8ay9EYg/t1x4 W6MPh8y0CsIJWYXUX/OG39WLPjcvF6jPQOkgyn3Kmw= X-Virus-Scanned: amavisd-new at lottspot.com Received: from mx0.lottspot.com ([127.0.0.1]) by localhost (mail.lottspot.com [127.0.0.1]) (amavisd-new, port 10024) with LMTP id LOICeQDPUtA8 for ; Fri, 14 Aug 2015 19:41:10 -0700 (PDT) Received: from arch_project.localnet (h69-131-58-73.nrfdvt.dsl.dynamic.tds.net [69.131.58.73]) by mx0.lottspot.com (Postfix) with ESMTPSA id 1E544403C6 for ; Fri, 14 Aug 2015 19:41:10 -0700 (PDT) From: James Lott To: freebsd-net@freebsd.org Subject: Re: Ethernet tunneling options under FreeBSD Date: Fri, 14 Aug 2015 19:40:45 -0700 Message-ID: <3236701.dypBHjs8Lg@arch_project> Organization: LottSpot User-Agent: KMail/4.14.10 (Linux/4.1.4-1-ARCH; KDE/4.14.10; x86_64; ; ) In-Reply-To: <55CE0659.6050206@freebsd.org> References: <55CD1CE6.2010502@lottspot.com> <55CE0659.6050206@freebsd.org> MIME-Version: 1.0 Content-Transfer-Encoding: 7Bit Content-Type: text/plain; charset="us-ascii" X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 15 Aug 2015 02:42:12 -0000 > you haven't really described the network well enough.. > try an ascii-art diagram (don't forget to set fixed width font :-) > a VPN required two ends.. one is FreeBSD... what's the other? The thing is, the "other" could be any number of operating systems. I'm looking for a tunneling protocol with good cross-platform representation, but the higher priority it enduring it tunnels ethernet frames. For the sake of example we can say the other end is a FreeBSD host, since FreeBSD is looking like the "lowest common denominator" on this topic. > if both ends are FreeBSD there are dozens of possibilities.. > for example: > ng_eif->netgraph->ppp->ipsec->ppp->netgraph->ng_eif > > ng_eif->ng_ksock(udp)->IPsec->ng_ksock->ng_eif > I'm not overly concerned with the host side interfaces. What I'm really concerned with is the tunneling protocol since that's what will need support on all of my platforms. Thus, a solution requiring netgraph on both ends is not an option in my case. > tap->ppp->ppp->tap I have not found any ppp implementations under FreeBSD which support BCP. To my understanding, that's the only method by which ethernet frames can be tunneled over ppp... if I'm wrong, please do correct me! I would love nothing more than to be wrong about that :) On Friday, August 14, 2015 23:16:41 Julian Elischer wrote: > On 8/14/15 6:40 AM, James Lott wrote: > > Hello list, > > > > I am in the process of planning a build out of a L2 VPN, in which > > I'd like to have my primary "switch" and DHCP server be a FreeBSD > > system. I would like to join each new host to the VPN by > > establishing an IP tunnel with the primary "switch" which transports > > ethernet frames over the tunnel. > > you haven't really described the network well enough.. > try an ascii-art diagram (don't forget to set fixed width font :-) > a VPN required two ends.. one is FreeBSD... what's the other? > > > So far, the only protocol I have found supported by FreeBSD which > > seems capable of this is EtherIP. As far as I can tell, it doesn't > > look like there is any support for L2TPv3, and none of the PPP > > implementations available appear to support BCP. > > > > I'm not completely opposed to using EtherIP, but if there is > > something more modern which will meet my needs, I would probably try > > that first. So my question becomes: > > > > * Does anyone know of a method supported under FreeBSD (other than > > EtherIP) for tunneling ethernet over IP that they may be able to > > suggest I check out? > > if both ends are FreeBSD there are dozens of possibilities.. > for example: > ng_eif->netgraph->ppp->ipsec->ppp->netgraph->ng_eif > > ng_eif->ng_ksock(udp)->IPsec->ng_ksock->ng_eif > > tap->ppp->ppp->tap > > > Thanks for any suggestions! > > _______________________________________________ > > freebsd-net@freebsd.org mailing list > > https://lists.freebsd.org/mailman/listinfo/freebsd-net > > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" > > _______________________________________________ > freebsd-net@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" -- James Lott From owner-freebsd-net@freebsd.org Sat Aug 15 03:05:18 2015 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 1333F9B989C for ; Sat, 15 Aug 2015 03:05:18 +0000 (UTC) (envelope-from outbackdingo@gmail.com) Received: from mail-io0-x234.google.com (mail-io0-x234.google.com [IPv6:2607:f8b0:4001:c06::234]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id CA95A11BE for ; Sat, 15 Aug 2015 03:05:17 +0000 (UTC) (envelope-from outbackdingo@gmail.com) Received: by iodb91 with SMTP id b91so102786753iod.1 for ; Fri, 14 Aug 2015 20:05:17 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=LfBQJxSHwQVd6q5ewOsJpr83Ct52wmRFTyHbCPWEMos=; b=lTsmZjFfqRDxiMRgwsrOFz8rxDoh8luOAp1klSroCAk/OmKvVyLf+tDUn08EUik2L5 VAtAdnC9aFZl20W/dm7kee9qj6Tot4qKNFFO5LR1Kl00OxR5tU0i1m8KNXViGvKGSqTV 9Es2UkvDDDOQ+T4EY112yx/ADDNBemKYeAkCYATQTFlqWvnCmSiB5m6nrEpJyk+DKgkB U2dTdCruV/xlr6MlvoGleo1N8vl1T4wwk7w+vFbiw0VrGUoD6wbG+cTx9kY+K/eGvl6S QaB0Y2T5molzmQRtj7wmM9RuRzQ+tKP9sMRlMxU7RYz6J1O1eA7iCdAEqEsdV0CebLrq Px5A== MIME-Version: 1.0 X-Received: by 10.107.46.162 with SMTP id u34mr51830255iou.124.1439607917097; Fri, 14 Aug 2015 20:05:17 -0700 (PDT) Received: by 10.107.7.29 with HTTP; Fri, 14 Aug 2015 20:05:17 -0700 (PDT) In-Reply-To: <3236701.dypBHjs8Lg@arch_project> References: <55CD1CE6.2010502@lottspot.com> <55CE0659.6050206@freebsd.org> <3236701.dypBHjs8Lg@arch_project> Date: Sat, 15 Aug 2015 13:05:17 +1000 Message-ID: Subject: Re: Ethernet tunneling options under FreeBSD From: Outback Dingo To: James Lott Cc: freebsd-net@freebsd.org Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 15 Aug 2015 03:05:18 -0000 On Sat, Aug 15, 2015 at 12:40 PM, James Lott wrote: > > you haven't really described the network well enough.. > > try an ascii-art diagram (don't forget to set fixed width font :-) > > a VPN required two ends.. one is FreeBSD... what's the other? > > The thing is, the "other" could be any number of operating systems. I'm > looking for a tunneling protocol with good cross-platform representation, > but > the higher priority it enduring it tunnels ethernet frames. > > For the sake of example we can say the other end is a FreeBSD host, since > FreeBSD is looking like the "lowest common denominator" on this topic. > > > if both ends are FreeBSD there are dozens of possibilities.. > > for example: > > ng_eif->netgraph->ppp->ipsec->ppp->netgraph->ng_eif > > > > ng_eif->ng_ksock(udp)->IPsec->ng_ksock->ng_eif > > > > I'm not overly concerned with the host side interfaces. What I'm really > concerned with is the tunneling protocol since that's what will need > support > on all of my platforms. Thus, a solution requiring netgraph on both ends is > not an option in my case. > > > tap->ppp->ppp->tap > > I have not found any ppp implementations under FreeBSD which support BCP. > To my understanding, that's the only method by which ethernet frames can be > tunneled over ppp... if I'm wrong, please do correct me! I would love > nothing > more than to be wrong about that :) > > On Friday, August 14, 2015 23:16:41 Julian Elischer wrote: > > On 8/14/15 6:40 AM, James Lott wrote: > > > Hello list, > > > > > > I am in the process of planning a build out of a L2 VPN, in which > > > I'd like to have my primary "switch" and DHCP server be a FreeBSD > > > system. I would like to join each new host to the VPN by > > > establishing an IP tunnel with the primary "switch" which transports > > > ethernet frames over the tunnel. > > > > you haven't really described the network well enough.. > > try an ascii-art diagram (don't forget to set fixed width font :-) > > a VPN required two ends.. one is FreeBSD... what's the other? > > > > > So far, the only protocol I have found supported by FreeBSD which > > > seems capable of this is EtherIP. As far as I can tell, it doesn't > > > look like there is any support for L2TPv3, and none of the PPP > > > implementations available appear to support BCP. > > > > > > I'm not completely opposed to using EtherIP, but if there is > > > something more modern which will meet my needs, I would probably try > > > that first. So my question becomes: > > > > > > * Does anyone know of a method supported under FreeBSD (other than > > > EtherIP) for tunneling ethernet over IP that they may be able to > > > suggest I check out? > > > > if both ends are FreeBSD there are dozens of possibilities.. > > for example: > > ng_eif->netgraph->ppp->ipsec->ppp->netgraph->ng_eif > > > > ng_eif->ng_ksock(udp)->IPsec->ng_ksock->ng_eif > > > > tap->ppp->ppp->tap > > > > > Thanks for any suggestions! > theres also N2N which is pretty nice, and well ZeroTierOne which is somewhat unique > > > _______________________________________________ > > > freebsd-net@freebsd.org mailing list > > > https://lists.freebsd.org/mailman/listinfo/freebsd-net > > > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" > > > > _______________________________________________ > > freebsd-net@freebsd.org mailing list > > https://lists.freebsd.org/mailman/listinfo/freebsd-net > > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" > > -- > James Lott > _______________________________________________ > freebsd-net@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" > From owner-freebsd-net@freebsd.org Sat Aug 15 03:34:14 2015 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 4F8A59B9BE2 for ; Sat, 15 Aug 2015 03:34:14 +0000 (UTC) (envelope-from james@lottspot.com) Received: from mx0.lottspot.com (sfo.lottspot.com [198.199.98.33]) by mx1.freebsd.org (Postfix) with ESMTP id 2E7841B94 for ; Sat, 15 Aug 2015 03:34:13 +0000 (UTC) (envelope-from james@lottspot.com) Received: from localhost (localhost [127.0.0.1]) by mail.lottspot.com (Postfix) with ESMTP id E156641295 for ; Fri, 14 Aug 2015 20:34:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=lottspot.com; h= content-type:content-type:content-transfer-encoding:mime-version :references:in-reply-to:user-agent:organization:message-id:date :date:subject:subject:from:from:received:received; s=mail; t= 1439609599; bh=0td/IVXI9e3uSHVEM2sDsnff8wjabvUHr10K+YOs5L8=; b=Y qPZGSIKUwNn3FeIttwNoZY9DumqkQHrdPzE4h5GAqylw2R28rJhrv2lsxj3+N9dk JEWJxNQRLZoxDq/KcYHXKKeklvc3VYZLlXWLyfNJ3RGcfjevvQT2mLKCJuAJj7GG eWpj+s6lYDmQ8kX9RhIPar1LTdDdF8DuKb0Y6RCpw0= X-Virus-Scanned: amavisd-new at lottspot.com Received: from mx0.lottspot.com ([127.0.0.1]) by localhost (mail.lottspot.com [127.0.0.1]) (amavisd-new, port 10024) with LMTP id jkd2XGzJa7Tr for ; Fri, 14 Aug 2015 20:33:19 -0700 (PDT) Received: from arch_project.localnet (h69-131-58-73.nrfdvt.dsl.dynamic.tds.net [69.131.58.73]) by mx0.lottspot.com (Postfix) with ESMTPSA id DA970403C6 for ; Fri, 14 Aug 2015 20:33:18 -0700 (PDT) From: James Lott To: freebsd-net@freebsd.org Subject: Re: Ethernet tunneling options under FreeBSD Date: Fri, 14 Aug 2015 20:32:55 -0700 Message-ID: <2628655.0T22OuP5Ng@arch_project> Organization: LottSpot User-Agent: KMail/4.14.10 (Linux/4.1.4-1-ARCH; KDE/4.14.10; x86_64; ; ) In-Reply-To: References: <55CD1CE6.2010502@lottspot.com> <3236701.dypBHjs8Lg@arch_project> MIME-Version: 1.0 Content-Transfer-Encoding: 7Bit Content-Type: text/plain; charset="us-ascii" X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 15 Aug 2015 03:34:14 -0000 n2n honestly looks wonderful, but it also appears to be dead... I'm trying to stay as close to the OS layer as possible with my options, so I would prefer to limit the role of comprehensive software like OpenVPN or what ZeroTierOne appears to be. I actually found this interesting github project, which provides a simple solution for what I'm trying to do... https://github.com/vsergeev/tinytaptunnel Unfortunately, it's written for Linux... and... in go... but the README at least gave me a couple more ideas to look into. Feel free to keep coming with the suggestions if anyone has anymore! This is great stuff On Saturday, August 15, 2015 13:05:17 Outback Dingo wrote: > On Sat, Aug 15, 2015 at 12:40 PM, James Lott wrote: > > > you haven't really described the network well enough.. > > > try an ascii-art diagram (don't forget to set fixed width font :-) > > > a VPN required two ends.. one is FreeBSD... what's the other? > > > > The thing is, the "other" could be any number of operating systems. I'm > > looking for a tunneling protocol with good cross-platform representation, > > but > > the higher priority it enduring it tunnels ethernet frames. > > > > For the sake of example we can say the other end is a FreeBSD host, since > > FreeBSD is looking like the "lowest common denominator" on this topic. > > > > > if both ends are FreeBSD there are dozens of possibilities.. > > > for example: > > > ng_eif->netgraph->ppp->ipsec->ppp->netgraph->ng_eif > > > > > > ng_eif->ng_ksock(udp)->IPsec->ng_ksock->ng_eif > > > > I'm not overly concerned with the host side interfaces. What I'm really > > concerned with is the tunneling protocol since that's what will need > > support > > on all of my platforms. Thus, a solution requiring netgraph on both ends > > is > > not an option in my case. > > > > > tap->ppp->ppp->tap > > > > I have not found any ppp implementations under FreeBSD which support BCP. > > To my understanding, that's the only method by which ethernet frames can > > be > > tunneled over ppp... if I'm wrong, please do correct me! I would love > > nothing > > more than to be wrong about that :) > > > > On Friday, August 14, 2015 23:16:41 Julian Elischer wrote: > > > On 8/14/15 6:40 AM, James Lott wrote: > > > > Hello list, > > > > > > > > I am in the process of planning a build out of a L2 VPN, in which > > > > I'd like to have my primary "switch" and DHCP server be a FreeBSD > > > > system. I would like to join each new host to the VPN by > > > > establishing an IP tunnel with the primary "switch" which transports > > > > ethernet frames over the tunnel. > > > > > > you haven't really described the network well enough.. > > > try an ascii-art diagram (don't forget to set fixed width font :-) > > > a VPN required two ends.. one is FreeBSD... what's the other? > > > > > > > So far, the only protocol I have found supported by FreeBSD which > > > > seems capable of this is EtherIP. As far as I can tell, it doesn't > > > > look like there is any support for L2TPv3, and none of the PPP > > > > implementations available appear to support BCP. > > > > > > > > I'm not completely opposed to using EtherIP, but if there is > > > > something more modern which will meet my needs, I would probably try > > > > that first. So my question becomes: > > > > > > > > * Does anyone know of a method supported under FreeBSD (other than > > > > EtherIP) for tunneling ethernet over IP that they may be able to > > > > suggest I check out? > > > > > > if both ends are FreeBSD there are dozens of possibilities.. > > > for example: > > > ng_eif->netgraph->ppp->ipsec->ppp->netgraph->ng_eif > > > > > > ng_eif->ng_ksock(udp)->IPsec->ng_ksock->ng_eif > > > > > > tap->ppp->ppp->tap > > > > > > > Thanks for any suggestions! > > theres also N2N which is pretty nice, and well ZeroTierOne which is > somewhat unique > > > > > _______________________________________________ > > > > freebsd-net@freebsd.org mailing list > > > > https://lists.freebsd.org/mailman/listinfo/freebsd-net > > > > To unsubscribe, send any mail to "freebsd-net- unsubscribe@freebsd.org" > > > > > > _______________________________________________ > > > freebsd-net@freebsd.org mailing list > > > https://lists.freebsd.org/mailman/listinfo/freebsd-net > > > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" > > > > -- > > James Lott > > _______________________________________________ > > freebsd-net@freebsd.org mailing list > > https://lists.freebsd.org/mailman/listinfo/freebsd-net > > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" -- James Lott From owner-freebsd-net@freebsd.org Sat Aug 15 03:46:09 2015 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id D20609B9D2C for ; Sat, 15 Aug 2015 03:46:09 +0000 (UTC) (envelope-from lists@jnielsen.net) Received: from webmail2.jnielsen.net (webmail2.jnielsen.net [50.114.224.20]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "webmail2.jnielsen.net", Issuer "freebsdsolutions.net" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 248821EB1 for ; Sat, 15 Aug 2015 03:46:08 +0000 (UTC) (envelope-from lists@jnielsen.net) Received: from [192.168.2.123] (c-50-160-123-105.hsd1.ut.comcast.net [50.160.123.105]) (authenticated bits=0) by webmail2.jnielsen.net (8.15.1/8.15.1) with ESMTPSA id t7F3JqEY013587 (version=TLSv1 cipher=ECDHE-RSA-AES256-SHA bits=256 verify=NO); Fri, 14 Aug 2015 21:19:55 -0600 (MDT) (envelope-from lists@jnielsen.net) X-Authentication-Warning: webmail2.jnielsen.net: Host c-50-160-123-105.hsd1.ut.comcast.net [50.160.123.105] claimed to be [192.168.2.123] Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (1.0) Subject: Re: vlan+bridge questions From: John Nielsen X-Mailer: iPhone Mail (12H143) In-Reply-To: Date: Fri, 14 Aug 2015 21:19:51 -0600 Cc: FreeBSD Net Content-Transfer-Encoding: quoted-printable Message-Id: <1468D6AA-1368-4B3E-B9A1-24D5B7489A02@jnielsen.net> References: To: Hooshang F X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 15 Aug 2015 03:46:09 -0000 > On Aug 14, 2015, at 11:57 AM, Hooshang F wrote: >=20 > We need to install a freebsd firewall (pf). The freebsd > box needs to be placed in bridge mode in the middle of a VLAN truck > link between 2 Cisco switches. The em0 and em1 ports > are connected to the trunk ports on the 2 switches. >=20 > We are going to: >=20 > 1- Define two vlan interfaces for vlan id X. > one with em0 as parent and the other on top of em1. > 2- Create a bridge interface. > 3- Add the two vlan interfaces as members of the bridge. > 4- Repeat 1-3 for every vlan id used in the network. >=20 > 2 questions: >=20 > 1- Is not there a simpler method which does not involve creating so > many vlans & bridges? For instance, is it possible to have > a truck interface which accepts 'all' vlan IDs (like cisco) instead > of creating two vlan interface per ID? >=20 > 2- How the untagged traffic should be bridged? Cisco switches > send out packets untagged if vlan ID is equal to the trunk port > 'native' vlan id. To bridge this packets, we should create > a bridge with em0 and em1 as members, but that will > effectively disables bridging on vlan interfaces. Right? Same answer for both questions: bridge the parent interfaces. If you need vl= an interfaces, create them as children of the single bridge interface.=20= From owner-freebsd-net@freebsd.org Sat Aug 15 13:05:05 2015 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id DD2D29BA841 for ; Sat, 15 Aug 2015 13:05:05 +0000 (UTC) (envelope-from barney_cordoba@yahoo.com) Received: from nm42-vm6.bullet.mail.ne1.yahoo.com (nm42-vm6.bullet.mail.ne1.yahoo.com [98.138.121.62]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 920AD1E6C for ; Sat, 15 Aug 2015 13:05:05 +0000 (UTC) (envelope-from barney_cordoba@yahoo.com) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1439643058; bh=GRfcLccQztD1BiPTNsPmsI4aBq8k6xsacO+ZiFsREgg=; h=Date:From:Reply-To:To:Cc:In-Reply-To:References:Subject:From:Subject; b=T/Ej4Kn2eAhB5JRbVTz+sE6LgK0R1vWAoQsXxZyMEgBr9lxSBoW6WsoJcTXr4SM7Jf3aF5BaSVaeO0IjwRSmb1wtzCJxzQq3S6Iuyy9bGAyFg/RdlFOXyGAajW6lHPPpWidw9PGi8a7VF7R+ScgHtUGM9f6YJMqRxfyhsMXq8MeZ9QE8tCFpkozfLH9EOZfVpKVQ1MsxK0JTNrnxSeK1kyO6xylrBA+fBwnCCv/qc10biPHvExgullkPu/UoQ9oH6GKtYkLak2vCuQxjO8RhH6DJwZ2WsKLBPc32Jnw81lFTD2EbAdzPra7Ge/IWJY00zuslKnxOIDcHQJK33d8g3Q== Received: from [127.0.0.1] by nm42.bullet.mail.ne1.yahoo.com with NNFMP; 15 Aug 2015 12:50:58 -0000 Received: from [98.138.226.177] by nm42.bullet.mail.ne1.yahoo.com with NNFMP; 15 Aug 2015 12:48:06 -0000 Received: from [98.138.87.1] by tm12.bullet.mail.ne1.yahoo.com with NNFMP; 15 Aug 2015 12:48:06 -0000 Received: from [127.0.0.1] by omp1001.mail.ne1.yahoo.com with NNFMP; 15 Aug 2015 12:48:06 -0000 X-Yahoo-Newman-Property: ymail-4 X-Yahoo-Newman-Id: 13674.98859.bm@omp1001.mail.ne1.yahoo.com X-YMail-OSG: .o16wKYVM1lJruaFS9eYxLNQT5ztQDtvKuPYG8.jTyeliGExxUkLcGJWdmyTjfc yC3VB1yVTHx.gAbS26cmbCoP9eABFNsqPHmRdl2VTpvzkKwMMqgseHeCBp21dceRP1r33Fn.W7.V DObRWt4NwYzL9TzyhIeUkJZEkg2C52AGYFRxh3a0uv..mewqZ3FN.81Mil7makZ0SzYFHnTLyZ1n W4aFVDXDSobjdiChXgJg8UWKp5uWLjsyo9QThq_cHbv4T6TuFdEKyXq5epq5EBEAuT6J7MLRI8eQ K7WvKSNDRXJ7IZk_VA9QfBLQySRYV4xJ7ffs4wBBC2QujPrv0REuCafi.w8du6YOB_4XCKiEgNTD 2sUOk33Zgb6NSSKeN5JPUCUcw_uAsL.HnV_MaUw0ALBQU7J4ar.le0SHaofcHPJovxKxekYRgKpM 7MV_i6uMjhsMCYj_Ns7CQM.YjRdYuFKqYm35gcCiwxwTjT39HwsYuDJT._AR97DuMN1hDVzZo3rA FL_Gr8tItBgBAvDeFNH0hTExi Received: by 98.138.101.163; Sat, 15 Aug 2015 12:48:05 +0000 Date: Sat, 15 Aug 2015 12:48:00 +0000 (UTC) From: Barney Cordoba Reply-To: Barney Cordoba To: Maxim Sobolev , Luigi Rizzo Cc: "Alexander V. Chernikov" , FreeBSD Net , Babak Farrokhi , "freebsd@intel.com" , =?UTF-8?Q?Jev_Bj=C3=B6rsell?= , =?UTF-8?Q?Olivier_Cochard-Labb=C3=A9?= Message-ID: <1344881217.5202288.1439642880348.JavaMail.yahoo@mail.yahoo.com> In-Reply-To: References: Subject: Re: Poor high-PPS performance of the 10G ixgbe(9) NIC/driver in FreeBSD 10.1 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 15 Aug 2015 13:05:06 -0000 I am laughing so hard that I had to open some windows to get more oxygen!= =C2=A0=20 On Friday, August 14, 2015 1:30 PM, Maxim Sobolev wrote: =20 Hi guys, unfortunately no, neither reduction of the number of queues from = 8 to 6 nor pinning interrupt rate at 20000 per queue have not made any difference. The card still goes kaboom at about 200Kpps no matter what. in fact I've gone bit further, and after the first spike went on an pushed interrupt rate even further down to 10000, but again no difference either, it still blows at the same mark. Although it did have effect on interrupt rate reduction from 190K to some 130K according to the systat -vm, so that the moderation itself seems to be working fine. We will try disabling IXGBE_FDIR tomorrow and see if it helps. http://sobomax.sippysoft.com/ScreenShot391.png <- systat -vm with max_interrupt_rate =3D 20000 right before overload http://sobomax.sippysoft.com/ScreenShot392.png <- systat -vm during issue unfolding (max_interrupt_rate =3D 10000) http://sobomax.sippysoft.com/ScreenShot394.png <- cpu/net monitoring, first two spikes are with max_interrupt_rate =3D 20000, the third one max_interrupt_rate =3D 10000 -Max On Wed, Aug 12, 2015 at 5:23 AM, Luigi Rizzo wrote: > As I was telling to maxim, you should disable aim because it only matches > the max interrupt rate to the average packet size, which is the last thin= g > you want. > > Setting the interrupt rate with sysctl (one per queue) gives you precise > control on the max rate and (hence, extra latency). 20k interrupts/s give > you 50us of latency, and the 2k slots in the queue are still enough to > absorb a burst of min-sized frames hitting a single queue (the os will > start dropping long before that level, but that's another story). > > Cheers > Luigi > > On Wednesday, August 12, 2015, Babak Farrokhi > wrote: > >> I ran into the same problem with almost the same hardware (Intel X520) >> on 10-STABLE. HT/SMT is disabled and cards are configured with 8 queues, >> with the same sysctl tunings as sobomax@ did. I am not using lagg, no >> FLOWTABLE. >> >> I experimented with pmcstat (RESOURCE_STALLS) a while ago and here [1] >> [2] you can see the results, including pmc output, callchain, flamegraph >> and gprof output. >> >> I am experiencing huge number of interrupts with 200kpps load: >> >> # sysctl dev.ix | grep interrupt_rate >> dev.ix.1.queue7.interrupt_rate: 125000 >> dev.ix.1.queue6.interrupt_rate: 6329 >> dev.ix.1.queue5.interrupt_rate: 500000 >> dev.ix.1.queue4.interrupt_rate: 100000 >> dev.ix.1.queue3.interrupt_rate: 50000 >> dev.ix.1.queue2.interrupt_rate: 500000 >> dev.ix.1.queue1.interrupt_rate: 500000 >> dev.ix.1.queue0.interrupt_rate: 100000 >> dev.ix.0.queue7.interrupt_rate: 500000 >> dev.ix.0.queue6.interrupt_rate: 6097 >> dev.ix.0.queue5.interrupt_rate: 10204 >> dev.ix.0.queue4.interrupt_rate: 5208 >> dev.ix.0.queue3.interrupt_rate: 5208 >> dev.ix.0.queue2.interrupt_rate: 71428 >> dev.ix.0.queue1.interrupt_rate: 5494 >> dev.ix.0.queue0.interrupt_rate: 6250 >> >> [1] http://farrokhi.net/~farrokhi/pmc/6/ >> [2] http://farrokhi.net/~farrokhi/pmc/7/ >> >> Regards, >> Babak >> >> >> Alexander V. Chernikov wrote: >> > 12.08.2015, 02:28, "Maxim Sobolev" : >> >> Olivier, keep in mind that we are not "kernel forwarding" packets, bu= t >> "app >> >> forwarding", i.e. the packet goes full way >> >> net->kernel->recvfrom->app->sendto->kernel->net, which is why we have >> much >> >> lower PPS limits and which is why I think we are actually benefiting >> from >> >> the extra queues. Single-thread sendto() in a loop is CPU-bound at >> about >> >> 220K PPS, and while running the test I am observing that outbound >> traffic >> >> from one thread is mapped into a specific queue (well, pair of queues >> on >> >> two separate adaptors, due to lagg load balancing action). And the pe= ak >> >> performance of that test is at 7 threads, which I believe corresponds >> to >> >> the number of queues. We have plenty of CPU cores in the box (24) wit= h >> >> HTT/SMT disabled and one CPU is mapped to a specific queue. This >> leaves us >> >> with at least 8 CPUs fully capable of running our app. If you look at >> the >> >> CPU utilization, we are at about 10% when the issue hits. >> > >> > In any case, it would be great if you could provide some profiling inf= o >> since there could be >> > plenty of problematic places starting from TX rings contention to some >> locks inside udp or even >> > (in)famous random entropy harvester.. >> > e.g. something like pmcstat -TS instructions -w1 might be sufficient t= o >> determine the reason >> >> ix0: >> port >> >> 0x6020-0x603f mem 0xc7c00000-0xc7dfffff,0xc7e04000-0xc7e07fff irq 40 = at >> >> device 0.0 on pci3 >> >> ix0: Using MSIX interrupts with 9 vectors >> >> ix0: Bound queue 0 to cpu 0 >> >> ix0: Bound queue 1 to cpu 1 >> >> ix0: Bound queue 2 to cpu 2 >> >> ix0: Bound queue 3 to cpu 3 >> >> ix0: Bound queue 4 to cpu 4 >> >> ix0: Bound queue 5 to cpu 5 >> >> ix0: Bound queue 6 to cpu 6 >> >> ix0: Bound queue 7 to cpu 7 >> >> ix0: Ethernet address: 0c:c4:7a:5e:be:64 >> >> ix0: PCI Express Bus: Speed 5.0GT/s Width x8 >> >> 001.000008 [2705] netmap_attach success for ix0 tx 8/4096 rx >> >> 8/4096 queues/slots >> >> ix1: >> port >> >> 0x6000-0x601f mem 0xc7a00000-0xc7bfffff,0xc7e00000-0xc7e03fff irq 44 = at >> >> device 0.1 on pci3 >> >> ix1: Using MSIX interrupts with 9 vectors >> >> ix1: Bound queue 0 to cpu 8 >> >> ix1: Bound queue 1 to cpu 9 >> >> ix1: Bound queue 2 to cpu 10 >> >> ix1: Bound queue 3 to cpu 11 >> >> ix1: Bound queue 4 to cpu 12 >> >> ix1: Bound queue 5 to cpu 13 >> >> ix1: Bound queue 6 to cpu 14 >> >> ix1: Bound queue 7 to cpu 15 >> >> ix1: Ethernet address: 0c:c4:7a:5e:be:65 >> >> ix1: PCI Express Bus: Speed 5.0GT/s Width x8 >> >> 001.000009 [2705] netmap_attach success for ix1 tx 8/4096 rx >> >> 8/4096 queues/slots >> >> >> >> On Tue, Aug 11, 2015 at 4:14 PM, Olivier Cochard-Labb=C3=A9 < >> olivier@cochard.me> >> >> wrote: >> >> >> >>>=C2=A0 On Tue, Aug 11, 2015 at 11:18 PM, Maxim Sobolev > > >> >>>=C2=A0 wrote: >> >>> >> >>>>=C2=A0 Hi folks, >> >>>> >> >>>>=C2=A0 =E2=80=8BHi, >> >>>=C2=A0 =E2=80=8B >> >>> >> >>>>=C2=A0 We've trying to migrate some of our high-PPS systems to a new >> hardware >> >>>>=C2=A0 that >> >>>>=C2=A0 has four X540-AT2 10G NICs and observed that interrupt time g= oes >> through >> >>>>=C2=A0 roof after we cross around 200K PPS in and 200K out (two port= s in >> LACP). >> >>>>=C2=A0 The previous hardware was stable up to about 350K PPS in and = 350K >> out. I >> >>>>=C2=A0 believe the old one was equipped with the I350 and had the >> identical LACP >> >>>>=C2=A0 configuration. The new box also has better CPU with more core= s >> (i.e. 24 >> >>>>=C2=A0 cores vs. 16 cores before). CPU itself is 2 x E5-2690 v3. >> >>>=C2=A0 =E2=80=8B200K PPS, and even 350K PPS are very low value indeed= . >> >>>=C2=A0 On a Intel Xeon L5630 (4 cores only) with one X540-AT2=E2=80= =8B >> >>> >> >>>=C2=A0 =E2=80=8B(then 2 10Gigabit ports)=E2=80=8B I've reached about = 1.8Mpps (fastforwarding >> >>>=C2=A0 enabled) [1]. >> >>>=C2=A0 But my setup didn't use lagg(4): Can you disable lagg configur= ation >> and >> >>>=C2=A0 re-measure your performance without lagg ? >> >>> >> >>>=C2=A0 Do you let Intel NIC drivers using 8 queues for port too? >> >>>=C2=A0 In my use case (forwarding smallest UDP packet size), I obtain= better >> >>>=C2=A0 behaviour by limiting NIC queues to 4 (hw.ix.num_queues or >> >>>=C2=A0 hw.ixgbe.num_queues, don't remember) if my system had 8 cores.= And >> this >> >>>=C2=A0 with Gigabit Intel[2] or Chelsio NIC [3]. >> >>> >> >>>=C2=A0 Don't forget to disable TSO and LRO too. >> >>> >> >>>=C2=A0 =E2=80=8BRegards, >> >>> >> >>>=C2=A0 Olivier >> >>> >> >>>=C2=A0 [1] >> >>> >> http://bsdrp.net/documentation/examples/forwarding_performance_lab_of_an= _ibm_system_x3550_m3_with_10-gigabit_intel_x540-at2#graphs >> >>>=C2=A0 [2] >> >>> >> http://bsdrp.net/documentation/examples/forwarding_performance_lab_of_a_= superserver_5018a-ftn4#graph1 >> >>>=C2=A0 [3] >> >>> >> http://bsdrp.net/documentation/examples/forwarding_performance_lab_of_a_= hp_proliant_dl360p_gen8_with_10-gigabit_with_10-gigabit_chelsio_t540-cr#red= ucing_nic_queues >> >> _______________________________________________ >> >> freebsd-net@freebsd.org mailing list >> >> http://lists.freebsd.org/mailman/listinfo/freebsd-net >> >> To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org= " >> > _______________________________________________ >> > freebsd-net@freebsd.org mailing list >> > http://lists.freebsd.org/mailman/listinfo/freebsd-net >> > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" >> _______________________________________________ >> freebsd-net@freebsd.org mailing list >> http://lists.freebsd.org/mailman/listinfo/freebsd-net >> To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" > > > > -- > -----------------------------------------+------------------------------- >=C2=A0 Prof. Luigi RIZZO, rizzo@iet.unipi.it=C2=A0 . Dip. di Ing. dell'Inf= ormazione >=C2=A0 http://www.iet.unipi.it/~luigi/ =C2=A0 =C2=A0 =C2=A0 . Universita` = di Pisa >=C2=A0 TEL=C2=A0 =C2=A0 =C2=A0 +39-050-2217533=C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 . via Diotisalvi 2 >=C2=A0 Mobile=C2=A0 +39-338-6809875=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 . 56122 PISA (Italy) > -----------------------------------------+------------------------------- > > _______________________________________________ freebsd-net@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org"