From owner-freebsd-net@FreeBSD.ORG Sat Apr 27 03:01:53 2013 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id C0EB5A74 for ; Sat, 27 Apr 2013 03:01:53 +0000 (UTC) (envelope-from barney_cordoba@yahoo.com) Received: from nm8-vm4.bullet.mail.ne1.yahoo.com (nm8-vm4.bullet.mail.ne1.yahoo.com [98.138.91.168]) by mx1.freebsd.org (Postfix) with ESMTP id 8F3621930 for ; Sat, 27 Apr 2013 03:01:53 +0000 (UTC) Received: from [98.138.90.53] by nm8.bullet.mail.ne1.yahoo.com with NNFMP; 27 Apr 2013 03:01:47 -0000 Received: from [98.138.87.7] by tm6.bullet.mail.ne1.yahoo.com with NNFMP; 27 Apr 2013 03:01:46 -0000 Received: from [127.0.0.1] by omp1007.mail.ne1.yahoo.com with NNFMP; 27 Apr 2013 03:01:46 -0000 X-Yahoo-Newman-Property: ymail-3 X-Yahoo-Newman-Id: 979025.68748.bm@omp1007.mail.ne1.yahoo.com Received: (qmail 99115 invoked by uid 60001); 27 Apr 2013 03:01:46 -0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1367031706; bh=jxvnoMAZmdpQYBCUvWh/R/ZKlJ6VuiHzpbLzVTooWhA=; h=X-YMail-OSG:Received:X-Rocket-MIMEInfo:X-Mailer:Message-ID:Date:From:Subject:To:Cc:MIME-Version:Content-Type:Content-Transfer-Encoding; b=w0Ri4XR/zp2qUXUHL+VNXYOsFi/jyInnvNDaCROqCFySZAFF00LaHNfsfiRqKkiCF0l/snkMmbl3hB0zzaF/12kFNLPvLA5lCoBihnLR/FCi++gb5AyvnodRzb48LGF+FC7zKKST64fvR7FGUYXgBVvudBG/wo9Ry6UE53EsjzM= DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=s1024; d=yahoo.com; h=X-YMail-OSG:Received:X-Rocket-MIMEInfo:X-Mailer:Message-ID:Date:From:Subject:To:Cc:MIME-Version:Content-Type:Content-Transfer-Encoding; b=v4U5km0DmIqYk9ANETPR/WJphOQYEEnBoUH4n2TV5pC5TgGk3xWkW7eNiMfTY0xBjWNGB1MyRTITUs0eDZd0LwIPfG22saRkqOtYzmoV49FlAha2GQAD/egQwRBdmyD6IVH6uEnDFfqz62yeDU+j+HWuOPjJvsgV940S3M7Ug3Q=; X-YMail-OSG: wZaFKoYVM1lFAabP.jPB8oN3mw37NbR9DjUg5pjPk58nS.5 I0OVXtmfyOLenIZGD1Cu1BZElDWhwWOen3YSwDpuJd3tIpizQknIVOfSk4fX O.KQ4QjE0Rl5L.J2qz5O6cxHl9frehmw1TVkLZ_4NZhTsyuKsx29KDsljiKk i.lTDVWXNwGUfJCWzPriyE9brZJhdZNn1FfmF.dbeiaNenlD2u_89UVKrgQl YVhOZERSiNo9aAOZjDWc8Am2A.F7Wxwt5m3.bOxF_gyyqEYdbbuDKacXMxZZ .keA4.nWoexIQodNm36bcsMRU_59CjU1_zaZdgMTjNKq6ka3pKIkXJxsiQic m6YJBUofQjJJBQrFQ6WzAIk1dK8nd74pxWO.s4JY.8y27Bee42b_1tqFl9fV QGusiJr_DA.wFNlJGznHW74i9N5Njkdw5gu0UjNbfPhiDnJJwL88sUt8vCh1 A6OT4ZKocX.5tW6FufmEA4CxRinXMf7PFdMmpj4kofnthnA.eRGD3m2.Dgi3 ZyUoO6Ps- Received: from [98.203.118.124] by web121602.mail.ne1.yahoo.com via HTTP; Fri, 26 Apr 2013 20:01:46 PDT X-Rocket-MIMEInfo: 002.001, DQotLS0gT24gRnJpLCA0LzI2LzEzLCBFcmljaCBXZWlsZXIgPHdlaWxlckBzb2UudWNzYy5lZHU.IHdyb3RlOg0KDQo.IEZyb206IEVyaWNoIFdlaWxlciA8d2VpbGVyQHNvZS51Y3NjLmVkdT4NCj4gU3ViamVjdDogUmU6IHBmIHBlcmZvcm1hbmNlPw0KPiBUbzogIkFuZHJlIE9wcGVybWFubiIgPGFuZHJlQGZyZWVic2Qub3JnPg0KPiBDYzogIlBhdWwgVGF0YXJza3kiIDxwYXVsQHNvZS51Y3NjLmVkdT4sIGZyZWVic2QtbmV0QGZyZWVic2Qub3JnDQo.IERhdGU6IEZyaWRheSwgQXByaWwgMjYsIDIwMTMsIDEBMAEBAQE- X-Mailer: YahooMailClassic/15.1.8 YahooMailWebService/0.8.141.536 Message-ID: <1367031706.98947.YahooMailClassic@web121602.mail.ne1.yahoo.com> Date: Fri, 26 Apr 2013 20:01:46 -0700 (PDT) From: Barney Cordoba Subject: Re: pf performance? To: Andre Oppermann , Erich Weiler MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: quoted-printable Cc: Paul Tatarsky , freebsd-net@freebsd.org X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 27 Apr 2013 03:01:53 -0000 --- On Fri, 4/26/13, Erich Weiler wrote: > From: Erich Weiler > Subject: Re: pf performance? > To: "Andre Oppermann" > Cc: "Paul Tatarsky" , freebsd-net@freebsd.org > Date: Friday, April 26, 2013, 12:04 PM > >> But the work pf does would > show up in 'system' on top right?=A0 So if I > >> see all my CPUs tied up 100% > >> in 'interrupts' and very little 'system', would it > be a reasonable > >> assumption to think that if I got > >> more CPU cores to handle the interrupts that > eventually I would see > >> 'system' load increase as the > >> interrupt load became faster to be handled?=A0 > And thus increase my > >> bandwidth? > >=20 > > Having the work of pf show up in 'interrupts' or > 'system' depends on the > > network driver and how it handles sending packets up > the stack.=A0 In most > > cases drivers deliver packets from interrupt context. >=20 > Ah, I see.=A0 Definitely appears for me in interrupts > then.=A0 I've got the mxge driver doing the work > here.=A0 So, given that I can spread out the interrupts > to every core (like, pin an interrupt queue to each core), I > can have all my cores work on the process.=A0 But seeing > as though the pf bit is still serialized I'm not sure that I > understand how it is serialized when many CPUs are handling > interrupts, and hence doing the work of pf as well?=A0 > Wouldn't that indicate that the work of pf is being handled > by many cores, as many cores are handling the interrupts? >=20 you're thinking exactly backwards. You're creating lock contention by having a bunch of receive processes going into a single threaded pf process. Think of it like a six lane highway that has 5 lanes closed a mile up the road. The result isn't that you go the same speed as a 1 lane highway; what you have is a parking lot. The only thing you're doing by spreading the interrupts is using up more cycles on more cores. What you *should* be doing, if you can engineer it, is use 1 path through the pf filter. You could have 4 queues feed a single process that dequeues and runs through the filter. The problem with that is that the pf process IS the bottleneck in that its slower than the receive processes, so you'd best just use the other cores to do userland stuff. You could use cpuset to make sure that no userland process uses the interrupt core, and dedicate 1 cpu to packet filtering. 1 modern CPU can easily handle a gig of traffic. There's no need to spread in most case. BC > Or are you saying that pf *is* being handled by many cores, > but just in a serialized nature?=A0 Like, packet 1 is > handled by core 0, then packet 2 is handled by core 1, > packet 3 is handled by core 4, etc?=A0 Such that even > though multiple cores are handling it, they are just doing > so serially and not in parallel?=A0 And if so, maybe it > still helps to have many cores? >=20 > Thanks for all the excellent info! >=20 > >> In other words, until I see like 100% system usage > in one core, I > >> would have room to grow? > >=20 > > You have room to grow if 'idle' is more than 0% and the > interrupts of > > the networks cards are running on different > cores.=A0 If one core gets > > all the interrupts a second idle core doesn't get the > chance to help > > out.=A0 IIRC the interrupt allocation to cores is > done at interrupt > > registration time or driver attach time.=A0 It can > be re-distributed > > at run time on most architecture but I'm not sure we > have an easily > > accessible API for that.