Date: Wed, 8 Apr 2009 04:48:23 -0700 (PDT) From: Barney Cordoba <barney_cordoba@yahoo.com> To: "H.Fazaeli" <fazaeli@sepehrs.com> Cc: freebsd-net@freebsd.org, Ivan Voras <ivoras@freebsd.org> Subject: Re: Advice on a multithreaded netisr patch? Message-ID: <477001.91824.qm@web63902.mail.re1.yahoo.com> In-Reply-To: <49DC3961.8090707@sepehrs.com>
next in thread | previous in thread | raw e-mail | index | archive | help
--- On Wed, 4/8/09, H.Fazaeli <fazaeli@sepehrs.com> wrote: > From: H.Fazaeli <fazaeli@sepehrs.com> > Subject: Re: Advice on a multithreaded netisr patch? > To: barney_cordoba@yahoo.com > Cc: freebsd-net@freebsd.org, "Ivan Voras" <ivoras@freebsd.org> > Date: Wednesday, April 8, 2009, 1:42 AM > Barney Cordoba wrote: > > > > > > --- On Tue, 4/7/09, Ivan Voras > <ivoras@freebsd.org> wrote: > > > > > >> From: Ivan Voras <ivoras@freebsd.org> > >> Subject: Re: Advice on a multithreaded netisr > patch? > >> To: freebsd-net@freebsd.org > >> Date: Tuesday, April 7, 2009, 5:59 PM > >> Barney Cordoba wrote: > >> > >> > >>> 1) Multiple TX queues are not supported. > There's > >>> > >> some hokey code to > >> > >>> test, but it doesn't properly separate > flows to > >>> > >> the queues. > >> > >>> 2) 2 Rx queues don't work, so only 1 and 4 > work > >>> 3) With 4 queues, it just sucks up CPU under > heavy > >>> > >> load on 4 cpus. It will > >> > >>> blow 4 cpus at a lower load than em will with > 1 > >>> 4) You'll need to fix DMA setup, as it > sets the > >>> > >> alignment requirement > >> > >>> to PAGE_SIZE. I haven't been able to > convince Jack > >>> > >> that its wrong, not > >> > >>> that I've tried very hard since its easy > to just > >>> > >> fix myself. > >> > >> Reading this thread it looks like the development > of both > >> Intel drivers > >> is a bit stalled, doesn't it? AFAIK the em > driver is > >> also > >> semi-officially abandoned, and both from my > experience and > >> others it > >> looks like new development and patches are being > rejected. > >> Time to shop > >> other hardware? > >> > > > > To be fair, the OS doesn't really support > multiqueue yet, or has > > for only a few hours, so lets not go crazy. > > > > It makes a lot more sense to have someone on the > "team" work with > > Jack on improving the performance and working out the > kinks. When > > I asked Jack about the poor performance of if_igb, he > indicated that > > Intel's position is that the drivers are > "just samples", which really > > doesn't give anyone much confidence that they want > to run their business > > on them. You already have Jack doing all of the hard > work; that is > > supporting the new-chip-per-week that intel puts out, > so it seems to > > me the best strategy would be to try to convince Intel > that its in > > their best interest to have drivers that work well so > people don't > > think that their hardware stinks. > > > > As an example, the Chelsio 10gb bypass card is $3495. > and an Intel > > card is ~$1000, so its a big win for the community as > a whole to have > > good intel drivers going forward. > > > > My work is commercially proprietary so I can't > share my code, but > > I can certainly share ideas on things that I've > tested and discovered. > > > > > can you provide more details on the improvements you > achieved? > > > Barney > > > > > > > > _______________________________________________ > > freebsd-net@freebsd.org mailing list > > http://lists.freebsd.org/mailman/listinfo/freebsd-net > > To unsubscribe, send any mail to > "freebsd-net-unsubscribe@freebsd.org" > > > > > > -- As all developers konw, programming is 90% learning and 10% code. So far, I've implemented multiqueue for 7.x and gotten everything to work for both igb and ixgbe. igb isn't all that interesting since em can easily handle 1 Gb/s; so ixgbe is really the goal. The igb and ixgbe are similar designs so the work is somewhat parallel. As of now, I'm working on separating the theory from the real world and getting a feel for which design techniques work best. I'm also *not* designing for a system that uses the stack (a filtering firewall type system), so the things that Robert talks about apply differently. A web server, for example, will likely only have 1 controller and will have many user threads; while a router or firewall will have 2 equally loaded NICs with few if any user threads. Its quite likely that completely different approaches are needed to optimize each. I'm at the point of testing design approaches. So the jury is out as what what can be achieved. What I can say is that multiqueue isn't a panacea or even desirable if its not designed correctly. Out of the box, increasing the number of queues just to "spread interrupts" doesn't seem to have any advantage; in fact it seems to make things worse in terms of utilization. I'm not entirely sure why as of yet. Barney
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?477001.91824.qm>