Date: Mon, 13 May 2013 18:08:50 -0700 (PDT) From: Barney Cordoba <barney_cordoba@yahoo.com> To: Hooman Fazaeli <hoomanfazaeli@gmail.com>, Adrian Chadd <adrian@freebsd.org> Cc: freebsd-net@freebsd.org, =?iso-8859-1?Q?Cl=E9ment_Hermann_=28nodens=29?= <nodens2099@gmail.com>, Eugene Grosbein <egrosbein@rdtc.ru> Subject: Re: High CPU interrupt load on intel I350T4 with igb on 8.3 Message-ID: <1368493730.55723.YahooMailClassic@web121601.mail.ne1.yahoo.com> In-Reply-To: <CAJ-Vmom47bzkFZKhb%2BT8Xb88jo1YxD=jh4EK5OgPDd9xn_x2gg@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
You have to admit there's a problem before you can fix it. If Eugene is=20 going to blame to bottleneck and no one is going to tell him he's wrong, then there is no discussion. The solution in this case is to use 1 queue, which was my suggestion many days ago.=20 The defaults are broken. The driver should default to 1 queue, and be tuned to the system environment. With 2 NICs in the box, the defaults are defective.=20 1 queue should always work. Other settings require tuning and an understanding of how things work.=20 I've had to support i350 so I've been playing with the driver a bit. It=20 works fine with lots of cores. But you have to have more cores than queues. 2 cards with 4 queues on a 6 physical core system gets into a contention problem at certain loads. I've also removed the cpu bindings, which is about all I'm free to disclose= . The driver needs a tuning doc as much as anything else. BC --- On Sat, 5/11/13, Adrian Chadd <adrian@freebsd.org> wrote: > From: Adrian Chadd <adrian@freebsd.org> > Subject: Re: High CPU interrupt load on intel I350T4 with igb on 8.3 > To: "Hooman Fazaeli" <hoomanfazaeli@gmail.com> > Cc: "Barney Cordoba" <barney_cordoba@yahoo.com>, ""Cl=E9ment Hermann (nod= ens)"" <nodens2099@gmail.com>, "Eugene Grosbein" <egrosbein@rdtc.ru>, freeb= sd-net@freebsd.org > Date: Saturday, May 11, 2013, 6:16 PM > Hi, >=20 > The motivation behind the locking scheme in igb in friends > is for a > very specific, userland-traffic-origin workload. >=20 > Sure, it may or may not work well for forwarding/filtering > workloads. >=20 > If you want to fix it, let's have a discussion about how to > do it, > followed by some patches to do so. >=20 >=20 >=20 >=20 > Adrian >=20 > On 11 May 2013 13:12, Hooman Fazaeli <hoomanfazaeli@gmail.com> > wrote: > > On 5/11/2013 8:26 PM, Barney Cordoba wrote: > >> Clearly you don't understand the problem. Your > logic is that because other drivers are defective also; > therefore its not a driver problem? The problem is caused by > a multi-threaded driver that > >> haphazardly launches tasks and that doesn't manage > the case that the rest of the system can't handle the load. > It's no different than a driver that barfs when mbuf > clusters are exhausted. The answer > >> isn't to increase memory or mbufs, even though that > may alleviate the problem. The answer is to fix the driver, > so that it doesn't crash the system for an event that is > wholly predictable. igb has > >> 1) too many locks and 2) exasperates the problem by > binding to cpus, which causes it to not only have to wait > for the lock to free, but also for a specific cpu to become > free. So it chugs along > >> happily until it encounters a bottleneck, at which > point it quickly blows up the entire system in a domino > effect. It needs to manage locks more efficiently, and also > to detect when the backup is > >> unmanageable. Ever since FreeBSD 5 the answer has > been "it's fixed in 7, or its fixed in 9, or it's fixed in > 10". There will always be bottlenecks, and no driver should > blow up the system no matter > >> what intermediate code may present a problem. Its > the driver's responsibility to behave and to drop packets if > necessary. BC > > > > And how the driver should behave? You suggest dropping > the packets. Even if we accept > > that dropping packets is a good strategy in all > configurations (which I doubt), the driver is > > definitely not the best place to implement it, since > that involves duplication of similar > > code between drivers. Somewhere like the Ethernet layer > is a much better choice to watch > > load of packets and drop them to prevent them to eat > all the cores. Furthermore, ignoring > > the fact that pf is not optimized for multi-processors > and blaming drivers for not adjusting > > themselves with the this pf's fault, is a bit unfair, I > believe. > > > > > > -- > > > > Best regards. > > Hooman Fazaeli > > > > _______________________________________________ > > freebsd-net@freebsd.org > mailing list > > http://lists.freebsd.org/mailman/listinfo/freebsd-net > > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" > _______________________________________________ > freebsd-net@freebsd.org > mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" >
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?1368493730.55723.YahooMailClassic>