Date: Fri, 19 Jun 2009 07:56:00 -0700 (PDT) From: Barney Cordoba <barney_cordoba@yahoo.com> To: freebsd-net@FreeBSD.org, Michael <freebsdusb@bindone.de> Subject: Re: kern/135222: [igb] low speed routing between two igb interfaces Message-ID: <436489.18496.qm@web63904.mail.re1.yahoo.com>
index | next in thread | raw e-mail
--- On Wed, 6/17/09, Michael <freebsdusb@bindone.de> wrote: > From: Michael <freebsdusb@bindone.de> > Subject: Re: kern/135222: [igb] low speed routing between two igb interfaces > To: freebsd-net@FreeBSD.org > Date: Wednesday, June 17, 2009, 9:40 PM > The following reply was made to PR > kern/135222; it has been noted by GNATS. > > From: Michael <freebsdusb@bindone.de> > To: Barney Cordoba <barney_cordoba@yahoo.com> > Cc: freebsd-gnats-submit@FreeBSD.org > Subject: Re: kern/135222: [igb] low speed routing between > two igb interfaces > Date: Thu, 18 Jun 2009 03:32:15 +0200 > > Barney Cordoba wrote: > > > > > > --- On Wed, 6/17/09, Michael <freebsdusb@bindone.de> > wrote: > > > >> From: Michael <freebsdusb@bindone.de> > >> Subject: Re: kern/135222: [igb] low speed routing > between two igb interfaces > >> To: "Barney Cordoba" <barney_cordoba@yahoo.com> > >> Cc: freebsd-net@FreeBSD.org > >> Date: Wednesday, June 17, 2009, 5:28 PM > >> Barney Cordoba wrote: > >>> > >>> --- On Fri, 6/12/09, Michael <freebsdusb@bindone.de> > >> wrote: > >>>> From: Michael <freebsdusb@bindone.de> > >>>> Subject: Re: kern/135222: [igb] low speed > routing > >> between two igb interfaces > >>>> To: freebsd-net@FreeBSD.org > >>>> Date: Friday, June 12, 2009, 5:50 AM > >>>> The following reply was made to PR > >>>> kern/135222; it has been noted by GNATS. > >>>> > >>>> From: Michael <freebsdusb@bindone.de> > >>>> To: Cc: freebsd-gnats-submit@FreeBSD.org > >>>> Subject: Re: kern/135222: [igb] low speed > routing > >> between > >>>> two igb interfaces > >>>> Date: Fri, 12 Jun 2009 11:45:47 +0200 > >>>> > >>>> The original poster > reported that the > >> suggested fix works > >>>> for him: > >>>> --- > >>>> Hello Michael, > >>>> > >>>> Thank you. It's > working. > >>>> > >>>> I consider it necessary > to put this into the > >> release > >>>> errata. > >>>> > >>>> > >>>> Mishustin Andrew wrote: > >>>> >> Number: > > >>>> 135222 > >>>> >> > Category: > >> kern > >>>> >> > Synopsis: > >> [igb] > >>>> low speed routing between two igb > interfaces > >>>> >> > Confidential: no > >>>> >> > Severity: > >> serious > >>>> >> > Priority: > >> medium > >>>> >> > Responsible: > >> freebsd-bugs > >>>> >> State: > > >> open > >>>> >> Quarter: > > >>>> >> > Keywords: > >> > >>>> >> Date-Required: > >>>> >> Class: > > >> sw-bug > >>>> >> > >> Submitter-Id: current-users > >>>> >> > Arrival-Date: Wed > >> Jun 03 > >>>> 18:30:01 UTC 2009 > >>>> >> Closed-Date: > >>>> >> Last-Modified: > >>>> >> Originator: > >> Mishustin > >>>> Andrew > >>>> >> Release: > > >> FreeBSD > >>>> 7.1-RELEASE amd64, FreeBSD 7.2-RELEASE > amd64 > >>>> >> Organization: > >>>> > HNT > >>>> >> Environment: > >>>> > FreeBSD test.hnt > 7.2-RELEASE FreeBSD > >> 7.2-RELEASE #12: > >>>> Thu Apr 30 18:28:15 MSD 20 > >>>> > 09 > admin@test.hnt:/usr/src/sys/amd64/compile/GENERIC > >>>> amd64 > >>>> >> Description: > >>>> > I made a FreeBSD > multiprocesor server > >> to act as > >>>> simple gateway. > >>>> > It use onboard > Intel 82575EB Dual-Port > >> Gigabit > >>>> Ethernet Controller. > >>>> > I observe traffic > speed near 400 > >> Kbit/s. > >>>> > I test both > interfaces separately - > >>>> > ftp client work at > speed near 1 Gbit/s > >> in both > >>>> directions. > >>>> > Then I change NIC > to old Intel "em" NIC > >> - gateway > >>>> work at speed near 1 Gbit/s. > >>>> > > >>>> > Looks like a bug in > igb driver have an > >> effect upon > >>>> forwarded traffic. > >>>> > > >>>> > If you try > >>>> > > hw.igb.enable_aim=0 > >>>> > The speed is near 1 > Mbit/s > >>>> > > >>>> > hw.igb.rxd, > hw.igb.txd, "ifconfig -tso" > >> has no > >>>> effect. > >>>> > > >>>> > Nothing in > messages.log > >>>> > > >>>> > netstat -m > >>>> > 516/1674/2190 mbufs > in use > >> (current/cache/total) > >>>> > 515/927/1442/66560 > mbuf clusters in > >> use > >>>> (current/cache/total/max) > >>>> > 515/893 > mbuf+clusters out of packet > >> secondary zone in > >>>> use (current/cache) > >>>> > 0/44/44/33280 4k > (page size) jumbo > >> clusters in use > >>>> (current/cache/total/max) > >>>> > 0/0/0/16640 9k > jumbo clusters in use > >>>> (current/cache/total/max) > >>>> > 0/0/0/8320 16k > jumbo clusters in use > >>>> (current/cache/total/max) > >>>> > 1159K/2448K/3607K > bytes allocated to > >> network > >>>> (current/cache/total) > >>>> > 0/0/0 requests for > mbufs denied > >>>> (mbufs/clusters/mbuf+clusters) > >>>> > 0/0/0 requests for > jumbo clusters > >> denied (4k/9k/16k) > >>>> > 0/0/0 sfbufs in use > (current/peak/max) > >>>> > 0 requests for > sfbufs denied > >>>> > 0 requests for > sfbufs delayed > >>>> > 0 requests for I/O > initiated by > >> sendfile > >>>> > 0 calls to protocol > drain routines > >>>> > > >>>> > I use only IPv4 > traffic. > >>>> > > >>>> >> How-To-Repeat: > >>>> > On machine with two > igb interfaces > >>>> > use rc.conf like > this: > >>>> > > >>>> > > hostname="test.test" > >>>> > > gateway_enable="YES" > >>>> > ifconfig_igb0="inet > 10.10.10.1/24" > >>>> > ifconfig_igb1="inet > 10.10.11.1/24" > >>>> > > >>>> > And try create > heavy traffic between > >> two networks. > >>>> >> Fix: > >>>> > > >>>> > > >>>> >> Release-Note: > >>>> >> Audit-Trail: > >>>> >> Unformatted: > >>>> > > >> _______________________________________________ > >>>> > freebsd-bugs@freebsd.org > >>> > >>> This is not a bug. Unless you consider poorly > written > >> drivers to be bugs. You need to provide your > tuning > >> parameters for the card as well otherwise there's > nothing to > >> learn. > >>> The issue is that the driver doesn't address > the > >> purpose of the controller; which is to utilize > >> multiprocessor systems more effectively. The > effect is that > >> lock contention actually makes things worse than > if you just > >> use a single task as em does. Until the > multiqueue drivers > >> are re-written to manage locks properly you are > best advised > >> to save your money and stick with em. > >>> You should get similar performance using 1 > queue as > >> with em. You could also force legacy > configuration by > >> forcing igb_setup_msix to return 0. Sadly, this > is the best > >> performance you will get from the stock driver. > >>> Barney > >>> > >>> Barney > >>> > >>> > >>> > >> I tried using 1 queue and it didn't make things > any better > >> (actually I'm > >> not sure if that worked at all). If it is > considered a bug > >> or not > >> doesn't really matter, what actually matters for > users (who > >> cannot > >> always chose which network controller will be > on-board) is > >> that they get > >> a least decent performance when doing IP > forwarding (and > >> not the > >> 5-50kb/s I've seen). You can get this out of the > >> controller, when > >> disabling lro through the sysctl. That's why I've > been > >> asking to put > >> this into the release errata section and/or at > least the > >> igb man page, > >> because the sysctl isn't documented anywhere. > Also the > >> fact, that tuning > >> the sysctl only affects the behaviour when it's > set on boot > >> might be > >> considered problematic. > >> > >> So at the very least, I think the following > should be > >> done: > >> 1. Document the sysctl in man igb(4) > >> 2. Put a known issues paragraph to man igb(4) > which > >> explains the issue > >> and what to put in sysctl.conf to stop this from > happening > >> 3. Add an entry to the release errata page about > this issue > >> (like I > >> suggested in one of my earlier emails) and > stating > >> something like "see > >> man igb(4) for details) > >> > >> This is not about using the controller to its > full > >> potential, but to > >> safe Joe Admin from spending days on figuring out > why the > >> machine is > >> forwarding packages slower than his BSD 2.x > machine did in > >> the 90s. > >> > >> cheers > >> Michael > > > > None of the offload crap should be enabled by > default. > > > > The real point is that "Joe Admin" shouldn't be using > controllers that have bad drivers at all. If you have to use > whatever hardware you have laying around, and don't have > enough flexibility to lay out $100 for a 2 port controller > that works to use with your $2000 server, than you need to > get your priorities in order. People go out and buy > redundant power supplies, high GHZ quad core processors and > gobs of memory and then they use whatever crappy onboard > controller they get no matter how poorly its suppo rted. Its > mindless. > > > > Barney > > > > > > > > How should anybody know that the controller is poorly > supported if there > is nothing in the documentation, release notes, man pages > or anywhere > else about this? > > The fact of the matter is that "the offload crap" _is_ > enabled by > default. The release is out, it claims to support the > controller. There > _is_ a workaround and I'm asked if somebody could document > this so users > will have a chance. I'm also not convinced that it is a > crappy > controller per se, but just poorly supported. We used > those a lot before > without any issues, unfortunately now we had touse IP > forwarding in a > machine that has that controller (it has 6 interfaces in > total, four em > ports and two igb ports, all of them are in use and I > don't feel like > hooking up the sodering iron). > > So bottomline: > I said, there is a problem with the driver, there is a > workaround and it > should be documented. > > You say, the driver is bad and nobody should use it and if > they do it's > their own damn fault. We won't do anything about it and > refuse to tell > anybody, because we are the only ones who should know. We > don't care if > people can actually use our software and still claim the > hardware is > actually supported. > > Your attitude is really contra productive (actually > googling around I > see you made similar statements in the past about > stupid people not > willing to spend xxx$ on whatever piece of hardware, so > maybe you're > just trolling). > > Michael Tuning the card to be brain-dead isn't really a workaround. I'm sorry that you're not able to understand, but you can't educate the woodchucks, so carry on and feel free to do whatever you wish. BChome | help
Want to link to this message? Use this
URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?436489.18496.qm>