Date: Fri, 19 Jun 2009 09:01:29 -0700 (PDT) From: Barney Cordoba <barney_cordoba@yahoo.com> To: freebsd-net@FreeBSD.org Subject: Re: kern/135222: [igb] low speed routing between two igb interfaces Message-ID: <904439.18775.qm@web63908.mail.re1.yahoo.com>
next in thread | raw e-mail | index | archive | help
--- On Fri, 6/19/09, Michael <freebsdusb@bindone.de> wrote: > From: Michael <freebsdusb@bindone.de> > Subject: Re: kern/135222: [igb] low speed routing between two igb interfaces > To: "Barney Cordoba" <barney_cordoba@yahoo.com> > Cc: freebsd-net@FreeBSD.org > Date: Friday, June 19, 2009, 11:38 AM > Barney Cordoba wrote: > > > > > > --- On Wed, 6/17/09, Michael <freebsdusb@bindone.de> > wrote: > > > >> From: Michael <freebsdusb@bindone.de> > >> Subject: Re: kern/135222: [igb] low speed routing > between two igb interfaces > >> To: freebsd-net@FreeBSD.org > >> Date: Wednesday, June 17, 2009, 9:40 PM > >> The following reply was made to PR > >> kern/135222; it has been noted by GNATS. > >> > >> From: Michael <freebsdusb@bindone.de> > >> To: Barney Cordoba <barney_cordoba@yahoo.com> > >> Cc: freebsd-gnats-submit@FreeBSD.org > >> Subject: Re: kern/135222: [igb] low speed routing > between > >> two igb interfaces > >> Date: Thu, 18 Jun 2009 03:32:15 +0200 > >> > >> Barney Cordoba wrote: > >> > > >> > > >> > --- On Wed, 6/17/09, Michael <freebsdusb@bindone.de> > >> wrote: > >> > > >> >> From: Michael <freebsdusb@bindone.de> > >> >> Subject: Re: kern/135222: [igb] low > speed routing > >> between two igb interfaces > >> >> To: "Barney Cordoba" <barney_cordoba@yahoo.com> > >> >> Cc: freebsd-net@FreeBSD.org > >> >> Date: Wednesday, June 17, 2009, > 5:28 PM > >> >> Barney Cordoba wrote: > >> >>> > >> >>> --- On Fri, 6/12/09, Michael > <freebsdusb@bindone.de> > >> >> wrote: > >> >>>> From: Michael <freebsdusb@bindone.de> > >> >>>> Subject: Re: kern/135222: > [igb] low speed > >> routing > >> >> between two igb interfaces > >> >>>> To: freebsd-net@FreeBSD.org > >> >>>> Date: Friday, June 12, > 2009, 5:50 AM > >> >>>> The following reply was > made to PR > >> >>>> kern/135222; it has been > noted by GNATS. > >> >>>> > >> >>>> From: Michael <freebsdusb@bindone.de> > >> >>>> To: Cc: freebsd-gnats-submit@FreeBSD.org > >> >>>> Subject: Re: kern/135222: > [igb] low speed > >> routing > >> >> between > >> >>>> two igb interfaces > >> >>>> Date: Fri, 12 Jun 2009 > 11:45:47 +0200 > >> >>>> > >> >>>> The > original poster > >> reported that the > >> >> suggested fix works > >> >>>> for him: > >> >>>> --- > >> >>>> Hello > Michael, > >> >>>> > >> >>>> Thank you. > It's > >> working. > >> >>>> > >> >>>> I consider > it necessary > >> to put this into the > >> >> release > >> >>>> errata. > >> >>>> > >> >>>> > >> >>>> Mishustin > Andrew wrote: > >> >>>> >> > Number: > >> > >> >>>> > 135222 > >> >>>> >> > >> Category: > >> >> kern > >> >>>> >> > >> Synopsis: > >> >> [igb] > >> >>>> low speed routing between > two igb > >> interfaces > >> >>>> >> > >> Confidential: no > >> >>>> >> > >> Severity: > >> >> serious > >> >>>> >> > >> Priority: > >> >> medium > >> >>>> >> > >> Responsible: > >> >> freebsd-bugs > >> >>>> >> > State: > >> > >> >> open > >> >>>> >> > Quarter: > >> > >> >>>> >> > >> Keywords: > >> >> > >> >>>> >> > Date-Required: > >> >>>> >> > Class: > >> > >> >> sw-bug > >> >>>> >> > >> >> > Submitter-Id: current-users > >> >>>> >> > >> Arrival-Date: Wed > >> >> Jun 03 > >> >>>> 18:30:01 UTC 2009 > >> >>>> >> > Closed-Date: > >> >>>> >> > Last-Modified: > >> >>>> >> > Originator: > >> >> Mishustin > >> >>>> Andrew > >> >>>> >> > Release: > >> > >> >> FreeBSD > >> >>>> 7.1-RELEASE amd64, FreeBSD > 7.2-RELEASE > >> amd64 > >> >>>> >> > Organization: > >> >>>> > HNT > >> >>>> >> > Environment: > >> >>>> > > FreeBSD test.hnt > >> 7.2-RELEASE FreeBSD > >> >> 7.2-RELEASE #12: > >> >>>> Thu Apr 30 18:28:15 MSD 20 > >> >>>> > 09 > >> admin@test.hnt:/usr/src/sys/amd64/compile/GENERIC > >> >>>> amd64 > >> >>>> >> > Description: > >> >>>> > I > made a FreeBSD > >> multiprocesor server > >> >> to act as > >> >>>> simple gateway. > >> >>>> > It > use onboard > >> Intel 82575EB Dual-Port > >> >> Gigabit > >> >>>> Ethernet Controller. > >> >>>> > I > observe traffic > >> speed near 400 > >> >> Kbit/s. > >> >>>> > I > test both > >> interfaces separately - > >> >>>> > ftp > client work at > >> speed near 1 Gbit/s > >> >> in both > >> >>>> directions. > >> >>>> > Then > I change NIC > >> to old Intel "em" NIC > >> >> - gateway > >> >>>> work at speed near 1 > Gbit/s. > >> >>>> > > >> >>>> > Looks > like a bug in > >> igb driver have an > >> >> effect upon > >> >>>> forwarded traffic. > >> >>>> > > >> >>>> > If > you try > >> >>>> > > >> hw.igb.enable_aim=0 > >> >>>> > The > speed is near 1 > >> Mbit/s > >> >>>> > > >> >>>> > > hw.igb.rxd, > >> hw.igb.txd, "ifconfig -tso" > >> >> has no > >> >>>> effect. > >> >>>> > > >> >>>> > > Nothing in > >> messages.log > >> >>>> > > >> >>>> > > netstat -m > >> >>>> > > 516/1674/2190 mbufs > >> in use > >> >> (current/cache/total) > >> >>>> > > 515/927/1442/66560 > >> mbuf clusters in > >> >> use > >> >>>> (current/cache/total/max) > >> >>>> > > 515/893 > >> mbuf+clusters out of packet > >> >> secondary zone in > >> >>>> use (current/cache) > >> >>>> > > 0/44/44/33280 4k > >> (page size) jumbo > >> >> clusters in use > >> >>>> (current/cache/total/max) > >> >>>> > > 0/0/0/16640 9k > >> jumbo clusters in use > >> >>>> (current/cache/total/max) > >> >>>> > > 0/0/0/8320 16k > >> jumbo clusters in use > >> >>>> (current/cache/total/max) > >> >>>> > > 1159K/2448K/3607K > >> bytes allocated to > >> >> network > >> >>>> (current/cache/total) > >> >>>> > 0/0/0 > requests for > >> mbufs denied > >> >>>> > (mbufs/clusters/mbuf+clusters) > >> >>>> > 0/0/0 > requests for > >> jumbo clusters > >> >> denied (4k/9k/16k) > >> >>>> > 0/0/0 > sfbufs in use > >> (current/peak/max) > >> >>>> > 0 > requests for > >> sfbufs denied > >> >>>> > 0 > requests for > >> sfbufs delayed > >> >>>> > 0 > requests for I/O > >> initiated by > >> >> sendfile > >> >>>> > 0 > calls to protocol > >> drain routines > >> >>>> > > >> >>>> > I use > only IPv4 > >> traffic. > >> >>>> > > >> >>>> >> > How-To-Repeat: > >> >>>> > On > machine with two > >> igb interfaces > >> >>>> > use > rc.conf like > >> this: > >> >>>> > > >> >>>> > > >> hostname="test.test" > >> >>>> > > >> gateway_enable="YES" > >> >>>> > > ifconfig_igb0="inet > >> 10.10.10.1/24" > >> >>>> > > ifconfig_igb1="inet > >> 10.10.11.1/24" > >> >>>> > > >> >>>> > And > try create > >> heavy traffic between > >> >> two networks. > >> >>>> >> > Fix: > >> >>>> > > >> >>>> > > >> >>>> >> > Release-Note: > >> >>>> >> > Audit-Trail: > >> >>>> >> > Unformatted: > >> >>>> > > >> >> > _______________________________________________ > >> >>>> > freebsd-bugs@freebsd.org > >> >>> > >> >>> This is not a bug. Unless you > consider poorly > >> written > >> >> drivers to be bugs. You need to > provide your > >> tuning > >> >> parameters for the card as well > otherwise there's > >> nothing to > >> >> learn. > >> >>> The issue is that the driver > doesn't address > >> the > >> >> purpose of the controller; which is > to utilize > >> >> multiprocessor systems more > effectively. The > >> effect is that > >> >> lock contention actually makes > things worse than > >> if you just > >> >> use a single task as em does. Until > the > >> multiqueue drivers > >> >> are re-written to manage locks > properly you are > >> best advised > >> >> to save your money and stick with > em. > >> >>> You should get similar > performance using 1 > >> queue as > >> >> with em. You could also force > legacy > >> configuration by > >> >> forcing igb_setup_msix to return 0. > Sadly, this > >> is the best > >> >> performance you will get from the > stock driver. > >> >>> Barney > >> >>> > >> >>> Barney > >> >>> > >> >>> > >> >>> > >> >> I tried using 1 queue and it didn't > make things > >> any better > >> >> (actually I'm > >> >> not sure if that worked at all). If > it is > >> considered a bug > >> >> or not > >> >> doesn't really matter, what > actually matters for > >> users (who > >> >> cannot > >> >> always chose which network > controller will be > >> on-board) is > >> >> that they get > >> >> a least decent performance when > doing IP > >> forwarding (and > >> >> not the > >> >> 5-50kb/s I've seen). You can get > this out of the > >> >> controller, when > >> >> disabling lro through the sysctl. > That's why I've > >> been > >> >> asking to put > >> >> this into the release errata > section and/or at > >> least the > >> >> igb man page, > >> >> because the sysctl isn't documented > anywhere. > >> Also the > >> >> fact, that tuning > >> >> the sysctl only affects the > behaviour when it's > >> set on boot > >> >> might be > >> >> considered problematic. > >> >> > >> >> So at the very least, I think the > following > >> should be > >> >> done: > >> >> 1. Document the sysctl in man > igb(4) > >> >> 2. Put a known issues paragraph to > man igb(4) > >> which > >> >> explains the issue > >> >> and what to put in sysctl.conf to > stop this from > >> happening > >> >> 3. Add an entry to the release > errata page about > >> this issue > >> >> (like I > >> >> suggested in one of my earlier > emails) and > >> stating > >> >> something like "see > >> >> man igb(4) for details) > >> >> > >> >> This is not about using the > controller to its > >> full > >> >> potential, but to > >> >> safe Joe Admin from spending days > on figuring out > >> why the > >> >> machine is > >> >> forwarding packages slower than his > BSD 2.x > >> machine did in > >> >> the 90s. > >> >> > >> >> cheers > >> >> Michael > >> > > >> > None of the offload crap should be > enabled by > >> default. > >> > > >> > The real point is that "Joe Admin" > shouldn't be using > >> controllers that have bad drivers at all. If you > have to use > >> whatever hardware you have laying around, and > don't have > >> enough flexibility to lay out $100 for a 2 port > controller > >> that works to use with your $2000 server, than you > need to > >> get your priorities in order. People go out and > buy > >> redundant power supplies, high GHZ quad core > processors and > >> gobs of memory and then they use whatever crappy > onboard > >> controller they get no matter how poorly its suppo > rted. Its > >> mindless. > >> > > >> > Barney > >> > > >> > > >> > > >> > >> How should anybody know that the controller > is poorly > >> supported if there > >> is nothing in the documentation, release > notes, man pages > >> or anywhere > >> else about this? > >> > >> The fact of the matter is that "the offload > crap" _is_ > >> enabled by > >> default. The release is out, it claims to > support the > >> controller. There > >> _is_ a workaround and I'm asked if somebody > could document > >> this so users > >> will have a chance. I'm also not convinced > that it is a > >> crappy > >> controller per se, but just poorly > supported. We used > >> those a lot before > >> without any issues, unfortunately now we had > touse IP > >> forwarding in a > >> machine that has that controller (it has 6 > interfaces in > >> total, four em > >> ports and two igb ports, all of them are in > use and I > >> don't feel like > >> hooking up the sodering iron). > >> > >> So bottomline: > >> I said, there is a problem with the driver, > there is a > >> workaround and it > >> should be documented. > >> > >> You say, the driver is bad and nobody should > use it and if > >> they do it's > >> their own damn fault. We won't do anything > about it and > >> refuse to tell > >> anybody, because we are the only ones who > should know. We > >> don't care if > >> people can actually use our software and > still claim the > >> hardware is > >> actually supported. > >> > >> Your attitude is really contra productive > (actually > >> googling around I > >> see you made similar statements in the > past about > >> stupid people not > >> willing to spend xxx$ on whatever piece of > hardware, so > >> maybe you're > >> just trolling). > >> > >> Michael > > > > Tuning the card to be brain-dead isn't really a > workaround. I'm sorry that you're not able to understand, > but you can't educate the woodchucks, so carry on and feel > free to do whatever you wish. > > > > BC > > > > > > > > Without tuning the card: 5kb/s, with tuning the card: > 50mb/s > That's the definition of a workaround, the fix would be > making lro work > correctly - in general I prefer a brain-dead card to a > brain-dead > mailing list subscriber. Welcome to the real world :) > > Anyway, I'll stop feeding you now, this is getting boring > and leads nowhere. > > I still think that this should be noted somewhere in the > docs, whoever > has permissions to commit might proceed in doing so... > _______________________________________________ > freebsd-net@freebsd.org > mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" > Ignorance is Bliss, as they say. BC
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?904439.18775.qm>