Date: Fri, 19 Jun 2009 17:38:27 +0200 From: Michael <freebsdusb@bindone.de> To: Barney Cordoba <barney_cordoba@yahoo.com> Cc: freebsd-net@FreeBSD.org Subject: Re: kern/135222: [igb] low speed routing between two igb interfaces Message-ID: <4A3BB0F3.80005@bindone.de> In-Reply-To: <436489.18496.qm@web63904.mail.re1.yahoo.com> References: <436489.18496.qm@web63904.mail.re1.yahoo.com>
next in thread | previous in thread | raw e-mail | index | archive | help
Barney Cordoba wrote: > > > --- On Wed, 6/17/09, Michael <freebsdusb@bindone.de> wrote: > >> From: Michael <freebsdusb@bindone.de> >> Subject: Re: kern/135222: [igb] low speed routing between two igb interfaces >> To: freebsd-net@FreeBSD.org >> Date: Wednesday, June 17, 2009, 9:40 PM >> The following reply was made to PR >> kern/135222; it has been noted by GNATS. >> >> From: Michael <freebsdusb@bindone.de> >> To: Barney Cordoba <barney_cordoba@yahoo.com> >> Cc: freebsd-gnats-submit@FreeBSD.org >> Subject: Re: kern/135222: [igb] low speed routing between >> two igb interfaces >> Date: Thu, 18 Jun 2009 03:32:15 +0200 >> >> Barney Cordoba wrote: >> > >> > >> > --- On Wed, 6/17/09, Michael <freebsdusb@bindone.de> >> wrote: >> > >> >> From: Michael <freebsdusb@bindone.de> >> >> Subject: Re: kern/135222: [igb] low speed routing >> between two igb interfaces >> >> To: "Barney Cordoba" <barney_cordoba@yahoo.com> >> >> Cc: freebsd-net@FreeBSD.org >> >> Date: Wednesday, June 17, 2009, 5:28 PM >> >> Barney Cordoba wrote: >> >>> >> >>> --- On Fri, 6/12/09, Michael <freebsdusb@bindone.de> >> >> wrote: >> >>>> From: Michael <freebsdusb@bindone.de> >> >>>> Subject: Re: kern/135222: [igb] low speed >> routing >> >> between two igb interfaces >> >>>> To: freebsd-net@FreeBSD.org >> >>>> Date: Friday, June 12, 2009, 5:50 AM >> >>>> The following reply was made to PR >> >>>> kern/135222; it has been noted by GNATS. >> >>>> >> >>>> From: Michael <freebsdusb@bindone.de> >> >>>> To: Cc: freebsd-gnats-submit@FreeBSD.org >> >>>> Subject: Re: kern/135222: [igb] low speed >> routing >> >> between >> >>>> two igb interfaces >> >>>> Date: Fri, 12 Jun 2009 11:45:47 +0200 >> >>>> >> >>>> The original poster >> reported that the >> >> suggested fix works >> >>>> for him: >> >>>> --- >> >>>> Hello Michael, >> >>>> >> >>>> Thank you. It's >> working. >> >>>> >> >>>> I consider it necessary >> to put this into the >> >> release >> >>>> errata. >> >>>> >> >>>> >> >>>> Mishustin Andrew wrote: >> >>>> >> Number: >> >> >>>> 135222 >> >>>> >> >> Category: >> >> kern >> >>>> >> >> Synopsis: >> >> [igb] >> >>>> low speed routing between two igb >> interfaces >> >>>> >> >> Confidential: no >> >>>> >> >> Severity: >> >> serious >> >>>> >> >> Priority: >> >> medium >> >>>> >> >> Responsible: >> >> freebsd-bugs >> >>>> >> State: >> >> >> open >> >>>> >> Quarter: >> >> >>>> >> >> Keywords: >> >> >> >>>> >> Date-Required: >> >>>> >> Class: >> >> >> sw-bug >> >>>> >> >> >> Submitter-Id: current-users >> >>>> >> >> Arrival-Date: Wed >> >> Jun 03 >> >>>> 18:30:01 UTC 2009 >> >>>> >> Closed-Date: >> >>>> >> Last-Modified: >> >>>> >> Originator: >> >> Mishustin >> >>>> Andrew >> >>>> >> Release: >> >> >> FreeBSD >> >>>> 7.1-RELEASE amd64, FreeBSD 7.2-RELEASE >> amd64 >> >>>> >> Organization: >> >>>> > HNT >> >>>> >> Environment: >> >>>> > FreeBSD test.hnt >> 7.2-RELEASE FreeBSD >> >> 7.2-RELEASE #12: >> >>>> Thu Apr 30 18:28:15 MSD 20 >> >>>> > 09 >> admin@test.hnt:/usr/src/sys/amd64/compile/GENERIC >> >>>> amd64 >> >>>> >> Description: >> >>>> > I made a FreeBSD >> multiprocesor server >> >> to act as >> >>>> simple gateway. >> >>>> > It use onboard >> Intel 82575EB Dual-Port >> >> Gigabit >> >>>> Ethernet Controller. >> >>>> > I observe traffic >> speed near 400 >> >> Kbit/s. >> >>>> > I test both >> interfaces separately - >> >>>> > ftp client work at >> speed near 1 Gbit/s >> >> in both >> >>>> directions. >> >>>> > Then I change NIC >> to old Intel "em" NIC >> >> - gateway >> >>>> work at speed near 1 Gbit/s. >> >>>> > >> >>>> > Looks like a bug in >> igb driver have an >> >> effect upon >> >>>> forwarded traffic. >> >>>> > >> >>>> > If you try >> >>>> > >> hw.igb.enable_aim=0 >> >>>> > The speed is near 1 >> Mbit/s >> >>>> > >> >>>> > hw.igb.rxd, >> hw.igb.txd, "ifconfig -tso" >> >> has no >> >>>> effect. >> >>>> > >> >>>> > Nothing in >> messages.log >> >>>> > >> >>>> > netstat -m >> >>>> > 516/1674/2190 mbufs >> in use >> >> (current/cache/total) >> >>>> > 515/927/1442/66560 >> mbuf clusters in >> >> use >> >>>> (current/cache/total/max) >> >>>> > 515/893 >> mbuf+clusters out of packet >> >> secondary zone in >> >>>> use (current/cache) >> >>>> > 0/44/44/33280 4k >> (page size) jumbo >> >> clusters in use >> >>>> (current/cache/total/max) >> >>>> > 0/0/0/16640 9k >> jumbo clusters in use >> >>>> (current/cache/total/max) >> >>>> > 0/0/0/8320 16k >> jumbo clusters in use >> >>>> (current/cache/total/max) >> >>>> > 1159K/2448K/3607K >> bytes allocated to >> >> network >> >>>> (current/cache/total) >> >>>> > 0/0/0 requests for >> mbufs denied >> >>>> (mbufs/clusters/mbuf+clusters) >> >>>> > 0/0/0 requests for >> jumbo clusters >> >> denied (4k/9k/16k) >> >>>> > 0/0/0 sfbufs in use >> (current/peak/max) >> >>>> > 0 requests for >> sfbufs denied >> >>>> > 0 requests for >> sfbufs delayed >> >>>> > 0 requests for I/O >> initiated by >> >> sendfile >> >>>> > 0 calls to protocol >> drain routines >> >>>> > >> >>>> > I use only IPv4 >> traffic. >> >>>> > >> >>>> >> How-To-Repeat: >> >>>> > On machine with two >> igb interfaces >> >>>> > use rc.conf like >> this: >> >>>> > >> >>>> > >> hostname="test.test" >> >>>> > >> gateway_enable="YES" >> >>>> > ifconfig_igb0="inet >> 10.10.10.1/24" >> >>>> > ifconfig_igb1="inet >> 10.10.11.1/24" >> >>>> > >> >>>> > And try create >> heavy traffic between >> >> two networks. >> >>>> >> Fix: >> >>>> > >> >>>> > >> >>>> >> Release-Note: >> >>>> >> Audit-Trail: >> >>>> >> Unformatted: >> >>>> > >> >> _______________________________________________ >> >>>> > freebsd-bugs@freebsd.org >> >>> >> >>> This is not a bug. Unless you consider poorly >> written >> >> drivers to be bugs. You need to provide your >> tuning >> >> parameters for the card as well otherwise there's >> nothing to >> >> learn. >> >>> The issue is that the driver doesn't address >> the >> >> purpose of the controller; which is to utilize >> >> multiprocessor systems more effectively. The >> effect is that >> >> lock contention actually makes things worse than >> if you just >> >> use a single task as em does. Until the >> multiqueue drivers >> >> are re-written to manage locks properly you are >> best advised >> >> to save your money and stick with em. >> >>> You should get similar performance using 1 >> queue as >> >> with em. You could also force legacy >> configuration by >> >> forcing igb_setup_msix to return 0. Sadly, this >> is the best >> >> performance you will get from the stock driver. >> >>> Barney >> >>> >> >>> Barney >> >>> >> >>> >> >>> >> >> I tried using 1 queue and it didn't make things >> any better >> >> (actually I'm >> >> not sure if that worked at all). If it is >> considered a bug >> >> or not >> >> doesn't really matter, what actually matters for >> users (who >> >> cannot >> >> always chose which network controller will be >> on-board) is >> >> that they get >> >> a least decent performance when doing IP >> forwarding (and >> >> not the >> >> 5-50kb/s I've seen). You can get this out of the >> >> controller, when >> >> disabling lro through the sysctl. That's why I've >> been >> >> asking to put >> >> this into the release errata section and/or at >> least the >> >> igb man page, >> >> because the sysctl isn't documented anywhere. >> Also the >> >> fact, that tuning >> >> the sysctl only affects the behaviour when it's >> set on boot >> >> might be >> >> considered problematic. >> >> >> >> So at the very least, I think the following >> should be >> >> done: >> >> 1. Document the sysctl in man igb(4) >> >> 2. Put a known issues paragraph to man igb(4) >> which >> >> explains the issue >> >> and what to put in sysctl.conf to stop this from >> happening >> >> 3. Add an entry to the release errata page about >> this issue >> >> (like I >> >> suggested in one of my earlier emails) and >> stating >> >> something like "see >> >> man igb(4) for details) >> >> >> >> This is not about using the controller to its >> full >> >> potential, but to >> >> safe Joe Admin from spending days on figuring out >> why the >> >> machine is >> >> forwarding packages slower than his BSD 2.x >> machine did in >> >> the 90s. >> >> >> >> cheers >> >> Michael >> > >> > None of the offload crap should be enabled by >> default. >> > >> > The real point is that "Joe Admin" shouldn't be using >> controllers that have bad drivers at all. If you have to use >> whatever hardware you have laying around, and don't have >> enough flexibility to lay out $100 for a 2 port controller >> that works to use with your $2000 server, than you need to >> get your priorities in order. People go out and buy >> redundant power supplies, high GHZ quad core processors and >> gobs of memory and then they use whatever crappy onboard >> controller they get no matter how poorly its suppo rted. Its >> mindless. >> > >> > Barney >> > >> > >> > >> >> How should anybody know that the controller is poorly >> supported if there >> is nothing in the documentation, release notes, man pages >> or anywhere >> else about this? >> >> The fact of the matter is that "the offload crap" _is_ >> enabled by >> default. The release is out, it claims to support the >> controller. There >> _is_ a workaround and I'm asked if somebody could document >> this so users >> will have a chance. I'm also not convinced that it is a >> crappy >> controller per se, but just poorly supported. We used >> those a lot before >> without any issues, unfortunately now we had touse IP >> forwarding in a >> machine that has that controller (it has 6 interfaces in >> total, four em >> ports and two igb ports, all of them are in use and I >> don't feel like >> hooking up the sodering iron). >> >> So bottomline: >> I said, there is a problem with the driver, there is a >> workaround and it >> should be documented. >> >> You say, the driver is bad and nobody should use it and if >> they do it's >> their own damn fault. We won't do anything about it and >> refuse to tell >> anybody, because we are the only ones who should know. We >> don't care if >> people can actually use our software and still claim the >> hardware is >> actually supported. >> >> Your attitude is really contra productive (actually >> googling around I >> see you made similar statements in the past about >> stupid people not >> willing to spend xxx$ on whatever piece of hardware, so >> maybe you're >> just trolling). >> >> Michael > > Tuning the card to be brain-dead isn't really a workaround. I'm sorry that you're not able to understand, but you can't educate the woodchucks, so carry on and feel free to do whatever you wish. > > BC > > > Without tuning the card: 5kb/s, with tuning the card: 50mb/s That's the definition of a workaround, the fix would be making lro work correctly - in general I prefer a brain-dead card to a brain-dead mailing list subscriber. Welcome to the real world :) Anyway, I'll stop feeding you now, this is getting boring and leads nowhere. I still think that this should be noted somewhere in the docs, whoever has permissions to commit might proceed in doing so...
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?4A3BB0F3.80005>