Skip site navigation (1)Skip section navigation (2)
Date:      Fri, 19 Jun 2009 17:38:27 +0200
From:      Michael <freebsdusb@bindone.de>
To:        Barney Cordoba <barney_cordoba@yahoo.com>
Cc:        freebsd-net@FreeBSD.org
Subject:   Re: kern/135222: [igb] low speed routing between two igb interfaces
Message-ID:  <4A3BB0F3.80005@bindone.de>
In-Reply-To: <436489.18496.qm@web63904.mail.re1.yahoo.com>
References:  <436489.18496.qm@web63904.mail.re1.yahoo.com>

next in thread | previous in thread | raw e-mail | index | archive | help
Barney Cordoba wrote:
> 
> 
> --- On Wed, 6/17/09, Michael <freebsdusb@bindone.de> wrote:
> 
>> From: Michael <freebsdusb@bindone.de>
>> Subject: Re: kern/135222: [igb] low speed routing between two igb interfaces
>> To: freebsd-net@FreeBSD.org
>> Date: Wednesday, June 17, 2009, 9:40 PM
>> The following reply was made to PR
>> kern/135222; it has been noted by GNATS.
>>
>> From: Michael <freebsdusb@bindone.de>
>> To: Barney Cordoba <barney_cordoba@yahoo.com>
>> Cc: freebsd-gnats-submit@FreeBSD.org
>> Subject: Re: kern/135222: [igb] low speed routing between
>> two igb interfaces
>> Date: Thu, 18 Jun 2009 03:32:15 +0200
>>
>>  Barney Cordoba wrote:
>>  > 
>>  > 
>>  > --- On Wed, 6/17/09, Michael <freebsdusb@bindone.de>
>> wrote:
>>  > 
>>  >> From: Michael <freebsdusb@bindone.de>
>>  >> Subject: Re: kern/135222: [igb] low speed routing
>> between two igb interfaces
>>  >> To: "Barney Cordoba" <barney_cordoba@yahoo.com>
>>  >> Cc: freebsd-net@FreeBSD.org
>>  >> Date: Wednesday, June 17, 2009, 5:28 PM
>>  >> Barney Cordoba wrote:
>>  >>>
>>  >>> --- On Fri, 6/12/09, Michael <freebsdusb@bindone.de>
>>  >> wrote:
>>  >>>> From: Michael <freebsdusb@bindone.de>
>>  >>>> Subject: Re: kern/135222: [igb] low speed
>> routing
>>  >> between two igb interfaces
>>  >>>> To: freebsd-net@FreeBSD.org
>>  >>>> Date: Friday, June 12, 2009, 5:50 AM
>>  >>>> The following reply was made to PR
>>  >>>> kern/135222; it has been noted by GNATS.
>>  >>>>
>>  >>>> From: Michael <freebsdusb@bindone.de>
>>  >>>> To: Cc: freebsd-gnats-submit@FreeBSD.org
>>  >>>> Subject: Re: kern/135222: [igb] low speed
>> routing
>>  >> between
>>  >>>> two igb interfaces
>>  >>>> Date: Fri, 12 Jun 2009 11:45:47 +0200
>>  >>>>
>>  >>>>   The original poster
>> reported that the
>>  >> suggested fix works
>>  >>>> for him:
>>  >>>>   ---
>>  >>>>   Hello Michael,
>>  >>>>   
>>  >>>>   Thank you. It's
>> working.
>>  >>>>   
>>  >>>>   I consider it necessary
>> to put this into the
>>  >> release
>>  >>>> errata.
>>  >>>>   
>>  >>>>   
>>  >>>>   Mishustin Andrew wrote:
>>  >>>>   >> Number: 
>>    
>>  >>>>     135222
>>  >>>>   >>
>> Category:   
>>  >>    kern
>>  >>>>   >>
>> Synopsis:   
>>  >>    [igb]
>>  >>>> low speed routing between two igb
>> interfaces
>>  >>>>   >>
>> Confidential:   no
>>  >>>>   >>
>> Severity:   
>>  >>    serious
>>  >>>>   >>
>> Priority:   
>>  >>    medium
>>  >>>>   >>
>> Responsible:   
>>  >> freebsd-bugs
>>  >>>>   >> State: 
>>      
>>  >>   open
>>  >>>>   >> Quarter: 
>>      
>>  >>>>   >>
>> Keywords:   
>>  >>    
>>  >>>>   >> Date-Required:
>>  >>>>   >> Class: 
>>      
>>  >>   sw-bug
>>  >>>>   >>
>>  >> Submitter-Id:   current-users
>>  >>>>   >>
>> Arrival-Date:   Wed
>>  >> Jun 03
>>  >>>> 18:30:01 UTC 2009
>>  >>>>   >> Closed-Date:
>>  >>>>   >> Last-Modified:
>>  >>>>   >> Originator: 
>>  >>    Mishustin
>>  >>>> Andrew
>>  >>>>   >> Release: 
>>      
>>  >> FreeBSD
>>  >>>> 7.1-RELEASE amd64, FreeBSD 7.2-RELEASE
>> amd64
>>  >>>>   >> Organization:
>>  >>>>   > HNT
>>  >>>>   >> Environment:
>>  >>>>   > FreeBSD test.hnt
>> 7.2-RELEASE FreeBSD
>>  >> 7.2-RELEASE #12:
>>  >>>> Thu Apr 30 18:28:15 MSD 20
>>  >>>>   > 09 
>>    admin@test.hnt:/usr/src/sys/amd64/compile/GENERIC
>>  >>>> amd64
>>  >>>>   >> Description:
>>  >>>>   > I made a FreeBSD
>> multiprocesor server
>>  >> to act as
>>  >>>> simple gateway.
>>  >>>>   > It use onboard
>> Intel 82575EB Dual-Port
>>  >> Gigabit
>>  >>>> Ethernet Controller.
>>  >>>>   > I observe traffic
>> speed near 400
>>  >> Kbit/s.
>>  >>>>   > I test both
>> interfaces separately -
>>  >>>>   > ftp client work at
>> speed near 1 Gbit/s
>>  >> in both
>>  >>>> directions.
>>  >>>>   > Then I change NIC
>> to old Intel "em" NIC
>>  >> - gateway
>>  >>>> work at speed near 1 Gbit/s.
>>  >>>>   > 
>>  >>>>   > Looks like a bug in
>> igb driver have an
>>  >> effect upon
>>  >>>> forwarded traffic.
>>  >>>>   > 
>>  >>>>   > If you try
>>  >>>>   >
>> hw.igb.enable_aim=0
>>  >>>>   > The speed is near 1
>> Mbit/s
>>  >>>>   > 
>>  >>>>   > hw.igb.rxd,
>> hw.igb.txd, "ifconfig -tso"
>>  >> has no
>>  >>>> effect.
>>  >>>>   > 
>>  >>>>   > Nothing in
>> messages.log
>>  >>>>   > 
>>  >>>>   > netstat -m
>>  >>>>   > 516/1674/2190 mbufs
>> in use
>>  >> (current/cache/total)
>>  >>>>   > 515/927/1442/66560
>> mbuf clusters in
>>  >> use
>>  >>>> (current/cache/total/max)
>>  >>>>   > 515/893
>> mbuf+clusters out of packet
>>  >> secondary zone in
>>  >>>> use (current/cache)
>>  >>>>   > 0/44/44/33280 4k
>> (page size) jumbo
>>  >> clusters in use
>>  >>>> (current/cache/total/max)
>>  >>>>   > 0/0/0/16640 9k
>> jumbo clusters in use
>>  >>>> (current/cache/total/max)
>>  >>>>   > 0/0/0/8320 16k
>> jumbo clusters in use
>>  >>>> (current/cache/total/max)
>>  >>>>   > 1159K/2448K/3607K
>> bytes allocated to
>>  >> network
>>  >>>> (current/cache/total)
>>  >>>>   > 0/0/0 requests for
>> mbufs denied
>>  >>>> (mbufs/clusters/mbuf+clusters)
>>  >>>>   > 0/0/0 requests for
>> jumbo clusters
>>  >> denied (4k/9k/16k)
>>  >>>>   > 0/0/0 sfbufs in use
>> (current/peak/max)
>>  >>>>   > 0 requests for
>> sfbufs denied
>>  >>>>   > 0 requests for
>> sfbufs delayed
>>  >>>>   > 0 requests for I/O
>> initiated by
>>  >> sendfile
>>  >>>>   > 0 calls to protocol
>> drain routines
>>  >>>>   > 
>>  >>>>   > I use only IPv4
>> traffic.
>>  >>>>   > 
>>  >>>>   >> How-To-Repeat:
>>  >>>>   > On machine with two
>> igb interfaces
>>  >>>>   > use rc.conf like
>> this:
>>  >>>>   > 
>>  >>>>   >
>> hostname="test.test"
>>  >>>>   >
>> gateway_enable="YES"
>>  >>>>   > ifconfig_igb0="inet
>> 10.10.10.1/24"
>>  >>>>   > ifconfig_igb1="inet
>> 10.10.11.1/24"
>>  >>>>   > 
>>  >>>>   > And try create
>> heavy traffic between
>>  >> two networks.
>>  >>>>   >> Fix:
>>  >>>>   > 
>>  >>>>   > 
>>  >>>>   >> Release-Note:
>>  >>>>   >> Audit-Trail:
>>  >>>>   >> Unformatted:
>>  >>>>   >
>>  >> _______________________________________________
>>  >>>>   > freebsd-bugs@freebsd.org
>>  >>>
>>  >>> This is not a bug. Unless you consider poorly
>> written
>>  >> drivers to be bugs. You need to provide your
>> tuning
>>  >> parameters for the card as well otherwise there's
>> nothing to
>>  >> learn.
>>  >>> The issue is that the driver doesn't address
>> the
>>  >> purpose of the controller; which is to utilize
>>  >> multiprocessor systems more effectively. The
>> effect is that
>>  >> lock contention actually makes things worse than
>> if you just
>>  >> use a single task as em does. Until the
>> multiqueue drivers
>>  >> are re-written to manage locks properly you are
>> best advised
>>  >> to save your money and stick with em.
>>  >>> You should get similar performance using 1
>> queue as
>>  >> with em. You could also force legacy
>> configuration by
>>  >> forcing igb_setup_msix to return 0. Sadly, this
>> is the best
>>  >> performance you will get from the stock driver.
>>  >>> Barney
>>  >>>
>>  >>> Barney
>>  >>>
>>  >>>
>>  >>>        
>>  >> I tried using 1 queue and it didn't make things
>> any better
>>  >> (actually I'm
>>  >> not sure if that worked at all). If it is
>> considered a bug
>>  >> or not
>>  >> doesn't really matter, what actually matters for
>> users (who
>>  >> cannot
>>  >> always chose which network controller will be
>> on-board) is
>>  >> that they get
>>  >> a least decent performance when doing IP
>> forwarding (and
>>  >> not the
>>  >> 5-50kb/s I've seen). You can get this out of the
>>  >> controller, when
>>  >> disabling lro through the sysctl. That's why I've
>> been
>>  >> asking to put
>>  >> this into the release errata section and/or at
>> least the
>>  >> igb man page,
>>  >> because the sysctl isn't documented anywhere.
>> Also the
>>  >> fact, that tuning
>>  >> the sysctl only affects the behaviour when it's
>> set on boot
>>  >> might be
>>  >> considered problematic.
>>  >>
>>  >> So at the very least, I think the following
>> should be
>>  >> done:
>>  >> 1. Document the sysctl in man igb(4)
>>  >> 2. Put a known issues paragraph to man igb(4)
>> which
>>  >> explains the issue
>>  >> and what to put in sysctl.conf to stop this from
>> happening
>>  >> 3. Add an entry to the release errata page about
>> this issue
>>  >> (like I
>>  >> suggested in one of my earlier emails) and
>> stating
>>  >> something like "see
>>  >> man igb(4) for details)
>>  >>
>>  >> This is not about using the controller to its
>> full
>>  >> potential, but to
>>  >> safe Joe Admin from spending days on figuring out
>> why the
>>  >> machine is
>>  >> forwarding packages slower than his BSD 2.x
>> machine did in
>>  >> the 90s.
>>  >>
>>  >> cheers
>>  >> Michael
>>  > 
>>  > None of the offload crap should be enabled by
>> default. 
>>  > 
>>  > The real point is that "Joe Admin" shouldn't be using
>> controllers that have bad drivers at all. If you have to use
>> whatever hardware you have laying around, and don't have
>> enough flexibility to lay out $100 for a 2 port controller
>> that works to use with your $2000 server, than you need to
>> get your priorities in order. People go out and buy
>> redundant power supplies, high GHZ quad core processors and
>> gobs of memory and then they use whatever crappy onboard
>> controller they get no matter how poorly its suppo rted. Its
>> mindless.
>>  > 
>>  > Barney
>>  > 
>>  > 
>>  >       
>>  
>>  How should anybody know that the controller is poorly
>> supported if there
>>  is nothing in the documentation, release notes, man pages
>> or anywhere
>>  else about this?
>>  
>>  The fact of the matter is that "the offload crap" _is_
>> enabled by
>>  default. The release is out, it claims to support the
>> controller. There
>>  _is_ a workaround and I'm asked if somebody could document
>> this so users
>>  will have a chance. I'm also not convinced that it is a
>> crappy
>>  controller per se, but just poorly supported. We used
>> those a lot before
>>  without any issues, unfortunately now we had touse IP
>> forwarding in a
>>  machine that has that controller (it has 6 interfaces in
>> total, four em
>>  ports and two igb ports, all of them are in use and I
>> don't feel like
>>  hooking up the sodering iron).
>>  
>>  So bottomline:
>>  I said, there is a problem with the driver, there is a
>> workaround and it
>>  should be documented.
>>  
>>  You say, the driver is bad and nobody should use it and if
>> they do it's
>>  their own damn fault. We won't do anything about it and
>> refuse to tell
>>  anybody, because we are the only ones who should know. We
>> don't care if
>>  people can actually use our software and still claim the
>> hardware is
>>  actually supported.
>>  
>>  Your attitude is really contra productive (actually
>> googling around I
>>  see  you made similar statements in the past about
>> stupid people not
>>  willing to spend xxx$ on whatever piece of hardware, so
>> maybe you're
>>  just trolling).
>>  
>>  Michael
> 
> Tuning the card to be brain-dead isn't really a workaround. I'm sorry that you're not able to understand, but you can't educate the woodchucks, so carry on and feel free to do whatever you wish.
> 
> BC
> 
> 
>       

Without tuning the card: 5kb/s, with tuning the card: 50mb/s
That's the definition of a workaround, the fix would be making lro work
correctly - in general I prefer a brain-dead card to a brain-dead
mailing list subscriber. Welcome to the real world :)

Anyway, I'll stop feeding you now, this is getting boring and leads nowhere.

I still think that this should be noted somewhere in the docs, whoever
has permissions to commit might proceed in doing so...



Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?4A3BB0F3.80005>