Date: Fri, 19 Jun 2009 09:01:29 -0700 (PDT) From: Barney Cordoba <barney_cordoba@yahoo.com> To: freebsd-net@FreeBSD.org Subject: Re: kern/135222: [igb] low speed routing between two igb interfaces Message-ID: <904439.18775.qm@web63908.mail.re1.yahoo.com>
next in thread | raw e-mail | index | archive | help
=0A=0A--- On Fri, 6/19/09, Michael <freebsdusb@bindone.de> wrote:=0A=0A> Fr= om: Michael <freebsdusb@bindone.de>=0A> Subject: Re: kern/135222: [igb] low= speed routing between two igb interfaces=0A> To: "Barney Cordoba" <barney_= cordoba@yahoo.com>=0A> Cc: freebsd-net@FreeBSD.org=0A> Date: Friday, June 1= 9, 2009, 11:38 AM=0A> Barney Cordoba wrote:=0A> > =0A> > =0A> > --- On Wed,= 6/17/09, Michael <freebsdusb@bindone.de>=0A> wrote:=0A> > =0A> >> From: Mi= chael <freebsdusb@bindone.de>=0A> >> Subject: Re: kern/135222: [igb] low sp= eed routing=0A> between two igb interfaces=0A> >> To: freebsd-net@FreeBSD.o= rg=0A> >> Date: Wednesday, June 17, 2009, 9:40 PM=0A> >> The following repl= y was made to PR=0A> >> kern/135222; it has been noted by GNATS.=0A> >>=0A>= >> From: Michael <freebsdusb@bindone.de>=0A> >> To: Barney Cordoba <barney= _cordoba@yahoo.com>=0A> >> Cc: freebsd-gnats-submit@FreeBSD.org=0A> >> Subj= ect: Re: kern/135222: [igb] low speed routing=0A> between=0A> >> two igb in= terfaces=0A> >> Date: Thu, 18 Jun 2009 03:32:15 +0200=0A> >>=0A> >>=A0 Barn= ey Cordoba wrote:=0A> >>=A0 > =0A> >>=A0 > =0A> >>=A0 > --- On Wed, 6/17/09= , Michael <freebsdusb@bindone.de>=0A> >> wrote:=0A> >>=A0 > =0A> >>=A0 >> F= rom: Michael <freebsdusb@bindone.de>=0A> >>=A0 >> Subject: Re: kern/135222:= [igb] low=0A> speed routing=0A> >> between two igb interfaces=0A> >>=A0 >>= To: "Barney Cordoba" <barney_cordoba@yahoo.com>=0A> >>=A0 >> Cc: freebsd-n= et@FreeBSD.org=0A> >>=A0 >> Date: Wednesday, June 17, 2009,=0A> 5:28 PM=0A>= >>=A0 >> Barney Cordoba wrote:=0A> >>=A0 >>>=0A> >>=A0 >>> --- On Fri, 6/1= 2/09, Michael=0A> <freebsdusb@bindone.de>=0A> >>=A0 >> wrote:=0A> >>=A0 >>>= > From: Michael <freebsdusb@bindone.de>=0A> >>=A0 >>>> Subject: Re: kern/13= 5222:=0A> [igb] low speed=0A> >> routing=0A> >>=A0 >> between two igb inter= faces=0A> >>=A0 >>>> To: freebsd-net@FreeBSD.org=0A> >>=A0 >>>> Date: Frida= y, June 12,=0A> 2009, 5:50 AM=0A> >>=A0 >>>> The following reply was=0A> ma= de to PR=0A> >>=A0 >>>> kern/135222; it has been=0A> noted by GNATS.=0A> >>= =A0 >>>>=0A> >>=A0 >>>> From: Michael <freebsdusb@bindone.de>=0A> >>=A0 >>>= > To: Cc: freebsd-gnats-submit@FreeBSD.org=0A> >>=A0 >>>> Subject: Re: kern= /135222:=0A> [igb] low speed=0A> >> routing=0A> >>=A0 >> between=0A> >>=A0 = >>>> two igb interfaces=0A> >>=A0 >>>> Date: Fri, 12 Jun 2009=0A> 11:45:47 = +0200=0A> >>=A0 >>>>=0A> >>=A0 >>>>=A0=A0=A0The=0A> original poster=0A> >> = reported that the=0A> >>=A0 >> suggested fix works=0A> >>=A0 >>>> for him:= =0A> >>=A0 >>>>=A0=A0=A0---=0A> >>=A0 >>>>=A0=A0=A0Hello=0A> Michael,=0A> >= >=A0 >>>>=A0=A0=A0=0A> >>=A0 >>>>=A0=A0=A0Thank you.=0A> It's=0A> >> workin= g.=0A> >>=A0 >>>>=A0=A0=A0=0A> >>=A0 >>>>=A0=A0=A0I consider=0A> it necessa= ry=0A> >> to put this into the=0A> >>=A0 >> release=0A> >>=A0 >>>> errata.= =0A> >>=A0 >>>>=A0=A0=A0=0A> >>=A0 >>>>=A0=A0=A0=0A> >>=A0 >>>>=A0=A0=A0Mis= hustin=0A> Andrew wrote:=0A> >>=A0 >>>>=A0=A0=A0>>=0A> Number: =0A> >>=A0 = =A0 =0A> >>=A0 >>>>=A0=0A> =A0=A0=A0135222=0A> >>=A0 >>>>=A0=A0=A0>>=0A> >>= Category:=A0=A0=A0=0A> >>=A0 >>=A0 =A0 kern=0A> >>=A0 >>>>=A0=A0=A0>>=0A> = >> Synopsis:=A0=A0=A0=0A> >>=A0 >>=A0 =A0 [igb]=0A> >>=A0 >>>> low speed ro= uting between=0A> two igb=0A> >> interfaces=0A> >>=A0 >>>>=A0=A0=A0>>=0A> >= > Confidential:=A0=A0=A0no=0A> >>=A0 >>>>=A0=A0=A0>>=0A> >> Severity:=A0=A0= =A0=0A> >>=A0 >>=A0 =A0 serious=0A> >>=A0 >>>>=A0=A0=A0>>=0A> >> Priority:= =A0=A0=A0=0A> >>=A0 >>=A0 =A0 medium=0A> >>=A0 >>>>=A0=A0=A0>>=0A> >> Respo= nsible:=A0=A0=A0=0A> >>=A0 >> freebsd-bugs=0A> >>=A0 >>>>=A0=A0=A0>>=0A> St= ate: =0A> >>=A0 =A0 =A0 =0A> >>=A0 >>=A0=A0=A0open=0A> >>=A0 >>>>=A0=A0=A0>= >=0A> Quarter: =0A> >>=A0 =A0 =A0 =0A> >>=A0 >>>>=A0=A0=A0>>=0A> >> Keyword= s:=A0=A0=A0=0A> >>=A0 >>=A0 =A0 =0A> >>=A0 >>>>=A0=A0=A0>>=0A> Date-Require= d:=0A> >>=A0 >>>>=A0=A0=A0>>=0A> Class: =0A> >>=A0 =A0 =A0 =0A> >>=A0 >>=A0= =A0=A0sw-bug=0A> >>=A0 >>>>=A0=A0=A0>>=0A> >>=A0 >>=0A> Submitter-Id:=A0=A0= =A0current-users=0A> >>=A0 >>>>=A0=A0=A0>>=0A> >> Arrival-Date:=A0=A0=A0Wed= =0A> >>=A0 >> Jun 03=0A> >>=A0 >>>> 18:30:01 UTC 2009=0A> >>=A0 >>>>=A0=A0= =A0>>=0A> Closed-Date:=0A> >>=A0 >>>>=A0=A0=A0>>=0A> Last-Modified:=0A> >>= =A0 >>>>=A0=A0=A0>>=0A> Originator: =0A> >>=A0 >>=A0 =A0 Mishustin=0A> >>= =A0 >>>> Andrew=0A> >>=A0 >>>>=A0=A0=A0>>=0A> Release: =0A> >>=A0 =A0 =A0 = =0A> >>=A0 >> FreeBSD=0A> >>=A0 >>>> 7.1-RELEASE amd64, FreeBSD=0A> 7.2-REL= EASE=0A> >> amd64=0A> >>=A0 >>>>=A0=A0=A0>>=0A> Organization:=0A> >>=A0 >>>= >=A0=A0=A0> HNT=0A> >>=A0 >>>>=A0=A0=A0>>=0A> Environment:=0A> >>=A0 >>>>= =A0=A0=A0>=0A> FreeBSD test.hnt=0A> >> 7.2-RELEASE FreeBSD=0A> >>=A0 >> 7.2= -RELEASE #12:=0A> >>=A0 >>>> Thu Apr 30 18:28:15 MSD 20=0A> >>=A0 >>>>=A0= =A0=A0> 09 =0A> >>=A0 =A0 admin@test.hnt:/usr/src/sys/amd64/compile/GENERIC= =0A> >>=A0 >>>> amd64=0A> >>=A0 >>>>=A0=A0=A0>>=0A> Description:=0A> >>=A0 = >>>>=A0=A0=A0> I=0A> made a FreeBSD=0A> >> multiprocesor server=0A> >>=A0 >= > to act as=0A> >>=A0 >>>> simple gateway.=0A> >>=A0 >>>>=A0=A0=A0> It=0A> = use onboard=0A> >> Intel 82575EB Dual-Port=0A> >>=A0 >> Gigabit=0A> >>=A0 >= >>> Ethernet Controller.=0A> >>=A0 >>>>=A0=A0=A0> I=0A> observe traffic=0A>= >> speed near 400=0A> >>=A0 >> Kbit/s.=0A> >>=A0 >>>>=A0=A0=A0> I=0A> test= both=0A> >> interfaces separately -=0A> >>=A0 >>>>=A0=A0=A0> ftp=0A> clien= t work at=0A> >> speed near 1 Gbit/s=0A> >>=A0 >> in both=0A> >>=A0 >>>> di= rections.=0A> >>=A0 >>>>=A0=A0=A0> Then=0A> I change NIC=0A> >> to old Inte= l "em" NIC=0A> >>=A0 >> - gateway=0A> >>=A0 >>>> work at speed near 1=0A> G= bit/s.=0A> >>=A0 >>>>=A0=A0=A0> =0A> >>=A0 >>>>=A0=A0=A0> Looks=0A> like a = bug in=0A> >> igb driver have an=0A> >>=A0 >> effect upon=0A> >>=A0 >>>> fo= rwarded traffic.=0A> >>=A0 >>>>=A0=A0=A0> =0A> >>=A0 >>>>=A0=A0=A0> If=0A> = you try=0A> >>=A0 >>>>=A0=A0=A0>=0A> >> hw.igb.enable_aim=3D0=0A> >>=A0 >>>= >=A0=A0=A0> The=0A> speed is near 1=0A> >> Mbit/s=0A> >>=A0 >>>>=A0=A0=A0> = =0A> >>=A0 >>>>=A0=A0=A0>=0A> hw.igb.rxd,=0A> >> hw.igb.txd, "ifconfig -tso= "=0A> >>=A0 >> has no=0A> >>=A0 >>>> effect.=0A> >>=A0 >>>>=A0=A0=A0> =0A> = >>=A0 >>>>=A0=A0=A0>=0A> Nothing in=0A> >> messages.log=0A> >>=A0 >>>>=A0= =A0=A0> =0A> >>=A0 >>>>=A0=A0=A0>=0A> netstat -m=0A> >>=A0 >>>>=A0=A0=A0>= =0A> 516/1674/2190 mbufs=0A> >> in use=0A> >>=A0 >> (current/cache/total)= =0A> >>=A0 >>>>=A0=A0=A0>=0A> 515/927/1442/66560=0A> >> mbuf clusters in=0A= > >>=A0 >> use=0A> >>=A0 >>>> (current/cache/total/max)=0A> >>=A0 >>>>=A0= =A0=A0>=0A> 515/893=0A> >> mbuf+clusters out of packet=0A> >>=A0 >> seconda= ry zone in=0A> >>=A0 >>>> use (current/cache)=0A> >>=A0 >>>>=A0=A0=A0>=0A> = 0/44/44/33280 4k=0A> >> (page size) jumbo=0A> >>=A0 >> clusters in use=0A> = >>=A0 >>>> (current/cache/total/max)=0A> >>=A0 >>>>=A0=A0=A0>=0A> 0/0/0/166= 40 9k=0A> >> jumbo clusters in use=0A> >>=A0 >>>> (current/cache/total/max)= =0A> >>=A0 >>>>=A0=A0=A0>=0A> 0/0/0/8320 16k=0A> >> jumbo clusters in use= =0A> >>=A0 >>>> (current/cache/total/max)=0A> >>=A0 >>>>=A0=A0=A0>=0A> 1159= K/2448K/3607K=0A> >> bytes allocated to=0A> >>=A0 >> network=0A> >>=A0 >>>>= (current/cache/total)=0A> >>=A0 >>>>=A0=A0=A0> 0/0/0=0A> requests for=0A> = >> mbufs denied=0A> >>=A0 >>>>=0A> (mbufs/clusters/mbuf+clusters)=0A> >>=A0= >>>>=A0=A0=A0> 0/0/0=0A> requests for=0A> >> jumbo clusters=0A> >>=A0 >> d= enied (4k/9k/16k)=0A> >>=A0 >>>>=A0=A0=A0> 0/0/0=0A> sfbufs in use=0A> >> (= current/peak/max)=0A> >>=A0 >>>>=A0=A0=A0> 0=0A> requests for=0A> >> sfbufs= denied=0A> >>=A0 >>>>=A0=A0=A0> 0=0A> requests for=0A> >> sfbufs delayed= =0A> >>=A0 >>>>=A0=A0=A0> 0=0A> requests for I/O=0A> >> initiated by=0A> >>= =A0 >> sendfile=0A> >>=A0 >>>>=A0=A0=A0> 0=0A> calls to protocol=0A> >> dra= in routines=0A> >>=A0 >>>>=A0=A0=A0> =0A> >>=A0 >>>>=A0=A0=A0> I use=0A> on= ly IPv4=0A> >> traffic.=0A> >>=A0 >>>>=A0=A0=A0> =0A> >>=A0 >>>>=A0=A0=A0>>= =0A> How-To-Repeat:=0A> >>=A0 >>>>=A0=A0=A0> On=0A> machine with two=0A> >>= igb interfaces=0A> >>=A0 >>>>=A0=A0=A0> use=0A> rc.conf like=0A> >> this:= =0A> >>=A0 >>>>=A0=A0=A0> =0A> >>=A0 >>>>=A0=A0=A0>=0A> >> hostname=3D"test= .test"=0A> >>=A0 >>>>=A0=A0=A0>=0A> >> gateway_enable=3D"YES"=0A> >>=A0 >>>= >=A0=A0=A0>=0A> ifconfig_igb0=3D"inet=0A> >> 10.10.10.1/24"=0A> >>=A0 >>>>= =A0=A0=A0>=0A> ifconfig_igb1=3D"inet=0A> >> 10.10.11.1/24"=0A> >>=A0 >>>>= =A0=A0=A0> =0A> >>=A0 >>>>=A0=A0=A0> And=0A> try create=0A> >> heavy traffi= c between=0A> >>=A0 >> two networks.=0A> >>=A0 >>>>=A0=A0=A0>>=0A> Fix:=0A>= >>=A0 >>>>=A0=A0=A0> =0A> >>=A0 >>>>=A0=A0=A0> =0A> >>=A0 >>>>=A0=A0=A0>>= =0A> Release-Note:=0A> >>=A0 >>>>=A0=A0=A0>>=0A> Audit-Trail:=0A> >>=A0 >>>= >=A0=A0=A0>>=0A> Unformatted:=0A> >>=A0 >>>>=A0=A0=A0>=0A> >>=A0 >>=0A> ___= ____________________________________________=0A> >>=A0 >>>>=A0=A0=A0> freeb= sd-bugs@freebsd.org=0A> >>=A0 >>>=0A> >>=A0 >>> This is not a bug. Unless y= ou=0A> consider poorly=0A> >> written=0A> >>=A0 >> drivers to be bugs. You = need to=0A> provide your=0A> >> tuning=0A> >>=A0 >> parameters for the card= as well=0A> otherwise there's=0A> >> nothing to=0A> >>=A0 >> learn.=0A> >>= =A0 >>> The issue is that the driver=0A> doesn't address=0A> >> the=0A> >>= =A0 >> purpose of the controller; which is=0A> to utilize=0A> >>=A0 >> mult= iprocessor systems more=0A> effectively. The=0A> >> effect is that=0A> >>= =A0 >> lock contention actually makes=0A> things worse than=0A> >> if you j= ust=0A> >>=A0 >> use a single task as em does. Until=0A> the=0A> >> multiqu= eue drivers=0A> >>=A0 >> are re-written to manage locks=0A> properly you ar= e=0A> >> best advised=0A> >>=A0 >> to save your money and stick with=0A> em= .=0A> >>=A0 >>> You should get similar=0A> performance using 1=0A> >> queue= as=0A> >>=A0 >> with em. You could also force=0A> legacy=0A> >> configurat= ion by=0A> >>=A0 >> forcing igb_setup_msix to return 0.=0A> Sadly, this=0A>= >> is the best=0A> >>=A0 >> performance you will get from the=0A> stock dr= iver.=0A> >>=A0 >>> Barney=0A> >>=A0 >>>=0A> >>=A0 >>> Barney=0A> >>=A0 >>>= =0A> >>=A0 >>>=0A> >>=A0 >>>=A0 =A0 =A0 =A0 =0A> >>=A0 >> I tried using 1 q= ueue and it didn't=0A> make things=0A> >> any better=0A> >>=A0 >> (actually= I'm=0A> >>=A0 >> not sure if that worked at all). If=0A> it is=0A> >> cons= idered a bug=0A> >>=A0 >> or not=0A> >>=A0 >> doesn't really matter, what= =0A> actually matters for=0A> >> users (who=0A> >>=A0 >> cannot=0A> >>=A0 >= > always chose which network=0A> controller will be=0A> >> on-board) is=0A>= >>=A0 >> that they get=0A> >>=A0 >> a least decent performance when=0A> do= ing IP=0A> >> forwarding (and=0A> >>=A0 >> not the=0A> >>=A0 >> 5-50kb/s I'= ve seen). You can get=0A> this out of the=0A> >>=A0 >> controller, when=0A>= >>=A0 >> disabling lro through the sysctl.=0A> That's why I've=0A> >> been= =0A> >>=A0 >> asking to put=0A> >>=A0 >> this into the release errata=0A> s= ection and/or at=0A> >> least the=0A> >>=A0 >> igb man page,=0A> >>=A0 >> b= ecause the sysctl isn't documented=0A> anywhere.=0A> >> Also the=0A> >>=A0 = >> fact, that tuning=0A> >>=A0 >> the sysctl only affects the=0A> behaviour= when it's=0A> >> set on boot=0A> >>=A0 >> might be=0A> >>=A0 >> considered= problematic.=0A> >>=A0 >>=0A> >>=A0 >> So at the very least, I think the= =0A> following=0A> >> should be=0A> >>=A0 >> done:=0A> >>=A0 >> 1. Document= the sysctl in man=0A> igb(4)=0A> >>=A0 >> 2. Put a known issues paragraph = to=0A> man igb(4)=0A> >> which=0A> >>=A0 >> explains the issue=0A> >>=A0 >>= and what to put in sysctl.conf to=0A> stop this from=0A> >> happening=0A> = >>=A0 >> 3. Add an entry to the release=0A> errata page about=0A> >> this i= ssue=0A> >>=A0 >> (like I=0A> >>=A0 >> suggested in one of my earlier=0A> e= mails) and=0A> >> stating=0A> >>=A0 >> something like "see=0A> >>=A0 >> man= igb(4) for details)=0A> >>=A0 >>=0A> >>=A0 >> This is not about using the= =0A> controller to its=0A> >> full=0A> >>=A0 >> potential, but to=0A> >>=A0= >> safe Joe Admin from spending days=0A> on figuring out=0A> >> why the=0A= > >>=A0 >> machine is=0A> >>=A0 >> forwarding packages slower than his=0A> = BSD 2.x=0A> >> machine did in=0A> >>=A0 >> the 90s.=0A> >>=A0 >>=0A> >>=A0 = >> cheers=0A> >>=A0 >> Michael=0A> >>=A0 > =0A> >>=A0 > None of the offload= crap should be=0A> enabled by=0A> >> default. =0A> >>=A0 > =0A> >>=A0 > Th= e real point is that "Joe Admin"=0A> shouldn't be using=0A> >> controllers = that have bad drivers at all. If you=0A> have to use=0A> >> whatever hardwa= re you have laying around, and=0A> don't have=0A> >> enough flexibility to = lay out $100 for a 2 port=0A> controller=0A> >> that works to use with your= $2000 server, than you=0A> need to=0A> >> get your priorities in order. Pe= ople go out and=0A> buy=0A> >> redundant power supplies, high GHZ quad core= =0A> processors and=0A> >> gobs of memory and then they use whatever crappy= =0A> onboard=0A> >> controller they get no matter how poorly its suppo=0A> = rted. Its=0A> >> mindless.=0A> >>=A0 > =0A> >>=A0 > Barney=0A> >>=A0 > =0A>= >>=A0 > =0A> >>=A0 >=A0 =A0 =A0=A0=A0=0A> >>=A0 =0A> >>=A0 How should anyb= ody know that the controller=0A> is poorly=0A> >> supported if there=0A> >>= =A0 is nothing in the documentation, release=0A> notes, man pages=0A> >> or= anywhere=0A> >>=A0 else about this?=0A> >>=A0 =0A> >>=A0 The fact of the m= atter is that "the offload=0A> crap" _is_=0A> >> enabled by=0A> >>=A0 defau= lt. The release is out, it claims to=0A> support the=0A> >> controller. The= re=0A> >>=A0 _is_ a workaround and I'm asked if somebody=0A> could document= =0A> >> this so users=0A> >>=A0 will have a chance. I'm also not convinced= =0A> that it is a=0A> >> crappy=0A> >>=A0 controller per se, but just poorl= y=0A> supported. We used=0A> >> those a lot before=0A> >>=A0 without any is= sues, unfortunately now we had=0A> touse IP=0A> >> forwarding in a=0A> >>= =A0 machine that has that controller (it has 6=0A> interfaces in=0A> >> tot= al, four em=0A> >>=A0 ports and two igb ports, all of them are in=0A> use a= nd I=0A> >> don't feel like=0A> >>=A0 hooking up the sodering iron).=0A> >>= =A0 =0A> >>=A0 So bottomline:=0A> >>=A0 I said, there is a problem with the= driver,=0A> there is a=0A> >> workaround and it=0A> >>=A0 should be docume= nted.=0A> >>=A0 =0A> >>=A0 You say, the driver is bad and nobody should=0A>= use it and if=0A> >> they do it's=0A> >>=A0 their own damn fault. We won't= do anything=0A> about it and=0A> >> refuse to tell=0A> >>=A0 anybody, beca= use we are the only ones who=0A> should know. We=0A> >> don't care if=0A> >= >=A0 people can actually use our software and=0A> still claim the=0A> >> ha= rdware is=0A> >>=A0 actually supported.=0A> >>=A0 =0A> >>=A0 Your attitude = is really contra productive=0A> (actually=0A> >> googling around I=0A> >>= =A0 see=A0 you made similar statements in the=0A> past about=0A> >> stupid = people not=0A> >>=A0 willing to spend xxx$ on whatever piece of=0A> hardwar= e, so=0A> >> maybe you're=0A> >>=A0 just trolling).=0A> >>=A0 =0A> >>=A0 Mi= chael=0A> > =0A> > Tuning the card to be brain-dead isn't really a=0A> work= around. I'm sorry that you're not able to understand,=0A> but you can't edu= cate the woodchucks, so carry on and feel=0A> free to do whatever you wish.= =0A> > =0A> > BC=0A> > =0A> > =0A> >=A0 =A0 =A0=A0=A0=0A> =0A> Without tuni= ng the card: 5kb/s, with tuning the card:=0A> 50mb/s=0A> That's the definit= ion of a workaround, the fix would be=0A> making lro work=0A> correctly - i= n general I prefer a brain-dead card to a=0A> brain-dead=0A> mailing list s= ubscriber. Welcome to the real world :)=0A> =0A> Anyway, I'll stop feeding = you now, this is getting boring=0A> and leads nowhere.=0A> =0A> I still thi= nk that this should be noted somewhere in the=0A> docs, whoever=0A> has per= missions to commit might proceed in doing so...=0A> _______________________= ________________________=0A> freebsd-net@freebsd.org=0A> mailing list=0A> h= ttp://lists.freebsd.org/mailman/listinfo/freebsd-net=0A> To unsubscribe, se= nd any mail to "freebsd-net-unsubscribe@freebsd.org"=0A> =0A=0AIgnorance is= Bliss, as they say.=0A=0ABC=0A=0A=0A
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?904439.18775.qm>