From owner-freebsd-net@FreeBSD.ORG Sun Aug 2 16:54:42 2009 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 348731065675 for ; Sun, 2 Aug 2009 16:54:42 +0000 (UTC) (envelope-from barney_cordoba@yahoo.com) Received: from web63904.mail.re1.yahoo.com (web63904.mail.re1.yahoo.com [69.147.97.119]) by mx1.freebsd.org (Postfix) with SMTP id E139F8FC12 for ; Sun, 2 Aug 2009 16:54:41 +0000 (UTC) (envelope-from barney_cordoba@yahoo.com) Received: (qmail 36099 invoked by uid 60001); 2 Aug 2009 16:54:41 -0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1249232081; bh=fLrAm4DQlf28GKf2HRbxazySzAzWB9GHzCLGxUw/GaQ=; h=Message-ID:X-YMail-OSG:Received:X-Mailer:Date:From:Subject:To:In-Reply-To:MIME-Version:Content-Type:Content-Transfer-Encoding; b=f4iwH/Poq1yWsxxiS/Wp/i3xdLIHEs2v/SbXD1tJy80e1R1Y4DfqxN7CU64Rsb4jofMGlSF6f/O18OnLQr1eNpEmb9ok63hnH57rmxkakqtbDAKVKBjX8eYtnl81PXVHnoeEfHpkGImQey9MkeG+/FyDv48BViiwyN7BdRw1Ehg= DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=s1024; d=yahoo.com; h=Message-ID:X-YMail-OSG:Received:X-Mailer:Date:From:Subject:To:In-Reply-To:MIME-Version:Content-Type:Content-Transfer-Encoding; b=QeRu2WVDZ5svQRM2JoMmjcV/R2B3V/pCn1ZyKwZfZObonHilVijxnUEUTVRJdeyeFcYtOHH42u18EtSjuqXNSQPMkYDwY3wjV/uDKumg9BHffDW3p1yCqmD39xtIQ5VoXm/d+Ut9AYBLfxveBJkMElyf2iRkCQiiklzUDsIlWuU=; Message-ID: <210006.36085.qm@web63904.mail.re1.yahoo.com> X-YMail-OSG: NFefiZ8VM1merjo3l7sHgZThJwOjsi.qJ2tr1AExyYe15DEK7k.lYDnwjGCKlh_CkrLvfblrhlE7j6ic8SVVmE1uWWSKRkwtnZSzrefSmuA6kZwQ8jj7KA1l77AFTDFlpdh9XSqx.DTEJ_US2Kr2WCeK.tDf8SMp0egYBrKTL_Y3aLrDaMlTQmXNTV25T9GzdQAehOFok5IZr3gmQaOrh09XHmqeCWrzxgT.Cb.x.YObE966f_HIB3atF1KhMYLP4bATPtTcze1LuiAXmKHNsonix4tTfVp.AxFvGJ00niedNwv3wcLgGkzySl2ll0uVUgG3DtMlpQW3 Received: from [66.176.162.245] by web63904.mail.re1.yahoo.com via HTTP; Sun, 02 Aug 2009 09:54:40 PDT X-Mailer: YahooMailClassic/6.0.19 YahooMailWebService/0.7.289.15 Date: Sun, 2 Aug 2009 09:54:40 -0700 (PDT) From: Barney Cordoba To: freebsd-net@freebsd.org, alexpalias-bsdnet@yahoo.com In-Reply-To: <11420.28890.qm@web56404.mail.re3.yahoo.com> MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: quoted-printable Cc: Subject: Re: em driver input errors X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 02 Aug 2009 16:54:42 -0000 =0A=0A--- On Sat, 8/1/09, alexpalias-bsdnet@yahoo.com wrote:=0A=0A> From: alexpalias-bsdnet@yahoo.com =0A> Subject: em driver input errors=0A> To: freebsd-net@freebsd= .org=0A> Date: Saturday, August 1, 2009, 9:05 AM=0A> Good day=0A> =0A> I'm = running a FreeBSD 7.2 router and I am seeing a lot of=0A> input errors on o= ne of the em interfaces (em0), coupled with=0A> (at approximately the same = times) much fewer errors on em1=0A> and em2.=A0 Monitoring is done with SNM= P from another=0A> machine, and the CPU load as reported via SNMP is mostly= =0A> below 30%, with a couple of spikes up to 35%.=0A> =0A> Software descri= ption:=0A> =0A> - FreeBSD 7.2-RELEASE-p2, amd64=0A> - bsnmpd with modules: = hostres and (from ports) snmp_ucd=0A> - quagga 0.99.12 (running only zebra = and bgpd)=0A> - netgraph (ng_ether and ng_netflow)=0A> =0A> Hardware descri= ption:=0A> =0A> - Dell machine, dual Xeon 3.20 GHz, 4 GB RAM=0A> - 2 x buil= t-in gigabit interfaces (em0, em1)=0A> - 1 x dual-port gigabit interface, P= CI-X (em2, em3) [see=0A> pciconf near the end]=0A> =0A> =0A> The machine re= ceives the global routing table ("netstat -nr=0A> | wc -l" gives 289115 cur= rently).=0A> =0A> All of the em interfaces are just configured "up", with= =0A> various vlan interfaces on them.=A0 Note that I use "kpps" to=0A> mean= "thousands of packets per second", sorry if that's the=0A> wrong shorthand= .=0A> =0A> - em0 sees a traffic of 10...22 kpps in, and 15...35 kpps=0A> ou= t.=A0 In bits, it's 30...120Mbits/s in, and=0A> 100...210Mbits/s out.=A0 Vl= ans configured are vlan100 and=0A> vlan200, and most of the traffic is on v= lan100 (vlan200 sees=0A> 4kpps in / 0.5kpps out maximum, with the average a= t about=0A> one third of this).=A0 em0 is the external interface, and its= =0A> traffic corresponds to the sum of traffic through em1 and=0A> em2=0A> = =0A> - em1 has 5 vlans, and sees about 22kpps in / 11kpps out=0A> (maximum)= =0A> =0A> - em2 has a single VLAN, and sees about 4...13kpps both in=0A> an= d out (almost equal in/out during most of the day)=0A> =0A> - em3 is a back= up interface, with 2 VLANS, and is the only=0A> one which has seen no error= s.=0A> =0A> Only the vlans on em0 are analyzed by ng_netflow, and the=0A> e= rrors I'm seeing have started appearing days before=0A> netgraph was even l= oaded in the kernel.=0A> =0A> Tuning done:=0A> =0A> /boot/loader.conf:=0A> = hw.em.rxd=3D4096=0A> hw.em.txd=3D4096=0A> =0A> Witout the above we were see= ing way more errors, now they=0A> are reduced, but still come in bursts of = over 1000 errors on=0A> em0.=0A> =0A> /etc/sysctl.conf:=0A> net.inet.ip.fas= tforwarding=3D1=0A> dev.em.0.rx_processing_limit=3D300=0A> dev.em.1.rx_proc= essing_limit=3D300=0A> dev.em.2.rx_processing_limit=3D300=0A> dev.em.3.rx_p= rocessing_limit=3D300=0A> =0A> Still seeing errros, after some searching th= e mailing lists=0A> we also added:=0A> =0A> # the four lines below are repe= ated for em1, em2, em3=0A> dev.em.0.rx_int_delay=3D0=0A> dev.em.0.rx_abs_in= t_delay=3D0=0A> dev.em.0.tx_int_delay=3D0=0A> dev.em.0.tx_abs_int_delay=3D0= =0A> =0A> Still getting errors, so I also added:=0A> =0A> net.inet.ip.intr_= queue_maxlen=3D4096=0A> net.route.netisr_maxqlen=3D1024=0A> =0A> and=0A> = =0A> kern.ipc.nmbclusters=3D655360=0A> =0A> =0A> Also tried with rx_process= ing_limit set to -1 on all em=0A> interfaces, still getting errors.=0A> =0A= > Looking at the shape of the error and packet graphs, there=0A> seems to b= e a correlation between the number of packets per=0A> second on em0 and the= height of the error "spikes" on the=0A> error graph.=A0 These spikes are s= pread throughout the day,=0A> with spaces (zones with no errors) of various= lengths (10=0A> minutes ... 2 hours spaces within the last 24 hours), but= =0A> sometimes there are errors even in the lowest kpps times of=0A> the da= y.=0A> =0A> em0 and em1 error times are correlated, with all errors on=0A> = the graph for em0 having a smaller corresponding error spike=0A> on em1 at = the same time, and sometimes an error spike on=0A> em2.=0A> =0A> The old ro= uter was seeing about the same traffic, and had=0A> em0, em1, re0 and re1 n= etwork cards, and was only seeing=0A> errors on the em cards.=A0 It was run= ning=0A> 7.2-PRERELEASE/i386=0A> =0A> =0A> Any suggestions would be greatly= appreciated.=A0 Please note=0A> that this is a live router, and I can't re= boot it (unless=0A> absolutely necessary).=A0 Tuning that can be applied wi= thout=0A> rebooting will be tried first.=0A> =0A> Here are some more detail= s:=0A> =0A> Trimmed output of netstat -ni (sorry if there are line=0A> brea= ks):=0A> Name=A0 =A0 Mtu Network=A0 =A0 =A0=A0=A0Address=A0 =A0 =A0 =A0 =A0= =A0=0A> =A0 Ipkts Ierrs=A0 =A0 Opkts Oerrs=A0 Coll=0A> em0=A0 =A0 1500 =A0 =A0 =A0 00:14:22:xx:xx:xx=0A> 19744458839 15494721 24284439443=A0 = =A0=A0=A00=A0 =A0=A0=A00=0A> em1=A0 =A0 1500 =A0 =A0 =A0 00:14:22:x= x:xx:xx=0A> 12832245469 123181 10105031790=A0 =A0=A0=A00=A0 =A0=A0=A00=0A> = em2=A0 =A0 1500 =A0 =A0 =A0 00:04:23:xx:xx:xx=0A> 12082552403 10964= 10339416865=A0 =A0=A0=A00=A0 =A0=A0=A00=0A> em3=A0 =A0 1500 =A0 = =A0 =A0 00:04:23:xx:xx:xx=0A> 79912337=A0 =A0=A0=A00 48178737=A0 =A0=A0=A00= =A0 =A0=A0=A00=0A> =0A> Relevant part of pciconf -vl:=0A> =0A> em0@pci0:6:7= :0: class=3D0x020000 card=3D0x016d1028=0A> chip=3D0x10768086 rev=3D0x05 hdr= =3D0x00=0A> =A0 =A0 vendor=A0 =A0=A0=A0=3D 'Intel Corporation'=0A> =A0 =A0 = device=A0 =A0=A0=A0=3D '82541EI Gigabit Ethernet=0A> Controller'=0A> =A0 = =A0 class=A0 =A0 =A0 =3D network=0A> =A0 =A0 subclass=A0=A0=A0=3D ethernet= =0A> em1@pci0:7:8:0: class=3D0x020000 card=3D0x016d1028=0A> chip=3D0x107680= 86 rev=3D0x05 hdr=3D0x00=0A> =A0 =A0 vendor=A0 =A0=A0=A0=3D 'Intel Corporat= ion'=0A> =A0 =A0 device=A0 =A0=A0=A0=3D '82541EI Gigabit Ethernet=0A> Contr= oller'=0A> =A0 =A0 class=A0 =A0 =A0 =3D network=0A> =A0 =A0 subclass=A0=A0= =A0=3D ethernet=0A> em2@pci0:9:4:0: class=3D0x020000 card=3D0x10128086=0A> = chip=3D0x10108086 rev=3D0x01 hdr=3D0x00=0A> =A0 =A0 vendor=A0 =A0=A0=A0=3D = 'Intel Corporation'=0A> =A0 =A0 device=A0 =A0=A0=A0=3D '82546EB Dual Port G= igabit Ethernet=0A> Controller (Copper)'=0A> =A0 =A0 class=A0 =A0 =A0 =3D n= etwork=0A> =A0 =A0 subclass=A0=A0=A0=3D ethernet=0A> em3@pci0:9:4:1: class= =3D0x020000 card=3D0x10128086=0A> chip=3D0x10108086 rev=3D0x01 hdr=3D0x00= =0A> =A0 =A0 vendor=A0 =A0=A0=A0=3D 'Intel Corporation'=0A> =A0 =A0 device= =A0 =A0=A0=A0=3D '82546EB Dual Port Gigabit Ethernet=0A> Controller (Copper= )'=0A> =A0 =A0 class=A0 =A0 =A0 =3D network=0A> =A0 =A0 subclass=A0=A0=A0= =3D ethernet=0A> =0A> Kernel messages after sysctl dev.em.0.stats=3D1:=0A> = (note that I've removed the lines which only showed zeros=0A> in the second= and third outputs)=0A> =0A> em0: Excessive collisions =3D 0=0A> em0: Seque= nce errors =3D 0=0A> em0: Defer count =3D 0=0A> em0: Missed Packets =3D 154= 35312=0A> em0: Receive No Buffers =3D 16446113=0A> em0: Receive Length Erro= rs =3D 0=0A> em0: Receive errors =3D 1=0A> em0: Crc errors =3D 2=0A> em0: A= lignment errors =3D 0=0A> em0: Collision/Carrier extension errors =3D 0=0A>= em0: RX overruns =3D 96826=0A> em0: watchdog timeouts =3D 0=0A> em0: RX MS= IX IRQ =3D 0 TX MSIX IRQ =3D 0 LINK MSIX IRQ =3D 0=0A> em0: XON Rcvd =3D 0= =0A> em0: XON Xmtd =3D 0=0A> em0: XOFF Rcvd =3D 0=0A> em0: XOFF Xmtd =3D 0= =0A> em0: Good Packets Rcvd =3D 19002068797=0A> em0: Good Packets Xmtd =3D = 23168462599=0A> em0: TSO Contexts Xmtd =3D 0=0A> em0: TSO Contexts Failed = =3D 0=0A> =0A> [later]=0A> em0: Excessive collisions =3D 0=0A> em0: Missed = Packets =3D 15459111=0A> em0: Receive No Buffers =3D 16447082=0A> em0: Rece= ive errors =3D 1=0A> em0: Crc errors =3D 2=0A> em0: RX overruns =3D 96835= =0A> em0: Good Packets Rcvd =3D 19165047284=0A> em0: Good Packets Xmtd =3D = 23386976960=0A> =0A> [later]=0A> em0: Excessive collisions =3D 0=0A> em0: M= issed Packets =3D 15470583=0A> em0: Receive No Buffers =3D 16447686=0A> em0= : Receive errors =3D 1=0A> em0: Crc errors =3D 2=0A> em0: RX overruns =3D 9= 6840=0A> em0: Good Packets Rcvd =3D 19255466068=0A> em0: Good Packets Xmtd = =3D 23519004546=0A> =0A=0A=0ANote that "most" pcix motherboards wire onboar= d NICs to 32bits and 33Mhz, mainly because its apparently easier to do so. = Its likely that your =0Aadd-on card is running at 64bits and 133Mhz. =0A=0A= 32bits/33Mhz isn't really fast enough to manage gigabit traffic flows, as= =0Aits max burst is only 1 Gb/s, so you really can't use them for any sort= =0Aof primary traffic flow. Check with you MB manufacturer as they usually= =0Adon't advertise it. =0A=0ABarney=0A=0A=0A