Date: Mon, 17 Dec 2007 12:57:05 -0500 From: Mark Fullmer <maf@eng.oar.net> To: Jeremy Chadwick <koitsu@freebsd.org> Cc: freebsd-net@FreeBSD.org, freebsd-stable@freebsd.org Subject: Re: Packet loss every 30.999 seconds Message-ID: <F62928F4-BA8D-4C0F-9C5E-217D70E36321@eng.oar.net> In-Reply-To: <20071217054305.GA18268@eos.sc1.parodius.com> References: <C921285B-91B3-43BD-B440-128EC60CE815@eng.oar.net> <20071217054305.GA18268@eos.sc1.parodius.com>
next in thread | previous in thread | raw e-mail | index | archive | help
Back to back test with no ethernet switch between two em interfaces, same result. The receiving side has been up > 1 day and exhibits the problem. These are also two different servers. The small gettimeofday() syscall tester also shows the same ~30 second pattern of high latency between syscalls. Receiver test application reports 3699 missed packets Sender netstat -i: (before test) em1 1500 <Link#2> 00:04:23:cf:51:b7 20 0 15975785 0 0 em1 1500 10.1/24 10.1.0.2 37 - 15975801 - - (after test) em1 1500 <Link#2> 00:04:23:cf:51:b7 22 0 25975822 0 0 em1 1500 10.1/24 10.1.0.2 39 - 25975838 - - total IP packets sent in during test = end - start 25975838-15975801 = 10000037 (expected, 1,000,000 packets test + overhead) Receiver netstat -i: (before test) em1 1500 <Link#2> 00:04:23:c4:cc:89 15975785 0 21 0 0 em1 1500 10.1/24 10.1.0.1 15969626 - 19 - - (after test) em1 1500 <Link#2> 00:04:23:c4:cc:89 25975822 0 23 0 0 em1 1500 10.1/24 10.1.0.1 25965964 - 21 - - total ethernet frames received during test = end - start 25975822-15975785 = 10000037 (as expected) total IP packets processed during test = end - start 25965964-15969626 = 9996338 (expecting 10000037) Missed packets = expected - received 10000037-9996338 = 3699 netstat -i accounts for the 3699 missed packets also reported by the application Looking closer at the tester output again shows the periodic ~30 second windows of packet loss. There's a second problem here in that packets are just disappearing before they make it to ip_input(), or there's a dropped packets counter I've not found yet. I can provide remote access to anyone who wants to take a look, this is very easy to duplicate. The ~ 1 day uptime before the behavior surfaces is not making this easy to isolate. -- mark On Dec 17, 2007, at 12:43 AM, Jeremy Chadwick wrote: > On Mon, Dec 17, 2007 at 12:21:43AM -0500, Mark Fullmer wrote: >> While trying to diagnose a packet loss problem in a RELENG_6 >> snapshot dated >> November 8, 2007 it looks like I've stumbled across a broken >> driver or >> kernel routine which stops interrupt processing long enough to >> severly >> degrade network performance every 30.99 seconds. >> >> Packets appear to make it as far as ether_input() then get lost. > > Are you sure this isn't being caused by something the switch is doing, > such as MAC/ARP cache clearing or LACP? I'm just speculating, but it > would be worthwhile to remove the switch from the picture (crossover > cable to the rescue). > > I know that at least in the case of fxp(4) and em(4), Jack Vogel does > some through testing of throughput using a professional/high-end > packet > generator (some piece of hardware, I forget the name...) > > -- > | Jeremy Chadwick jdc at > parodius.com | > | Parodius Networking http:// > www.parodius.com/ | > | UNIX Systems Administrator Mountain View, > CA, USA | > | Making life hard for others since 1977. PGP: > 4BD6C0CB | > > _______________________________________________ > freebsd-stable@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-stable > To unsubscribe, send any mail to "freebsd-stable- > unsubscribe@freebsd.org" >
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?F62928F4-BA8D-4C0F-9C5E-217D70E36321>