Date: Mon, 22 Jan 2007 21:50:49 +1100 (EST) From: Bruce Evans <bde@zeta.org.au> To: Max Laier <max@love2party.net> Cc: freebsd-net@FreeBSD.org Subject: Re: slow writes on nfs with bge devices Message-ID: <20070122211609.M28527@delplex.bde.org> In-Reply-To: <20070122175630.I7870@besplex.bde.org> References: <20070121155510.C23922@delplex.bde.org> <200701210809.27770.max@love2party.net> <20070121215054.C24780@delplex.bde.org> <200701212033.34914.max@love2party.net> <20070122140144.R7272@besplex.bde.org> <20070122175630.I7870@besplex.bde.org>
next in thread | previous in thread | raw e-mail | index | archive | help
On Mon, 22 Jan 2007, Bruce Evans wrote: > % ... > > Then it got to here with only relatively minor glitches: > > % 16:41:12.947085 IP phiplex.bde.org.1735978781 > besplex.bde.org.nfs: 1472 > write fh 1028,575312/94220 8192 bytes @ 0x2d6000 > % 16:41:12.947086 IP phiplex.bde.org > besplex.bde.org: udp > % 16:41:12.947087 IP phiplex.bde.org > besplex.bde.org: udp > % 16:41:12.947088 IP phiplex.bde.org > besplex.bde.org: udp > % 16:41:12.947089 IP phiplex.bde.org > besplex.bde.org: udp > % 16:41:12.947090 IP phiplex.bde.org > besplex.bde.org: udp > % 16:41:12.947861 IP phiplex.bde.org.1735978421 > besplex.bde.org.nfs: 1472 > write fh 1028,575312/94220 8192 bytes @ 0x6000 > > It's having problems writing 0x6000. This is a retry. Just for unreported > packet loss? Perhaps my switch or cable is the problem. Nevermind / 2. The switch or cable certain has problems. The only problem in FreeBSD or the bge driver, if any, may be that errors are not detected except as completely lost packets. Swapping cables gives the following strange behaviours: - fxp <-> bge (5705) works perfectly (at only 100 MBps of course) with a direct link (11.83e6 B/S), but not through the switch (cheap 1Gbps switch). - fxp <-> bge (5701) works better with a direct link, but imperfectly with the same cables that work for the 5705. nfs write speed increases from ~1 MB/S to ~8 MB/S. The link is autonegotiated correctly as 100baseTX full-duplex. With this configuration, errors seem to be detected correctly. nfs writes cause input errors (fxp -> bge at a an almost constant rate of almost exactly 1 per 500 packets. (500 may be significant since it the bge rx ring sizes is 512.) - on reconnecting the 5701 through the switch, no errors were detected when the link is autonegotited to 1000BaseTX full-duplex, but when it is manually configured to 100baseTX full-duplex, some input errors but at a highly variable rate not nearly as many as high 1 in 500. A few packet blasting tests on the 5701 didn't cause any input errors at 100baseTX. I haven't figured out the type of the input errors. No output errors were reported for any of this. - when the 5701 was in the server, exhaustive packet blasting tests at 1000basetX caused a large number of input errors whenever the rx interrupt moderation was configured to just a little more aggressively than the default, in condtions where the tx is very active. (The rx ring is supposed to have size 512, but had an effective size of only 20 under tx load, apparently due to misconfiguration of contention with tx resources.) Packets were just dropped on input. Bruce
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20070122211609.M28527>