Date: Sun, 21 Nov 2004 21:21:10 -0800 From: Sean McNeil <sean@mcneil.com> To: Matthew Dillon <dillon@apollo.backplane.com> Cc: Shunsuke SHINOMIYA <shino@fornext.org> Subject: Re: Re[4]: serious networking (em) performance (ggate and NFS) problem Message-ID: <1101100870.16086.16.camel@server.mcneil.com> In-Reply-To: <200411220442.iAM4gduM053764@apollo.backplane.com> References: <20041121205158.45CE.SHINO@fornext.org> <200411220038.iAM0c7JQ052589@apollo.backplane.com> <20041122104527.5204.SHINO@fornext.org> <200411220442.iAM4gduM053764@apollo.backplane.com>
next in thread | previous in thread | raw e-mail | index | archive | help
--=-ak8TFvWEot9EfT6qjJ5l Content-Type: text/plain Content-Transfer-Encoding: quoted-printable On Sun, 2004-11-21 at 20:42 -0800, Matthew Dillon wrote: > : Yes, I knew that adjusting TCP window size is important to use up a lin= k. > : However I wanted to show adjusting the parameters of Interrupt > : Moderation affects network performance. > : > : And I think a packet loss was occured by enabled Interrupt Moderation. > : The mechanism of a packet loss in this case is not cleared, but I think > : inappropriate TCP window size is not the only reason. >=20 > Packet loss is not likely, at least not for the contrived tests we > are doing because GiGE links have hardware flow control (I'm fairly > sure). I have to disagree. Packet loss is likely according to some of my tests. With the re driver, no change except placing a 100BT setup with no packet loss to a gigE setup (both linksys switches) will cause serious packet loss at 20Mbps data rates. I have discovered the only way to get good performance with no packet loss was to 1) Remove interrupt moderation 2) defrag each mbuf that comes in to the driver. Doing both of these, I get excellent performance without any packet loss. All my testing has been with UDP packets, however, and nothing was checked for TCP. > One could calculate the worst case small-packet build up in the recei= ve > ring. I'm not sure what the minimum pad for GiGE is, but lets say it= 's > 64 bytes. Then the packet rate would be around 1.9M pps or 244 packe= ts > per interrupt at a moderation frequency of 8000 hz. The ring is 256 > packets. But, don't forget the hardware flow control! The switch > has some buffering too. >=20 > hmm... me thinks I now understand why 8000 was chosen as the default = :-) >=20 > I would say that this means packet loss due to the interrupt moderati= on > is highly unlikely, at least in theory, but if one were paranoid one > might want to use a higher moderation frequency, say 16000 hz, to be = sure. Your calculations are based on the mbufs being a particular size, no? What happens if they are seriously defragmented? Is this what you mean by "small-packet"? Are you assuming the mbufs are as small as they get? How small can they go? 1 byte? 1 MTU? Increasing the interrupt moderation frequency worked on the re driver, but it only made it marginally better. Even without moderation, however, I could lose packets without m_defrag. I suspect that there is something in the higher level layers that is causing the packet loss. I have no explanation why m_defrag makes such a big difference for me, but it does. I also have no idea why a 20Mbps UDP stream can lose data over gigE phy and not lose anything over 100BT... without the above mentioned changes that is. --=-ak8TFvWEot9EfT6qjJ5l Content-Type: application/pgp-signature; name=signature.asc Content-Description: This is a digitally signed message part -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.6 (FreeBSD) iD8DBQBBoXdGyQsGN30uGE4RAmV0AKCW9MeUkuqtCuvyfD9J5ryRMCW2dACfbGA3 /SNWm1+5hpl2j+Z2dcIQ9QM= =dDLI -----END PGP SIGNATURE----- --=-ak8TFvWEot9EfT6qjJ5l--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?1101100870.16086.16.camel>