Date: Sun, 10 Mar 2013 23:22:28 +0100 From: Andre Oppermann <andre@freebsd.org> To: Garrett Wollman <wollman@bimajority.org> Cc: freebsd-net@freebsd.org, Jack Vogel <jfvogel@gmail.com> Subject: Re: Limits on jumbo mbuf cluster allocation Message-ID: <513D07A4.4010503@freebsd.org> In-Reply-To: <20796.8842.691923.899276@hergotha.csail.mit.edu> References: <20793.36593.774795.720959@hergotha.csail.mit.edu> <51399926.6020201@freebsd.org> <CAFOYbc=x7U-s70KvcZJdrVP6v-On716qMi=HN1P2Kj%2Bd_K972A@mail.gmail.com> <20794.6692.191898.682241@hergotha.csail.mit.edu> <513A2887.2010408@freebsd.org> <CAFOYbc=7iROKzUwnB0fMR=ix8VFo%2BONfG=NX43jeF7jkp74JhQ@mail.gmail.com> <20796.8842.691923.899276@hergotha.csail.mit.edu>
next in thread | previous in thread | raw e-mail | index | archive | help
On 10.03.2013 07:04, Garrett Wollman wrote: > <<On Fri, 8 Mar 2013 12:13:28 -0800, Jack Vogel <jfvogel@gmail.com> said: > >> Yes, in the past the code was in this form, it should work fine Garrett, >> just make sure >> the 4K pool is large enough. > > [Andre Oppermann's patch:] >>> if (adapter->max_frame_size <= 2048) > adapter-> rx_mbuf_sz = MCLBYTES; >>> - else if (adapter->max_frame_size <= 4096) >>> + else > adapter-> rx_mbuf_sz = MJUMPAGESIZE; >>> - else if (adapter->max_frame_size <= 9216) >>> - adapter->rx_mbuf_sz = MJUM9BYTES; >>> - else >>> - adapter->rx_mbuf_sz = MJUM16BYTES; > > So I tried exactly this, and it certainly worked insofar as only 4k > clusters were allocated, but NFS performance went down precipitously > (to fewer than 100 ops/s where normally it would be doing 2000 > ops/s). I took a tcpdump while it was in this state, which I will try > to make some sense of when I get back to the office. (It wasn't > livelocked; in fact, the server was mostly idle, but responses would > take seconds rather than milliseconds -- assuming the client could > even successfully mount the server at all, which the Debian > automounter frequently refused to do.) This is very weird and unlikely to come from the 4k mbufs by itself considering they are in heavy use in the write() path. Such a high delay smells like an issue in either the driver dealing with multiple mbufs per packet or nfs having a problem with it. > I ended up reverting back to the old kernel (which I managed to lose > the sources for), and once I get my second server up, I will try to do > some more testing to see if I can identify the source of the problem. -- Andre
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?513D07A4.4010503>