Date: Mon, 08 Apr 2002 15:55:11 -0700 From: Terry Lambert <tlambert2@mindspring.com> To: Andrew Gallatin <gallatin@cs.duke.edu> Cc: freebsd-hackers@freebsd.org Subject: Re: performance of mbufs vs contig buffers? Message-ID: <3CB21FCF.6B018811@mindspring.com> References: <15538.5971.620626.548508@grasshopper.cs.duke.edu>
next in thread | previous in thread | raw e-mail | index | archive | help
Andrew Gallatin wrote: > After updating the firmware on our our 2 gigabit nic to allow enough > scatter entries per packet to stock the 9K (jumbo frame) receive > rings with cluster mubfs rather than contigmalloc'ed buffers(*), I > noticed a dramatic performance decrease: netperf TCP_STREAM > performance dropped from 1.6Gb/sec to 1.2Gb/sec. [ ... ] > Is it possible that my problems are being caused by cache misses in > on cluster mbufs occuring when copying out to userspace as another > packet is being DMA'ed up? I'd thought that since the cache line size > is 32 bytes, I'd be pretty much equally screwed either way. [ ... ] > Does anybody have any ideas why contig malloc'ed buffers are so much > quicker? Instrument m_pullup(), and see how much it's being called in both cases. Probably you are seeing the 2 byte misalignment of the TCP payload in the the ethernet packet. My other guess would be that the clusters you are dealing with are non-contiguous. This has both scatter/gather implications, and cache-line implications when using them. Having thought about this problem before, I think that what you probably need is to chunk the buffers up, and treat them as M_EXT type mbufs (e.g. go with contigmalloc). To be able to use "generic" mbufs for this, what's really needed is the ability to have variable size mbufs. At the very least, I think a single mbuf should be of a size so that the MTU fits inside it. Fixing this would be a large amount of work, and the gain is uncertain. You can get a minor idea of the available gain by looking at the Tigon II firmware changes to use page based buffer allocations, per Bill Paul & Co.. -- Terry To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-hackers" in the body of the message
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?3CB21FCF.6B018811>