Date: Mon, 15 Dec 2008 19:23:23 +0000 (GMT) From: Robert Watson <rwatson@FreeBSD.org> To: Yony Yossef <yonyossef.lists@gmail.com> Cc: freebsd-net@freebsd.org Subject: Re: About netstat -m: What is "mbuf+clusters out of packet secondary zone in use" ? Message-ID: <alpine.BSF.1.10.0812151916390.80196@fledge.watson.org> In-Reply-To: <alpine.BSF.1.10.0812151828470.80196@fledge.watson.org> References: <000501c95ebe$709bddb0$220f000a@mtl.com> <alpine.BSF.1.10.0812151828470.80196@fledge.watson.org>
next in thread | previous in thread | raw e-mail | index | archive | help
On Mon, 15 Dec 2008, Robert Watson wrote: > On Mon, 15 Dec 2008, Yony Yossef wrote: > >> I'm testing an Ethernet driver on FreeBSD 6.3. >> >> Running netstat -m during an ethernt stress test I see that the >> "mbuf+clusters out of packet secondary zone in use" number is growing >> gradualy. Problem is it never goes down after I stop the test, so it's >> pushing the "mbufs in use" up until the following stress test iterations >> reach the OS limit. What does this number mean? I realized that I didn't quite answer your question here, so to be more specific: the FreeBSD mbuf allocator has a number of slab allocator zones it can draw from, depending on the type of request. These are enumerated in the vmstat -z output: robert@fledge:~> vmstat -z | head -1 ; vmstat -z | grep -i mbuf ITEM SIZE LIMIT USED FREE REQUESTS FAILURES mbuf_packet: 256, 0, 262, 222, 202448978, 0 mbuf: 256, 0, 9, 527, 686757049, 0 mbuf_cluster: 2048, 25600, 484, 304, 4017957, 0 mbuf_jumbo_pagesize: 4096, 12800, 7, 298, 18924376, 0 mbuf_jumbo_9k: 9216, 6400, 0, 0, 0, 0 mbuf_jumbo_16k: 16384, 3200, 0, 0, 0, 0 mbuf_ext_refcnt: 4, 0, 0, 406, 8873247, 0 The 'mbuf' zone allocates simple mbufs from a cache. The various cluster zones allocate cluster storage of various sizes, from the 2k default cluster size up to various jumbo sizes used when sending large amounts of data or when jumbograms are configured for a network interface that supports them. The mbuf_packet (mbuf+cluster) zone is a special zone allocating mbufs with pre-attached clusters, which was an optimization that seemed to help a lot a few years ago. In particular, you don't have to enter the memory allocator twice in order to find an mbuf with a cluster. There are some downsides, not least more complicated book-keeping, and some issues with how to account for and manage memory across the various caches. Notice that all mbuf_packet FREE (cached) clusters are billed as used clusters for the mbuf_cluster zone, as while UMA knows that mbuf_packet counts as a regular mbuf allocation, it doesn't know about the cluster hooked off it; netstat -m adjusts the reported cluster allocation for this before printing stats. Recently, we've been discussing moving to a variable-size mbuf scheme supporting large mbuf sizes with embedded storage, which seems to improve memory efficient quite a bit. There are some downsides -- one issue is that it somewhat complicates reference-counted cluster use (although not much); another is that data is no longer page-aligned which will make page-flipping less convenient on the receiver. Currently, if the hardware supports header-spitting, you can imagine the data going fairly easily into appropriately sized clusters (especially if the hardware supports LRO directly), and then flipping the pages into user memory on receive. With mbufs stuck on the front end of the page, not so much. Robert N M Watson Computer Laboratory University of Cambridge > > It seems likely that one of two things is happening: > > (1) a leak of mbuf + clusters > (2) a bug in stats on mbuf + clusters > > Could you paste the output of "vmstat -z | grep -i mbuf" into an e-mail? > That's the underlying vm stats from the zone used to generate netstat -m > output, and might shed more light on things. > > Robert N M Watson > Computer Laboratory > University of Cambridge > >> >> >> 506391/126009/632400 mbufs in use (current/cache/total) >> 141035/121109/262144/262144 mbuf clusters in use (current/cache/total/max) >> 131054/610 mbuf+clusters out of packet secondary zone in use >> (current/cache) >> >> >> Thanks >> Yony >> _______________________________________________ >> freebsd-net@freebsd.org mailing list >> http://lists.freebsd.org/mailman/listinfo/freebsd-net >> To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" >> > _______________________________________________ > freebsd-net@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" >
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?alpine.BSF.1.10.0812151916390.80196>