From owner-freebsd-net@FreeBSD.ORG Mon Dec 15 19:23:25 2008 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 4805D1065670 for ; Mon, 15 Dec 2008 19:23:25 +0000 (UTC) (envelope-from rwatson@FreeBSD.org) Received: from cyrus.watson.org (cyrus.watson.org [65.122.17.42]) by mx1.freebsd.org (Postfix) with ESMTP id 0874F8FC08 for ; Mon, 15 Dec 2008 19:23:25 +0000 (UTC) (envelope-from rwatson@FreeBSD.org) Received: from fledge.watson.org (fledge.watson.org [65.122.17.41]) by cyrus.watson.org (Postfix) with ESMTP id 1EB3546B1A; Mon, 15 Dec 2008 14:23:24 -0500 (EST) Date: Mon, 15 Dec 2008 19:23:23 +0000 (GMT) From: Robert Watson X-X-Sender: robert@fledge.watson.org To: Yony Yossef In-Reply-To: Message-ID: References: <000501c95ebe$709bddb0$220f000a@mtl.com> User-Agent: Alpine 1.10 (BSF 962 2008-03-14) MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: freebsd-net@freebsd.org Subject: Re: About netstat -m: What is "mbuf+clusters out of packet secondary zone in use" ? X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 15 Dec 2008 19:23:25 -0000 On Mon, 15 Dec 2008, Robert Watson wrote: > On Mon, 15 Dec 2008, Yony Yossef wrote: > >> I'm testing an Ethernet driver on FreeBSD 6.3. >> >> Running netstat -m during an ethernt stress test I see that the >> "mbuf+clusters out of packet secondary zone in use" number is growing >> gradualy. Problem is it never goes down after I stop the test, so it's >> pushing the "mbufs in use" up until the following stress test iterations >> reach the OS limit. What does this number mean? I realized that I didn't quite answer your question here, so to be more specific: the FreeBSD mbuf allocator has a number of slab allocator zones it can draw from, depending on the type of request. These are enumerated in the vmstat -z output: robert@fledge:~> vmstat -z | head -1 ; vmstat -z | grep -i mbuf ITEM SIZE LIMIT USED FREE REQUESTS FAILURES mbuf_packet: 256, 0, 262, 222, 202448978, 0 mbuf: 256, 0, 9, 527, 686757049, 0 mbuf_cluster: 2048, 25600, 484, 304, 4017957, 0 mbuf_jumbo_pagesize: 4096, 12800, 7, 298, 18924376, 0 mbuf_jumbo_9k: 9216, 6400, 0, 0, 0, 0 mbuf_jumbo_16k: 16384, 3200, 0, 0, 0, 0 mbuf_ext_refcnt: 4, 0, 0, 406, 8873247, 0 The 'mbuf' zone allocates simple mbufs from a cache. The various cluster zones allocate cluster storage of various sizes, from the 2k default cluster size up to various jumbo sizes used when sending large amounts of data or when jumbograms are configured for a network interface that supports them. The mbuf_packet (mbuf+cluster) zone is a special zone allocating mbufs with pre-attached clusters, which was an optimization that seemed to help a lot a few years ago. In particular, you don't have to enter the memory allocator twice in order to find an mbuf with a cluster. There are some downsides, not least more complicated book-keeping, and some issues with how to account for and manage memory across the various caches. Notice that all mbuf_packet FREE (cached) clusters are billed as used clusters for the mbuf_cluster zone, as while UMA knows that mbuf_packet counts as a regular mbuf allocation, it doesn't know about the cluster hooked off it; netstat -m adjusts the reported cluster allocation for this before printing stats. Recently, we've been discussing moving to a variable-size mbuf scheme supporting large mbuf sizes with embedded storage, which seems to improve memory efficient quite a bit. There are some downsides -- one issue is that it somewhat complicates reference-counted cluster use (although not much); another is that data is no longer page-aligned which will make page-flipping less convenient on the receiver. Currently, if the hardware supports header-spitting, you can imagine the data going fairly easily into appropriately sized clusters (especially if the hardware supports LRO directly), and then flipping the pages into user memory on receive. With mbufs stuck on the front end of the page, not so much. Robert N M Watson Computer Laboratory University of Cambridge > > It seems likely that one of two things is happening: > > (1) a leak of mbuf + clusters > (2) a bug in stats on mbuf + clusters > > Could you paste the output of "vmstat -z | grep -i mbuf" into an e-mail? > That's the underlying vm stats from the zone used to generate netstat -m > output, and might shed more light on things. > > Robert N M Watson > Computer Laboratory > University of Cambridge > >> >> >> 506391/126009/632400 mbufs in use (current/cache/total) >> 141035/121109/262144/262144 mbuf clusters in use (current/cache/total/max) >> 131054/610 mbuf+clusters out of packet secondary zone in use >> (current/cache) >> >> >> Thanks >> Yony >> _______________________________________________ >> freebsd-net@freebsd.org mailing list >> http://lists.freebsd.org/mailman/listinfo/freebsd-net >> To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" >> > _______________________________________________ > freebsd-net@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" >