From owner-freebsd-net@FreeBSD.ORG Mon Dec 16 14:35:10 2013 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id A8021A5E; Mon, 16 Dec 2013 14:35:10 +0000 (UTC) Received: from out4-smtp.messagingengine.com (out4-smtp.messagingengine.com [66.111.4.28]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 7AA8C14CD; Mon, 16 Dec 2013 14:35:10 +0000 (UTC) Received: from compute5.internal (compute5.nyi.mail.srv.osa [10.202.2.45]) by gateway1.nyi.mail.srv.osa (Postfix) with ESMTP id 97A6420E05; Mon, 16 Dec 2013 09:35:00 -0500 (EST) Received: from web3 ([10.202.2.213]) by compute5.internal (MEProxy); Mon, 16 Dec 2013 09:35:00 -0500 DKIM-Signature: v=1; a=rsa-sha1; c=relaxed/relaxed; d= messagingengine.com; h=message-id:from:to:mime-version :content-transfer-encoding:content-type:subject:date; s=smtpout; bh=A9n63p96CGowysF88oow4a/swNw=; b=hVw1JTEDnmWJTSFE4F7XFkulpYms HxACgrzQqal94kissY/gcxWGTkA/eB930FU9bwaRXJ09dTzBNu053Oxh8aBZqgVw tqWQ3LI38XF8Sjr5bHGJPIjmjUaf0MGOUGcgwYxBDkUIwJqe6RdogC/sbTfd309o GAN+qLsQnIw63lw= Received: by web3.nyi.mail.srv.osa (Postfix, from userid 99) id 52C83115EE8; Mon, 16 Dec 2013 09:35:00 -0500 (EST) Message-Id: <1387204500.12061.60192349.19EAE1B4@webmail.messagingengine.com> X-Sasl-Enc: 6jYsIDzJXFCcPwwYCIHUdBMQroS8kAtvrkoR0P5S3+Ga 1387204500 From: Mark Felder To: freebsd-net@freebsd.org, freebsd-stable@freebsd.org MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Type: text/plain X-Mailer: MessagingEngine.com Webmail Interface - ajax-7f24e732 Subject: 10.0-RC1: bad mbuf leak? Date: Mon, 16 Dec 2013 08:35:00 -0600 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 16 Dec 2013 14:35:10 -0000 Hi all, I think I'm experiencing a bad mbuf leak or something of the sort and I don't know how to diagnose this further. I have a machine at home that is mostly used for transcoding video for viewing on my TV via the multimedia/plexmediaserver port. This software runs in a jail and gets the actual files from my NAS via NFSv4. It's a pretty simple setup and sits idle unless I am watching TV. Between the 10.0-BETAs and the 10.0-RC1 did something network related that could affect mbufs change? Ever since I upgraded this machine to RC1 it has been "crashing", which I diagnosed as actually being an mbuf exhaustion. Raising the mbufs brings it back to life, and it does mention the exhaustion on the system console. Last night, for example, I rebooted the machine and it has been sitting mostly idle. I wake up this morning to see this: # vmstat -z ITEM SIZE LIMIT USED FREE REQ FAIL SLEEP mbuf_packet: 256, 6511095, 1023, 1727, 8322474, 0, 0 mbuf: 256, 6511095, 2811247, 1563,56000603,121933, 0 mbuf_cluster: 2048, 1017358, 2750, 0, 2750,2740, 0 mbuf_jumbo_page: 4096, 508679, 0, 152, 2831466, 137, 0 # netstat -m 812270/3290/2815560 mbufs in use (current/cache/total) 1023/1727/2750/1017358 mbuf clusters in use (current/cache/total/max) 1023/1727 mbuf+clusters out of packet secondary zone in use (current/cache) 0/152/152/508679 4k (page size) jumbo clusters in use (current/cache/total/max) 0/0/0/150719 9k jumbo clusters in use (current/cache/total/max) 0/0/0/84779 16k jumbo clusters in use (current/cache/total/max) 705113K/4884K/709998K bytes allocated to network (current/cache/total) 121933/2740/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters) 0/0/0 requests for mbufs delayed (mbufs/clusters/mbuf+clusters) 0/0/0 requests for jumbo clusters delayed (4k/9k/16k) 137/0/0 requests for jumbo clusters denied (4k/9k/16k) 0 requests for sfbufs denied 0 requests for sfbufs delayed 0 requests for I/O initiated by sendfile The network interface is em(4). Things I've tried: - restarting all software/services including the jail - down/up the network interface The only thing that works is rebooting. Also, the only possible "strange" part of this setup is that the NFS mounts used by the jail are not direct. They're actually nullfs mounted into the jail as I want access to them outside of the jail as well. Not sure if nullfs+nfs could do something this bizarre. If anyone has any hints on what I can do to track this down it would be appreciated.