From owner-freebsd-fs@freebsd.org Sat Dec 31 14:59:18 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id CE3BCC99E35 for ; Sat, 31 Dec 2016 14:59:18 +0000 (UTC) (envelope-from julien@perdition.city) Received: from relay-b01.edpnet.be (relay-b01.edpnet.be [212.71.1.221]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "edpnet.email", Issuer "Go Daddy Secure Certificate Authority - G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 812701C81 for ; Sat, 31 Dec 2016 14:59:18 +0000 (UTC) (envelope-from julien@perdition.city) X-ASG-Debug-ID: 1483195173-0a7ff5320b678620001-dE2xID Received: from mordor.lan ([213.219.148.14]) by relay-b01.edpnet.be with ESMTP id ApuKWn9EeUmyR531 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Sat, 31 Dec 2016 15:39:34 +0100 (CET) X-Barracuda-Envelope-From: julien@perdition.city X-Barracuda-Effective-Source-IP: UNKNOWN[213.219.148.14] X-Barracuda-Apparent-Source-IP: 213.219.148.14 Date: Sat, 31 Dec 2016 15:39:32 +0100 From: Julien Cigar To: freebsd-fs@FreeBSD.org Subject: nfsuserd + jails mbufs leak ? Message-ID: <20161231143932.GS15696@mordor.lan> X-ASG-Orig-Subj: nfsuserd + jails mbufs leak ? MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="6pbY/KU4ayLo+qis" Content-Disposition: inline User-Agent: Mutt/1.7.1 (2016-10-04) X-Barracuda-Connect: UNKNOWN[213.219.148.14] X-Barracuda-Start-Time: 1483195173 X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384 X-Barracuda-URL: https://212.71.1.221:443/cgi-mod/mark.cgi X-Barracuda-Scan-Msg-Size: 1736 X-Virus-Scanned: by bsmtpd at edpnet.be X-Barracuda-BRTS-Status: 1 X-Barracuda-Bayes: INNOCENT GLOBAL 0.5000 1.0000 0.0000 X-Barracuda-Spam-Score: 0.00 X-Barracuda-Spam-Status: No, SCORE=0.00 using global scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=6.0 tests= X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.35471 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 31 Dec 2016 14:59:18 -0000 --6pbY/KU4ayLo+qis Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable Hello, I just upgraded a bunch of machines from 10.0 to 10.3. Those machines have a lot of jails with NFS shares (mounted on the HOST), for example: jcigar@HOST:~/ > mount|grep -i 'filer' filer.prod.lan:/pictures/collections on /usr/jails/www2/filer/pictures/collection (nfs, read-only, nfsv4acls) filer.prod.lan:/webapps on /usr/jails/www2/filer/webapps (nfs, nfsv4acls) filer.prod.lan:/documents on /usr/jails/www2/filer/documents (nfs, read-only, nfsv4acls) filer.prod.lan:/apache on /usr/jails/www2/filer/apache (nfs, nfsv4acls) filer.prod.lan:/geoserver on /usr/jails/java2/filer/geoserver (nfs, nfsv4acls) HOSTs and NFS server have vfs.nfsd.server_min_nfsvers=3D4 vfs.nfsd.server_max_nfsvers=3D4 (but it's probably not related to the problem) and nfsuserd is runnings on each HOST (with no special ${nfsuserd_flags}) On the hosts since I upgraded to 10.3 I'm seeing tons of: Dec 31 14:29:33 duvel nfsuserd:[675]: req from ip=3D0xc0a80a21 port=3D618 It is not clear to me yet why I'm getting this, but from what I=20 understand it's because requests are not coming from 127.0.0.1 but from the jail ip (192.168.10.x in my case).. Parallel to this I'm observing a constant increase of mbufs usage so that at some point in time I'm getting an exhaustion of mbufs: [zone: mbuf] kern.ipc.nmbufs limit reached =2E. and the HOST needs a reboot :( If I'm stopping nfsuserd I don't see any mbuf leakage anymore Any idea ? :) Thanks, Julien --=20 Julien Cigar Belgian Biodiversity Platform (http://www.biodiversity.be) PGP fingerprint: EEF9 F697 4B68 D275 7B11 6A25 B2BB 3710 A204 23C0 No trees were killed in the creation of this message. However, many electrons were terribly inconvenienced. --6pbY/KU4ayLo+qis Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQIzBAABCgAdFiEE7vn2l0to0nV7EWolsrs3EKIEI8AFAlhnwyEACgkQsrs3EKIE I8C2tQ//eLltljnBIZiCLHV5j99toF6UO/lmYINWxl5WVxG2UENSo0AVNHhW92b6 yZmwg/KG3EKoT8DShSe0LRg2LA2GSrDYk1x/s7szfdZo8Og5QtOvUaKGm6cpk/OK XP4gNXrTWCiQsmRxUkdtvjsjqnXQ2UOjZ5Bbt0Q2yvpiMiV1tdYLYLIyRn0fu6Ag 60zcGKKfP1utI+kpD+QcplUEhyxAuDU/Ze2sOaTFvtlnmFkV2Gyt0/L43TWkQbUU FMNiTreD4913U2hI5CSao7uDjtC6ZMQLp8vG+ktLfJllo7wU4DeIFo/T7rgdTO73 Rwg1Zc0JVGlvWuwgVHsB90QVMJ+NzsAxWTAvuelwxUeAzDzotm/fMvAdEmPUgODl GnDJh3qISmfOotyQti3GZE7dEhsAzD9ULBAxFBs8u1OEsgRL/MFXNO9bEC8A0RLs IhsvUEu8OkKickH/rWnuUD5+k60gJlj3DObniWDCRxCNR2iwCsmCBbl10uSfaI3C EYbE+8oU6YdntnR12Fx4BHSgxPqOI4DZEuNXEKGynR40GFJKa5PrONJnUPxRf/Z1 NNdBsuAcTjvTDNPZpaMhp47dCafud/zMlNr1nlEheI0/NnKrzCB5THMq5ITORUIP HSybirKOAktMzgUdGiIHBZvKV1zReNP3DjSbqvvzWxSza0ZFX/M= =vAfY -----END PGP SIGNATURE----- --6pbY/KU4ayLo+qis--