From owner-freebsd-fs@freebsd.org Sat Dec 31 22:34:03 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 80291C99EAB for ; Sat, 31 Dec 2016 22:34:03 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from NAM02-CY1-obe.outbound.protection.outlook.com (mail-cys01nam02on0073.outbound.protection.outlook.com [104.47.37.73]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "Microsoft IT SSL SHA2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 382CD109C for ; Sat, 31 Dec 2016 22:34:02 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from YTXPR01MB0189.CANPRD01.PROD.OUTLOOK.COM (10.165.218.133) by YTXPR01MB0189.CANPRD01.PROD.OUTLOOK.COM (10.165.218.133) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P384) id 15.1.817.10; Sat, 31 Dec 2016 22:01:18 +0000 Received: from YTXPR01MB0189.CANPRD01.PROD.OUTLOOK.COM ([10.165.218.133]) by YTXPR01MB0189.CANPRD01.PROD.OUTLOOK.COM ([10.165.218.133]) with mapi id 15.01.0817.009; Sat, 31 Dec 2016 22:01:17 +0000 From: Rick Macklem To: Julien Cigar , "freebsd-fs@FreeBSD.org" Subject: Re: nfsuserd + jails mbufs leak ? Thread-Topic: nfsuserd + jails mbufs leak ? Thread-Index: AQHSY3aEeZOV3aKrOkKpZJz6LX9A86EimXdV Date: Sat, 31 Dec 2016 22:01:17 +0000 Message-ID: References: <20161231143932.GS15696@mordor.lan> In-Reply-To: <20161231143932.GS15696@mordor.lan> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: spf=none (sender IP is ) smtp.mailfrom=rmacklem@uoguelph.ca; x-ms-office365-filtering-correlation-id: 95c695a4-304b-4085-7ec9-08d431c88c16 x-microsoft-antispam: UriScan:;BCL:0;PCL:0;RULEID:(22001);SRVR:YTXPR01MB0189; x-microsoft-exchange-diagnostics: 1; YTXPR01MB0189; 7:i1YLTMOUd33rNU7FXUDD4Bib4jDE4h5ZBVrSkWzbsrIwiglIVp/Wl96rsgnpYwYjogaK3gWsIU8A3Ud+z8NJw930JZlmFxOQ0Y1HDciQ3h5a1xqTCKyQ29Tinh0cC+tfWXeWDQ2YOtyI8gU+x5eJzXSCQr0VQkz4bRwW0nzCBoeWdZSGZZom2ZuMMY8lr3zVuDwPrVokqZXJu+zvkFSPsOHCIsWoCljOKiodYhIunXQdZVux+PsSfiNCHIkop5MMN9DOhVD9TlVHukK1vpO4/h0gh3w6yzdZShJ9WYh5N10MJPuk9HVpoLk8yxJY6+eFbYd3WutvKROs4gbpVOt2tabhHnXuy+E5XNTaBEsniwZ/QuJdIulODcflLkvuEHB8aHD2ibsj2/asWxBI2TmwogIqQpKR+tUaVUWZ3/CNJhf0LfapmYavwLCgvDlggq8jeAtp2hm4jfrHF1OhQSJiIg== x-microsoft-antispam-prvs: x-exchange-antispam-report-test: UriScan:; x-exchange-antispam-report-cfa-test: BCL:0; PCL:0; RULEID:(6040375)(2401047)(8121501046)(5005006)(3002001)(10201501046)(6041248)(20161123564025)(20161123562025)(20161123555025)(20161123560025)(6072148); SRVR:YTXPR01MB0189; BCL:0; PCL:0; RULEID:; SRVR:YTXPR01MB0189; x-forefront-prvs: 0173C6D4D5 x-forefront-antispam-report: SFV:NSPM; SFS:(10009020)(6009001)(7916002)(39450400003)(199003)(24454002)(189002)(3660700001)(68736007)(81166006)(8936002)(3280700002)(81156014)(305945005)(74316002)(55016002)(106116001)(106356001)(105586002)(101416001)(9686002)(8676002)(2906002)(33656002)(2501003)(2900100001)(102836003)(2950100002)(97736004)(6436002)(189998001)(5001770100001)(107886002)(92566002)(74482002)(77096006)(7696004)(5660300001)(6506006)(229853002)(122556002)(54356999)(76176999)(50986999)(575784001)(86362001)(38730400001); DIR:OUT; SFP:1101; SCL:1; SRVR:YTXPR01MB0189; H:YTXPR01MB0189.CANPRD01.PROD.OUTLOOK.COM; FPR:; SPF:None; PTR:InfoNoRecords; A:1; MX:1; LANG:en; received-spf: None (protection.outlook.com: uoguelph.ca does not designate permitted sender hosts) spamdiagnosticoutput: 1:99 spamdiagnosticmetadata: NSPM Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: uoguelph.ca X-MS-Exchange-CrossTenant-originalarrivaltime: 31 Dec 2016 22:01:17.3472 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: be62a12b-2cad-49a1-a5fa-85f4f3156a7d X-MS-Exchange-Transport-CrossTenantHeadersStamped: YTXPR01MB0189 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 31 Dec 2016 22:34:03 -0000 Julien Cigar wrote: >I just upgraded a bunch of machines from 10.0 to 10.3. Those machines >have a lot of jails with NFS shares (mounted on the HOST), for example: [stuff snipped] >On the hosts since I upgraded to 10.3 I'm seeing tons of: >Dec 31 14:29:33 duvel nfsuserd:[675]: req from ip=3D0xc0a80a21 port=3D618 > >It is not clear to me yet why I'm getting this, but from what I >understand it's because requests are not coming from 127.0.0.1 but from >the jail ip (192.168.10.x in my case).. Parallel to this I'm observing a >constant increase of mbufs usage so that at some point in time I'm >getting an exhaustion of mbufs: Yes, nfsuserd only accepts upcalls from 127.0.0.1 (about the only time a reserved port# actually means anything). As such, it doesn't work when you have jails. I came up with a patch that switched nfsuserd to using AF_LOCAL, but the person testing this at the time had hangs, so I never committed it to head,= etc. (I don't know what the mbuf leak is, but since it is busted and useless, I = don't think fixing the leak is necessary.;-) Your choices are: 1 - As you mentioned, don't run nfsuserd. If you are using FreeBSD and/or L= inux clients, this should work once you set vfs.nfsd.enablestringtouid (or = whatever it is called) to 1. 2 - I had a patch that added an option to nfsuserd that allowed specificati= on of a different ip address (unfortunately, I don't have this old patch han= dy, but might be able to find it at the end of this week. Email if you want me= to find it.) rick