From owner-freebsd-current@freebsd.org Wed May 24 20:40:03 2017 Return-Path: Delivered-To: freebsd-current@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 88CB8D8093E for ; Wed, 24 May 2017 20:40:03 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from CAN01-TO1-obe.outbound.protection.outlook.com (mail-eopbgr670076.outbound.protection.outlook.com [40.107.67.76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "Microsoft IT SSL SHA2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 44DB4100F for ; Wed, 24 May 2017 20:40:02 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from YTXPR01MB0189.CANPRD01.PROD.OUTLOOK.COM (10.165.218.133) by YTXPR01MB0189.CANPRD01.PROD.OUTLOOK.COM (10.165.218.133) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256_P256) id 15.1.1101.14; Wed, 24 May 2017 20:40:01 +0000 Received: from YTXPR01MB0189.CANPRD01.PROD.OUTLOOK.COM ([10.165.218.133]) by YTXPR01MB0189.CANPRD01.PROD.OUTLOOK.COM ([10.165.218.133]) with mapi id 15.01.1101.022; Wed, 24 May 2017 20:40:01 +0000 From: Rick Macklem To: "freebsd-current@freebsd.org" Subject: NFS client performance degradation when SMP enabled Thread-Topic: NFS client performance degradation when SMP enabled Thread-Index: AQHS1MwG2fxodwGtMEW4vi34w36XTw== Date: Wed, 24 May 2017 20:40:00 +0000 Message-ID: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: freebsd.org; dkim=none (message not signed) header.d=none;freebsd.org; dmarc=none action=none header.from=uoguelph.ca; x-ms-publictraffictype: Email x-microsoft-exchange-diagnostics: 1; YTXPR01MB0189; 7:THMuKrhZVtYzQ1l2WwDU6RzP2lOXTpPZVXXqVxqhDkT+5pnldiYwtVFhYtm/KeYjJDQG5kfgHfwGFSx63KgGITpbpdtSV4vPIrOYbxC4yJq0+V9SDwbZ9iGgyUjl0qwxPb9VZQzA2FngUvdwdO2gxL+UqNZ1vihWGGmm120F/XREt7ZIWvDpduZrS918Txp+VGWNy196h7Jf6LOzKy2IY0j3KeSsbXmWAKQ0teecHZ5dFu0XlgvyIlz64ilTa5EdvgqOqPh51ZzfqUamNylnXPpx6gcYAkarK0thEswSu/ZB0HmgKwOnmBdG9AiMYSpvOKxpc8zYAqGta92OloDUVQ== x-ms-traffictypediagnostic: YTXPR01MB0189: x-ms-office365-filtering-correlation-id: 8ddcbe9a-1121-4eb2-d64c-08d4a2e50cba x-microsoft-antispam: UriScan:; BCL:0; PCL:0; RULEID:(22001)(2017030254075)(201703131423075)(201703031133081)(201702281549075); SRVR:YTXPR01MB0189; x-microsoft-antispam-prvs: x-exchange-antispam-report-test: UriScan:(158342451672863); x-exchange-antispam-report-cfa-test: BCL:0; PCL:0; RULEID:(6040450)(2401047)(5005006)(8121501046)(10201501046)(3002001)(93006095)(93001095)(6041248)(20161123564025)(20161123555025)(201703131423075)(201702281529075)(201702281528075)(201703061421075)(201703061406153)(20161123558100)(20161123560025)(20161123562025)(6072148); SRVR:YTXPR01MB0189; BCL:0; PCL:0; RULEID:; SRVR:YTXPR01MB0189; x-forefront-prvs: 031763BCAF x-forefront-antispam-report: SFV:NSPM; SFS:(10009020)(6009001)(39400400002)(39840400002)(39850400002)(39410400002)(39450400003)(55016002)(6506006)(5640700003)(2351001)(81166006)(33656002)(25786009)(50986999)(8676002)(54356999)(5660300001)(189998001)(8936002)(86362001)(74482002)(6436002)(77096006)(102836003)(6916009)(2501003)(122556002)(7696004)(74316002)(305945005)(2906002)(2900100001)(3280700002)(478600001)(9686003)(110136004)(53936002)(3660700001)(38730400002); DIR:OUT; SFP:1101; SCL:1; SRVR:YTXPR01MB0189; H:YTXPR01MB0189.CANPRD01.PROD.OUTLOOK.COM; FPR:; SPF:None; MLV:sfv; LANG:en; spamdiagnosticoutput: 1:99 spamdiagnosticmetadata: NSPM Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: uoguelph.ca X-MS-Exchange-CrossTenant-originalarrivaltime: 24 May 2017 20:40:00.9633 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: be62a12b-2cad-49a1-a5fa-85f4f3156a7d X-MS-Exchange-Transport-CrossTenantHeadersStamped: YTXPR01MB0189 X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 24 May 2017 20:40:03 -0000 Without boring you with too much detail, I have been doing development/test= ing of pNFS stuff (mostly server side) on a 1 year old kernel (Apr. 12, 2016). When I recently carried the code across to a recent kernel, everything seem= ed to work, but performance was much slower. After some fiddling around, it appears to be on the NFS client side and not= hing in the NFS client code seemed to be causing it. (RPC counts were almost exactly th= e same, for example. I tried reverting r316532 and disabling vfs.nfs.use_buf_pager.= Neither made a significant difference.) I made most of the performance degradation go away by disabling SMP on the = client. Here's some elapsed times for kernel builds with everything the same except= for which kernel and SMP enabled/disabled (amd64 client machine). 1 year old kernel, SMP enabled - 100minutes recent kernel, SMP disabled - 113minutes recent kernel, SMP enabled - 148minutes (The builds were all of the same kernel sources. When I say "1 year old" vs= "recent" I am referring to which kernel was booted for the test run.) All I can think of is that some change in the last year has resulted in an = increase in something like interrupt latency or context switch latency that has caused = this? Anyone have an idea what this might be caused by or any tunables to fool wi= th beyond disabling SMP (which I suspect won't be a popular answer to "how to = fix slow NFS";-). I haven't yet tried fiddling with interrupt moderation on the net interface= , but the tests all used the same settings. rick=