From owner-freebsd-current@freebsd.org Sat May 27 22:20:51 2017 Return-Path: Delivered-To: freebsd-current@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 8562BD85201 for ; Sat, 27 May 2017 22:20:51 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from CAN01-TO1-obe.outbound.protection.outlook.com (mail-eopbgr670058.outbound.protection.outlook.com [40.107.67.58]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "Microsoft IT SSL SHA2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 3CF261F2A; Sat, 27 May 2017 22:20:50 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from YTXPR01MB0189.CANPRD01.PROD.OUTLOOK.COM (10.165.218.133) by YTXPR01MB0191.CANPRD01.PROD.OUTLOOK.COM (10.165.218.135) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256_P256) id 15.1.1124.9; Sat, 27 May 2017 22:20:48 +0000 Received: from YTXPR01MB0189.CANPRD01.PROD.OUTLOOK.COM ([10.165.218.133]) by YTXPR01MB0189.CANPRD01.PROD.OUTLOOK.COM ([10.165.218.133]) with mapi id 15.01.1124.015; Sat, 27 May 2017 22:20:48 +0000 From: Rick Macklem To: "freebsd-current@freebsd.org" CC: Andriy Gapon , "cem@freebsd.org" , "jeff@freebsd.org" , Ryan Stone Subject: Re: NFS client perf. degradation when SCHED_ULE is used (was when SMP enabled) Thread-Topic: NFS client perf. degradation when SCHED_ULE is used (was when SMP enabled) Thread-Index: AQHS1mmgxqkdxhAY5U2DZ8FjIm51zqIIvdIk Date: Sat, 27 May 2017 22:20:48 +0000 Message-ID: References: In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: FreeBSD.org; dkim=none (message not signed) header.d=none;FreeBSD.org; dmarc=none action=none header.from=uoguelph.ca; x-ms-publictraffictype: Email x-microsoft-exchange-diagnostics: 1; YTXPR01MB0191; 7:ukXOdVXaqXVm67Ucc3IrI7lD8mCP/a3msZIq6S+GPzbOuOBfmFK0MXgvjJgnoPz2IEdO93DOKoyZBNmwNKmUVTNSF4cnrfbbylwk5jvAB2An9U1QgzfPk7QgyRKoyMxkPgzFB4BSn4NqVcHUAHZprv9zq/+kqN7XOHPfu+k26XLnXs3yc+7aC8F4P7TMxg4lzev/Ol5B0KnqECcaUStrvr4BoluH8vggjsDqJiAZ86iAkJP79w+5vKzXOyswsZV5cmrscKf/ARroIleZiN5ydompmOpvw+t9BpCKITdnYvXOxGtix0G5PZouRSIUVAozuc5VmS0ETEaLDLrzSdufNA== x-ms-traffictypediagnostic: YTXPR01MB0191: x-ms-office365-filtering-correlation-id: 1976c55f-f1f3-4ce5-90df-08d4a54ea0b9 x-microsoft-antispam: UriScan:; BCL:0; PCL:0; RULEID:(22001)(2017030254075)(201703131423075)(201703031133081); SRVR:YTXPR01MB0191; x-microsoft-antispam-prvs: x-exchange-antispam-report-test: UriScan:; x-exchange-antispam-report-cfa-test: BCL:0; PCL:0; RULEID:(6040450)(2401047)(5005006)(8121501046)(93006095)(93001095)(10201501046)(3002001)(6041248)(201703131423075)(201702281529075)(201702281528075)(201703061421075)(201703061406153)(20161123558100)(20161123560025)(20161123555025)(20161123562025)(20161123564025)(6072148); SRVR:YTXPR01MB0191; BCL:0; PCL:0; RULEID:; SRVR:YTXPR01MB0191; x-forefront-prvs: 0320B28BE1 x-forefront-antispam-report: SFV:NSPM; SFS:(10009020)(6009001)(39410400002)(39840400002)(39400400002)(39450400003)(24454002)(8936002)(8676002)(2351001)(81166006)(2900100001)(305945005)(74316002)(74482002)(5660300001)(86362001)(3280700002)(2906002)(229853002)(122556002)(50986999)(76176999)(33656002)(54356999)(77096006)(450100002)(25786009)(2501003)(4326008)(189998001)(14454004)(2950100002)(6916009)(3660700001)(5640700003)(7696004)(54906002)(55016002)(53936002)(6506006)(9686003)(110136004)(38730400002)(478600001)(102836003)(6246003)(6436002); DIR:OUT; SFP:1101; SCL:1; SRVR:YTXPR01MB0191; H:YTXPR01MB0189.CANPRD01.PROD.OUTLOOK.COM; FPR:; SPF:None; MLV:sfv; LANG:en; spamdiagnosticoutput: 1:99 spamdiagnosticmetadata: NSPM Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: uoguelph.ca X-MS-Exchange-CrossTenant-originalarrivaltime: 27 May 2017 22:20:48.7380 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: be62a12b-2cad-49a1-a5fa-85f4f3156a7d X-MS-Exchange-Transport-CrossTenantHeadersStamped: YTXPR01MB0191 X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 27 May 2017 22:20:51 -0000 I wrote: >To briefly summarize the previous post related to perf. degradation when r= unning a >recent kernel... >- kernel build running 1yr old kernel took 100minutes >- same kernel build running recent kernel 148minutes >(ie. Almost a 50% degradation.) >As noted in the last post, I got rid of most of the degradation by disabli= ng SMP. > >- same kernel build running recent kernel with SCHED_4BSD 104minutes > After poking at this some more, it appears that r312426 is the main cause o= f this degradation. Doing SMP enabled test runs using SCHED_ULE running the recent kernel, I go= t: - recent kernel (as above) 148minutes - with r312426 reverted 122minutes - with the "obvious change" mentioned in r312426's commit message, using (flags & SW_TYPE_MASK) =3D=3D SWT_RELINQUISH instead of (flag & SWT_RELI= NQUISH) 121minutes So, I'd say either reverting the patch or replacing it with the "obvious ch= ange" mentioned in the commit message will at least mostly fix the problem. I actually suspect that setting "preempt" for SWT_IDLE and/or SWT_IWAIT is = what is needed to be the pre-r312426 performance, since those are the ones that SWT_RELINQUISH doesn't match. (There is also SWT_PREEMPT, but that was handled by the r312426 patch.) I also tested: ((flags & SW_PREEMPT) !=3D 0 || (flags & SW_TYPE_MASK) =3D=3D SWT_IDLE = || (flags & SW_TYPE_MASK) =3D=3D SWT_IWAIT) and it also resulted in 121minutes I still get better perf. from SCHED_4BSD of 104minutes, but I usually see b= etter performance for SCHED_4BSD, so I think this is expected. I know nothing about SCHED_ULE, so I don't think I can do more, unless some= one wants me to try a different patch? rick=