From owner-freebsd-current@freebsd.org Sat Jun 3 12:25:08 2017 Return-Path: Delivered-To: freebsd-current@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id BE8F3BF0BA9 for ; Sat, 3 Jun 2017 12:25:08 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from CAN01-TO1-obe.outbound.protection.outlook.com (mail-eopbgr670081.outbound.protection.outlook.com [40.107.67.81]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "Microsoft IT SSL SHA2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 5621B6ABC3; Sat, 3 Jun 2017 12:25:07 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from YTXPR01MB0189.CANPRD01.PROD.OUTLOOK.COM (10.165.218.133) by YTXPR01MB0190.CANPRD01.PROD.OUTLOOK.COM (10.165.218.134) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256_P256) id 15.1.1124.9; Sat, 3 Jun 2017 12:25:05 +0000 Received: from YTXPR01MB0189.CANPRD01.PROD.OUTLOOK.COM ([10.165.218.133]) by YTXPR01MB0189.CANPRD01.PROD.OUTLOOK.COM ([10.165.218.133]) with mapi id 15.01.1124.021; Sat, 3 Jun 2017 12:25:05 +0000 From: Rick Macklem To: Colin Percival , "freebsd-current@freebsd.org" CC: Andriy Gapon , "cem@freebsd.org" , "jeff@freebsd.org" , Ryan Stone Subject: Re: NFS client perf. degradation when SCHED_ULE is used (was when SMP enabled) Thread-Topic: NFS client perf. degradation when SCHED_ULE is used (was when SMP enabled) Thread-Index: AQHS1mmgxqkdxhAY5U2DZ8FjIm51zqIIvdIkgAFvXYaACHL5gIAAeZGt Date: Sat, 3 Jun 2017 12:25:04 +0000 Message-ID: References: , <0100015c6c549e3d-228427b4-2734-4ab5-9eef-88fc9ae71f9a-000000@email.amazonses.com> In-Reply-To: <0100015c6c549e3d-228427b4-2734-4ab5-9eef-88fc9ae71f9a-000000@email.amazonses.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: FreeBSD.org; dkim=none (message not signed) header.d=none;FreeBSD.org; dmarc=none action=none header.from=uoguelph.ca; x-ms-publictraffictype: Email x-microsoft-exchange-diagnostics: 1; YTXPR01MB0190; 7:PBYuF1+LjcoqmpWoEUHCpqz0QLUtXO++L6I6MQ6exsHJ3a0KHyQLhtnCCWqb1kAiCqR9kPM4qy3VHXwV5ElRea8urddil8cOrcJ/eMKVIHkCSwmEUKDSYiBoMMUWbstufRxMygx7XVH/PXkGKYzmfZ3eUvap0e6Rlt2t2Ckj2aodxaimqyzpzd1cb5CjDuaj4DWcZ7ILXSTWZXJYArJlsRh7m9KXfDNbUn9TIzJPbrZ+oR13WBKV3tVOaMecUPmeYEg9aXv1ws+3Z+OJAQcyl0bLTq/JMA+sn6UgmT6wGnrnPQ3ghH7UV79FaNajwLRWZiq7cy2YN8sDEyqhsLLN0g== x-ms-traffictypediagnostic: YTXPR01MB0190: x-ms-office365-filtering-correlation-id: 83af8286-1264-4527-d14e-08d4aa7b90a6 x-microsoft-antispam: UriScan:; BCL:0; PCL:0; RULEID:(22001)(2017030254075)(201703131423075)(201703031133081)(201702281549075); SRVR:YTXPR01MB0190; x-microsoft-antispam-prvs: x-exchange-antispam-report-test: UriScan:; x-exchange-antispam-report-cfa-test: BCL:0; PCL:0; RULEID:(100000700101)(100105000095)(100000701101)(100105300095)(100000702101)(100105100095)(6040450)(2401047)(5005006)(8121501046)(93006095)(93001095)(3002001)(100000703101)(100105400095)(10201501046)(6041248)(20161123555025)(20161123562025)(20161123564025)(20161123560025)(201703131423075)(201702281529075)(201702281528075)(201703061421075)(201703061406153)(20161123558100)(6072148)(100000704101)(100105200095)(100000705101)(100105500095); SRVR:YTXPR01MB0190; BCL:0; PCL:0; RULEID:(100000800101)(100110000095)(100000801101)(100110300095)(100000802101)(100110100095)(100000803101)(100110400095)(100000804101)(100110200095)(100000805101)(100110500095); SRVR:YTXPR01MB0190; x-forefront-prvs: 0327618309 x-forefront-antispam-report: SFV:NSPM; SFS:(10009020)(6009001)(39400400002)(39410400002)(39450400003)(39850400002)(39840400002)(24454002)(76176999)(2950100002)(53936002)(6246003)(74482002)(55016002)(189998001)(5660300001)(38730400002)(305945005)(74316002)(6506006)(77096006)(7696004)(102836003)(229853002)(6436002)(2900100001)(2906002)(3280700002)(14454004)(25786009)(3660700001)(4326008)(50986999)(54356999)(8676002)(122556002)(33656002)(9686003)(8936002)(54906002)(81166006)(2501003)(93886004)(86362001); DIR:OUT; SFP:1101; SCL:1; SRVR:YTXPR01MB0190; H:YTXPR01MB0189.CANPRD01.PROD.OUTLOOK.COM; FPR:; SPF:None; MLV:sfv; LANG:en; spamdiagnosticoutput: 1:99 spamdiagnosticmetadata: NSPM Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: uoguelph.ca X-MS-Exchange-CrossTenant-originalarrivaltime: 03 Jun 2017 12:25:04.8758 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: be62a12b-2cad-49a1-a5fa-85f4f3156a7d X-MS-Exchange-Transport-CrossTenantHeadersStamped: YTXPR01MB0190 X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 03 Jun 2017 12:25:08 -0000 Colin Percival wrote: >On 05/28/17 13:16, Rick Macklem wrote: >> cperciva@ is running a highly parallelized buuildworld and he sees bette= r >> slightly better elapsed times and much lower system CPU for SCHED_ULE. >> >> As such, I suspect it is the single threaded, processes mostly sleeping = waiting >> for I/O case that is broken. >> I suspect this is how many people use NFS, since a highly parallelized m= ake would >> not be a typical NFS client task, I think? > >Running `make buildworld -j36` on an EC2 "c4.8xlarge" instance (36 vCPUs, = 60 >GB RAM, 10 GbE) with GENERIC-NODEBUG, ULE has a slight edge over 4BSD: > >GENERIC-NODEBUG, SCHED_4BSD: > 1h14m12.48s real 6h25m44.59s user 1h4m53.42s sys > 1h15m25.48s real 6h25m12.20s user 1h4m34.23s sys > 1h13m34.02s real 6h25m14.44s user 1h4m09.55s sys > 1h13m44.04s real 6h25m08.60s user 1h4m40.21s sys > 1h14m59.69s real 6h25m53.13s user 1h4m55.20s sys > 1h14m24.00s real 6h24m59.29s user 1h5m37.31s sys > >GENERIC-NODEBUG, SCHED_ULE: > 1h13m00.61s real 6h02m47.59s user 26m45.89s sys > 1h12m30.18s real 6h01m39.97s user 26m16.45s sys > 1h13m08.43s real 6h01m46.94s user 26m39.20s sys > 1h12m18.94s real 6h02m26.80s user 27m39.71s sys > 1h13m21.38s real 6h00m46.13s user 27m14.96s sys > 1h12m01.80s real 6h02m24.48s user 27m18.37s sys > >Running `make buildworld -j2` on an E2 "m4.large" instance (2 vCPUs, 8 GB = RAM, >~ 500 Mbps network), 4BSD has a slight edge over ULE on real and sys >time but is slightly worse on user time: > >GENERIC-NODEBUG, SCHED_4BSD: > 6h29m25.17s real 7h2m56.02s user 14m52.63s sys > 6h29m36.82s real 7h2m58.19s user 15m14.21s sys > 6h28m27.61s real 7h1m38.24s user 14m56.91s sys > 6h27m05.42s real 7h1m38.57s user 15m04.31s sys > >GENERIC-NODEBUG, SCHED_ULE: > 6h34m19.41s real 6h59m43.99s user 18m8.62s sys > 6h33m55.08s real 6h58m44.91s user 18m4.31s sys > 6h34m49.68s real 6h56m03.58s user 17m49.83s sys > 6h35m22.14s real 6h58m12.62s user 17m52.05s sys Doing these test runs, but on the 36v CPU system would be closer to what I was testing. My tests do not use "-j" and run on an 8core chunk of real hardware. >Note that in both cases there is lots of idle time (although far more in t= he >-j36 case); this is partly due to a lack of parallelism in buildworld, but >largely due to having /usr/obj mounted on Amazon EFS. > >These differences all seem within the range which could result from cache >effects due to threads staying on one CPU rather than bouncing around; so >whatever Rick is tripping over, it doesn't seem to be affecting these test= s. Yep. Thanks for doing the testing, rick