From nobody Sun Mar 27 15:06:50 2022 X-Original-To: freebsd-stable@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 0AD9C1A4D76E for ; Sun, 27 Mar 2022 15:07:00 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from CAN01-QB1-obe.outbound.protection.outlook.com (mail-eopbgr660064.outbound.protection.outlook.com [40.107.66.64]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "DigiCert Cloud Services CA-1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4KRK1G3w0Cz4s3J for ; Sun, 27 Mar 2022 15:06:58 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=aMBnFPvvj06cTwDuRovAEypVqFMbbBj4rWkvaEVz66Arqb/lVp1gcNoSNQ3hOfZwCGR39Kmb0ys8q64h1Cfp0nnuOxMVdOXWIFu9c3FC1hGLKiyEXYdWdQVEuyPZFPjaz56IMLv53pKZlVerMO9eBlzCQ4VXO2grb8RPhbwKmO45P/he7VkyIzZuV27yZ1MPa8ZEmIlDr7CyZ0S9hgidALhXYBgosGrVxcHy/GwTEmzQQZgSAGKq6ZtUgElPzQQOjYetAz0O9h8LFkTs5YusoZCW7xyYUQQ4qRoHs3CplRicMKeDBuTuRwBHmiDml39Zt82NuyClCLOQiQpHtmW56Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=hg6Pxh2kdVIoHdIvCj7bgIM0KX1sqrrr0mDtAllhSSE=; b=oJb9H9YdydGT1fKA5sXJ6oWC3fgdh2jYO//7gcaTYaLucgv2jdoZGg50f4L8Yle5gN+sh45nq++aqJUHPqxJ5VfmGm4wqeAeP8sPPgykuBgBgnVDm7oJVPZSnfbBV9w1ROqY+Ru2nBFBbNTsCdLRGMLwiKRYSIrYY2X8ZnNpldXRnbORA6ZEZKHZDD0XW3A1e2nmbQUL2M70YrLnWqPSwxTq0cIhJCC64A8HvLoftJpATx9wW0OiV0qUdWFl50MFSymQPEvzeaDSdK/tNyOiP60K01eSSh6eBBduyaOwuEnCAa/YgV9iwdK0rqgB9ezD/ytBB1SPaIjzXgSJxtXypQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=uoguelph.ca; dmarc=pass action=none header.from=uoguelph.ca; dkim=pass header.d=uoguelph.ca; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=uoguelph.ca; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=hg6Pxh2kdVIoHdIvCj7bgIM0KX1sqrrr0mDtAllhSSE=; b=FNrklRuktFoyESUdR6GMzCs/g9RcXp3m549fW6J0lfFGFv+qUA7mSMpa42vqdVHv2OJj8keOlr7bpjr37t4MT4Y2Z+Ebku1ZJYQj5jiAXBzt0nv2i9H5LCZG5gb6/lQYi8FZWbgx9rfpP0aaobTqKUhv4j77HFSi65CHDgu3+YQTfhswmvJbxP5rK0LfLl6+J0LKepsoBz1iER8LyVHA5k7tOmHhrQ0vDVELW2HgAkJvUnmaWuZRGmg9022vQIRvfmnrL/nz/mRmg9JOoMDFU7RWH7AZOK7gkKZ9Cr1qJ89B3gNr+tkVJJKTr9pAh/cPWwXxPZYsZ9WlPuV+hZIf3g== Received: from YT2PR01MB9730.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:b01:de::14) by YT1PR01MB2971.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:b01:12::16) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5102.16; Sun, 27 Mar 2022 15:06:51 +0000 Received: from YT2PR01MB9730.CANPRD01.PROD.OUTLOOK.COM ([fe80::39d7:e98e:fd00:2591]) by YT2PR01MB9730.CANPRD01.PROD.OUTLOOK.COM ([fe80::39d7:e98e:fd00:2591%6]) with mapi id 15.20.5102.022; Sun, 27 Mar 2022 15:06:50 +0000 From: Rick Macklem To: Yoshihiro Ota CC: mike tancsa , freebsd-stable Subject: Re: nfsd becomes slow when machine CPU usage is at or over 100% on STABLE/13 Thread-Topic: nfsd becomes slow when machine CPU usage is at or over 100% on STABLE/13 Thread-Index: AQHYM7vqHmVJAWfo2UaVD6ISojss76y3HTBUgBEa/mGAAAvigIAAiheugAOiHQCABv/NpQ== Date: Sun, 27 Mar 2022 15:06:50 +0000 Message-ID: References: <20220309034601.ea3135e31aec3ffb2623f145@j.email.ne.jp> <20220318211850.67b77d43b3a02043c3819bf3@j.email.ne.jp> <9b2935cd-5f70-d2a2-a328-004fa3e7b3f8@sentex.net> <9aee6d0f-e1c9-8c98-7352-7a0ff3e3d147@sentex.net> <20220323001002.35f7a7d660654757178f0518@j.email.ne.jp> In-Reply-To: <20220323001002.35f7a7d660654757178f0518@j.email.ne.jp> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: suggested_attachment_session_id: e5fc65eb-76e9-d123-9463-eed23b595a3b x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: 3392482d-bbfa-4104-5d7c-08da10036c07 x-ms-traffictypediagnostic: YT1PR01MB2971:EE_ x-microsoft-antispam-prvs: x-ms-exchange-senderadcheck: 1 x-ms-exchange-antispam-relay: 0 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: 3Ixtz7AROhP4Hp4+qyQpMgRDnI7IclpCFLOIjm1D4xlRxMkVANMiUQt9INdQPlQVgTPy9bYMXKTHIhuKSksl40QrpHXcpuOqjpbT0mwL8NebZH0csdMTLI+TaitGFXGOPJZhJoc2gtrFvlZW4XWUqknLCl0HBkMhGnlYg3Gz+5H0oUSM2rJKXic55rIIuv5yQQEI6Rf5+PB0kI/IiGWuMkjMF2MXnFRlfX32ooQdDvZwkdVMmaDMxD3OszyCVZ86aiSA1SP/l7rgHjGKZBmjyIoOpqUd8+hzXfbFHCHAlhJZ8X2iWa4lDfgWOhedVTIf2QTj9mvlbBEmW2d+6LMOWsacmKXTcblLZf4K6gEK7dgmJMK9JH7zFam8UhPrfcwLY01mRyE1vt3bRQlT0RyEKSub/6hloAmNOXFOjpY3XdiHf889oQVYg6uHH6kf5ONbtwX8Rl9DVJpVflkMvi9QW802UN1JiAn19OddWfnOQNt2hb9YmDBD2mz1usgU+a9Be978cV4qvWQuAG12K0favdmFjMkG7bvOyWnHqyN4hqcGOWJ/AT78+w9B/N4T6H2x6Zz8shGjBjspJji8T7wHr4gJ3BRsnzqk8krCHyXYUjoY2PN6NWCM6j+OSnYimiZhbereNCVIAOB97gpa8YHEF27KGyEyXYnqvwVkP6UxHdsYpUCZFUtXG0RnOLAvdmolCUnYqkkAqNfFm3q/c+gERA== x-forefront-antispam-report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:YT2PR01MB9730.CANPRD01.PROD.OUTLOOK.COM;PTR:;CAT:NONE;SFS:(13230001)(366004)(91956017)(64756008)(66476007)(83380400001)(76116006)(66446008)(2906002)(316002)(786003)(9686003)(66556008)(52536014)(33656002)(66946007)(122000001)(5660300002)(54906003)(6916009)(71200400001)(6506007)(8936002)(86362001)(53546011)(38100700002)(7696005)(8676002)(186003)(38070700005)(55016003)(508600001)(4326008);DIR:OUT;SFP:1101; x-ms-exchange-antispam-messagedata-chunkcount: 2 x-ms-exchange-antispam-messagedata-0: =?iso-8859-1?Q?jsukog0NAf5sUG/9t7C0S/y3I25Kccllfec3Vrn1X8JddJCR1Z4vm8OnSC?= =?iso-8859-1?Q?KqRu6oJEZLMRL7wfCZ+ZRL6feGgxTohUkfOFeHtn3BxSik9g4EkFyeWEGn?= =?iso-8859-1?Q?WkXqcKDdm2NREUiP8rdRjcAFxbQiMpblS99N6K+Rj2kcjsjNVJv5RvSY1V?= =?iso-8859-1?Q?XYANmQq2jhNqcTewiEFPOvjTn7vq4MegVw/i3vWq8flimEFctrpUkLdVFE?= =?iso-8859-1?Q?6zNp5ZZ2sHKZDMinfgO0qYoxW/wDrJZOXwJmR45JM0xrCs+smqvQ+7Fgzw?= =?iso-8859-1?Q?F00z38K+xS8oEOGi3+U7xSqcAt6moYl+ceGjs8K9LmcvN0bvR2iB6v6mPA?= =?iso-8859-1?Q?uA7x69Na8HBk/kA4uy9P1tB/NKPJmpUcEdbK+Y85thYk/xXJ+Y+7Lp/exe?= =?iso-8859-1?Q?WOxhmUwHjggCHVdpzuXf8qMcBa1sZMBBvkD7tiFYfmbn7yE2Ok0W2QmikQ?= =?iso-8859-1?Q?ScgFpyIV2VEi9lI2v4gzwLKNPiTc0yQY7kHp9NCh86man1nXp1ZBUf7ZPo?= =?iso-8859-1?Q?5FrHGp+M0vAsenNtU9z7mihhjXSMbWAxL2RSI8+MC0DkBgDK2K5X+HlVg0?= =?iso-8859-1?Q?a8Wp2uuB0SbPk2WtFUErPdLZjJXAx9//3UaOLHgZR3PWNQbIh+j6sF/TNY?= =?iso-8859-1?Q?ptwaXytqNQuFU1lUX0ZW2yY3SR2j5owkE/7ntLLaphyGsN97AUYqk7PR5E?= =?iso-8859-1?Q?K9Axzx7VP2Lr56LUh1+TGWI0GaV+q6dj2bWtSkVQKMb9DpZ8VVFIR33nRw?= =?iso-8859-1?Q?RPASg1SgzBnpYkb3hS6g6u1OlXpyaJ7yxR/6Rh5X17qEQAs4iQ/dW0xLRo?= =?iso-8859-1?Q?3lllE9Y+1z9fddCeV8VjpqIYYRKnh6jkQzixEZUCnULY+rXBnNo9LxZwel?= =?iso-8859-1?Q?B7aYRUxRv1qwRyLw7UtgpnkiaFr49MKhTwhk26xXFP14pMamXjUWV9LOas?= =?iso-8859-1?Q?ZRttsAaBMAVAt2f/qzeZ2JReAL9f8JWEPpb2Bd406DTvhMHN0t9774JVg+?= =?iso-8859-1?Q?LJ81JJNs2LVkCZ2VUOr+Zl6+NJ872SmXkIbkkAEo3sW5RvlvVgiw5U+xfp?= =?iso-8859-1?Q?vt/4B6Z6WIs2NEQcK/j3ZORDbdGAyYmgyU81M4bbaAJQjXlb72CatM/oRF?= =?iso-8859-1?Q?D4+EOEaX2owKSTTjnb8CKnoitWjVAXuXEolOGCy5UlCMB+phUgk/4OPCcw?= =?iso-8859-1?Q?MMFelJKJi17IO/Q3nhWoa9Rjx/hoT1q8q8KDu2Nuwk3mHk4vOeWcNpPhIZ?= =?iso-8859-1?Q?+2UbvJWbPbjVWWbDO7UE5Fiz6WvjUpllEgJqIItpXAlmtebrK5s/L+6v2N?= =?iso-8859-1?Q?fm8PHdJiYUnyIw6DEW7qJbLoxzHzQUkZN5dqWG4BNq9xCoFbWCwzEwOMNT?= =?iso-8859-1?Q?RGDQpRTwJD9heXLqZQtaVHELv0B8WHPN+sw9HGvvwfRzNL3aFtLi2T9Qv4?= =?iso-8859-1?Q?QLsCUC874rq0hOQgD8yKWXc+yJFZO5Q5MTFEjRlhtsxe/TjyL5iGBtfSVz?= =?iso-8859-1?Q?RKRJ3ZOAnyYoTv5y1ioldx0I7etf26TZBuNnRunSTbjDj/3TtwjqwAIx16?= =?iso-8859-1?Q?+/uX15nHWZCfqh39bqtlUSThRno54xB/EMhCZjt9XBQ7jgJruECkzXzjsB?= =?iso-8859-1?Q?UIrpYXZ832LGI7mHAHrQUKXmoc5h/q8jPUHdod3DSvdh6ja6nMUUlQVVOa?= =?iso-8859-1?Q?2sNmnrIdeSaE7m9f1vu79t4AkpIPUyev3CQY9tnXePRNvpIbROPNNU17bh?= =?iso-8859-1?Q?XuH1mt4Ez1w5Pm7ryZ5OjaBlg6+hIvjxy8uzxNfZmFDICWuGo82IuBYi/E?= =?iso-8859-1?Q?GUS+D18AwyYs5ZyUUPB1fpg6H1WE83ABjs/yGIU1FN+GaW/+bU6yPFOL/J?= =?iso-8859-1?Q?6R?= x-ms-exchange-antispam-messagedata-1: mbPD7HXKlPg9wA== Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable List-Id: Production branch of FreeBSD source code List-Archive: https://lists.freebsd.org/archives/freebsd-stable List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-stable@freebsd.org X-BeenThere: freebsd-stable@freebsd.org MIME-Version: 1.0 X-OriginatorOrg: uoguelph.ca X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: YT2PR01MB9730.CANPRD01.PROD.OUTLOOK.COM X-MS-Exchange-CrossTenant-Network-Message-Id: 3392482d-bbfa-4104-5d7c-08da10036c07 X-MS-Exchange-CrossTenant-originalarrivaltime: 27 Mar 2022 15:06:50.8777 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: be62a12b-2cad-49a1-a5fa-85f4f3156a7d X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: vgQvSPfhuSA6Jv+qJ81z26APu0pFOaCsbWy6i6nxzrUKgxdzk8elH7H1W5EQD7BoaIXIq+/Ytb5CitduGJXbLA== X-MS-Exchange-Transport-CrossTenantHeadersStamped: YT1PR01MB2971 X-Rspamd-Queue-Id: 4KRK1G3w0Cz4s3J X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=uoguelph.ca header.s=selector2 header.b=FNrklRuk; arc=pass ("microsoft.com:s=arcselector9901:i=1"); dmarc=pass (policy=none) header.from=uoguelph.ca; spf=pass (mx1.freebsd.org: domain of rmacklem@uoguelph.ca designates 40.107.66.64 as permitted sender) smtp.mailfrom=rmacklem@uoguelph.ca X-Spamd-Result: default: False [-4.70 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; R_DKIM_ALLOW(-0.20)[uoguelph.ca:s=selector2]; FREEFALL_USER(0.00)[rmacklem]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; R_SPF_ALLOW(-0.20)[+ip4:40.107.0.0/16]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[text/plain]; NEURAL_SPAM_SHORT(0.30)[0.295]; DWL_DNSWL_LOW(-1.00)[uoguelph.ca:dkim]; RCVD_COUNT_THREE(0.00)[3]; TO_MATCH_ENVRCPT_SOME(0.00)[]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[uoguelph.ca:+]; DMARC_POLICY_ALLOW(-0.50)[uoguelph.ca,none]; RCVD_IN_DNSWL_NONE(0.00)[40.107.66.64:from]; MLMMJ_DEST(0.00)[freebsd-stable]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:8075, ipnet:40.104.0.0/14, country:US]; ARC_ALLOW(-1.00)[microsoft.com:s=arcselector9901:i=1]; RWL_MAILSPIKE_POSSIBLE(0.00)[40.107.66.64:from] X-ThisMailContainsUnwantedMimeParts: N Yoshihiro Ota wrote:=0A= > I've been building default kernel, that is GENERIC for releng and stable= =0A= > branches.=0A= >=0A= > I see GENERIC-NODEBUG on head but don't seem to find it on releng/=0A= > nor stable/=0A= > I grepped /usr/src/* for std.nodebug but don't seem have a match...=0A= > I wonder how nodebug kicks in releng.=0A= I do not know what magic is used so that the releng kernels build=0A= without debugging.=0A= =0A= However, if you see this line:=0A= WARNING: WITNESS option enabled, expect reduced performance.=0A= in the first screen when booting, you are running a kernel with debugging= =0A= built into it.=0A= =0A= rick=0A= =0A= I added "enable DDB" to i386 kernel but didn't touch amd64 kernel.=0A= Both amd64 and i386 looks okay with releng/13.1.=0A= =0A= Hiro=0A= =0A= On Sun, 20 Mar 2022 20:45:30 +0000=0A= Rick Macklem wrote:=0A= =0A= > mike tancsa wrote:=0A= > > On 3/20/2022 7:43 AM, mike tancsa wrote:=0A= > >> On 3/18/2022 9:18 PM, Yoshihiro Ota wrote:=0A= > >>> I had built several versions between releng/13.0 branch point to=0A= > >>> stable/13 (before releng/13.1 was created) and all of them had such= =0A= > >> performance degrade.=0A= > >>>=0A= > >>> I started suspecting stable debug options and thus built releng/13.1= =0A= > >>> and tested.=0A= > >>> I don't see NFS slowdown unlike stable/13.=0A= > >>> releng/13.0 and releng/12.2 were also fine.=0A= > >>=0A= > >> Hi,=0A= > >>=0A= > >> I would think there is very little difference (if any) between=0A= > >> releng/13.1 and stable/13 right now. Are you sure stable/13 suffers= =0A= > >> from this issue you are seeing ?=0A= > The sources may be almost the same, but the build is not.=0A= > See /usr/src/sys/conf/std.nodebug.=0A= >=0A= > I'm assuming his releng/13.1 build created a non-debug kernel.=0A= > Debug kernels do spit out "expect reduced performance" if i recall=0A= > correctly. It sounds like he found an example of this.=0A= >=0A= > rick=0A= >=0A= > >=0A= > >=0A= > These look to be the only files touched below.=0A= >=0A= > 0{cage}% git diff remotes/origin/releng/13.1..remotes/origin/stable/13 |= =0A= > grep '^\-\-'=0A= > --- a/contrib/tzcode/stdtime/ctime.3=0A= > --- a/lib/libc/gen/time.3=0A= > --- a/lib/libcasper/services/cap_net/cap_net.c=0A= > --- a/lib/libpfctl/libpfctl.c=0A= > --- a/lib/libpfctl/libpfctl.h=0A= > --- a/libexec/rc/rc.d/dumpon=0A= > --- a/release/pkg_repos/release-dvd.conf=0A= > --- a/sbin/devd/devd.conf=0A= > --- a/sbin/ipf/common/ipf.h=0A= > --- a/sbin/ipf/libipf/printactivenat.c=0A= > --- a/sbin/ipf/libipf/printstate.c=0A= > --- a/sbin/pfctl/pfctl.c=0A= > --- a/sbin/pfctl/pfctl_optimize.c=0A= > --- a/share/man/man4/Makefile=0A= > --- a/share/man/man4/netmap.4=0A= > --- /dev/null=0A= > --- a/share/man/man4/vale.4=0A= > --- a/share/man/man9/crypto_buffer.9=0A= > --- a/stand/efi/libefi/efi_console.c=0A= > --- a/stand/i386/libi386/vidconsole.c=0A= > --- a/sys/arm64/include/pcpu.h=0A= > --- a/sys/cddl/contrib/opensolaris/uts/common/dtrace/fasttrap.c=0A= > --- a/sys/cddl/contrib/opensolaris/uts/intel/dtrace/fasttrap_isa.c=0A= > --- a/sys/conf/newvers.sh=0A= > --- a/sys/crypto/armv8/armv8_crypto.c=0A= > --- a/sys/crypto/armv8/armv8_crypto.h=0A= > --- a/sys/crypto/armv8/armv8_crypto_wrap.c=0A= > --- a/sys/dev/netmap/netmap.c=0A= > --- a/sys/dev/netmap/netmap_bdg.c=0A= > --- a/sys/dev/netmap/netmap_kern.h=0A= > --- a/sys/dev/netmap/netmap_vale.c=0A= > --- a/sys/i386/i386/machdep.c=0A= > --- a/sys/kern/kern_rmlock.c=0A= > --- a/sys/kern/sys_process.c=0A= > --- a/sys/kern/vfs_cache.c=0A= > --- a/sys/kern/vfs_subr.c=0A= > --- a/sys/modules/if_epair/Makefile=0A= > --- a/sys/modules/linuxkpi/Makefile=0A= > --- a/sys/net/if_epair.c=0A= > --- a/sys/opencrypto/cryptodev.h=0A= > --- a/sys/riscv/include/cpufunc.h=0A= > --- a/sys/riscv/include/pmap.h=0A= > --- a/sys/riscv/include/pte.h=0A= > --- a/sys/riscv/include/riscvreg.h=0A= > --- a/sys/riscv/include/vmparam.h=0A= > --- a/sys/riscv/riscv/elf_machdep.c=0A= > --- a/sys/riscv/riscv/locore.S=0A= > --- a/sys/riscv/riscv/pmap.c=0A= > --- a/sys/sys/param.h=0A= > --- a/sys/x86/x86/mp_x86.c=0A= > --- a/usr.bin/diff/pr.c=0A= > --- a/usr.bin/touch/touch.c=0A= > 0{cage}%=0A= >=0A= >=0A= >=0A= =0A=