From nobody Sun Mar 20 20:45:30 2022 X-Original-To: freebsd-stable@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id CC1D31A2E7C2 for ; Sun, 20 Mar 2022 20:45:50 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from CAN01-QB1-obe.outbound.protection.outlook.com (mail-eopbgr660051.outbound.protection.outlook.com [40.107.66.51]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "DigiCert Cloud Services CA-1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4KM8sT36S7z3MHL for ; Sun, 20 Mar 2022 20:45:49 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=EO7Ddejdzhp+wlNv0HCeunQYkbUMxbXfmOvAEwrFCujeSf4lqOHB0zsxoIK9BjIW8W6CoZWstVd8sPrQH48Jv8Jtpc9H86CfezXAmZI/nUmJFOCNn9EQ39WL/692sJtwzcYtjMCFLSkT5v6RjAHh0cLsRddPD6p7qVVOB7e1ZOXXvTVx5avdLMcQXjh3s69axfRvTNkMtuGX/qa+3gyEV5mQjVfps4UZhUi77aIcDHy3zaXUGuGjxNYqfEQ3V8cxGFsgB0MYzJ1rNX7GGncwFcJUK0kRgO5E5o9f5A7AN9m9fUmwXJ6Yy9IZ0jo838P9pnSCPQPoZdCD7aCCmE/2jw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=AsW5G1oL5nOZIX4J6ma+SqX+GZcY/yaeQJBKj+RfTys=; b=m0jJ+mkwEdsqowLI8NBIbJdbE/ppkf6ofLnBViB351Y+o+A+oddvkCZzd6AJWjlezqbPABHH8kie3nY0tLsGkoFuaACuNfQqOM411G6z63eVY8xg3O2AHxkX8D0Aa4cV0h0GnmRZgcc8NCnszMhKQqwArPwv9lVmymF5YAJ4LjATsO9ALnOnNwZNv2jrHhAwkHl7xEvkHfvhmxubKQHis4sJFlR6vZ84QAUfsiOuCDGNJ3jQwdfJmHxeUR5aiRCQ6XbpqtQaG1PipcWyALMY8rMc9JVeZXMuGwdI/U5kfwgU1DIQxfFPR++DKRjJdMngbJWOASi3f+gE1onJcVgd8w== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=uoguelph.ca; dmarc=pass action=none header.from=uoguelph.ca; dkim=pass header.d=uoguelph.ca; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=uoguelph.ca; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=AsW5G1oL5nOZIX4J6ma+SqX+GZcY/yaeQJBKj+RfTys=; b=V+tUHCT0bVUXTvag75miTWIBgLf/uSwXn7yMjkBlrH9cWVLwDNVjuMuv7BTeamvb0XBLATGCpLiSBCqVCgQO/07O+G/cARXgF60sZUeT821OJ5NcrLQxdjUOCy1fmx51qOV7XaznEaMkUJRLw4vfkRQpBHcZY7pWTG26yTwDK9wfvmmhf0P3fv2RtMuzrtP4wrV4eJ0jNDvxx/YA7JKysPD89taSoc/OniShlu4hFlCIBWZZ02nqBeQGwvQlKcpnscBPO+u7xXEWdPgFr1W7XYvcEXLgyVdK7MR/QOHS3IBDranUEdXF1aJ/TGlJe6EtQ2/XuEkL2t0BpUmhAhl2qw== Received: from YT2PR01MB9730.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:b01:de::14) by YQBPR0101MB4540.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:c01:17::14) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5081.14; Sun, 20 Mar 2022 20:45:42 +0000 Received: from YT2PR01MB9730.CANPRD01.PROD.OUTLOOK.COM ([fe80::39d7:e98e:fd00:2591]) by YT2PR01MB9730.CANPRD01.PROD.OUTLOOK.COM ([fe80::39d7:e98e:fd00:2591%6]) with mapi id 15.20.5081.022; Sun, 20 Mar 2022 20:45:31 +0000 From: Rick Macklem To: mike tancsa , Yoshihiro Ota CC: freebsd-stable Subject: Re: nfsd becomes slow when machine CPU usage is at or over 100% on STABLE/13 Thread-Topic: nfsd becomes slow when machine CPU usage is at or over 100% on STABLE/13 Thread-Index: AQHYM7vqHmVJAWfo2UaVD6ISojss76y3HTBUgBEa/mGAAAvigIAAiheu Date: Sun, 20 Mar 2022 20:45:30 +0000 Message-ID: References: <20220309034601.ea3135e31aec3ffb2623f145@j.email.ne.jp> <20220318211850.67b77d43b3a02043c3819bf3@j.email.ne.jp> <9b2935cd-5f70-d2a2-a328-004fa3e7b3f8@sentex.net> <9aee6d0f-e1c9-8c98-7352-7a0ff3e3d147@sentex.net> In-Reply-To: <9aee6d0f-e1c9-8c98-7352-7a0ff3e3d147@sentex.net> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: suggested_attachment_session_id: 437c476a-8cc2-a171-9935-1e9a73d4b2b4 x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: d640a784-e6d9-4d68-6dc8-08da0ab292d4 x-ms-traffictypediagnostic: YQBPR0101MB4540:EE_ x-microsoft-antispam-prvs: x-ms-exchange-senderadcheck: 1 x-ms-exchange-antispam-relay: 0 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: Wdb+C8nMOVr47vTrrqJ9o+FmsXDaqUn0SaWjOKMkDkVNzvq05QK8C8MiKcOM/VsvZloJ/HXNOAPcMpexBtXqv2QWFqCxieCbgZ7usvLPU0J4K6SBEVeONeo660+Dq9JoMd+PHChCMVQ0y6JVDg6Khts0k8IE/AldXN7avSEw46DXon1fRXU4bt5ukbhcVqLbOcClaJOWiLXsZ4YXpWGHzbuTiH9vTlE4NzP8xgiGOjmgFNKYWsaVtD3DlWz0bvP1TPmjPQ9A7/MZjhr2Hk0sBmFRJ/5KwetEXrEfEavGDRNQ30h24nE9gCIztH+nXFhGY/FC+x5e9qazHNLTXE7UTwNSt9MQdvWxYRT0hZ9CbRcoJ6/E33i7CAqX4sCT0G/s27RvNgA/TEEluF0+7xWuWrsvOA8A9cjSE7T/omxvU190wtHjClBQNBGZBdJxMIHn9TZ9V8B8JW/zvqsBcqf8zA39YBMzUQ0xN4GUXJ0H8hMj0qQJSHwJ93RfOIZYkI5qy1Clde5AWYO3KB5Eq3jB6yrsTlXzZtK280alLkNxVMhJrMVxoihXtVSuqeWtohc6sjGu8PKnIufCcG61NbpgGI7I3p9fIxWvFQHrtypnZweUN0ZbFSlM/U1G1wH9OQjViT3n6QZ088gAw6INIi3i4eps6Z69q+HTJambtIQWRFhdBq0bq5A2kaJX0J0rlOOPr5uWYvaf/rI4eMcNLTjWXQ== x-forefront-antispam-report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:YT2PR01MB9730.CANPRD01.PROD.OUTLOOK.COM;PTR:;CAT:NONE;SFS:(13230001)(366004)(110136005)(71200400001)(38100700002)(9686003)(38070700005)(186003)(55016003)(6506007)(7696005)(33656002)(53546011)(52536014)(86362001)(786003)(8936002)(83380400001)(66446008)(64756008)(76116006)(66556008)(508600001)(66476007)(66946007)(5660300002)(122000001)(91956017)(4326008)(8676002)(2906002)(316002);DIR:OUT;SFP:1101; x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: =?iso-8859-1?Q?/3VR6YS4JBgaRCHb90Y282fhV1WFhFCvNIz5skitT8OGv6g+IjB+ZWQDTW?= =?iso-8859-1?Q?rSgqQ4uvxIi2OO9bOdKiAW0NmFCggeT6lovtliSq0D0Sq2tFFoHkwZ1C7P?= =?iso-8859-1?Q?arACthJUTGamNbHOGdIf1XHNm+2hyKrOgnV2qIA3PMb2BoTSKOIQRns2mM?= =?iso-8859-1?Q?qZBVfE1KHbQHruNyCuVTZdW0eb0bzh5dncWVDOSA6N1CgSlZrohl2qnpxW?= =?iso-8859-1?Q?9jCWEVbum2pNUco/up2mQ3Kc7kQ2m0rkmkdbAAgfxtlsZl4+Ll8kvH6ViW?= =?iso-8859-1?Q?FdYsXqnaL28EMsYXtkrWV9hkJheJ3DOl7C45zNlwBbvR1xFpECxGkahsIQ?= =?iso-8859-1?Q?o+0MTZdwcv2EcQ/JRjtNAdBlsNXUD5gkFcyh3ozpO/P/u/iFNvR4jUVoxM?= =?iso-8859-1?Q?DseJwkVHEXvRUOqwTfOM9u9OnHyNWNc1Y9WWZyQK5WtWm4LoJUfBgOkPd6?= =?iso-8859-1?Q?X5fwCdOAizuSuHIPDySTrbQHHq2bBoZSdG/DNU8ZyhU1RKLJf8QYls5nU9?= =?iso-8859-1?Q?0TwATXL57S5rcDaDyY/ozhhRvJDXdy6MZDkpaiebhLOBj98P0HG7Be0atP?= =?iso-8859-1?Q?aW1tl3a39DMLmhmwTcw/1f+2fdi8Q9jUjzdjpUWbE+lP2AqbdS6S1MIW0P?= =?iso-8859-1?Q?DBVjhunxFjtfB0l8JaEKZm9BfLSQ5to4k9f+IHhsi3T2tbDyWAZ4mZuD9N?= =?iso-8859-1?Q?zM4sjhAsiHXEH97LTXnZvQpSXU4Eo/qjFxZHG64ur3yrqV/F19omhrblCj?= =?iso-8859-1?Q?AeGJUTI4nYATu3s5bhG9BhTuOkaC8mjqfNM8DMBexMQsIoTlTnonTRAMVY?= =?iso-8859-1?Q?D6/oIddriN1RizpWxFenUOY6kSBCA4L2SdoSnielrcOR5WCSJylMQs439I?= =?iso-8859-1?Q?QiBUkpxVJYtwzuRSed5g2hdFKRQ5EAmPB7P4xeK3AOyDyWS21uqMBwpKRL?= =?iso-8859-1?Q?/E4MecPdlBHZM6anFgdOiU6qhdQtruxtarOX5Ub0o0SXt/EpfCAFZi06UE?= =?iso-8859-1?Q?2o9kRK7OT+EXdGVUF7HIHfvvyiQddMHSa0lvOlYTF0M+D7qAcFC9t8sKLN?= =?iso-8859-1?Q?Xwm65bWmuU8u8+SvcjYwSvJjBLAAHHB4QwEz7eZ+SoCbg2vaMtFAhyvUL9?= =?iso-8859-1?Q?GLTNcl2cQ3FUWdmOJyhVWTYQupaqNHVA5vm0yi17i+QssIGA+vg0c6ofzq?= =?iso-8859-1?Q?Hhio/Kceuhv8WRstqMvu6sikmvuh4GrCM6qEX7KTJB14K78vqclrOt//rp?= =?iso-8859-1?Q?DC67io9ewDz5bss9R3kpJLzNmatpmgOQxUmdM2p66SCsSH98JrMz6MAtkO?= =?iso-8859-1?Q?eEOyquNVmQW9K2B65jjEYivF9Yso5U0K/H79/cgITnMqV/+j32l7i2xJVi?= =?iso-8859-1?Q?6G2RYlaXjTeomKxUZ1+OqnNKv/PAJ9rQ7B2rfbTzUg6izdPNgrL3WbU93s?= =?iso-8859-1?Q?sAKYJ2295bIyvHVzNDuHKs7Qn4zzjtZz7sOD5yg970yQhDGeDQ++vtI5ko?= =?iso-8859-1?Q?eh8HJHyj9QDdJWaHspsMoTpKZl1D4geaN4vn6/I6q8LBE2ppea5qlKNXyT?= =?iso-8859-1?Q?BGAoLfLWO/2PJ1XAV1BNYzziRivX?= Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable List-Id: Production branch of FreeBSD source code List-Archive: https://lists.freebsd.org/archives/freebsd-stable List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-stable@freebsd.org X-BeenThere: freebsd-stable@freebsd.org MIME-Version: 1.0 X-OriginatorOrg: uoguelph.ca X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: YT2PR01MB9730.CANPRD01.PROD.OUTLOOK.COM X-MS-Exchange-CrossTenant-Network-Message-Id: d640a784-e6d9-4d68-6dc8-08da0ab292d4 X-MS-Exchange-CrossTenant-originalarrivaltime: 20 Mar 2022 20:45:30.9415 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: be62a12b-2cad-49a1-a5fa-85f4f3156a7d X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: 126CFIo/jdpZ0qy+VvdR3NSdq5KdnALH0DDfnLlFadEBB2VLE/YR5ivygQkTKOKNZPNMGJV5qbTMm6Yka8k4dg== X-MS-Exchange-Transport-CrossTenantHeadersStamped: YQBPR0101MB4540 X-Rspamd-Queue-Id: 4KM8sT36S7z3MHL X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=uoguelph.ca header.s=selector2 header.b=V+tUHCT0; arc=pass ("microsoft.com:s=arcselector9901:i=1"); dmarc=pass (policy=none) header.from=uoguelph.ca; spf=pass (mx1.freebsd.org: domain of rmacklem@uoguelph.ca designates 40.107.66.51 as permitted sender) smtp.mailfrom=rmacklem@uoguelph.ca X-Spamd-Result: default: False [-4.00 / 15.00]; R_DKIM_ALLOW(-0.20)[uoguelph.ca:s=selector2]; FREEFALL_USER(0.00)[rmacklem]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; R_SPF_ALLOW(-0.20)[+ip4:40.107.0.0/16]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[text/plain]; NEURAL_SPAM_MEDIUM(1.00)[1.000]; DWL_DNSWL_LOW(-1.00)[uoguelph.ca:dkim]; RCVD_COUNT_THREE(0.00)[3]; TO_MATCH_ENVRCPT_SOME(0.00)[]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[uoguelph.ca:+]; DMARC_POLICY_ALLOW(-0.50)[uoguelph.ca,none]; RCVD_IN_DNSWL_NONE(0.00)[40.107.66.51:from]; NEURAL_HAM_SHORT(-1.00)[-1.000]; MLMMJ_DEST(0.00)[freebsd-stable]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:8075, ipnet:40.104.0.0/14, country:US]; ARC_ALLOW(-1.00)[microsoft.com:s=arcselector9901:i=1]; RWL_MAILSPIKE_POSSIBLE(0.00)[40.107.66.51:from] X-ThisMailContainsUnwantedMimeParts: N mike tancsa wrote:=0A= > On 3/20/2022 7:43 AM, mike tancsa wrote:=0A= >> On 3/18/2022 9:18 PM, Yoshihiro Ota wrote:=0A= >>> I had built several versions between releng/13.0 branch point to=0A= >>> stable/13 (before releng/13.1 was created) and all of them had such=0A= >> performance degrade.=0A= >>>=0A= >>> I started suspecting stable debug options and thus built releng/13.1=0A= >>> and tested.=0A= >>> I don't see NFS slowdown unlike stable/13.=0A= >>> releng/13.0 and releng/12.2 were also fine.=0A= >>=0A= >> Hi,=0A= >>=0A= >> I would think there is very little difference (if any) between=0A= >> releng/13.1 and stable/13 right now. Are you sure stable/13 suffers=0A= >> from this issue you are seeing ?=0A= The sources may be almost the same, but the build is not.=0A= See /usr/src/sys/conf/std.nodebug.=0A= =0A= I'm assuming his releng/13.1 build created a non-debug kernel.=0A= Debug kernels do spit out "expect reduced performance" if i recall=0A= correctly. It sounds like he found an example of this.=0A= =0A= rick=0A= =0A= >=0A= >=0A= These look to be the only files touched below.=0A= =0A= 0{cage}% git diff remotes/origin/releng/13.1..remotes/origin/stable/13 |=0A= grep '^\-\-'=0A= --- a/contrib/tzcode/stdtime/ctime.3=0A= --- a/lib/libc/gen/time.3=0A= --- a/lib/libcasper/services/cap_net/cap_net.c=0A= --- a/lib/libpfctl/libpfctl.c=0A= --- a/lib/libpfctl/libpfctl.h=0A= --- a/libexec/rc/rc.d/dumpon=0A= --- a/release/pkg_repos/release-dvd.conf=0A= --- a/sbin/devd/devd.conf=0A= --- a/sbin/ipf/common/ipf.h=0A= --- a/sbin/ipf/libipf/printactivenat.c=0A= --- a/sbin/ipf/libipf/printstate.c=0A= --- a/sbin/pfctl/pfctl.c=0A= --- a/sbin/pfctl/pfctl_optimize.c=0A= --- a/share/man/man4/Makefile=0A= --- a/share/man/man4/netmap.4=0A= --- /dev/null=0A= --- a/share/man/man4/vale.4=0A= --- a/share/man/man9/crypto_buffer.9=0A= --- a/stand/efi/libefi/efi_console.c=0A= --- a/stand/i386/libi386/vidconsole.c=0A= --- a/sys/arm64/include/pcpu.h=0A= --- a/sys/cddl/contrib/opensolaris/uts/common/dtrace/fasttrap.c=0A= --- a/sys/cddl/contrib/opensolaris/uts/intel/dtrace/fasttrap_isa.c=0A= --- a/sys/conf/newvers.sh=0A= --- a/sys/crypto/armv8/armv8_crypto.c=0A= --- a/sys/crypto/armv8/armv8_crypto.h=0A= --- a/sys/crypto/armv8/armv8_crypto_wrap.c=0A= --- a/sys/dev/netmap/netmap.c=0A= --- a/sys/dev/netmap/netmap_bdg.c=0A= --- a/sys/dev/netmap/netmap_kern.h=0A= --- a/sys/dev/netmap/netmap_vale.c=0A= --- a/sys/i386/i386/machdep.c=0A= --- a/sys/kern/kern_rmlock.c=0A= --- a/sys/kern/sys_process.c=0A= --- a/sys/kern/vfs_cache.c=0A= --- a/sys/kern/vfs_subr.c=0A= --- a/sys/modules/if_epair/Makefile=0A= --- a/sys/modules/linuxkpi/Makefile=0A= --- a/sys/net/if_epair.c=0A= --- a/sys/opencrypto/cryptodev.h=0A= --- a/sys/riscv/include/cpufunc.h=0A= --- a/sys/riscv/include/pmap.h=0A= --- a/sys/riscv/include/pte.h=0A= --- a/sys/riscv/include/riscvreg.h=0A= --- a/sys/riscv/include/vmparam.h=0A= --- a/sys/riscv/riscv/elf_machdep.c=0A= --- a/sys/riscv/riscv/locore.S=0A= --- a/sys/riscv/riscv/pmap.c=0A= --- a/sys/sys/param.h=0A= --- a/sys/x86/x86/mp_x86.c=0A= --- a/usr.bin/diff/pr.c=0A= --- a/usr.bin/touch/touch.c=0A= 0{cage}%=0A= =0A= =0A= =0A=