From nobody Wed Mar 23 04:10:02 2022 X-Original-To: freebsd-stable@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id B750C1A32575 for ; Wed, 23 Mar 2022 04:09:04 +0000 (UTC) (envelope-from ota@j.email.ne.jp) Received: from mail11.asahi-net.or.jp (mail11.asahi-net.or.jp [202.224.55.51]) by mx1.freebsd.org (Postfix) with ESMTP id 4KNZby24d4z3GY2 for ; Wed, 23 Mar 2022 04:09:02 +0000 (UTC) (envelope-from ota@j.email.ne.jp) Received: from e20.advok.com (pool-96-225-64-148.nwrknj.fios.verizon.net [96.225.64.148]) (Authenticated sender: NR2Y-OOT) by mail11.asahi-net.or.jp (Postfix) with ESMTPSA id 201F02C782; Wed, 23 Mar 2022 13:08:53 +0900 (JST) Date: Wed, 23 Mar 2022 00:10:02 -0400 From: Yoshihiro Ota To: Rick Macklem Cc: mike tancsa , freebsd-stable Subject: Re: nfsd becomes slow when machine CPU usage is at or over 100% on STABLE/13 Message-Id: <20220323001002.35f7a7d660654757178f0518@j.email.ne.jp> In-Reply-To: References: <20220309034601.ea3135e31aec3ffb2623f145@j.email.ne.jp> <20220318211850.67b77d43b3a02043c3819bf3@j.email.ne.jp> <9b2935cd-5f70-d2a2-a328-004fa3e7b3f8@sentex.net> <9aee6d0f-e1c9-8c98-7352-7a0ff3e3d147@sentex.net> X-Mailer: Sylpheed 3.7.0 (GTK+ 2.24.33; amd64-portbld-freebsd12.2) List-Id: Production branch of FreeBSD source code List-Archive: https://lists.freebsd.org/archives/freebsd-stable List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-stable@freebsd.org X-BeenThere: freebsd-stable@freebsd.org Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: 4KNZby24d4z3GY2 X-Spamd-Bar: / Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of ota@j.email.ne.jp designates 202.224.55.51 as permitted sender) smtp.mailfrom=ota@j.email.ne.jp X-Spamd-Result: default: False [-0.70 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-0.98)[-0.975]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; MV_CASE(0.50)[]; R_SPF_ALLOW(-0.20)[+ip4:202.224.55.0/24]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[email.ne.jp]; NEURAL_HAM_LONG(-0.97)[-0.966]; NEURAL_SPAM_SHORT(0.94)[0.943]; TO_MATCH_ENVRCPT_SOME(0.00)[]; TO_DN_ALL(0.00)[]; MLMMJ_DEST(0.00)[freebsd-stable]; RCVD_NO_TLS_LAST(0.10)[]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:4685, ipnet:202.224.32.0/19, country:JP]; RCVD_COUNT_TWO(0.00)[2]; MID_RHS_MATCH_FROM(0.00)[]; RECEIVED_SPAMHAUS_PBL(0.00)[96.225.64.148:received] X-ThisMailContainsUnwantedMimeParts: N I've been building default kernel, that is GENERIC for releng and stable branches. I see GENERIC-NODEBUG on head but don't seem to find it on releng/ nor stable/ I grepped /usr/src/* for std.nodebug but don't seem have a match... I wonder how nodebug kicks in releng. I added "enable DDB" to i386 kernel but didn't touch amd64 kernel. Both amd64 and i386 looks okay with releng/13.1. Hiro On Sun, 20 Mar 2022 20:45:30 +0000 Rick Macklem wrote: > mike tancsa wrote: > > On 3/20/2022 7:43 AM, mike tancsa wrote: > >> On 3/18/2022 9:18 PM, Yoshihiro Ota wrote: > >>> I had built several versions between releng/13.0 branch point to > >>> stable/13 (before releng/13.1 was created) and all of them had such > >> performance degrade. > >>> > >>> I started suspecting stable debug options and thus built releng/13.1 > >>> and tested. > >>> I don't see NFS slowdown unlike stable/13. > >>> releng/13.0 and releng/12.2 were also fine. > >> > >> Hi, > >> > >> I would think there is very little difference (if any) between > >> releng/13.1 and stable/13 right now. Are you sure stable/13 suffers > >> from this issue you are seeing ? > The sources may be almost the same, but the build is not. > See /usr/src/sys/conf/std.nodebug. > > I'm assuming his releng/13.1 build created a non-debug kernel. > Debug kernels do spit out "expect reduced performance" if i recall > correctly. It sounds like he found an example of this. > > rick > > > > > > These look to be the only files touched below. > > 0{cage}% git diff remotes/origin/releng/13.1..remotes/origin/stable/13 | > grep '^\-\-' > --- a/contrib/tzcode/stdtime/ctime.3 > --- a/lib/libc/gen/time.3 > --- a/lib/libcasper/services/cap_net/cap_net.c > --- a/lib/libpfctl/libpfctl.c > --- a/lib/libpfctl/libpfctl.h > --- a/libexec/rc/rc.d/dumpon > --- a/release/pkg_repos/release-dvd.conf > --- a/sbin/devd/devd.conf > --- a/sbin/ipf/common/ipf.h > --- a/sbin/ipf/libipf/printactivenat.c > --- a/sbin/ipf/libipf/printstate.c > --- a/sbin/pfctl/pfctl.c > --- a/sbin/pfctl/pfctl_optimize.c > --- a/share/man/man4/Makefile > --- a/share/man/man4/netmap.4 > --- /dev/null > --- a/share/man/man4/vale.4 > --- a/share/man/man9/crypto_buffer.9 > --- a/stand/efi/libefi/efi_console.c > --- a/stand/i386/libi386/vidconsole.c > --- a/sys/arm64/include/pcpu.h > --- a/sys/cddl/contrib/opensolaris/uts/common/dtrace/fasttrap.c > --- a/sys/cddl/contrib/opensolaris/uts/intel/dtrace/fasttrap_isa.c > --- a/sys/conf/newvers.sh > --- a/sys/crypto/armv8/armv8_crypto.c > --- a/sys/crypto/armv8/armv8_crypto.h > --- a/sys/crypto/armv8/armv8_crypto_wrap.c > --- a/sys/dev/netmap/netmap.c > --- a/sys/dev/netmap/netmap_bdg.c > --- a/sys/dev/netmap/netmap_kern.h > --- a/sys/dev/netmap/netmap_vale.c > --- a/sys/i386/i386/machdep.c > --- a/sys/kern/kern_rmlock.c > --- a/sys/kern/sys_process.c > --- a/sys/kern/vfs_cache.c > --- a/sys/kern/vfs_subr.c > --- a/sys/modules/if_epair/Makefile > --- a/sys/modules/linuxkpi/Makefile > --- a/sys/net/if_epair.c > --- a/sys/opencrypto/cryptodev.h > --- a/sys/riscv/include/cpufunc.h > --- a/sys/riscv/include/pmap.h > --- a/sys/riscv/include/pte.h > --- a/sys/riscv/include/riscvreg.h > --- a/sys/riscv/include/vmparam.h > --- a/sys/riscv/riscv/elf_machdep.c > --- a/sys/riscv/riscv/locore.S > --- a/sys/riscv/riscv/pmap.c > --- a/sys/sys/param.h > --- a/sys/x86/x86/mp_x86.c > --- a/usr.bin/diff/pr.c > --- a/usr.bin/touch/touch.c > 0{cage}% > > >