From nobody Sat Jul 16 14:57:01 2022 X-Original-To: freebsd-stable@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4LlWpy3RX6z4Wm5R for ; Sat, 16 Jul 2022 15:09:30 +0000 (UTC) (envelope-from pmc@citylink.dinoex.sub.org) Received: from uucp.dinoex.org (uucp.dinoex.org [IPv6:2a0b:f840::12]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "uucp.dinoex.sub.de", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4LlWpx26Ryz3Y1h for ; Sat, 16 Jul 2022 15:09:28 +0000 (UTC) (envelope-from pmc@citylink.dinoex.sub.org) Received: from uucp.dinoex.org (uucp.dinoex.org [IPv6:2a0b:f840:0:0:0:0:0:12]) by uucp.dinoex.org (8.17.1/8.17.1) with ESMTPS id 26GF94tY010059 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NO); Sat, 16 Jul 2022 17:09:05 +0200 (CEST) (envelope-from pmc@citylink.dinoex.sub.org) Received: (from uucp@localhost) by uucp.dinoex.org (8.17.1/8.17.1/Submit) with UUCP id 26GF94LT010058; Sat, 16 Jul 2022 17:09:04 +0200 (CEST) (envelope-from pmc@citylink.dinoex.sub.org) Received: from gate.intra.daemon.contact (gate-e [192.168.98.2]) by citylink.dinoex.sub.de (8.16.1/8.16.1) with ESMTP id 26GEvvQU061769; Sat, 16 Jul 2022 16:57:57 +0200 (CEST) (envelope-from peter@gate.intra.daemon.contact) Received: from gate.intra.daemon.contact (gate-e [192.168.98.2]) by gate.intra.daemon.contact (8.16.1/8.16.1) with ESMTPS id 26GEv1eL061608 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NO); Sat, 16 Jul 2022 16:57:02 +0200 (CEST) (envelope-from peter@gate.intra.daemon.contact) Received: (from peter@localhost) by gate.intra.daemon.contact (8.16.1/8.16.1/Submit) id 26GEv154061597; Sat, 16 Jul 2022 16:57:01 +0200 (CEST) (envelope-from peter) Date: Sat, 16 Jul 2022 16:57:01 +0200 From: Peter To: Rick Macklem Cc: "freebsd-stable@freebsd.org" Subject: Re: nfs stalls client: nfsrv_cache_session: no session Message-ID: References: List-Id: Production branch of FreeBSD source code List-Archive: https://lists.freebsd.org/archives/freebsd-stable List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-stable@freebsd.org X-BeenThere: freebsd-stable@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-Milter: Spamilter (Reciever: uucp.dinoex.org; Sender-ip: 0:0:2a0b:f840::; Sender-helo: uucp.dinoex.org;) X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (uucp.dinoex.org [IPv6:2a0b:f840:0:0:0:0:0:12]); Sat, 16 Jul 2022 17:09:07 +0200 (CEST) X-Rspamd-Queue-Id: 4LlWpx26Ryz3Y1h X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of pmc@citylink.dinoex.sub.org designates 2a0b:f840::12 as permitted sender) smtp.mailfrom=pmc@citylink.dinoex.sub.org X-Spamd-Result: default: False [-3.30 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-0.999]; R_SPF_ALLOW(-0.20)[+mx]; MIME_GOOD(-0.10)[text/plain]; RCPT_COUNT_TWO(0.00)[2]; MLMMJ_DEST(0.00)[freebsd-stable@freebsd.org]; R_DKIM_NA(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; TO_DN_EQ_ADDR_SOME(0.00)[]; DMARC_NA(0.00)[sub.org]; ASN(0.00)[asn:205376, ipnet:2a0b:f840::/32, country:DE]; ARC_NA(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_COUNT_FIVE(0.00)[5]; TO_DN_SOME(0.00)[]; FROM_HAS_DN(0.00)[]; RCVD_TLS_LAST(0.00)[] X-ThisMailContainsUnwantedMimeParts: N On Sat, Jul 16, 2022 at 01:43:11PM +0000, Rick Macklem wrote: ! Peter wrote: ! > Hija, ! > I have a problem with NFSv4: ! > ! > The configuration: ! > Server Rel. 13.1-RC2 ! > nfs_server_enable="YES" ! > nfs_server_flags="-u -t --minthreads 2 --maxthreads 20 -h ..." ! Allowing it to go down to 2 threads is very low. I've never even ! tried to run a server with less than 4 threads. Since kernel threads ! don't generate much overhead, I'd suggest replacing the ! minthreads/maxthreads with "-n 32" for a very small server. Okay. This normally used for building ports, quarterly or so, and writes go to a local filesystem. Only when something doesn't build and I start manual tests, then the default /usr/ports nfs share might get the writes. With Rel. 13 I think I should move the whole thing to virt-9p filesystems, on occasion. ! > mountd_enable="YES" ! > mountd_flags="-S -p 803 -h ..." ! > rpc_lockd_enable="YES" ! > rpc_lockd_flags="-h ..." ! > rpc_statd_enable="YES" ! > rpc_statd_flags="-h ..." ! > rpcbind_enable="YES" ! > rpcbind_flags="-h ..." ! > nfsv4_server_enable="YES" ! > sysctl vfs.nfs.enable_uidtostring=1 ! > sysctl vfs.nfsd.enable_stringtouid=1 ! > ! > Client bhyve Rel. 13.1-RELEASE on the same system ! > nfs_client_enable="YES" ! > nfs_access_cache="600" ! > nfs_bufpackets="32" ! > nfscbd_enable="YES" ! > ! > Mount-options: nfsv4,readahead=1,rw,async ! I would expect the behaviour you are seeing for "intr" and/or "soft" ! mounts, but since you are not using those, I don't know how you ! broke the session? (10052 is NFSERR_BADSESSION) ! You might want to do "nfsstat -m" on the client to see what options ! were actually negotiated for the mount and then check that neither ! "soft" nor "intr" are there. I killed that client after I found no way out. Normally it looks like this: nfsv4,minorversion=2,tcp,resvport,nconnect=1,hard,cto,sec=sys,acdirmin=3, acdirmax=60,acregmin=5,acregmax=60,nametimeo=60,negnametimeo=60,rsize=65536, wsize=65536,readdirsize=65536,readahead=1,wcommitsize=16777216,timeout=120, retrans=2147483647 ! I suspect that the recovery thread in the client (called "nfscl") is ! somehow wedged and cannot do the recovery from the bad session, These were present, two of them - I remember seeing "D" flag, but this seem to always be the case. ! If increasing the number of nfsd threads in the server doesn't resolve ! the problem, I'd guess it is some network weirdness caused by how ! the bhyve instance is networked to its host. (I always use bridging ! for bhyve instances and do NFS mounts, but I don't work those ! mounts hard.) They attach to a netgraph bridge: https://gitr.daemon.contact/sysup/tree/subr_virt.sh#n84 ! Btw, "umount -N " on the client will normally get rid ! of a hung mount, although it can take a couple of minutes to complete. Ups, I missed that! I only remembered -f, which didn't work. Thanks! PMc