From nobody Mon May 25 08:01:42 2026 X-Original-To: stable@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4gP7dJ5Sgwz6dsR6 for ; Mon, 25 May 2026 08:01:56 +0000 (UTC) (envelope-from danny@cs.huji.ac.il) Received: from kabab.cs.huji.ac.il (kabab.cs.huji.ac.il [132.65.116.210]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4gP7dH1KR1z3mZT for ; Mon, 25 May 2026 08:01:55 +0000 (UTC) (envelope-from danny@cs.huji.ac.il) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=cs.huji.ac.il header.s=57791128 header.b=uUsPe3CU; dkim=none ("invalid DKIM record") header.d=cse.huji.ac.il header.s=57791128 header.b=zNSj8oba; dmarc=pass (policy=none) header.from=huji.ac.il; spf=none (mx1.freebsd.org: domain of danny@cs.huji.ac.il has no SPF policy when checking 132.65.116.210) smtp.mailfrom=danny@cs.huji.ac.il DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=cs.huji.ac.il; s=57791128; h=References:To:Cc:In-Reply-To:Date:Subject:Mime-Version:Content-Type:Message-Id:From; bh=4PFqPcf7w1xza2l3lg3yWWCtkj0eJ45icalmEv3ohGs=; b=uUsPe3CUVGK88Mci/+7w3o3n+hOQmpNkRpujNbyXJXJN+JaEY3/aAapjZcb2FGrsN8U7xg6sPJmz7BmVRxFwfOLNdezwSu7TN4OcitS3OFz52GcUwY6l/h+KK2N5wbl4CEKtdKTOJBFzDzdGreuqA24fjiqu9xCQLgkk0B0SGWNlJa1l8xFjM8pE+Uc7I7UxiF1iPLggGF4qJW/FQjZhQlgCvaqGqYPrGOIPf5eTzQ5+ZnudD/ZmTm6aG7BMXyYl62obtpvDFAJ7nDGuGPUyaLdViIjPGaEr6ZHzSkkBj5GRm0utoe5Zq1EhJi6iTuTZHgbbjJy+6A4guBJfldFUAA==; DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=cse.huji.ac.il; s=57791128; h=References:To:Cc:In-Reply-To:Date:Subject:Mime-Version:Content-Type:Message-Id:From; bh=4PFqPcf7w1xza2l3lg3yWWCtkj0eJ45icalmEv3ohGs=; b=zNSj8obaom3IxyIsEeVhfuokszipKKxoalo/Do1FUi1EBqnQpMhpn4KqQm8LTj9sAKIEdAszK/esv3nulmDUSNHoHAnV3tRXCMZkWFo1z8srGzVzVbdygqvFKtF7Y000AY41ygLatMItksbnRMsAqfSYr20oXXtnW85mXgYW0x9YgDjXhIkVDB1JgQkKu3ELBMEbN2ScbTj6Z6LOwfsRC1Xnjllq92jNiPJh+Kf3K5/GR0PPtVPfbo6cC96caCZaUGhJCvUefa7rv1rteEU55ntD0BAEo4rgBJAFaMifvQg23rF09QpEgvvd/5CRyaO/N0wNZJpL6Z9kToFEjMyUqg==; Received: from bach.cs.huji.ac.il ([132.65.80.20] helo=smtpclient.apple) by kabab.cs.huji.ac.il with esmtp id 1wRQFt-000FgS-83; Mon, 25 May 2026 11:01:45 +0300 From: Daniel Braniss Message-Id: <0ED1A6F2-3454-4DA0-BF32-78016DB39C97@cs.huji.ac.il> Content-Type: multipart/alternative; boundary="Apple-Mail=_C13EB2B1-F371-4DE8-94CF-BA1F9319BF1C" List-Id: Production branch of FreeBSD source code List-Archive: https://lists.freebsd.org/archives/freebsd-stable List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: freebsd-stable@freebsd.org Sender: owner-freebsd-stable@FreeBSD.org List-Id: List-Post: List-Help: List-Subscribe: List-Unsubscribe: List-Owner: Precedence: list Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3696.120.41.1.10\)) Subject: Re: 15.1 diskless hangs Date: Mon, 25 May 2026 11:01:42 +0300 In-Reply-To: Cc: "Bjoern A. Zeeb" , Freebsd-stable List To: Rick Macklem References: <6D7CAB2A-7308-457F-9925-DBAD476B8E3F@cs.huji.ac.il> <42ns45s-94s2-32s2-710-41oprnp719q@mnoonqbm.arg> X-Mailer: Apple Mail (2.3696.120.41.1.10) X-Spamd-Result: default: False [-5.10 / 15.00]; DWL_DNSWL_MED(-2.00)[huji.ac.il:dkim]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-0.997]; DMARC_POLICY_ALLOW(-0.50)[huji.ac.il,none]; MV_CASE(0.50)[]; ONCE_RECEIVED(0.20)[]; R_DKIM_ALLOW(-0.20)[cs.huji.ac.il:s=57791128]; MIME_GOOD(-0.10)[multipart/alternative,text/plain,multipart/mixed]; DKIM_MIXED(0.00)[]; HAS_ATTACHMENT(0.00)[]; R_SPF_NA(0.00)[no SPF record]; RCVD_TLS_LAST(0.00)[]; FREEMAIL_TO(0.00)[gmail.com]; TO_DN_ALL(0.00)[]; R_DKIM_PERMFAIL(0.00)[cse.huji.ac.il:s=57791128]; DKIM_TRACE(0.00)[cs.huji.ac.il:+,cse.huji.ac.il:~]; RCVD_COUNT_ONE(0.00)[1]; MIME_TRACE(0.00)[0:+,1:+,2:+,3:~,4:~,5:~]; TO_MATCH_ENVRCPT_SOME(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; MLMMJ_DEST(0.00)[stable@freebsd.org]; TAGGED_RCPT(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; FREEFALL_USER(0.00)[danny]; ASN(0.00)[asn:378, ipnet:132.64.0.0/15, country:IL]; ARC_NA(0.00)[] X-Spamd-Bar: ----- X-Rspamd-Queue-Id: 4gP7dH1KR1z3mZT --Apple-Mail=_C13EB2B1-F371-4DE8-94CF-BA1F9319BF1C Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 > On 22 May 2026, at 15:38, Rick Macklem wrote: >=20 > On Thu, May 21, 2026 at 10:24=E2=80=AFPM Daniel Braniss = > wrote: >>=20 >>=20 >>=20 >>> On 21 May 2026, at 22:02, Bjoern A. Zeeb = wrote: >>>=20 >>> On Thu, 21 May 2026, Daniel Braniss wrote: >>>=20 >>>> have several bhives and my workstation, running diskless, the = server is also freebsd, >>>> and very often, they hang on nfs, (I assume trying to access the = root). >>>> having them boot locally, and everything else, ie. /usr/local, home = directory are nfs mounted without issues. >>>>=20 >>>> i=E2=80=99ll try and do a tcpdump but in the meantime any isights = are welcome. >>>=20 >>> When do they hang? During boot? Or during operation? >>=20 >> during normal operation. >>>=20 >>> If they hang on an interactive command try ^T and see. >>=20 >> since root is nfs mounted, and it=E2=80=99s hung, nothing but power = cycling works. >>>=20 >>> If tcpdump doesn't help much I'd start turning off checksum offloads = and the like >>> along the path (unclear if your host is your nfs server or not) to = see if that helps. >>=20 >> some more info: >> the server is running 14.3 (i don=E2=80=99t think this is relevant) >> the server us also providing /usr/local >> if the host is running with a local root, all is fine. >> virtual hosts, i.e bhive, also hang under similar configuration. >>=20 >> so now i'm running a tcpdump on the server, and will probably have = more info. > That should give you more information. A few things to note: > - 15.1 has the capability of using NFSv4 for root, but that requires = some > careful configuration. > - Assuming your root is NFSv3 (you'll see that in the packet capture > in wireshark), make sure you are not running nfsuserd(8) or gssd(8). only NFS3=20 > Both of these will try and look up user/group names in the = passwd/group > file and this can hang the system when they do the upcall from within > the NFS client (which would have to access the files). >=20 > rick so it hanged, im attaching the relevant tcpdump, what is weird, and I must have forgotten much,=20 nrnd is the server, chamsa is the client. it seems nrnd is requesting arp too often =E2=80=A6 is the TTL on the arp reseted when a packet is received, or it just times out? here is the abridged version: >=20 >>=20 >> thanks, >> danny >>=20 >>>=20 >>> /bz >>>=20 >>> -- >>> Bjoern A. Zeeb = r15:7 --Apple-Mail=_C13EB2B1-F371-4DE8-94CF-BA1F9319BF1C Content-Type: multipart/mixed; boundary="Apple-Mail=_71187085-3739-44C3-8D45-06941C6FC8ED" --Apple-Mail=_71187085-3739-44C3-8D45-06941C6FC8ED Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8

On 22 May 2026, at 15:38, Rick Macklem <rick.macklem@gmail.com> wrote:

On Thu, May 21, 2026 at 10:24=E2=80=AFPM Daniel Braniss = <danny@cs.huji.ac.il> wrote:



On 21 May 2026, at 22:02, Bjoern A. Zeeb <bzeeb-lists@lists.zabbadoz.net> wrote:
On Thu, 21 May 2026, Daniel Braniss wrote:

have several bhives and = my workstation, running diskless, the server is also freebsd,
and very often, they hang on nfs, (I assume trying to access = the root).
having them boot locally, and everything else, = ie. /usr/local, home directory are nfs mounted without issues.

i=E2=80=99ll try and do a tcpdump but in the = meantime any isights are welcome.

When do they hang?  During boot? Or during operation?

during normal operation.

If they = hang on an interactive command try ^T and see.

since root is nfs mounted, and = it=E2=80=99s hung, nothing but power cycling works.

If = tcpdump doesn't help much I'd start turning off checksum offloads and = the like
along the path (unclear if your host is your nfs = server or not) to see if that helps.

some more info:
the server is running 14.3 (i = don=E2=80=99t think this is relevant)
the server us also = providing /usr/local
if the host is running with a local = root, all is fine.
virtual hosts, i.e bhive, also hang = under similar configuration.

so now i'm = running a tcpdump on the server, and will probably have more info.
That should give you more information. A few things to = note:
- 15.1 has = the capability of using NFSv4 for root, but that requires some
  careful = configuration.
- Assuming your root is NFSv3 (you'll see that in the packet = capture
 in = wireshark), make sure you are not running nfsuserd(8) or = gssd(8).
only NFS3 
 Both of these will try and = look up user/group names in the passwd/group
 file and this can hang the = system when they do the upcall from within
 the NFS client (which would have to access the = files).

rick

so it = hanged,
im attaching the relevant tcpdump,
what is = weird, and I must have forgotten much, 
nrnd is the = server, chamsa is the client.
it seems nrnd is requesting arp = too often =E2=80=A6
is the TTL on the arp reseted when a = packet is received, or
it just times out?
here is = the abridged version:
= --Apple-Mail=_71187085-3739-44C3-8D45-06941C6FC8ED Content-Disposition: attachment; filename=hang Content-Type: application/octet-stream; x-unix-mode=0644; name="hang" Content-Transfer-Encoding: 7bit ... 00:25:22.032739 IP chamsa.cs.huji.ac.il.719 > nrnd.cs.huji.ac.il.nfsd: Flags [P.], seq 12239740:12239852, ack 9978805, win 2448, options [nop,nop,TS val 1166402375 ecr 891578467], length 112: NFS request xid 31608280 108 access fh 1670,404256/2 NFS_ACCESS_FULL 00:25:22.032776 IP nrnd.cs.huji.ac.il.nfsd > chamsa.cs.huji.ac.il.719: Flags [P.], seq 9978805:9978929, ack 12239852, win 29128, options [nop,nop,TS val 891579469 ecr 1166402375], length 124: NFS reply xid 31608280 reply ok 120 access c 0003 00:25:22.072237 IP chamsa.cs.huji.ac.il.719 > nrnd.cs.huji.ac.il.nfsd: Flags [.], ack 9978929, win 2448, options [nop,nop,TS val 1166402415 ecr 891579469], length 0 00:30:46.258902 ARP, Request who-has chamsa.cs.huji.ac.il tell router-340-01.cs.huji.ac.il, length 46 00:30:47.305012 ARP, Request who-has chamsa.cs.huji.ac.il tell router-340-01.cs.huji.ac.il, length 46 00:30:48.351810 ARP, Request who-has chamsa.cs.huji.ac.il tell router-340-01.cs.huji.ac.il, length 46 00:31:24.000926 IP nrnd.cs.huji.ac.il.nfsd > chamsa.cs.huji.ac.il.msexch-routing: Flags [F.], seq 10967253, ack 13744768, win 29128, options [nop,nop,TS val 1165956368 ecr 2800100369], length 0 00:31:24.204280 IP nrnd.cs.huji.ac.il.nfsd > chamsa.cs.huji.ac.il.719: Flags [F.], seq 9978929, ack 12239852, win 29128, options [nop,nop,TS val 891941641 ecr 1166402415], length 0 00:31:24.249273 IP nrnd.cs.huji.ac.il.nfsd > chamsa.cs.huji.ac.il.msexch-routing: Flags [F.], seq 10967253, ack 13744768, win 29128, options [nop,nop,TS val 1165956617 ecr 2800100369], length 0 00:31:24.485273 IP nrnd.cs.huji.ac.il.nfsd > chamsa.cs.huji.ac.il.719: Flags [F.], seq 9978929, ack 12239852, win 29128, options [nop,nop,TS val 891941922 ecr 1166402415], length 0 00:31:24.547273 IP nrnd.cs.huji.ac.il.nfsd > chamsa.cs.huji.ac.il.msexch-routing: Flags [F.], seq 10967253, ack 13744768, win 29128, options [nop,nop,TS val 1165956915 ecr 2800100369], length 0 00:31:24.847273 IP nrnd.cs.huji.ac.il.nfsd > chamsa.cs.huji.ac.il.719: Flags [F.], seq 9978929, ack 12239852, win 29128, options [nop,nop,TS val 891942284 ecr 1166402415], length 0 00:31:24.943274 IP nrnd.cs.huji.ac.il.nfsd > chamsa.cs.huji.ac.il.msexch-routing: Flags [F.], seq 10967253, ack 13744768, win 29128, options [nop,nop,TS val 1165957311 ecr 2800100369], length 0 00:31:25.371274 IP nrnd.cs.huji.ac.il.nfsd > chamsa.cs.huji.ac.il.719: Flags [F.], seq 9978929, ack 12239852, win 29128, options [nop,nop,TS val 891942808 ecr 1166402415], length 0 00:31:25.535277 IP nrnd.cs.huji.ac.il.nfsd > chamsa.cs.huji.ac.il.msexch-routing: Flags [F.], seq 10967253, ack 13744768, win 29128, options [nop,nop,TS val 1165957903 ecr 2800100369], length 0 00:31:26.219276 IP nrnd.cs.huji.ac.il.nfsd > chamsa.cs.huji.ac.il.719: Flags [F.], seq 9978929, ack 12239852, win 29128, options [nop,nop,TS val 891943656 ecr 1166402415], length 0 00:31:26.519274 IP nrnd.cs.huji.ac.il.nfsd > chamsa.cs.huji.ac.il.msexch-routing: Flags [F.], seq 10967253, ack 13744768, win 29128, options [nop,nop,TS val 1165958887 ecr 2800100369], length 0 00:31:27.715274 IP nrnd.cs.huji.ac.il.nfsd > chamsa.cs.huji.ac.il.719: Flags [F.], seq 9978929, ack 12239852, win 29128, options [nop,nop,TS val 891945152 ecr 1166402415], length 0 00:31:28.287275 IP nrnd.cs.huji.ac.il.nfsd > chamsa.cs.huji.ac.il.msexch-routing: Flags [F.], seq 10967253, ack 13744768, win 29128, options [nop,nop,TS val 1165960655 ecr 2800100369], length 0 00:31:30.507275 IP nrnd.cs.huji.ac.il.nfsd > chamsa.cs.huji.ac.il.719: Flags [F.], seq 9978929, ack 12239852, win 29128, options [nop,nop,TS val 891947944 ecr 1166402415], length 0 00:31:31.623276 IP nrnd.cs.huji.ac.il.nfsd > chamsa.cs.huji.ac.il.msexch-routing: Flags [F.], seq 10967253, ack 13744768, win 29128, options [nop,nop,TS val 1165963991 ecr 2800100369], length 0 00:31:35.891276 IP nrnd.cs.huji.ac.il.nfsd > chamsa.cs.huji.ac.il.719: Flags [F.], seq 9978929, ack 12239852, win 29128, options [nop,nop,TS val 891953328 ecr 1166402415], length 0 00:31:38.095275 IP nrnd.cs.huji.ac.il.nfsd > chamsa.cs.huji.ac.il.msexch-routing: Flags [F.], seq 10967253, ack 13744768, win 29128, options [nop,nop,TS val 1165970463 ecr 2800100369], length 0 00:31:46.459274 IP nrnd.cs.huji.ac.il.nfsd > chamsa.cs.huji.ac.il.719: Flags [F.], seq 9978929, ack 12239852, win 29128, options [nop,nop,TS val 891963896 ecr 1166402415], length 0 00:31:50.839277 IP nrnd.cs.huji.ac.il.nfsd > chamsa.cs.huji.ac.il.msexch-routing: Flags [F.], seq 10967253, ack 13744768, win 29128, options [nop,nop,TS val 1165983207 ecr 2800100369], length 0 00:32:07.395274 IP nrnd.cs.huji.ac.il.nfsd > chamsa.cs.huji.ac.il.719: Flags [F.], seq 9978929, ack 12239852, win 29128, options [nop,nop,TS val 891984832 ecr 1166402415], length 0 00:32:16.127274 IP nrnd.cs.huji.ac.il.nfsd > chamsa.cs.huji.ac.il.msexch-routing: Flags [F.], seq 10967253, ack 13744768, win 29128, options [nop,nop,TS val 1166008495 ecr 2800100369], length 0 00:32:41.415271 IP nrnd.cs.huji.ac.il.nfsd > chamsa.cs.huji.ac.il.msexch-routing: Flags [F.], seq 10967253, ack 13744768, win 29128, options [nop,nop,TS val 1166033783 ecr 2800100369], length 0 00:32:49.067272 IP nrnd.cs.huji.ac.il.nfsd > chamsa.cs.huji.ac.il.719: Flags [F.], seq 9978929, ack 12239852, win 29128, options [nop,nop,TS val 892026504 ecr 1166402415], length 0 00:33:06.703270 IP nrnd.cs.huji.ac.il.nfsd > chamsa.cs.huji.ac.il.msexch-routing: Flags [F.], seq 10967253, ack 13744768, win 29128, options [nop,nop,TS val 1166059071 ecr 2800100369], length 0 00:33:30.739268 IP nrnd.cs.huji.ac.il.nfsd > chamsa.cs.huji.ac.il.719: Flags [F.], seq 9978929, ack 12239852, win 29128, options [nop,nop,TS val 892068176 ecr 1166402415], length 0 00:33:31.991267 IP nrnd.cs.huji.ac.il.nfsd > chamsa.cs.huji.ac.il.msexch-routing: Flags [R.], seq 10967254, ack 13744768, win 0, options [nop,nop,TS val 1166084359 ecr 2800100369], length 0 00:34:12.411265 IP nrnd.cs.huji.ac.il.nfsd > chamsa.cs.huji.ac.il.719: Flags [F.], seq 9978929, ack 12239852, win 29128, options [nop,nop,TS val 892109848 ecr 1166402415], length 0 00:34:54.083261 IP nrnd.cs.huji.ac.il.nfsd > chamsa.cs.huji.ac.il.719: Flags [R.], seq 9978930, ack 12239852, win 0, options [nop,nop,TS val 892151520 ecr 1166402415], length 0 00:41:00.234041 ARP, Request who-has chamsa.cs.huji.ac.il tell router-340-01.cs.huji.ac.il, length 46 00:41:01.270312 ARP, Request who-has chamsa.cs.huji.ac.il tell router-340-01.cs.huji.ac.il, length 46 00:41:02.296492 ARP, Request who-has chamsa.cs.huji.ac.il tell router-340-01.cs.huji.ac.il, length 46 00:42:15.240343 ARP, Request who-has chamsa.cs.huji.ac.il tell router-340-01.cs.huji.ac.il, length 46 00:42:16.256438 ARP, Request who-has chamsa.cs.huji.ac.il tell router-340-01.cs.huji.ac.il, length 46 ... --Apple-Mail=_71187085-3739-44C3-8D45-06941C6FC8ED Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=us-ascii



thanks,
       danny


/bz

--
Bjoern A. = Zeeb =             &n= bsp;           &nbs= p;            =             &n= bsp;  r15:7
= --Apple-Mail=_71187085-3739-44C3-8D45-06941C6FC8ED-- --Apple-Mail=_C13EB2B1-F371-4DE8-94CF-BA1F9319BF1C--