From nobody Thu Aug 24 14:02:30 2023 X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4RWlCZ17nPz4rCW3 for ; Thu, 24 Aug 2023 14:02:50 +0000 (UTC) (envelope-from rick.macklem@gmail.com) Received: from mail-pl1-x630.google.com (mail-pl1-x630.google.com [IPv6:2607:f8b0:4864:20::630]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4RWlCX60dsz4FTm for ; Thu, 24 Aug 2023 14:02:48 +0000 (UTC) (envelope-from rick.macklem@gmail.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20221208 header.b=Vky2hfJI; spf=pass (mx1.freebsd.org: domain of rick.macklem@gmail.com designates 2607:f8b0:4864:20::630 as permitted sender) smtp.mailfrom=rick.macklem@gmail.com; dmarc=pass (policy=none) header.from=gmail.com Received: by mail-pl1-x630.google.com with SMTP id d9443c01a7336-1c0d0bf18d7so1118475ad.0 for ; Thu, 24 Aug 2023 07:02:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1692885764; x=1693490564; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=pYaGu/EzMxoy4BF/QZQ8TL4zxC3pYjjbkswAJWILKWA=; b=Vky2hfJIQzNlXKsoaP7xuELvkUn4KNJs0ZtW9DBJeU7dPoLjnShuSnHF6gxgMlPlEq R60jiJQP120EQ0R6Tex+vIcf7k3Y9GVkxLVCVSEIpNwqKF87zDlc7jn50kPUMLyfxFct Z7S4y1ELxFt+dPCvZd38tFT7xP02pX5mcWm+aN5ieLw32fJTU4ZUXaf1AJxYiG/gGjx7 DRLwvQkSmQULIp/764U8FCNshkuZt+bM42pIOZsG2UTYNz/WfmqjW213V4iNAn9nlAhY T0WgPSu36/2TTzBKrFqzuHrVNptLWGGO1uoJ5Mx/3grcGtBh3jNpcsWTEb86NAgk5oVy o8Nw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1692885764; x=1693490564; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=pYaGu/EzMxoy4BF/QZQ8TL4zxC3pYjjbkswAJWILKWA=; b=ila00M/jcukRPcKFlLEHErXjUwoHjMz76MVFdyxVq8Jbe5XyIEkJFtOLYI+8uOdzuM JCpw1DDn/w0w4XHaXsi5f4nEDe6VgrSEpQRXRJ6B4jrL1iYT5fE6G4KxnCcJy5ybIqu7 VXYCsPk7vNrlZY6/kONJM4keOR7Hlz5qXJJiUzh0feR5GQNowyhLvafmbwJl/MnAgJsI AmJPeLVX9oVUK3ESZ9Dey9SDzqg1sKFa0/alt0qJQwVxEFqUPgOpENYGb+AP2uC8/Mwh hOpI+VNlXCbqOYP0+XlAcLw3gSQAd3vNRd0TAMbmDQ9oeHHFAWhXRKC4zz5LDkVfcYgw 8LXg== X-Gm-Message-State: AOJu0YxJAuAEEF5m6K+U96ii9VIHoSIfp4TDpmE5OeufZI9bKrq1DeEq hcyVx+2DALFMUnch6IAio+FHeubOeB9Lm7sk+HHB//M= X-Google-Smtp-Source: AGHT+IEnDU+P1+5hM4/XSfHVFgu032PxIJ+TgyrI7QdYIMo28RN6xVrEMIphbWnb4SkZxvFdyQ7NAdMinWsEYTLbSYo= X-Received: by 2002:a17:902:f68d:b0:1bd:ca21:c85 with SMTP id l13-20020a170902f68d00b001bdca210c85mr14286736plg.69.1692885764489; Thu, 24 Aug 2023 07:02:44 -0700 (PDT) List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 References: In-Reply-To: From: Rick Macklem Date: Thu, 24 Aug 2023 07:02:30 -0700 Message-ID: Subject: Re: NFS client hang on 13.2-RELEASE-p2 on file locking / wrong interface selected To: J David Cc: FreeBSD FS Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Spamd-Result: default: False [-4.00 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-1.000]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20221208]; MIME_GOOD(-0.10)[text/plain]; FROM_HAS_DN(0.00)[]; TAGGED_RCPT(0.00)[]; FREEMAIL_FROM(0.00)[gmail.com]; ARC_NA(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::630:from]; PREVIOUSLY_DELIVERED(0.00)[freebsd-fs@freebsd.org]; BLOCKLISTDE_FAIL(0.00)[2607:f8b0:4864:20::630:server fail]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; TAGGED_FROM(0.00)[]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; DKIM_TRACE(0.00)[gmail.com:+]; RCPT_COUNT_TWO(0.00)[2]; TO_DN_ALL(0.00)[]; MID_RHS_MATCH_FROMTLD(0.00)[]; MLMMJ_DEST(0.00)[freebsd-fs@freebsd.org]; RCVD_COUNT_ONE(0.00)[1]; FROM_EQ_ENVFROM(0.00)[]; FREEMAIL_TO(0.00)[gmail.com]; FREEMAIL_ENVFROM(0.00)[gmail.com]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_LAST(0.00)[] X-Spamd-Bar: --- X-Rspamd-Queue-Id: 4RWlCX60dsz4FTm The NLM is a fundamentally broken protocol and has never been described by an RFC. I strongly recommend against using it. If the locks do not need to be visible to other clients, the "nolockd" mount option should work. Otherwise consider switching the mounts to NFSv4.1/4.2. (For either of these cases, the rpc.lockd and rpc.statd daemons no longer need to be run.) I am not familiar with the NLM code and have no interest in trying to change it. rick On Wed, Aug 23, 2023 at 8:56=E2=80=AFPM J David w= rote: > > Hello, > > We are seeing NFS hangs on FreeBSD 13.2-RELEASE-p2 clients talking to > a Debian bookworm NFS server using NFSv3. > > Whenever a process attempts to lock a file on an NFS mount, for example: > > lockf x sleep 3 > > That process hangs in state "nlmrcv" and goes to 100% CPU. > > I found huge numbers of exchanges like this via tcpdump: > > 03:05:41.432581 IP 172.17.200.2.998 > 172.17.250.10.50516: UDP, length 17= 2 > 0x0000: 4500 00c8 9afb 0000 4011 c4f9 ac11 c802 E.......@....... > 0x0010: ac11 fa0a 03e6 c554 00b4 1af6 04ed ac8b .......T........ > 0x0020: 0000 0000 0000 0002 0001 86b5 0000 0004 ................ > 0x0030: 0000 0002 0000 0001 0000 001c 64e6 c8e0 ............d... > 0x0040: 0000 0002 6332 0000 0001 bb87 0001 bb87 ....c2.......... > 0x0050: 0000 0001 0000 61a8 0000 0000 0000 0000 ......a......... > 0x0060: 0000 0004 3873 0200 0000 0000 0000 0001 ....8s.......... > 0x0070: 0000 0002 6332 0000 0000 0020 0100 0601 ....c2.......... > 0x0080: a0da 65c4 00d0 6316 0000 0000 0000 0000 ..e...c......... > 0x0090: 0a00 0a00 0000 0000 c733 0800 0000 0009 .........3...... > 0x00a0: 3130 3030 3031 4063 3200 0000 0001 86a1 100001@c2....... > 0x00b0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ > 0x00c0: 0000 0000 0000 0025 .......% > 03:05:41.432632 IP 172.17.250.10.50516 > 172.17.200.2.998: UDP, length 20 > 0x0000: 4500 0030 4675 4000 4011 da17 ac11 fa0a E..0Fu@.@....... > 0x0010: ac11 c802 c554 03e6 001c 1a5e 04ed ac8b .....T.....^.... > 0x0020: 0000 0001 0000 0001 0000 0001 0000 0001 ................ > 03:05:41.432647 IP 172.17.200.2.998 > 172.17.250.10.50516: UDP, length 17= 2 > 0x0000: 4500 00c8 9afc 0000 4011 c4f8 ac11 c802 E.......@....... > 0x0010: ac11 fa0a 03e6 c554 00b4 1af6 04ed ac8c .......T........ > 0x0020: 0000 0000 0000 0002 0001 86b5 0000 0004 ................ > 0x0030: 0000 0002 0000 0001 0000 001c 64e6 c8e0 ............d... > 0x0040: 0000 0002 6332 0000 0001 bb87 0001 bb87 ....c2.......... > 0x0050: 0000 0001 0000 61a8 0000 0000 0000 0000 ......a......... > 0x0060: 0000 0004 3973 0200 0000 0000 0000 0001 ....9s.......... > 0x0070: 0000 0002 6332 0000 0000 0020 0100 0601 ....c2.......... > 0x0080: a0da 65c4 00d0 6316 0000 0000 0000 0000 ..e...c......... > 0x0090: 0a00 0a00 0000 0000 c733 0800 0000 0009 .........3...... > 0x00a0: 3130 3030 3031 4063 3200 0000 0001 86a1 100001@c2....... > 0x00b0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ > 0x00c0: 0000 0000 0000 0025 .......% > 03:05:41.432697 IP 172.17.250.10.50516 > 172.17.200.2.998: UDP, length 20 > 0x0000: 4500 0030 4676 4000 4011 da16 ac11 fa0a E..0Fv@.@....... > 0x0010: ac11 c802 c554 03e6 001c 1a5e 04ed ac8c .....T.....^.... > 0x0020: 0000 0001 0000 0001 0000 0001 0000 0001 ................ > > A huge number of these are exchanged. Like, several million over the > span of a couple of minutes. > > Then the FreeBSD client system becomes unresponsive with these console me= ssages: > > [nl_neigh] rtnl_lle_event: error allocating group writer > [nl_neigh] rtnl_lle_event: error allocating group writer > [nl_neigh] rtnl_lle_event: error allocating group writer > [zone: mbuf] kern.ipc.nmbufs limit reached > [nl_neigh] rtnl_lle_event: error allocating group writer > [nl_neigh] rtnl_lle_event: error allocating group writer > [nl_neigh] rtnl_lle_event: error allocating group writer > > Now, while 172.17.200.2 is an IP address on the client and > 172.17.250.10 is an IP address on the server, that's the wrong subnet. > The filesystem is mounted over a dedicated VLAN for NFS which has the > IPs 172.20.200.2 and 172.20.250.10. So whatever this traffic is, it's > using the wrong interface. > > I was able to work around this by swapping the 172.17.0.0 network from > the NFS server. But that's not exactly an optimal solution. > > The main problem may be on the Debian side. I was able to workaround > the issue by swapping the order of the network interfaces on the > Debian side so the NFS VLAN was the "first" interface. That suggests > to me that something on the Debian side is choosing the first > interface instead of the right interface. > > So I'll pursue that. But it'd sure be nice if the FreeBSD client > didn't hang in this situation. > > Does anyone know what might be happening here or have any other > insight that might help me track this down? > > Thanks for any advice! >