From nobody Sat Sep 30 22:06:35 2023 X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4RyhBw1rM0z4vt75 for ; Sat, 30 Sep 2023 22:06:48 +0000 (UTC) (envelope-from rick.macklem@gmail.com) Received: from mail-pj1-x1033.google.com (mail-pj1-x1033.google.com [IPv6:2607:f8b0:4864:20::1033]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4RyhBv1z1Sz3XhS for ; Sat, 30 Sep 2023 22:06:47 +0000 (UTC) (envelope-from rick.macklem@gmail.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20230601 header.b=VgKgyFRL; spf=pass (mx1.freebsd.org: domain of rick.macklem@gmail.com designates 2607:f8b0:4864:20::1033 as permitted sender) smtp.mailfrom=rick.macklem@gmail.com; dmarc=pass (policy=none) header.from=gmail.com Received: by mail-pj1-x1033.google.com with SMTP id 98e67ed59e1d1-27752a1e184so9490131a91.3 for ; Sat, 30 Sep 2023 15:06:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1696111605; x=1696716405; darn=freebsd.org; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=AAtRQ2HZmhyylgr3KwIuda+WANkUvRWmChnrtyU0kW8=; b=VgKgyFRLACtl+ws+kQLvRqIE/hdD9XMMa2AqQ3Q9VLVYKa7OlfkAgJ4KtdXpwimj31 T3TJNFhSfyHy96OA0lXOGfUXita7t/OlGLguGw4iRHOQIIfXmgLN5onB7Y0LrggefqHn kZyStwuCP3NYTnlv8TfqJ3Qr/u70DKGK/6rWoZAf1hndL/V+lkY4lqXivqYGnnMzpPry O/5a57S/BqUA1of0JoHDizVN6gkubhXbXoUBT/kPgBquLA++SdpJNkGH0Njypphdc02j svxBZrLyh0TnQ0mmavQRw3WJqeGxhZVzV/hLiTTJKcB9zGxORp7vf7tKzi2vzhBBw0tf TD/g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1696111605; x=1696716405; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=AAtRQ2HZmhyylgr3KwIuda+WANkUvRWmChnrtyU0kW8=; b=BltJ7oGrXo/O9jMJZo0givcIKQ2VaJdE4aLR6Tkd70Tlvva/yiRjQkPH7Mfp7tVRa5 IhStc+ToCQmVw/q5Ia4vEnEyOwAV/T0h0w1FNitlkVZ4PVR9DiviPA27bqm8o3aS7xkH u+J/GcIquSnU33Di5amEPFMAMO184mvvLd3QdoOpySmBjK8hrxOcQav/Vv5am8vhf8yp nbMjh86n3xiaj0TQvJyPrFvUtpQDPJ14/cVCsEcZ1i5JPgYUzJEeZGb+PSMMyJJjehrP YgT0fQ+Ldvo0PQ/rmzcD9vBNufCZwbQ98/19lsPDBC9+C71meNa5d9cKHimsG5p2plub +PsA== X-Gm-Message-State: AOJu0YzYZs6W8qUyDgWvcmZ9GiSigBOYMNsHaY4ky58z0A568b4S4a+N lAZqko00XkbEwdiumRJaOtQycDRaVCHFrfWFXQ== X-Google-Smtp-Source: AGHT+IHSM3jeqR2YxWk5/spEiIqJuOrjkDt6BBe3fENoa0VqAQqvigAzhFQCP+CxjBIT8Fyha/GKL5u6NWlNPOeJo74= X-Received: by 2002:a17:90a:b38a:b0:263:f630:228f with SMTP id e10-20020a17090ab38a00b00263f630228fmr7779285pjr.23.1696111605314; Sat, 30 Sep 2023 15:06:45 -0700 (PDT) List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 References: In-Reply-To: From: Rick Macklem Date: Sat, 30 Sep 2023 15:06:35 -0700 Message-ID: Subject: Re: FreeBSD 13.2 NFS client mount hangs To: J David Cc: FreeBSD FS Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Spamd-Bar: --- X-Spamd-Result: default: False [-3.90 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-0.999]; NEURAL_HAM_SHORT(-0.90)[-0.900]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20230601]; MIME_GOOD(-0.10)[text/plain]; FROM_HAS_DN(0.00)[]; TAGGED_RCPT(0.00)[]; FREEMAIL_FROM(0.00)[gmail.com]; ARC_NA(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::1033:from]; PREVIOUSLY_DELIVERED(0.00)[freebsd-fs@freebsd.org]; BLOCKLISTDE_FAIL(0.00)[2607:f8b0:4864:20::1033:server fail]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; TAGGED_FROM(0.00)[]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; DKIM_TRACE(0.00)[gmail.com:+]; RCPT_COUNT_TWO(0.00)[2]; TO_DN_ALL(0.00)[]; MID_RHS_MATCH_FROMTLD(0.00)[]; MLMMJ_DEST(0.00)[freebsd-fs@freebsd.org]; RCVD_COUNT_ONE(0.00)[1]; FROM_EQ_ENVFROM(0.00)[]; FREEMAIL_TO(0.00)[gmail.com]; FREEMAIL_ENVFROM(0.00)[gmail.com]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_LAST(0.00)[] X-Rspamd-Queue-Id: 4RyhBv1z1Sz3XhS On Fri, Sep 29, 2023 at 5:50=E2=80=AFPM J David w= rote: > > I have noticed a new (to me) hang on FreeBSD NFS client machines > running 13.2-RELEASE-p2. > > It's happened twice this week to Apache processes. It's the root EUID > process and it appears to happen while the process is starting up or > reconfiguring. I.e., while it's reading the configs. > > The configs are not on NFS storage. But the vhost document roots are. > > The process ps looks like this: > > 0 19557 19548 3 25 5 25248 12036 nfstry DN - 0:12.85 > /usr/local/apache/2.4/bin/httpd -D FOREGROUND -f > /usr/local/apache/2.4/conf/httpd.conf > > The procstat -kk looks like: > > PID TID COMM TDNAME KSTACK > 19557 100341 httpd - mi_switch+0xc2 > sleepq_timedwait+0x2f _sleep+0x1ce clnt_vc_call+0x866 > clnt_reconnect_call+0x626 newnfs_request+0xc36 nfscl_request+0x5a > nfsrpc_getattr+0xbb nfs_close+0x489 vop_sigdefer+0x2b > VOP_CLOSE_APV+0x1c vn_close1+0x16a vn_closefile+0x3d _fdrop+0x11 > closef+0x24b closefp_impl+0x69 amd64_syscall+0x10c > fast_syscall_common+0xf8 This is just waiting for a reply for the Close RPC. > > The process slowly gains CPU time (a few hundredths per minute) but is > immune to kill -9 so it doesn't seem to be coming out of the kernel at > any point. > > I tried running procstat -kk every few seconds to see if I would get > anything different to show what it's doing. Most are the same as > above, but I also got this: > > 19557 100341 httpd - mi_switch+0xc2 > sleepq_timedwait+0x2f _sleep+0x1ce nfs_catnap+0x47 > newnfs_request+0x14b3 nfscl_request+0x5a nfsrpc_getattr+0xbb > nfs_close+0x489 vop_sigdefer+0x2b VOP_CLOSE_APV+0x1c vn_close1+0x16a > vn_closefile+0x3d _fdrop+0x11 closef+0x24b closefp_impl+0x69 > amd64_syscall+0x10c fast_syscall_common+0xf8 This one is sleeping for a short time before retrying an RPC. Although I cannot be 100% sure, it is probably one that received a NFS4ERR_DELAY reply from the server. Fairly recent versions of the Linux server hand out delegations. Imho delegations are pretty useless. I have seen reports on the linux-nfs@vger.kernel.org related to Close and Delegation Recall resulting in repeated NFS4ERR_DELAY replies. --> I'd suggest you try and disable delegations. I do not know how to do this on the Linux server, but not running the nfscbd(8) daemon should stop the= m from being issued. (No nfscbd(8) implies no callbacks and no callbacks should imply no delegations being issued. If the Linux server still issues delegations when the nfscbd(8) is not running (and was not running when the mount was done), it is broken. The FreeBSD client currently does not accept NFS4ERR_DELAY for Close. If t= he Linux server is replying NFS4ERR_DELAY for Close, all bets are off. > > (This differs starting at the newnfs_request after nfscl_request+0x5a.) > > I started unmounting NFS filesystems until I hit one where umount > hung. An ls on that filesystem also hung. However, an ls of that > filesystem from another client machine worked fine, so it does appear > to be a client-side issue rather than a server problem. umount -f > also hung. umount -N did unmount it very quickly and that caused all > the hanging umounts and the > httpd process to exit immediately. Yes, "umount -N is the way (and the only way) to get rid of hung NFS mounts. > > I didn't find anything good in the syslog or dmesg. The only thing > related to nfs are a handful of "nfsv4 err=3D10068" that look like they > were way back near when the system booted (about 5 days ago). Hmm, interesting. 10068 is NFS4ERR_RETRY_UNCACHED_REP. I have never seen (and do not recall anyone else reporting) this error return. - The RFC says it can be replied when a retry of the same RPC with the same session slot/seqid is received and the reply is not cached. To be honest, I do not know how the FreeBSD NFSv4.1/4.2 client will handle this? I will have to look at the code to see if this can happen afte= r a new TCP connection is established and outstanding RPCs are retried. --> If this can happen, the client code needs to be patched to retry the RP= C with a different session slot or same session slot, but adavnced seqi= d. Basically, I suspect the FreeBSD client is broken for handling this case, which I have never seen before. > > The mount flags are: > > nfsv4,minorversion=3D2,oneopenown,tcp,resvport,nconnect=3D1,hard,cto,sec= =3Dsys,acdirmin=3D3,acdirmax=3D60,acregmin=3D5,acregmax=3D60,nametimeo=3D60= ,negnametimeo=3D60,rsize=3D65536,wsize=3D65536,readdirsize=3D65536,readahea= d=3D1,wcommitsize=3D16777216,timeout=3D120,retrans=3D2147483647 > > Is there any other information I could provide or try to catch next > time that would help debug this? I'd suggest you first check network connetcivity. Both NFS client and NFS server should be able to ping each other. If that is the case, then I'd suggest you capture packets. On the FreeBSD end: # tcpdump -s 0 -w out.pcap host Let this run for a while and then pull out.pcap into wireshark and see what traffic is going between the NFS client and server. (Unlike tcpdump, wireshark does know how to decode NFS properly.) rick > > Thanks! > From nobody Sun Oct 1 14:53:13 2023 X-Original-To: fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Rz6X93RWMz4vPTk for ; Sun, 1 Oct 2023 14:53:13 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Rz6X92G2Vz4bbr for ; Sun, 1 Oct 2023 14:53:13 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1696171993; a=rsa-sha256; cv=none; b=S9KfOd9IldlgW5Ayb4O1SZIEBzSX6RNEnNeMBbRzOqWfrkOM+aUNm+l2OPI6Cdjx+SliJi kwvOADSfJyqxqaqRBBGeQpIopOusX59hv8CABstLd3CkwgLyoN2wK+BXj/uS0CUZfr7KZu MrNcIv7m7FFf6dqN3Lt2v8KeUBSEtXsEMC2TaV3vmTRNYDfEdUY7ZqKU3/9g+G8mulRDGZ Vrd1TbLtUnPbmCqh/YblGadQKoX0sI5pi7TmBDnWIxMCl6JVxUKIW/AghZYWJALt9gMdKN FVDLZ4O8wvP4L7fOU1bkVhBYNh/KMcLm3rEPiG0PnWH/ZLTUNwchsAcJmYl2Pg== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1696171993; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=lQ/JMQBHRmv0ZJ/oHfumnekgb0K49LlKD1ZOKleVXGM=; b=If+1a85mevpEgAgzq/miCqPynSeptsMyRnsNphIfkjXo0o1wavv4HWKS+suCiQRfQA34+r NPnj0HAZakLk6JsodDAUkBSnSWh8iADbLbODpgIYMUAK0X5SxJDfw0SymeX477u00Uj9AD eihi9vqv+J4eiH4Vjuj+wRPMJ5ZC8/JZXwBo7oOvRLq4qdTaJgBYIdhkr5WjuEqPe0zSWV k8ew6ONOHcsFYhgSVAGWhCtG9Tbb2MJPIPsc/MAk2RMAZ/viQABKYNBfmejBM0D3voQId7 IhG/BxAaUzCM8anLfNl8g06HzFzoJHUN5wNXEGu4fx8nN/YsfZytqy666dhi1g== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4Rz6X91KqyzYwc for ; Sun, 1 Oct 2023 14:53:13 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 391ErDUA098580 for ; Sun, 1 Oct 2023 14:53:13 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 391ErDP8098579 for fs@FreeBSD.org; Sun, 1 Oct 2023 14:53:13 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 257514] Can not embed ZFS into Kernel on RelEng/13.0 i386 Date: Sun, 01 Oct 2023 14:53:13 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 13.0-RELEASE X-Bugzilla-Keywords: needs-qa X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: grahamperrin@gmail.com X-Bugzilla-Status: Open X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: cc assigned_to keywords bug_status Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D257514 Graham Perrin changed: What |Removed |Added ---------------------------------------------------------------------------- CC|grahamperrin@freebsd.org | Assignee|bugs@FreeBSD.org |fs@FreeBSD.org Keywords| |needs-qa Status|New |Open --- Comment #2 from Graham Perrin --- Is this reproducible with releng/13.2 or releng/14.0? (In reply to CrazyMihey from comment #1) > UnFortunately, NoOne answers My Question =E2=86=91, so I dicided to Try t= o=20 > Fix This Annoying Bug MySelf. =E2=80=A6 I'm sorry, although (^Triage) please note that use of Bugzilla here is norm= ally reserved for actionable bug reports. Mailing lists and the like are often better for questions and answers; . --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Sun Oct 1 15:21:14 2023 X-Original-To: fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Rz78W6BVWz4vRjW for ; Sun, 1 Oct 2023 15:21:15 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Rz78W3jH2z3F04 for ; Sun, 1 Oct 2023 15:21:15 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1696173675; a=rsa-sha256; cv=none; b=Exuv1krvapqG7zCqoJypd8iKXPx72QF7OrGa6WW/Ii+4DdJHYN/Jp3ZNpJSaxI/tU39Ac6 RNmQmeVYrFwPnigkC+ngoXx2v8d5ZlZwuqiKWNNvJWeZ7yG9Xcfsya4tLyW7scIY21paDu N+g0buZCNOdAao2wodzcftoq028GnyKLduEfZJoPDXr2rxA0Kos9f06tAuJGr7Bh16nE5K VXqLgnbOOdBfnSJzfbLRzQKRNX+q5weABt1YPurwROPPDoqO4Sj2iOeE86tN3kF1duoklI Sf6gWueuYuVIVr2+SnPkAJvytxCyfYqTS49c3WqSD2l87MwHo7JGv/ZoLuxcSw== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1696173675; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=32LsGrxSlJMpuF9xkmLoPjx7KLmT1InmxCe+obZDoks=; b=udp7QCjjX1SFKLl5HW2DoOsXvnJuvUkOL7p8Hl1L5vbS5yaynLVGYEM9Vl7h7UaZMCx+PX JkTMvgFDPEGFGwWj2/UbkMwNqxvDAVhr4P5tb8he8HBgXSQJrxz+0x8c5cAAbcIGfZAYRC rsRWuwzWMRylpeo5x3u5F5WU0RUrnKPu/ZgP8kS9ldKGUqOaiXsUBvyBaH+r8BB4CNjyay 13gAUj+iUktbs6jsMwTMlPhvmDSEFX4v7p36Qci0CZypubxIzkpAD+Uw4CzV+0VXvML3o7 564humXnawQ2dA6/sBlQ1MMfppGhteoslP0RJMOWKQ0LRPV5IB5qEQrsKKxCeA== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4Rz78W2nTzzZnw for ; Sun, 1 Oct 2023 15:21:15 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 391FLFfd034309 for ; Sun, 1 Oct 2023 15:21:15 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 391FLFVw034308 for fs@FreeBSD.org; Sun, 1 Oct 2023 15:21:15 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 206328] Crash on shutdown with swap on NFS file (with patch) Date: Sun, 01 Oct 2023 15:21:14 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: CURRENT X-Bugzilla-Keywords: crash X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: grahamperrin@gmail.com X-Bugzilla-Status: Open X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: bug_status keywords Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D206328 Graham Perrin changed: What |Removed |Added ---------------------------------------------------------------------------- Status|New |Open Keywords|patch |crash --- Comment #5 from Graham Perrin --- Sorry for the lack of attention in recent years. Are crashes reproducible with any recently active branch of FreeBSD? --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Sun Oct 1 16:47:29 2023 X-Original-To: fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Rz9413Xf7z4vXNt for ; Sun, 1 Oct 2023 16:47:29 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Rz9410DQSz3SFl for ; Sun, 1 Oct 2023 16:47:29 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1696178849; a=rsa-sha256; cv=none; b=knMJA6E6eIXv2efI26g3PpW9UwcO5y/9zfo6kj+8m3935Yb91W1mZYOQp236kYkz6+si7O SoVeCpp0zhKO4Ihgp/bcQEyyjMaM7kP4nRUpcSshSEd2DoBIS3a5ITZDOJuLVpj19y//T/ ueU8qA0OghFhYX6LK97ElqL7hpEJ0kTFJzB8q6HV/YtIjEDPEpa4GxX2+FoYSSObZjJFQa JNfJownkVCJVAzildVhtWCGJ83CA4j4SaGzycOmxdmtBSR+LrhMaqZkuXO60GtsN6OJQO7 hxf4UPmE17quXJnlwXOhtGrEtyxi2n5H3/iTj4lf7btAlmz6E1KmNUvnfHVz+Q== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1696178849; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=vdF/2TWnDmcfKha+6xVdR42uydDxQAGLIFfnWKWWWOQ=; b=HEestWQw6IauCI/DXtPVIdaL/7Vv64BJxSdy9zcnkmox69LbVuo4RlN9aJWyNqvdaLYQ6D x06hmEORpcPvASh8gq/pRksX6jYAzdLc/kQGsYmciHGBc7meRyBtCjJJgh7cmDwjIQOI5C y+1NKxoeDT/MKSUq06GB4Uf//ZBEgVjbfTzygWH9Znep0t2dEktUHFz55+5Cn8LcL5Omzm EzUY08SLDkEZVmi0fzn2DKqzhWbokXX4MOiJsSvhI38jxEidxnvUv8pNJR6HW0JSikw33f jB9gBtzQy34IReecZjDrmxMSxBd+IUJuUJpKcfyZLBzgRANUcD/5tt49CtqdXQ== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4Rz9406Q0fzd9X for ; Sun, 1 Oct 2023 16:47:28 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 391GlSgg074162 for ; Sun, 1 Oct 2023 16:47:28 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 391GlSJQ074161 for fs@FreeBSD.org; Sun, 1 Oct 2023 16:47:28 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 251610] rc.d/zpool runs before ada(4) attaches Date: Sun, 01 Oct 2023 16:47:29 +0000 X-Bugzilla-Reason: CC X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: bin X-Bugzilla-Version: 13.0-STABLE X-Bugzilla-Keywords: needs-qa X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: grahamperrin@gmail.com X-Bugzilla-Status: Open X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: rc@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: keywords cc bug_status bug_file_loc short_desc Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D251610 Graham Perrin changed: What |Removed |Added ---------------------------------------------------------------------------- Keywords|patch |needs-qa CC| |fs@FreeBSD.org Status|New |Open URL| |https://github.com/freebsd/ | |freebsd-src/blob/main/libex | |ec/rc/rc.d/dumpon Summary|[patch] rc.d/zpool runs |rc.d/zpool runs before |before ada(4) attaches |ada(4) attaches --- Comment #6 from Graham Perrin --- (In reply to Harald Schmalzbauer from comment #0) > =E2=80=A6 probably also true for nvd(4) backed root systems, =E2=80=A6 Is the situation significantly different with 15.0-CURRENT?=20 , a few things catch m= y eye. (In reply to Harald Schmalzbauer from comment #1) % git -C /usr/src pull --ff-only freebsd main >From https://git.freebsd.org/src * branch main -> FETCH_HEAD Already up to date. % git -C /usr/src apply --check --verbose /tmp/222477.patch Checking patch libexec/rc/rc.d/dumpon... Hunk #1 succeeded at 47 (offset 12 lines). %=20 (In reply to Hauke Fath from comment #4) Re: for refer= ence only,=20 % sysctl -d kern.cam.boot_delay kern.cam.scsi_delay kern.cam.boot_delay: Bus registration wait time kern.cam.scsi_delay: Delay to allow devices to settle after a SCSI bus reset (ms) % --=20 You are receiving this mail because: You are on the CC list for the bug.= From nobody Sun Oct 1 21:00:18 2023 X-Original-To: fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4RzGgk69Ylz4vpLq for ; Sun, 1 Oct 2023 21:00:18 +0000 (UTC) (envelope-from bugzilla-noreply@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4RzGgk3wk7z4Sqd for ; Sun, 1 Oct 2023 21:00:18 +0000 (UTC) (envelope-from bugzilla-noreply@FreeBSD.org) ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1696194018; a=rsa-sha256; cv=none; b=GxSvZxsIub5TOIV/NHtfHh78y6W6eDV/vH7aER+e+XGSE2sZvoE3oUuIf+Ol+SBEKU3jC/ w0QJfbWTlNmm1oERdMjOlS4TSXxKs0op1tyO4mY3VIDN0jhql3ZOAT6VUpwvVBTYhTXOoK BZw939nymfaJnk9LoflHE0RabA/+jkyGtBHo9uTmfD1sPO2oO2vECYbMsBPmncnjpZwmw+ kkwkcMkoz/fAYUxgknhNvOOk0ROr2BiHEtXuDVdkk4G0oqtdwYy8DbGbo/VwfxqeqcpoRM CA9rEHTcpQEPsucnzqJiCtVW2NHsw/cUVev3q3FxTQqg6WksRjTJNHDa2Ep44A== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1696194018; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=9514LgGsEOTDtARkIHH7S4xQbNRYPyrnJW9+CzOvVDQ=; b=Pat/3BJLUzYM7JVebQt78SLv6oGoiry7pcLnMQmBuHg4++m7M4709EUyhfH39b9UU+vydm ICJ18HwKAGREPpCmki347PvFIGszNbLSo7cQUMO16UopSRdh0seqqlByHUXY/ulcTmsf79 vjFtZ2Dk25CX4GErXZwSzFHRe5/pf72aBcmqFffgd4Mw5c06pcCkND0rWEKrYoFGeC+qxM UAvghQCv9zlFtLL8QQguJlsltn7TMEvdnZZx76s2/S4C2ZoejCnRJfdY6HbUYolO+9cRYN UnpHam7JBAf/Dx8+3YAIWhUCX5dBDve68vMOqB41kTFRA1K/EWOZXaLf0VO2gw== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4RzGgk2WB5zl6w for ; Sun, 1 Oct 2023 21:00:18 +0000 (UTC) (envelope-from bugzilla-noreply@FreeBSD.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 391L0IeB087227 for ; Sun, 1 Oct 2023 21:00:18 GMT (envelope-from bugzilla-noreply@FreeBSD.org) Received: (from bugzilla@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 391L0Ije087226 for fs@FreeBSD.org; Sun, 1 Oct 2023 21:00:18 GMT (envelope-from bugzilla-noreply@FreeBSD.org) Message-Id: <202310012100.391L0Ije087226@kenobi.freebsd.org> X-Authentication-Warning: kenobi.freebsd.org: bugzilla set sender to bugzilla-noreply@FreeBSD.org using -f From: bugzilla-noreply@FreeBSD.org To: fs@FreeBSD.org Subject: Problem reports for fs@FreeBSD.org that need special attention Date: Sun, 1 Oct 2023 21:00:18 +0000 List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="16961940180.e7BaEc.85029" Content-Transfer-Encoding: 7bit --16961940180.e7BaEc.85029 Date: Sun, 1 Oct 2023 21:00:18 +0000 MIME-Version: 1.0 Content-Type: text/plain; charset="UTF-8" To view an individual PR, use: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=(Bug Id). The following is a listing of current problems submitted by FreeBSD users, which need special attention. These represent problem reports covering all versions including experimental development code and obsolete releases. Status | Bug Id | Description ------------+-----------+--------------------------------------------------- Open | 237067 | ZFS: Crash in vdev_dtl_reassess when using GELI w Open | 244692 | gjournal: Does not support TRIM Open | 269503 | docs.freebsd.org: default vfs.zfs.arc.meta_limit Open | 271384 | zfs_load is not suitably documented Open | 226130 | ZFS: solaris assert: zrl->zr_refcount == 0 (0x1 = 5 problems total for which you should take action. --16961940180.e7BaEc.85029 Date: Sun, 1 Oct 2023 21:00:18 +0000 MIME-Version: 1.0 Content-Type: text/html; charset="UTF-8" 
The following is a listing of current problems submitted by FreeBSD users,
which need special attention. These represent problem reports covering
all versions including experimental development code and obsolete releases.

Status      |    Bug Id | Description
------------+-----------+---------------------------------------------------
Open        |    237067 | ZFS: Crash in vdev_dtl_reassess when using GELI w
Open        |    244692 | gjournal: Does not support TRIM
Open        |    269503 | docs.freebsd.org: default vfs.zfs.arc.meta_limit
Open        |    271384 | zfs_load is not suitably documented
Open        |    226130 | ZFS: solaris assert: zrl->zr_refcount == 0 (0x1 =

5 problems total for which you should take action.
--16961940180.e7BaEc.85029-- From nobody Mon Oct 2 03:01:31 2023 X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4RzQhp0Vf7z4w9bW for ; Mon, 2 Oct 2023 03:01:46 +0000 (UTC) (envelope-from jdavidlists@gmail.com) Received: from mail-ot1-x32b.google.com (mail-ot1-x32b.google.com [IPv6:2607:f8b0:4864:20::32b]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4RzQhm6dzzz3JFn for ; Mon, 2 Oct 2023 03:01:44 +0000 (UTC) (envelope-from jdavidlists@gmail.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20230601 header.b=hn7efpTQ; spf=pass (mx1.freebsd.org: domain of jdavidlists@gmail.com designates 2607:f8b0:4864:20::32b as permitted sender) smtp.mailfrom=jdavidlists@gmail.com; dmarc=pass (policy=none) header.from=gmail.com Received: by mail-ot1-x32b.google.com with SMTP id 46e09a7af769-6c615df24c0so4909351a34.1 for ; Sun, 01 Oct 2023 20:01:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1696215703; x=1696820503; darn=freebsd.org; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=g7wzO7WbjwY2U3Txw3RTsN1TprQF4vORrLfrmN2i4yw=; b=hn7efpTQFonpIEKKlz+6ZLIocRGhVp/FkFxUL3ZuzdYYxpIEHfPyk4EIUvshW+o+CL tDRMkoU2N3rabQlnciZSvrwcr4tI7ym2qBKi0QF/+IER7rSwbSzCtNQEokpzqleUnBUF GzhjCyyHbxxIiawrFKDcht7XOJuR7n7crkz5e/k6QGHMC43jzDKmiUwkug2zYpxFeipV dqWZqwRRnVhW3qMXeaJ5+fbJ4YyT76wDIjuBr0PUmfstEUbAUlRAaRCTSN6QD/s8ZvDz RkSmiADJNSM82fgGk6V1suq+ZdQqR9wuy9ToJ4zo6gj3+h8AQ1kXBRzVR78QCJpn00kI Zhmg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1696215703; x=1696820503; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=g7wzO7WbjwY2U3Txw3RTsN1TprQF4vORrLfrmN2i4yw=; b=hvapc5c+vbIwoDERTcy37Af30xWuZQ8QqL8XwJZBau0uHNbYoImmeomNvmgN6Facj2 w/50kdNIWQ9e/S3rpyx2OOeWNVICnDlwhBy35F35tYziUDiJDPUYdvPGByY7Eaiez0dF AR4UbZKrRc6pwQlO+pgW2UD9Lf3oHSXKBTsKMRkTwTaar/OyyE+pkTWWB/QxXx+Cytxt TkxM8TlTBBPXgzaNge0yaZabYBzV7zaiBKxvIzUf+p3jdR7pjnQp9ellK/bZuSpC58/c scWLy3tTZUM4CALNBTJFcO9Hzs4CPv+TlHbNUH+Dwey9MfnKG0O2aAIjchK9muKMD7qV cW7Q== X-Gm-Message-State: AOJu0YzSrQJz8gp29K4ADYFV/kKYLvc9vfKvSmLop2LI0qzwcRcFVXIJ 1Gh9LRz3Vj9yvmKWfSbcOm+lWC4qo/rxuojq4qE9FA2Z X-Google-Smtp-Source: AGHT+IH7Lz2bsgXRlXVmEy1N1FWU+B2jkCJOUaji3ss9W4t8nenXrc8mQ+UBkedmH21ynkCII1PWna4aO4qUduAH/vM= X-Received: by 2002:a05:6358:9189:b0:134:f326:e819 with SMTP id j9-20020a056358918900b00134f326e819mr12296377rwa.29.1696215702980; Sun, 01 Oct 2023 20:01:42 -0700 (PDT) List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 References: In-Reply-To: From: J David Date: Sun, 1 Oct 2023 23:01:31 -0400 Message-ID: Subject: Re: FreeBSD 13.2 NFS client mount hangs To: Rick Macklem Cc: FreeBSD FS Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Spamd-Bar: --- X-Spamd-Result: default: False [-3.99 / 15.00]; NEURAL_HAM_SHORT(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-0.99)[-0.991]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36:c]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20230601]; MIME_GOOD(-0.10)[text/plain]; FROM_HAS_DN(0.00)[]; TAGGED_RCPT(0.00)[]; FREEMAIL_FROM(0.00)[gmail.com]; PREVIOUSLY_DELIVERED(0.00)[freebsd-fs@freebsd.org]; ARC_NA(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::32b:from]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; MLMMJ_DEST(0.00)[freebsd-fs@freebsd.org]; DKIM_TRACE(0.00)[gmail.com:+]; RCPT_COUNT_TWO(0.00)[2]; TO_DN_ALL(0.00)[]; MID_RHS_MATCH_FROMTLD(0.00)[]; TAGGED_FROM(0.00)[]; RCVD_COUNT_ONE(0.00)[1]; FROM_EQ_ENVFROM(0.00)[]; FREEMAIL_TO(0.00)[gmail.com]; FREEMAIL_ENVFROM(0.00)[gmail.com]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_LAST(0.00)[] X-Rspamd-Queue-Id: 4RzQhm6dzzz3JFn On Sat, Sep 30, 2023 at 6:06=E2=80=AFPM Rick Macklem wrote: > --> I'd suggest you try and disable delegations. I do not know how to > do this on > the Linux server, but not running the nfscbd(8) daemon should stop t= hem > from being issued. The nfscbd daemon is not running on any of the clients. > If the Linux server still > issues delegations How would I determine that? > Hmm, interesting. 10068 is NFS4ERR_RETRY_UNCACHED_REP. > I have never seen (and do not recall anyone else reporting) this error > return. It was days earlier than this issue and *might* be related to someone rebooting an NFS server the client is connected to at that time. Hard to tell for sure without timestamps. > I'd suggest you first check network connetcivity. Both NFS client and NFS > server should be able to ping each other. The network connectivity is OK; network interface error counts are monitored with munin on all systems, and don't appear to be an issue. On that specific system's dedicated interface for NFS: Name Mtu Network Address Ipkts Ierrs Idrop Opkts Oerrs Coll net1 1500 (redacted) 18754677570 0 0 18598354347 0 0 Also, in both cases (which occurred with different client & server pairs), there were three different directories from the server mounted on the client. One was affected. The other two worked fine. Listing files and directories. Locking. Calculating hashes of large files. Creating/writing files. Etc. E.g. /a /b and /c were mounted from nfs.server and /a and /b were fine while /c was hung. (And, as I mentioned before but seems worth repeating here for completeness, other FreeBSD clients had /c mounted with no problems while it was hung on this machine.) > If that is the case, then I'd suggest you capture packets. On the FreeBSD > end: > # tcpdump -s 0 -w out.pcap host > Let this run for a while and then pull out.pcap into wireshark and see wh= at > traffic is going between the NFS client and server. > (Unlike tcpdump, wireshark does know how to decode NFS properly.) If/when the issue happens again, I will attempt to do this and report back. Thanks! From nobody Mon Oct 2 12:40:40 2023 X-Original-To: fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4RzgXm42nsz4w2sF for ; Mon, 2 Oct 2023 12:40:40 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4RzgXm1ZKNz3P8b for ; Mon, 2 Oct 2023 12:40:40 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1696250440; a=rsa-sha256; cv=none; b=vaY/aWcSO6Lfr1U+NyM43Z8WICjfloQJaqIyPWksJLJhzEwi3KR1LALQHvV5CKP7lRp9PF lr1w7WaKBeSxPQpMGviDXH3oDKPOVCBXdj5M+Yvsx1LDyDMbhDC1V6lI59V9MAOpUjn2cI 1bdL9xjmRpOPUNf/Fg2yAe4iTBGhNJIPaoBHikbszJCTYYusuy/tQiuUrbFj4FVVBUnYnP wfVa0lwHRe6WRJPUftu+/Oe93aQG/MahRbuSR5Bg6yisZbPfWuVkFJTFtxGk1a6sOgHckZ NP+aSqQfefyjo53vVK3tUvwGq3jmXrjuv9bimVSHRAA/RlejZd+dQepOSrXtJA== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1696250440; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=GZO0+XRk3VKfmTFuVg94ajtkrOIQHiyYYnkWSijzic0=; b=U6Aahk50QtAC9K5qvF9a77688BE2wsA632CriynCXtJ0hdTxgyVY3BNxT2TiwXHzch3WKJ IOPgscWr72JfyNo8iL7Cdht76M7avobeOVRPc6KEtcFaJErU1+qdQjtRO+16/JHt6xe7eh i54JsrD1l+aLYQ5xe64ANPkLYsPex1nLPRsMfFKLsF7T0j92QI7gidXZZOwYqtnpHf1Y8z fBXjwhYFxKXi5LEOPRxsqokqQE2W6XZ43JHA3qf2iNYx74s8rLcRPAyV9V/3DiLWV9iO1k Z/wCgfGPf9ACB+KkEJVHy6oNlp+Q2wDDDhtDsxR2L8LNypguyCrHJSyZTFUDFg== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4RzgXm0fT1z1Bnm for ; Mon, 2 Oct 2023 12:40:40 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 392CeeUE016704 for ; Mon, 2 Oct 2023 12:40:40 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 392Ceee2016703 for fs@FreeBSD.org; Mon, 2 Oct 2023 12:40:40 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 273953] panic: vfs_remount_ro: mp is not busied Date: Mon, 02 Oct 2023 12:40:40 +0000 X-Bugzilla-Reason: CC X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 15.0-CURRENT X-Bugzilla-Keywords: crash X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: markj@FreeBSD.org X-Bugzilla-Status: Closed X-Bugzilla-Resolution: FIXED X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: kib@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: bug_status assigned_to resolution cc Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D273953 Mark Johnston changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Open |Closed Assignee|bugs@FreeBSD.org |kib@FreeBSD.org Resolution|--- |FIXED CC| |markj@FreeBSD.org --- Comment #6 from Mark Johnston --- I'll presume that the bug is fixed now. --=20 You are receiving this mail because: You are on the CC list for the bug.= From nobody Mon Oct 2 14:18:26 2023 X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Rzjjf6Tg1z4w7SW; Mon, 2 Oct 2023 14:18:30 +0000 (UTC) (envelope-from grahamperrin@gmail.com) Received: from mail-wm1-x32b.google.com (mail-wm1-x32b.google.com [IPv6:2a00:1450:4864:20::32b]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Rzjjf0cBGz3blJ; Mon, 2 Oct 2023 14:18:30 +0000 (UTC) (envelope-from grahamperrin@gmail.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20230601 header.b=gS76whN0; spf=pass (mx1.freebsd.org: domain of grahamperrin@gmail.com designates 2a00:1450:4864:20::32b as permitted sender) smtp.mailfrom=grahamperrin@gmail.com; dmarc=pass (policy=none) header.from=gmail.com Received: by mail-wm1-x32b.google.com with SMTP id 5b1f17b1804b1-40537481094so169520915e9.0; Mon, 02 Oct 2023 07:18:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1696256308; x=1696861108; darn=freebsd.org; h=content-transfer-encoding:in-reply-to:autocrypt:content-language :references:to:subject:from:user-agent:mime-version:date:message-id :from:to:cc:subject:date:message-id:reply-to; bh=Lf4QLtYuUDoidsdLmdg4Wpc//Av3qJ2DTVWNmSjdUVk=; b=gS76whN0ruOvL/3w2DCj9oioUjRklcue1mF2suNGgSXaPFVMpB7Ts/TpHODlQDBsGR yb/veM44La5fgL1uTYkSvaNDnoIMGLB1z9YsBU4qloKRP6yP5ErP7tbvm1ATtRYZKrfo VL1r/JphWfuk6alFEfpTug2z4pb2wWyqtgK9jeMfU68kuH8HKGh7sQT/MOVbwszmEkE4 sTm/JioGx63RB+3EOPKOBg0V3SOmkSGzEGveCt2ZyrZ7CDOfpWNcrIzWmRM49kjTPsxL tjUfW+cglUtBxg9ZCQaWR9wPz153adxTKVP49gKHwS/vIpPf8M8HWdw6x5gE4yJXjK5O eB0A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1696256308; x=1696861108; h=content-transfer-encoding:in-reply-to:autocrypt:content-language :references:to:subject:from:user-agent:mime-version:date:message-id :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=Lf4QLtYuUDoidsdLmdg4Wpc//Av3qJ2DTVWNmSjdUVk=; b=GaSI/qDTjygjbezYuTULW9XrNkq/yzHHp7rSQ+z7XxJ+GNVktbWN8ckt33hXjJ6rR6 ckuekrww2kNYRBFeStVBsRH5A79BiL2EGiLbnUsfxeNi7WsWWqnm8gG2AOY9NkuH0E/6 YFDG1iCwz36JtQVV507DLOXLMq5AWJVr6lZxR+RnJPAEXUZS/jP8s3et4hWasD1bAXOh Cu5Bi4TbWacT3W6ct57A+V+MI/CPO9KYWoCStoDcpdzUcIzwuaCwt39niKG/w2WP3V4y jYN+ebZf4KR0JGlr4qX17pSvaniSt9s9hi1isfvJ6DRgHnylc2cBrlM6x5zH03pWpTxy X0Fg== X-Gm-Message-State: AOJu0Yz9oJVFBIhYnFoYm3n6va5cFJ7fRqggbGzXTq3l+Fu1qlKPUU2v K0xfUcodlc5D4n++1+EcOQDPNDEym9uHQg== X-Google-Smtp-Source: AGHT+IFHbQBNujWu7GcYTKKdX2j8C6xPxKOxPhQEUq2zXGgfyjGp90ucnn+TqFNvBAzSPUa2JHJ+KA== X-Received: by 2002:a1c:720c:0:b0:402:8896:bb7b with SMTP id n12-20020a1c720c000000b004028896bb7bmr10718385wmc.6.1696256307612; Mon, 02 Oct 2023 07:18:27 -0700 (PDT) Received: from [192.168.1.6] (80-42-66-93.dynamic.dsl.as9105.com. [80.42.66.93]) by smtp.gmail.com with ESMTPSA id s12-20020a7bc38c000000b0040588d85b3asm7334253wmj.15.2023.10.02.07.18.26 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Mon, 02 Oct 2023 07:18:27 -0700 (PDT) Message-ID: <98646959-dd13-4a01-8a16-7b82d64c7ac3@gmail.com> Date: Mon, 2 Oct 2023 15:18:26 +0100 List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla Thunderbird From: Graham Perrin Subject: Occasional suspend/sleep failures: race? USB, HP dock, ZFS, three L2ARC devices. To: freebsd-fs@freebsd.org, FreeBSD CURRENT References: <5a642c3d-6b32-4614-ad7d-40f72b92e537@gmail.com> Content-Language: en-US Autocrypt: addr=grahamperrin@gmail.com; keydata= xsFNBGKYt7ABEAClu83dJ3ZKfVgPOk9YKRv0Z+dl2b88+k9R4vwAmElgguYdKE7yhnQNhhWM v9vi6AFrBMc2oJdVHJ2OrXfwpELBFIgiSMEWNsC4e+Z3HtSajcl+pFZsP7ciiSoycj/w3wIV kAZoVGbhyIbNG7fbCEJ8q81TbfsGypV3bRmbZVvGNecBguYiooBtz2Qht1p3itXMkIA6P9pS YDl+6QddZLyUUAjAnFv2QDoYSHLnaDUWw4oONZsB0SKVu8jMIBh4uJZoYEOvdvc9jQQdOpA2 CAgA6ulfm42Ikr9lKBUUCtjqiWAhJ7iXOTyHAIdR4Mf8alCE6tdTq6dHdIt+GktTY7oYNyL2 3aD3C7I5waU0SFXvJcOMG10QLfwYQMOQoYQ9XJ0U5A28WYiDcylDdUWT7SappP1e1ZMeJWWO y14mxxNzHaJSI4rK8P/p5tp3Q7SSC4k5gMh9zKba3K2ApCWNbVLGvXsJeQkZZNvu70tE81ey AHI5iZcB6D7WaHysBUmsKaEpbcmm1ZThTnGL0SHEl5to5Jab5Fg6O+Cnly5sVz5lX/v8Aosx kKNei7SCVqXOVtteQeGxWbXWbhPgbMyc0Gi3DuxBI/yvJ43k/rJysQlLGLWfJx/UXprwLluC PDK9EvKEB+fD1Z349uzp1sKr3ihpySbyKI8fpudftnAz4EsoCwARAQABzSZHcmFoYW0gUGVy cmluIDxncmFoYW1wZXJyaW5AZ21haWwuY29tPsLBlAQTAQoAPhYhBFk/5bLDBwftvJcvCrdn SG9KGNQLBQJimMMBAhsDBQkFo5qABQsJCAcDBRUKCQgLBRYDAgEAAh4FAheAAAoJELdnSG9K GNQLbHAQAJi998y42bEbq5HmABYovmAEtQj33YSUWyc9QRmAHpN8Er3lTKsgmZcVChB5Fu/d go2oYynDjlVpA7+wiSmg4AG78mOYbg/e19XMhrH0keDKqZXFkU+G7agR0mF09qvpQZ9MTJYZ 2u7FtytZK665UfipOdV8eGn2hFC/WynjUwEzKyryBgbbLAEbfOPeZNry4h2ZPWbtTvx/PE/V X3Vh2oGqYx69DCGz+0xEhy62ZKbkX5SL8LUf/1WViyCVzsHasFxmFxYPWIfBy8ayQ7xapz7M cSXSQyu4oDT4qh9eZiGP9/aAcZKHcV6t9y77JGhUJ/5O1sANKMa3YhgimE+Z86LHYa1IH774 PHj1nAXBwS+Cj/1l/NQoQcyjvOj8zuCsMJVaLMb6B46YsReP4+3yBLpyeBC//t6zWPbgAkWW VjROC0dXUAMTFpnA6NZe3UghG+Nc4fnCLGOhc2nyWFYHIaYV6Hv1ITFSem9DdeNnR1CFm1VM TJ7i7TuqYM+WZTkoUsTf4c46hS/ZNJZSCxh0s9yYr+BYk3XBbd+ElaZ1dJE6cuSVdw15+P2h DnprurxC4byl4YFkn+UAVvQsOgeq6aSHLOHX0weYu1OLoiPYsTdyGhne72+kDhEEdFD5aHdQ PFrbQIrqWLV0a04++0ZwGpNvXtgnWhDdAQJDwGsSSwbLzsFNBGKYt7ABEADRb1tZuh7DPYET 0wK6fe7owbYgM+RfKhmcrGgR2HI9M2q6+0WKF/ITnggWdIW2Ecc4z2boLz/cwvPGCS7/YxZM 61KklGCwuS7q1s04XnHDWHuFxfXQPzAdVmNO3bYoMZbJjHXs6sB2u5ksiwPwaMAWWaGkviSj c5pwvHCiTmX5vH5CBj/Vi+5ESyX38vK4JM5S/m4ouI/6M9biyFgimV+v3vVyCxJCT1gI9g4o GIh1qq5S433b1fihn4yHPf8XOKyBpA/QcwLONViBqJL5nnOxpsh344rNxn2R7CcRzzicOV+e 2IbMem4lwNWQlZKoRotKXZi9LqN5mynSBYqAUdoZum0QinWT9F22B0Qex5PH1zAt9i2W91Vd kcPB3LwkRXj07ycRtsSzpgPA6fLc6AsoWFslHl8kVOO5eJIA4xhjlPa+W8lguQHZ0iX+5uAv 2eAgXR2swADuHPuENNFStmsgAMl8OOOgtq75yA5TpyIzxMuXV9Nmp0VfIaUM/IdLdmxhc1pC c320l5fYMHVLFAReWEbSj2QH8YzWfpXHIegutWWYEbH9SiDXgS9KoKmCJV/Qa+x6/b8y3pOZ vnIbCDaynC2Yr50s8gRa9kb54JE8Z+p8r16U3SEsK3PtUi0RF0e51danCVHrrE6/Hat2XUO/ 6nnYgVgFOrLao6Gh/VMs8wARAQABwsF8BBgBCgAmFiEEWT/lssMHB+28ly8Kt2dIb0oY1AsF AmKYt7ACGwwFCQWjmoAACgkQt2dIb0oY1Av7qg//YjCZg8VXyMzXssgIQpROKKqh5V0UBSQl rM3tq4tWhyg0HVMugQj0Om+iNPsEEOGHkm6tyhHMzlKGpAc/l0iAM+8twIyg44Yo5+DcfFXr OMTbTw9T9jDsWOkOBksxy29iYhgpqpWdDBnhXvrJp/FNAiX8CfzrIOZeFPydDoEiKBEXAxfe a9o5J/JeVnZiUeoiFe7i68nZGsb4JxhPczNfqW12t0Ll5/ibjszg5BgjXiLao0KqbWNh4bS5 CVwH90Or+5qqWgzWPeBiuz+rN2QXE/V/fL44GEj1YKASCqmaiYRgjoRFubz1aq1wCXMXY3Iq d4525rscUgS7HBxbblnyTodUPaamN/2nSzcmE/Pkx8MApDSgZCIhs0RTAg+/AoX4HULV1rSE TQwMrBEQt84Tw5W5rHsvXKr4ZEsJUpbPLWYTISsp23nHR+vZtL/Ug+OWCmHC7X7D21xk/xVJ 4sA1RLJBKdCHtnyA4Unv/kNS1KVGxHnITVyw1a71QJADu4qsdtM5u6CyYUhqhM1oseWtV6j+ Qi8KC/G4C3AgZf06fe2fVl42z2grTabL4bC6FQXMwTX2dsm5NakWjUCmUL8uwsQE7ZA4zKxo EYI1YV9q1birpzncYRupr1qnMoggMUHWq0IBYshFQrEO8PeVUZBw7/GfAeh3argdw2Qu748T Cyw= In-Reply-To: <5a642c3d-6b32-4614-ad7d-40f72b92e537@gmail.com> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-Spamd-Bar: --- X-Spamd-Result: default: False [-3.05 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20230601]; R_SPF_ALLOW(-0.20)[+ip6:2a00:1450:4000::/36:c]; MIME_GOOD(-0.10)[text/plain]; NEURAL_HAM_SHORT(-0.06)[-0.063]; XM_UA_NO_VERSION(0.01)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; FREEFALL_USER(0.00)[grahamperrin]; FROM_HAS_DN(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; TO_MATCH_ENVRCPT_ALL(0.00)[]; ARC_NA(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[2a00:1450:4864:20::32b:from]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; MID_RHS_MATCH_FROM(0.00)[]; MLMMJ_DEST(0.00)[freebsd-fs@freebsd.org,freebsd-current@freebsd.org]; REDIRECTOR_URL(0.00)[goo.gl]; DKIM_TRACE(0.00)[gmail.com:+]; FREEMAIL_FROM(0.00)[gmail.com]; TO_DN_SOME(0.00)[]; SUBJECT_HAS_QUESTION(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; RCVD_TLS_LAST(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; FREEMAIL_ENVFROM(0.00)[gmail.com]; MIME_TRACE(0.00)[0:+]; RCVD_COUNT_TWO(0.00)[2] X-Rspamd-Queue-Id: 4Rzjjf0cBGz3blJ On 31/08/2023 19:42, Graham Perrin wrote (to freebsd-current alone, ): > I have a suspend.sh script that aims to take three cache devices > offline before sleep of the computer: > > % grep -v -e '# ' /etc/rc.suspend | uniq | grep -B 3 -A 2 suspend.sh > #!/bin/sh > # > > /usr/local/sbin/suspend.sh > >         echo "Usage: $0 [apm|acpi] [standby,suspend|1-4]" > % grep -v -e '# ' /usr/local/sbin/suspend.sh | uniq > #!/bin/sh > > while mount | grep Transcend 2>&1; do >    zpool export Transcend >    sleep 5 > done > > zpool offline august gpt/cache1-august > zpool offline august gpt/cache2-august > zpool offline august gpt/cache3-august > > sync > > killall pulseaudio > > while fstat | grep -e dsp -e mixer 2>&1; do >    fstat | grep -e dsp -e mixer | cut -w -f 3 | while read pid; >       do kill -15 "$pid" >    done > done > > sysctl hw.snd.default_unit=1 > > % > > > … On the morning of Tuesday 26th September, I predicted a failure (internal HDD ada1 was busy with a buildworld), so I took a screenshot before attempting to sleep the notebook (HP ZBook 17 G2). The album at begins with cropped and non-cropped versions of this shot (08:07:51). 08:10: the failure to suspend, photographed. L2ARC device gpt/cache1-august at the foot of the screen. 08:11: the HP dock. Two of four USB ports at the side occupied by flash drives, both Kingston DataTraveler 3.0. gpt/cache1-august above, gpt/cache2-august below. 08:12: the screen, after physically disconnecting various peripherals but not the dock. At the foot of the screen: a Kingston DataTraveler 2.0, which provides gpt/cache3-august. This drive was at the side of the notebook (was not docked). 08:13: the screen, after removing the HP ZBook from the HP dock. REMARKABLE: neither of the DataTraveler 3.0 devices (in the side of the dock) appears in any photograph. Is this a race condition? As if the OS, or ZFS, prematurely lost the ability to handle the docked storage devices on USB. Can I do anything to make the sleep-related script (above) stronger, to reduce the risk of failure? Might things be more reliable _without_ acpi_hp(4)? FreeBSD 15.0-CURRENT on the morning of 26th September would have been n265350-72d97e1dd9cc: % bectl list -c creation | tail -n 5 n265350-72d97e1dd9cc-c -      -          882M  2023-09-19 12:45 n265350-72d97e1dd9cc-d -      -          775M  2023-09-23 12:09 n265538-915af883221a-a -      -          256M  2023-09-26 16:30 n265538-915af883221a-b -      -          257M  2023-09-28 09:05 n265538-915af883221a-c NR     /          503G  2023-09-29 23:34 % % grep acpi_hp /boot/loader.conf acpi_hp_load="YES" # dev.acpi_hp.0.verbose=1 % man 4 acpi_hp % experimental, etc. From nobody Mon Oct 2 23:08:35 2023 X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4RzxTX0hthz4wbpx for ; Mon, 2 Oct 2023 23:08:48 +0000 (UTC) (envelope-from rick.macklem@gmail.com) Received: from mail-oi1-x22b.google.com (mail-oi1-x22b.google.com [IPv6:2607:f8b0:4864:20::22b]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4RzxTW2fNXz3bk2 for ; Mon, 2 Oct 2023 23:08:47 +0000 (UTC) (envelope-from rick.macklem@gmail.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20230601 header.b=X02AUZSd; spf=pass (mx1.freebsd.org: domain of rick.macklem@gmail.com designates 2607:f8b0:4864:20::22b as permitted sender) smtp.mailfrom=rick.macklem@gmail.com; dmarc=pass (policy=none) header.from=gmail.com Received: by mail-oi1-x22b.google.com with SMTP id 5614622812f47-3ae55c2e12bso221816b6e.0 for ; Mon, 02 Oct 2023 16:08:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1696288125; x=1696892925; darn=freebsd.org; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=5XfvAE7ksFCBaPOsZqL9T6ZwsuztRJgQXBwMBl5XHyw=; b=X02AUZSd3Y91W58Fp8kBikbvTodCvQnI8zOCX5pGXxQarusM2t4VaNrm+Lrxnfz/Z8 /eLAzutBNcW4XecBdCL/xe43ji0sNJtnaL3CWJcaqBr9TKkAaQBpMgqpn807cribDoY5 KN6nNBIX4dZ1oTp7R/0bwx+f2JoBtZckV1cfmI8gnnJ3P0nGjwbyWzlQ+HMfHdfV39pb ixPyQfTp66lMr1mbt4N/odAeHNH7kWvF0MqqZikeX1tJgWD6MtjsB19x3UZCmoE/8XgJ Du2eMc/IxpqpUEmOFja6tqFSJaL9eVXI9xabDW8KNj2zZIzkAhiB1JIpAEtb9mQtB2Dd 5f8w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1696288125; x=1696892925; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=5XfvAE7ksFCBaPOsZqL9T6ZwsuztRJgQXBwMBl5XHyw=; b=k6hjW9T8zIqwbPhpQlJK4vscKcsCnvW/IO4Ay/vsN9rE5493+GOH+vCZU0IMlRbP8z PvBFPkDO/n5NEtjby8Yhhfn9uKXHArPgP2xtKq+HFaqcol7hj7Qh9bxb2p60fg2B4IrB Ul8G/2QxDgi4rsC4OPBB5FWCn3StvEXpvUjbm2JU22WQiPy7e7ZDfl18ZqbsgGuxO5Bi jrLk/Q5hHmsFzmT5LYIz0ARxwYvvNTkJbCVTuBPKesArKOpVDCN2OvwQVHcBc8Okeb64 j9cM9xCS3p0x2VLlkjydXeAIrb5nyOldGo0jzFF8qLEk1xnTzj9rtPHdkO/WWuh1+1ro HHTA== X-Gm-Message-State: AOJu0YzDUkpoLs+iRxn4rRvLMF+W99JzbSwMQJC7llyCWQ1RDGExIj1G mYau2dcHlPl8ux2Jg0uX3fd3/P3h9cK/wJgdYkq6YYI= X-Google-Smtp-Source: AGHT+IFrgZmxmQpP9TtT0/EH6DRqdwuGLKX4/Z+BjIohieqf0l/RSPlKhWBCUvRH0z42rbjEaPG7IrudPmyuKxoGG3g= X-Received: by 2002:a05:6808:1448:b0:3a9:11cb:d5a6 with SMTP id x8-20020a056808144800b003a911cbd5a6mr17026280oiv.32.1696288125039; Mon, 02 Oct 2023 16:08:45 -0700 (PDT) List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 References: In-Reply-To: From: Rick Macklem Date: Mon, 2 Oct 2023 16:08:35 -0700 Message-ID: Subject: Re: FreeBSD 13.2 NFS client mount hangs To: J David Cc: FreeBSD FS Content-Type: multipart/mixed; boundary="00000000000021f1f20606c3dbed" X-Spamd-Bar: -- X-Spamd-Result: default: False [-2.19 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-0.91)[-0.906]; NEURAL_SPAM_SHORT(0.71)[0.711]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20230601]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; MIME_GOOD(-0.10)[multipart/mixed,text/plain]; FROM_HAS_DN(0.00)[]; FREEMAIL_FROM(0.00)[gmail.com]; TAGGED_RCPT(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::22b:from]; ARC_NA(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[freebsd-fs@freebsd.org]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; TAGGED_FROM(0.00)[]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; MID_RHS_MATCH_FROMTLD(0.00)[]; HAS_ATTACHMENT(0.00)[]; MLMMJ_DEST(0.00)[freebsd-fs@freebsd.org]; RCPT_COUNT_TWO(0.00)[2]; FREEMAIL_TO(0.00)[gmail.com]; FREEMAIL_ENVFROM(0.00)[gmail.com]; RCVD_COUNT_ONE(0.00)[1]; MIME_TRACE(0.00)[0:+,1:+,2:~]; FROM_EQ_ENVFROM(0.00)[]; RCVD_TLS_LAST(0.00)[] X-Rspamd-Queue-Id: 4RzxTW2fNXz3bk2 --00000000000021f1f20606c3dbed Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On Sun, Oct 1, 2023 at 8:01=E2=80=AFPM J David wr= ote: > > On Sat, Sep 30, 2023 at 6:06=E2=80=AFPM Rick Macklem wrote: > > --> I'd suggest you try and disable delegations. I do not know how to > > do this on > > the Linux server, but not running the nfscbd(8) daemon should stop= them > > from being issued. > > The nfscbd daemon is not running on any of the clients. > > > If the Linux server still > > issues delegations > > How would I determine that? nfsstat -E -c and then look at the number under "Delegs". It is a current count of delegations, so if it remains 0 over time, no delegations are being issued. > > > Hmm, interesting. 10068 is NFS4ERR_RETRY_UNCACHED_REP. > > I have never seen (and do not recall anyone else reporting) this error > > return. > > It was days earlier than this issue and *might* be related to someone > rebooting an NFS server the client is connected to at that time. Hard > to tell for sure without timestamps. Yes. This would occur after a NFS server reboot or some other occurrence that causes the client to do a TCP reconnect. I have attached a small patch which should make the NFS client handle this error correctly. rick > > > I'd suggest you first check network connetcivity. Both NFS client and N= FS > > server should be able to ping each other. > > The network connectivity is OK; network interface error counts are > monitored with munin on all systems, and don't appear to be an issue. > > On that specific system's dedicated interface for NFS: > > Name Mtu Network Address Ipkts Ierrs Idrop > Opkts Oerrs Coll > net1 1500 (redacted) 18754677570 0 0 > 18598354347 0 0 > > Also, in both cases (which occurred with different client & server > pairs), there were three different directories from the server mounted > on the client. > > One was affected. The other two worked fine. Listing files and > directories. Locking. Calculating hashes of large files. > Creating/writing files. Etc. > > E.g. /a /b and /c were mounted from nfs.server and /a and /b were fine > while /c was hung. (And, as I mentioned before but seems worth > repeating here for completeness, other FreeBSD clients had /c mounted > with no problems while it was hung on this machine.) > > > > If that is the case, then I'd suggest you capture packets. On the FreeB= SD > > end: > > # tcpdump -s 0 -w out.pcap host > > Let this run for a while and then pull out.pcap into wireshark and see = what > > traffic is going between the NFS client and server. > > (Unlike tcpdump, wireshark does know how to decode NFS properly.) > > If/when the issue happens again, I will attempt to do this and report bac= k. > > Thanks! --00000000000021f1f20606c3dbed Content-Type: application/octet-stream; name="replyuncached.patch" Content-Disposition: attachment; filename="replyuncached.patch" Content-Transfer-Encoding: base64 Content-ID: X-Attachment-Id: f_ln9i58ge0 LS0tIHN5cy9mcy9uZnMvbmZzX2NvbW1vbmtycGMuYy5yZXBseXVuY2FjaGVkCTIwMjMtMTAtMDEg MTY6NDc6MDQuMDI2ODcwMDAwIC0wNzAwCisrKyBzeXMvZnMvbmZzL25mc19jb21tb25rcnBjLmMJ MjAyMy0xMC0wMSAxNjo1MDoxOC41Nzc1MTQwMDAgLTA3MDAKQEAgLTEyOTQsNyArMTI5NCw4IEBA IHRyeWFnYWluOgogCQkJICAgICBuZC0+bmRfcHJvY251bSAhPSBORlNQUk9DX0xPQ0tVKSkpIHx8 CiAJCQkgICAgKG5kLT5uZF9yZXBzdGF0ID09IE5GU0VSUl9ERUxBWSAmJgogCQkJICAgICAobmQt Pm5kX2ZsYWcgJiBORF9ORlNWNCkgPT0gMCkgfHwKLQkJCSAgICBuZC0+bmRfcmVwc3RhdCA9PSBO RlNFUlJfUkVTT1VSQ0UpIHsKKwkJCSAgICBuZC0+bmRfcmVwc3RhdCA9PSBORlNFUlJfUkVTT1VS Q0UgfHwKKwkJCSAgICBuZC0+bmRfcmVwc3RhdCA9PSBORlNFUlJfUkVUUllVTkNBQ0hFRFJFUCkg ewogCQkJCS8qIENsaXAgYXQgTkZTX1RSWUxBVEVSREVMLiAqLwogCQkJCWlmICh0aW1lc3BlY2Nt cCgmdHJ5bGF0ZXJfZGVsYXksCiAJCQkJICAgICZuZnNfdHJ5bGF0ZXJfbWF4LCA+KSkK --00000000000021f1f20606c3dbed-- From nobody Tue Oct 3 01:29:59 2023 X-Original-To: fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4S00cS59sRz4vVhs for ; Tue, 3 Oct 2023 01:30:00 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4S00cS46mXz4JvW for ; Tue, 3 Oct 2023 01:30:00 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1696296600; a=rsa-sha256; cv=none; b=ECZ315tX5zjdH4IYxU4YTAQ1jUfFJkpvjJcE2ZmJf83E37jbv04oJjF2pw4GpeYXUAEfor V0mNBde0ZbvOY7jYqcpwv0Jp5Hi3/UjWfOZxYpPgJ7TdED751jyBXyDOWnYtoB2p2K13oK tIdu4q5vsDRPUK8YRsGlv+JsPMId3hh1NG2UN1Pg5Vru5XWuL9lXxEMWInhwC7/mgimcin DCX96t2fgSKk24haUFZJ8Es3BRWGIyod/o61D5KtKqWoLUCk9XEwuxsn5i2RpYxPRX+Tgu 3Ub5/487pjIljhJD79uTmgQ1m4EibyTkvTLn5BtjVVox3VTWyVmuupF+sQuR2w== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1696296600; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=SbqKX9s8a0+VFZ23NqgTHwgs11s+7iJaSKf1dBOqYtM=; b=V/eJCsKgITtju67Mt2eZW+jwNkpKxkvxGHU9kw7ZUwBzPfk86sgy9cIa3mujG5Zt/Y6WbL PL7q8p6e0rGYZRhfVCpqNrixMiqJSzl66axFKWtoCWvwDBfz+2FcwYumPIq1Yh/TE3uQ2a jciMQU18nXVq1FZ7+dXp8PZ/Sv0VltinpV86NBB6s6FiPPxX1qK48fTZWow9Sbk7wKQqL6 i5n0RaL/ySQO3UrlqnpPF1iaXGojg5/H9OBjjh9YEIQUzOmeefmngx1J0hH+URMJnLF6VJ QffCTmo2FF2axXGD7jpcBMaOGzpje/OijhKP5OY3lMm/aa5m6jgxQfTb5CtisA== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4S00cS3B4KzbTh for ; Tue, 3 Oct 2023 01:30:00 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 3931U07E049564 for ; Tue, 3 Oct 2023 01:30:00 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 3931U0mb049563 for fs@FreeBSD.org; Tue, 3 Oct 2023 01:30:00 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: =?UTF-8?B?W0J1ZyAyNzE5NDVdIHBhbmljOiBkZWFkbHJlc190ZF9zbGVlcF9x?= =?UTF-8?B?OiBwb3NzaWJsZSBkZWFkbG9jayBkZXRlY3RlZCBmb3Ig4oCmIChzdGF0KSB8?= =?UTF-8?B?IChwb3N0Z3JlcykgfCAoTFRPIHRvbWwuNTRkMjgxNzEtYykgLCBibG9ja2Vk?= =?UTF-8?B?IGZvciDigKYgdGlja3Mg4oCTIGR1cmluZyBwb3VkcmllcmUtYnVsayg4KQ==?= Date: Tue, 03 Oct 2023 01:29:59 +0000 X-Bugzilla-Reason: CC AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: CURRENT X-Bugzilla-Keywords: crash, needs-patch X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: junovitch@freebsd.org X-Bugzilla-Status: Open X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D271945 --- Comment #30 from Jason Unovitch --- The pull requests cited earlier between 15107 and 15122 landed in stable/14 before the releng/14.0 branch (https://cgit.freebsd.org/src/commit/?h=3Dstable/14&id=3Df789381671a3f97496= d496d8f996a67eaa8b1a33). I've seen everything perform well since so hopefully anyone else out there= is seeing the same. --=20 You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug.= From nobody Tue Oct 3 19:18:28 2023 X-Original-To: fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4S0SKK0Kxcz4vsk8 for ; Tue, 3 Oct 2023 19:18:29 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4S0SKJ6Bynz4WHy for ; Tue, 3 Oct 2023 19:18:28 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1696360708; a=rsa-sha256; cv=none; b=PKrwMYJUfnHCB5MQ+eXm0AD8nxnwPtHbD4SYragRWW60Brmi+089AeV6VC+MwY7JAGTfcZ tXzUXOc13cJXDBSHLwNwqW4yiZ3/cgJq2yHK6GGfDH98OIr80vJUGWQRyMr+/PYUbf7Gex tJqAHYzvP9+6KjScDx19Dxzy4EEtnNhNZGHyibxzmTjkcdHsj3djhxnCnhVf+w7fG24py9 JGhYFpMCRWe3kx2vFCRveyf+xqWyk/knxCuEx3orfIABBCC/78KTzfG8skE3vx0DrnPB/S +fY+k1gBoRAtb2oyEfEMC+cHfhYkAAkK55F3GoR5caWDf/3V5/1tj6BaOGitag== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1696360708; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=phJjrg/zwEnmx9DpoSDwF2O7UJSvV4gdXE3oeuNICM8=; b=uS3nCaknGknUEA+RWVowgxQu97NmXl+0Ld92XCkGIsu0eY5qDdZfA7fmgDUCCsVrDRqwZo VYQypU4PTB+2blYEx4k8/l9UsZdO6gFoVdnN/rLNz3SCcJ0UrjG3hH4PydWcnBI5VpmRne alv94RtXVQ/N/sccCg221sEZ4K3HrM1E3HbPcMbWaaeksSQohDUKmi3gH5hmPrOXyElih6 L5lrV+vhRjScdHi5Ck0p47RseATqra0eQ6z750nvfKhN3KZ0Ws8Kvru9/W4UvrHaF8nVzo Bwl6cyXFb7xFHcf/HH145nnr2bQHGmycacUsuEVIfRj2PQEOFoI+MdwR/gDEPw== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4S0SKJ5Fl1z16H2 for ; Tue, 3 Oct 2023 19:18:28 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 393JIS0j025406 for ; Tue, 3 Oct 2023 19:18:28 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 393JISMs025405 for fs@FreeBSD.org; Tue, 3 Oct 2023 19:18:28 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 272325] Page fault, possible ZFS related Date: Tue, 03 Oct 2023 19:18:28 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 13.2-RELEASE X-Bugzilla-Keywords: crash, needs-qa X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: grahamperrin@gmail.com X-Bugzilla-Status: Open X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: bug_file_loc keywords see_also Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D272325 Graham Perrin changed: What |Removed |Added ---------------------------------------------------------------------------- URL|https://bugs.freebsd.org/bu | |gzilla/show_bug.cgi?id=3D1743 | |72 | Keywords| |needs-qa See Also| |https://bugs.freebsd.org/bu | |gzilla/show_bug.cgi?id=3D1= 743 | |72 --- Comment #9 from Graham Perrin --- Do all pools scrub without error?=20 How is S.M.A.R.T. status for each of the hard disk drives? (In reply to porsolic from comment #0) > Similar PR from a decade ago: #174372 >From bug 174372 comment 4 (the recent closure):=20 > =E2=8B=AF I'm sure the other issue that was linked is unrelated to this. --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Tue Oct 3 19:21:09 2023 X-Original-To: fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4S0SNP6ML1z4vtGx for ; Tue, 3 Oct 2023 19:21:09 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4S0SNP4qsbz4X2D for ; Tue, 3 Oct 2023 19:21:09 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1696360869; a=rsa-sha256; cv=none; b=I2YFJQrKv07dNcs/brJTWqbM980TOjiuEU5TqeCFGS+Ex8uvGs20W8BxPke7HJ7zJQlSP3 zeZStKvytJNBS71b0kF7Fx6U382DGjBcrhl7C8tSWyijPWqCadfpKFRHSo1uYjeiMK4nIZ WYDaSNfFRzR+D9Icee0jMWdYGgTLNNJPuJoN/dUaNquM++cGYE6rqlPX7h7Mep/gmz9osY u/MtolOTdvMsIPGQH5zIBK9527kN5OCCGbDIAJIHDUh4syw2stBtrYS0f/bIY+PbNjytJS cPm1igbSqQn1rpHXmtB6jqtqwdnd4YFDUStxxZ82rX79f3skL9NkYJgiFHRgtw== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1696360869; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=rOV8p4cRRohHi5JKhk/40XzRd24e5Tm+zoR6rzu3BH0=; b=qKu5Kj6GvjX9fCnpdB1hMue9JcU5P7ytmjWWYcZiLemkwWzzPDh+Pty1IOH6RZ4Gtj6r99 sz2ufLqjrhsV8/v1I9zvIbujFUF0OcMCdwcDxpuB7+9VoYLRio9lkPYV3aw9dF1O/HkJcn CkIVlE6kr4n/u0IgrLiYjkJdF4TIzTb6ZfnMDpyRkdeNX9hphPra5fvqJ4B2u3zQoGGUN5 PGCuDVGouf4B7HMKgNUp8avh8JlLLzp06JygOOWMyEaB7Dm9CI88OWe7Oli11tU4MrhOge eguHUKlISPdddftwVFDmnVZIHYVRKaU+SSV2WykivXbfVeP9kS34uiCWCDll8Q== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4S0SNP3wfqz16H7 for ; Tue, 3 Oct 2023 19:21:09 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 393JL9RX030086 for ; Tue, 3 Oct 2023 19:21:09 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 393JL9ts030085 for fs@FreeBSD.org; Tue, 3 Oct 2023 19:21:09 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 272325] Page fault, possible ZFS related Date: Tue, 03 Oct 2023 19:21:09 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 13.2-RELEASE X-Bugzilla-Keywords: crash, needs-qa X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: grahamperrin@gmail.com X-Bugzilla-Status: Open X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: see_also Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D272325 Graham Perrin changed: What |Removed |Added ---------------------------------------------------------------------------- See Also|https://bugs.freebsd.org/bu | |gzilla/show_bug.cgi?id=3D1743 | |72 | --- Comment #10 from Graham Perrin --- (Sorry, the repeat addition of 174372 was a slip of the fingers. I'll repeat the removal. Apologies for the noise.) --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Wed Oct 4 13:44:12 2023 X-Original-To: fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4S0ws90mG6z4vy36 for ; Wed, 4 Oct 2023 13:44:13 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4S0ws85V10z3bQ1 for ; Wed, 4 Oct 2023 13:44:12 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1696427052; a=rsa-sha256; cv=none; b=v//CuuTKYObseKQcGPuY6yBw7zdccqU6xnT7vDBmay0bqv+kwvZ3Mwa3M6TEV7hmByFoHt TKlyYAC1rgepBG/nHoTe3gWEeNUhafGU8HVNnPh+Ev5KwIEUCZUJN/nKoKnDMainqIL8dV wU6KG0+q7lDA4/HDg0JI+/5whBOQXus2zXBOSv8i+eNrq9v7rLc8k6c382iqnVKxJBf4aN N6HlJB5CJRviZpwWaVtw1ErnerAmoCPqo3iTmWCeUU7IcuqGBp4MVerbRwZ8gGgqKJeadw uJLv96djWKZBad7hCVqxlzwDdZz+P+Cmbjetsay7CNVTd/+I8Zjl5YKKF/uXZw== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1696427052; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=dbJZ7A0Y71XaYz1I9km++isFMI4sGd0/U+jHtXBEla0=; b=HHCs9U9+OxtKNWkDgW6mNUgh3r3QWDr2uhJ4j+9mDhD+WYog+bwRxI85Uouq6q1gfKAPVJ xMyxkUEKeI/mvOQDEhfOqmxoopExl2ZLb+9WY+NclQ6K8zEW5TnsfnsvMpQLh0Bez+19h3 e5U5FpgIqrFVPsEbcUc0KAhXvzKsajfARfnSjd8hjbpcjwBkaQ9ak/lSX5vsn+Bd82gl5Z CZ3DDaoX5KWNKPla0Wp6x13sGv8DSLaZOyU4afqSwhTe8CNobIus+4r+Ik35BqnV9knP7v BOWjiVe77gioh9kgb8qQvsClUN5zYa2fiKYyHp4B//HW76Rtoy9yrLAp/NbeQQ== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4S0ws84b0VzfYW for ; Wed, 4 Oct 2023 13:44:12 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 394DiC9G078068 for ; Wed, 4 Oct 2023 13:44:12 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from bugzilla@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 394DiCj9078067 for fs@FreeBSD.org; Wed, 4 Oct 2023 13:44:12 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: bugzilla set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 264166] mount_msdosfs: Fails to mount glabel label: mount_msdosfs: /dev/label/MACOME_CONFIG: Invalid argument Date: Wed, 04 Oct 2023 13:44:12 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 13.1-STABLE X-Bugzilla-Keywords: needs-patch, needs-qa X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: commit-hook@FreeBSD.org X-Bugzilla-Status: Closed X-Bugzilla-Resolution: Works As Intended X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: fs@FreeBSD.org X-Bugzilla-Flags: maintainer-feedback? mfc-stable13? mfc-stable12- X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D264166 --- Comment #13 from commit-hook@FreeBSD.org --- A commit in branch stable/13 references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=3D2464d8c5e296a02de882310398e730772= 25cac00 commit 2464d8c5e296a02de882310398e73077225cac00 Author: Mark Johnston AuthorDate: 2022-05-26 13:48:17 +0000 Commit: Mark Johnston CommitDate: 2023-10-04 13:43:22 +0000 glabel.8: Warn against using generic labels on a shared device Also suggest against creating a generic label on a device which already contains a filesystem. PR: 264166 Reviewed by: imp, delphij, Pau Amma MFC after: 1 week Differential Revision: https://reviews.freebsd.org/D35326 (cherry picked from commit 81f36fbc98dd74ca923938e0329919d426811b0c) lib/geom/label/glabel.8 | 22 +++++++++++++++++++++- 1 file changed, 21 insertions(+), 1 deletion(-) --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Wed Oct 4 13:44:17 2023 X-Original-To: fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4S0wsF6RfVz4vxrH for ; Wed, 4 Oct 2023 13:44:17 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4S0wsF3SfSz3btG for ; Wed, 4 Oct 2023 13:44:17 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1696427057; a=rsa-sha256; cv=none; b=AiSS2bXkWOQeexGHezgmQVHTfYj2e2J+TQ28iCikz173uiU6CrF2kQ6X+lWXBqewgdaQvA UX8AzzoX3cdg5lFnr/RQcmvsTOVdQg1hlzhsT9nhPpMrU4ntYgy35Y+0biorz7+HbY4R+u 6yAKNa2SB3ydyAkuIDYQ4t/NnpvlYHQ+I4fY6XrcjeaNp5+IwFXPEwR4/Pqnqc+LwQULRG j0WCj3bB5+7K6GlaerrgfwR+urHG727B6Xaoh5/DKCfpk1l/FcIo2VEPwi8xk/HP/aDkj7 rHkjewCSBuGjIjCz8m1pr5Eqd4jwYdZA4ibkgicQfPz2YaqDQZnVzbFLMFpdrw== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1696427057; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=7SLK/BvSvzSxTn1mI1Da44V1vyUwjcFXdbMIH6M5sVQ=; b=O6fgxWhvDxpQlNZeQl8robysMQFBKBZAcJlSlDK7u7yulNcH7idOxn7DqsEDCak8H54+ww sGDvJeJwcAbHj4Zqjb/pn+SMgkkUpicBMVffeZm+xw+oUSvJ/hG6SkbH+ldL4oADvfNtjK ulsoIuqrJXz/VRdO9hLCYAr/kw0mwxzmlTl/BeiHO3c8+lvG2V17VyOo8gbGQMDLGIk3KG IBQByE6QIIMH0EYXSSpFuMWiJjo1GASCheSkP7l9rUeEz2BLUFhbVNZWULjjcoer5nKcxe UfhOzW2BM9Zo0nOXUDV1JxGjXY2gZcsih3A9ckDCQRMcm9E2YNZsN1uVC+pXnw== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4S0wsF23jdzfvt for ; Wed, 4 Oct 2023 13:44:17 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 394DiHp5078193 for ; Wed, 4 Oct 2023 13:44:17 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from bugzilla@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 394DiHha078192 for fs@FreeBSD.org; Wed, 4 Oct 2023 13:44:17 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: bugzilla set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 264166] mount_msdosfs: Fails to mount glabel label: mount_msdosfs: /dev/label/MACOME_CONFIG: Invalid argument Date: Wed, 04 Oct 2023 13:44:17 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 13.1-STABLE X-Bugzilla-Keywords: needs-patch, needs-qa X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: commit-hook@FreeBSD.org X-Bugzilla-Status: Closed X-Bugzilla-Resolution: Works As Intended X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: fs@FreeBSD.org X-Bugzilla-Flags: maintainer-feedback? mfc-stable13? mfc-stable12- X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D264166 --- Comment #14 from commit-hook@FreeBSD.org --- A commit in branch stable/14 references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=3Db11568083f09f48caf11d78d25ab58fc6= 5793bcf commit b11568083f09f48caf11d78d25ab58fc65793bcf Author: Mark Johnston AuthorDate: 2022-05-26 13:48:17 +0000 Commit: Mark Johnston CommitDate: 2023-10-04 13:41:29 +0000 glabel.8: Warn against using generic labels on a shared device Also suggest against creating a generic label on a device which already contains a filesystem. PR: 264166 Reviewed by: imp, delphij, Pau Amma MFC after: 1 week Differential Revision: https://reviews.freebsd.org/D35326 (cherry picked from commit 81f36fbc98dd74ca923938e0329919d426811b0c) lib/geom/label/glabel.8 | 22 +++++++++++++++++++++- 1 file changed, 21 insertions(+), 1 deletion(-) --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Thu Oct 5 02:16:59 2023 X-Original-To: fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4S1FYm1L1mz4vrP8 for ; Thu, 5 Oct 2023 02:17:00 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4S1FYm05lPz3PKk for ; Thu, 5 Oct 2023 02:17:00 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1696472220; a=rsa-sha256; cv=none; b=Vsqn6HHe53/KGnrXIZpEso+1FL+FnVQxzKS1Y1sGJAs/5+m3mgPPdbhQSbjGgnV2MypJ/t 4XMHJ9qfs5XyiAOsX9RbQLYHdnsY0ak1M1lF/2NUgJ5fg/ua1Le3i3G/htbmT2O2/plpm5 L9p2ZjkEZXwIm2P8RPS/hXzLvKEOPbgGgS53XiLmVCS4ZHhjbpyNYVU35DwpUwsA7ibx4v VRpgiutn9WDTBVvSX7hWzTg6souhqgL4ueLXur1vqW0bqyz7RE6G55sAFT0ejRLLJ8qnGV zG8bhH3W3v/D9J67rbzvCoeARsd3izO4zvjRqBLaT8hA8UfgasdPqPa5ii6fEA== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1696472220; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=vMOZyTx6lnNyB0tYH464lxvl+pOoRIMMzZ0XUH3TdTI=; b=VD976yUgsy+oRe3yeuRiRfvZLd12ldkw7gbEdLHOfCd3rjPZABo/z5EokmnvVmOeI31jxI /DlZ/VD1zjfrzjdJyknbgh243MtwRZEbzOP+Cr6ZJkpFt6Ex1L6fWiHFRQtRevPogbl+Y9 LJmRD2hbbKd1kBfnkg8q8WznACcw4OFhEB6YzHE+nHoV2shtp+fh/BPgvFCUmUgm50ZniL qcuqp09YckIlfiYr5PKsKE3Xwh2PDUlMAP+Y3XLoknHOLSJzH6p3LT91uMdsZKddhDKySV TwMlnU/DiTarFJx7oUbdTu51qTvtjRduekJOnt3yh3+aZTt6dXamGQvNjVlPVA== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4S1FYl6K96z12db for ; Thu, 5 Oct 2023 02:16:59 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 3952GxVQ057649 for ; Thu, 5 Oct 2023 02:16:59 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 3952Gx3I057648 for fs@FreeBSD.org; Thu, 5 Oct 2023 02:16:59 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 274271] zfs deadlock while/after writing files Date: Thu, 05 Oct 2023 02:16:59 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 13.2-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: linimon@FreeBSD.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: assigned_to Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D274271 Mark Linimon changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|bugs@FreeBSD.org |fs@FreeBSD.org --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Fri Oct 6 00:54:00 2023 X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4S1qgd66Rwz4wQhr for ; Fri, 6 Oct 2023 00:54:05 +0000 (UTC) (envelope-from markjdb@gmail.com) Received: from mail-qk1-x72b.google.com (mail-qk1-x72b.google.com [IPv6:2607:f8b0:4864:20::72b]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4S1qgd103Wz3dDK; Fri, 6 Oct 2023 00:54:05 +0000 (UTC) (envelope-from markjdb@gmail.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20230601 header.b=Iz6l0tIY; spf=pass (mx1.freebsd.org: domain of markjdb@gmail.com designates 2607:f8b0:4864:20::72b as permitted sender) smtp.mailfrom=markjdb@gmail.com; dmarc=none Received: by mail-qk1-x72b.google.com with SMTP id af79cd13be357-7740cedd4baso108869785a.2; Thu, 05 Oct 2023 17:54:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1696553644; x=1697158444; darn=freebsd.org; h=content-disposition:mime-version:message-id:subject:cc:to:from:date :sender:from:to:cc:subject:date:message-id:reply-to; bh=eIzg/AdZ/rRVdAxYrrIzUeizHYDVr2N6bfG3Hy6jcWg=; b=Iz6l0tIYuZNQt/5Mv6VvjqxTOwRTDnHclSQfvu9ULZj6y2MiOpfZpRbdJFjZ4bsOlH Db1pkyw0b7GLFwfGrSLSzy2lg40cp4ROkGLSPv3MXoAQWvCk0kB18iLBUI4tAhp6B9KR WKEHEzvIp3Nh4m/zmMc0UYYkTLlpubDEk1v0acpWB13Q3olZYesWS1GsPw7VqDltruYW ziCax7RPeppamm0FmhCrIlq74ZawviwDholaZLVxhOVfmCQx7kjFi/xaKMDc6iQjAfYt wmmQAfO5jQbVbeW3J0BcClt3O50g80CADk0YeXHJ4gH2T+SMXtXq1DxW7BlpajTCc1EB WLcQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1696553644; x=1697158444; h=content-disposition:mime-version:message-id:subject:cc:to:from:date :sender:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=eIzg/AdZ/rRVdAxYrrIzUeizHYDVr2N6bfG3Hy6jcWg=; b=xBrBBuYoCL72HmDI4h1fwNXnMKTt4jdtZ5IY7iTohHrmcpGDSbunn7r2R8cB+/mRJ+ MZg492lSeP89yZWGUNRRFaoVqEO7LnasPnoiV/VTgg4d8ADKUK8IO0HPyBUW/TVpu++C cU21NEYimYEMlnPP/13bE2GOK6Y8e2kwOYKFkIAHkL+3YQrKI5Tfie5jo0u0LMY5m/Up bLghRhJyaRMrcTBL4EE8lVjpjpajjk+jC1x7mJKzu0wjglXC+6NUr97no62p/swTU2YM y55EWjhZAwm1ONwU8meIIUmsAaAkWkzBnw77QFzwb6T+VEKSXSw6Ei03tzVAil7w7WQl dGQA== X-Gm-Message-State: AOJu0YziJ2AeaFM9+I3Bh/S6YJvHdtuPLnFwnETOXW5y7EgiWKKFWXl3 YACl86XjNc+GaZoEgsrq+48+4hEnWUo= X-Google-Smtp-Source: AGHT+IGNDs+iVOI6z5XcXrOLVyIf1uy7bcR5K41aXjr6au9C1gMITuR5JuuceS8oIR4B8BZza1A1lg== X-Received: by 2002:a05:620a:e9a:b0:774:1494:bb0f with SMTP id w26-20020a05620a0e9a00b007741494bb0fmr7092744qkm.60.1696553643704; Thu, 05 Oct 2023 17:54:03 -0700 (PDT) Received: from nuc (192-0-220-237.cpe.teksavvy.com. [192.0.220.237]) by smtp.gmail.com with ESMTPSA id x13-20020a05620a14ad00b007756736aed7sm894190qkj.106.2023.10.05.17.54.02 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 05 Oct 2023 17:54:03 -0700 (PDT) Date: Thu, 5 Oct 2023 20:54:00 -0400 From: Mark Johnston To: freebsd-fs@freebsd.org Cc: mav@freebsd.org Subject: arc_prune thread consuming 100% CPU Message-ID: List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline X-Spamd-Bar: -- X-Spamd-Result: default: False [-2.65 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.95)[-0.951]; MID_RHS_NOT_FQDN(0.50)[]; FORGED_SENDER(0.30)[markj@freebsd.org,markjdb@gmail.com]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20230601]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; MIME_GOOD(-0.10)[text/plain]; MLMMJ_DEST(0.00)[freebsd-fs@freebsd.org]; MIME_TRACE(0.00)[0:+]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::72b:from]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; FREEMAIL_ENVFROM(0.00)[gmail.com]; RCPT_COUNT_TWO(0.00)[2]; RCVD_TLS_LAST(0.00)[]; FROM_NEQ_ENVFROM(0.00)[markj@freebsd.org,markjdb@gmail.com]; FROM_HAS_DN(0.00)[]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; TO_DN_NONE(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; DMARC_NA(0.00)[freebsd.org]; RCVD_COUNT_TWO(0.00)[2] X-Rspamd-Queue-Id: 4S1qgd103Wz3dDK Hi, Sometime in the past two months I started seeing a problem with ZFS on the latest FreeBSD main on my desktop. The symptom is that the arc_prune thread, which attempts to shrink the vnode cache, consumes 100% CPU even when the system is idle. I was confused at first, since dtrace shows that 1) arc_prune_task() is getting called several dozen times per second, but 2) arc_prune_async() is not getting called. But now I see that the taskq zone has ~250000 allocations, and the number is slowly shrinking. So, I believe arc_evict() was called many times in quick succession at some point, and it queued up a massive amount of work that is slowly draining. Could there be some regression in the arc_evict() changes that were made earlier this year in commit a8d83e2a24de6? If so, I'm not sure why I'm only seeing it now, it looks like that commit was merged to FreeBSD in April. From nobody Fri Oct 6 17:47:51 2023 X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4S2G9d0pdsz4vyyH for ; Fri, 6 Oct 2023 17:48:05 +0000 (UTC) (envelope-from jdavidlists@gmail.com) Received: from mail-pg1-x52d.google.com (mail-pg1-x52d.google.com [IPv6:2607:f8b0:4864:20::52d]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4S2G9c2ywGz4bjv for ; Fri, 6 Oct 2023 17:48:04 +0000 (UTC) (envelope-from jdavidlists@gmail.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20230601 header.b=ZyIV2ubq; spf=pass (mx1.freebsd.org: domain of jdavidlists@gmail.com designates 2607:f8b0:4864:20::52d as permitted sender) smtp.mailfrom=jdavidlists@gmail.com; dmarc=pass (policy=none) header.from=gmail.com Received: by mail-pg1-x52d.google.com with SMTP id 41be03b00d2f7-5855333fbadso1634030a12.1 for ; Fri, 06 Oct 2023 10:48:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1696614483; x=1697219283; darn=freebsd.org; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=F//jLIJFu1mvbKvAzXQBAb3cfMCz9X7yMb4BAA2dLC4=; b=ZyIV2ubq6dRuPMo7mZbXKNZnGlaAkOk/XjiHk1u2FYKb91VonYNrQiBsnNC0/WTFcY eiOuNSFwumkaMUdQrTp/1BOH2UCEq+O8PLOzREF9dtHrM4EBtF0eRwAotimBhgWCv6PL Njmn5Q8MWvznMmHiP/FxqzkVODKpIHnl6I0R3dUg9hMO4qDpGwW1sdoK7jBvimWZPRh5 xSLmaMkZdDYRkTBPMULvHOt9HxWza3JgdgcPKYtn5tU8BK5C7db5jOxr0OQIXi65CFjG x1hTt3tRJ9nOiiIogvDLjlxpilhHPTm7lC54cpKC2qXYdNDl3Y+9lD+TyfKhglmfiPbj iKZg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1696614483; x=1697219283; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=F//jLIJFu1mvbKvAzXQBAb3cfMCz9X7yMb4BAA2dLC4=; b=Rov2r/pWbbHAMiBhkycn+8N5KX3Thm5W4bTEMl/pM4EKUa6tL99zUl+cdoRBLks6r+ 66k1YmVUID/0XpS7DgwcanAKnloSfa4yQFjOQYXbON7mjgPHiwX9z/kpv4aOtCOgeQcj EjfEIBx9DjqpsHBhPd1nagb4GH2sur80+Tjc1SRYLresF5diVSPTsIcZ7t+0i2zwGsw5 2V5ax7qNBuFfzGJPlRWiA5AgCFumF159DJR2Bj30wIihTgrqGp0bHU+YR6WtltxoOuZM 9hzAHcVaWROFQnr6P8gA9O/jiMjzDoUPAycGFF2blSnI7G5ByRiZcv1yAw0XMNHc9BmR Hl1Q== X-Gm-Message-State: AOJu0Yw3HmTStOCp2MxWPF+/f7wpV0P+SGTv+GZrwoExHBZilcKzOQWN KatgBV65FT+w7LHjHY75XZg7jL+JpJ7+uJepe+Btz4O+WSo= X-Google-Smtp-Source: AGHT+IEErR9HO/R7/4ifSfeWKsJOHQKySzT4jdcqFwH2UnqNs7jYfWA+5UQ8GWxJ40buzQZ3RmZLqhMUWnjjDxdKdLU= X-Received: by 2002:a17:90b:1bc7:b0:26d:4ab3:fe11 with SMTP id oa7-20020a17090b1bc700b0026d4ab3fe11mr8185172pjb.24.1696614482782; Fri, 06 Oct 2023 10:48:02 -0700 (PDT) List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 References: In-Reply-To: From: J David Date: Fri, 6 Oct 2023 13:47:51 -0400 Message-ID: Subject: Re: FreeBSD 13.2 NFS client mount hangs To: Rick Macklem Cc: FreeBSD FS Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Spamd-Bar: --- X-Spamd-Result: default: False [-3.99 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.99)[-0.986]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20230601]; MIME_GOOD(-0.10)[text/plain]; FROM_HAS_DN(0.00)[]; TAGGED_RCPT(0.00)[]; FREEMAIL_FROM(0.00)[gmail.com]; PREVIOUSLY_DELIVERED(0.00)[freebsd-fs@freebsd.org]; ARC_NA(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::52d:from]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; MLMMJ_DEST(0.00)[freebsd-fs@freebsd.org]; DKIM_TRACE(0.00)[gmail.com:+]; RCPT_COUNT_TWO(0.00)[2]; TO_DN_ALL(0.00)[]; MID_RHS_MATCH_FROMTLD(0.00)[]; TAGGED_FROM(0.00)[]; RCVD_COUNT_ONE(0.00)[1]; FROM_EQ_ENVFROM(0.00)[]; FREEMAIL_TO(0.00)[gmail.com]; FREEMAIL_ENVFROM(0.00)[gmail.com]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_LAST(0.00)[] X-Rspamd-Queue-Id: 4S2G9c2ywGz4bjv On Mon, Oct 2, 2023 at 7:08=E2=80=AFPM Rick Macklem wrote: > > The nfscbd daemon is not running on any of the clients. > > > > > If the Linux server still > > > issues delegations > > > > How would I determine that? > nfsstat -E -c > and then look at the number under "Delegs". It is a current count of > delegations, so if it remains 0 over time, no delegations are being issue= d. But if this is done from a client that is not running nfscbd, isn't it pretty well guaranteed to be zero? Checking all the clients I can find, "Deleg" is zero on all of them. On about half, "DelegRet" is nonzero but small (1-100), but I don't know what that is or if it's related. > I have attached a small patch which should make the NFS client handle > this error correctly. I will look for a way to try this patch, but the clients in this case are all managed with freebsd-update and don't have enough disk space to build a kernel locally, so it may be tricky. > > > # tcpdump -s 0 -w out.pcap host > > > Let this run for a while and then pull out.pcap into wireshark and se= e what > > > traffic is going between the NFS client and server. > > > (Unlike tcpdump, wireshark does know how to decode NFS properly.) > > > > If/when the issue happens again, I will attempt to do this and report b= ack. I am also working on getting access to Wireshark. In the interim, it did happen again, so the best I can do is put a little bit of tcpdump output here: https://pastebin.com/UDrphwr5 . I can't vouch for "correct" but it does mostly seem to decode the NFS packe= ts. It seems to loop the same couple of actions with long delays (15 seconds) between retries: This sequence: +0.0000s: Client -> server xid 1205841201 getattr fh 0,7/2 ("Getattr" in packet body) +1.4106s: Client -> server xid 1205841202 getattr fh 0,5/2 ("Renew" in packet body) +0.0002s: Server -> client xid 1205841202 getattr LNK 12231267145 ids 1/53 sz 0 ("Renew" in packet body) +3.8001s: Server -> client xid 1205841201 getattr ERROR: Request couldn't be completed in time ("Getattr" in packet body) Repeats after 15 seconds: +15.0090s: client -> server 1205841203 getattr fh 0,7/2 ("Getattr" in packet body) ... etc The "fh 0,7/2" and "fh 0,5/2" seem to be consistent each time. The xid (transaction/request ID?) increments each time. Maybe that will provide a lucky flash of insight in the interim. Thanks! From nobody Sat Oct 7 00:12:10 2023 X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4S2Qj40tPJz4w77r for ; Sat, 7 Oct 2023 00:12:24 +0000 (UTC) (envelope-from rick.macklem@gmail.com) Received: from mail-pj1-x102f.google.com (mail-pj1-x102f.google.com [IPv6:2607:f8b0:4864:20::102f]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4S2Qj33M3Jz4YLZ for ; Sat, 7 Oct 2023 00:12:23 +0000 (UTC) (envelope-from rick.macklem@gmail.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20230601 header.b="U/mRQ7ML"; spf=pass (mx1.freebsd.org: domain of rick.macklem@gmail.com designates 2607:f8b0:4864:20::102f as permitted sender) smtp.mailfrom=rick.macklem@gmail.com; dmarc=pass (policy=none) header.from=gmail.com Received: by mail-pj1-x102f.google.com with SMTP id 98e67ed59e1d1-278eaffd81dso1966537a91.0 for ; Fri, 06 Oct 2023 17:12:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1696637542; x=1697242342; darn=freebsd.org; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=USrbJOBF1BmWR5xp08bZiAs/J+8kUe2LHIy6uZYQoaA=; b=U/mRQ7ML6ihnWXXT9S3m++eaOjUag2knFESYSlIJ9JOZ8k4Nczp5jg8ouJ9RGf67KY hQ0yGP9BBX63Q1Rytc4ZmonAqNl+9awghuy5Yt4Kr3pm/4lO2hROSSMIktQwG4YiTLm/ XCBQpDfldKlgBq22AwyGLEDnv6gmX9WI6+wJ2C7edmnFmk2B5G4elG24c7hAo64g7+zN a+7gvAG75A8kmaeU+ECU8deiqdHuZn5yKoV/UPL+2yvE4Vl3odPVmTZLdr5lQLquxkzH yDGRRi02rnoWnCAu1+O6MhXDM5/0bXpHKEiKEedCEKqsMm+7vX7UCikir8jZfNEQl+/x nwzA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1696637542; x=1697242342; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=USrbJOBF1BmWR5xp08bZiAs/J+8kUe2LHIy6uZYQoaA=; b=SugyGLlv/+0nqzJhCQvml7DI/NOTg3BHnqAuVUOGKJ/t3Nyyls0QGMVX/30CHhIbYv YTetVOxtPoud6CVDjqlC0HM0qI/c1HEFpOw8SUcUmOZxreGzc61IrrAgxKF6mReFiO7/ Il7mau1BsAq+QfSCa+fmdDNGVNnD3WM1GltGyms4gaoVl9JWX2iBmKIWe0droxsFUBbi uLBbXRx9NN1rSnowpfOzJRHhKk015FRFmEk+LLAXAS1YtkreEGjQGruSDoInIPZoHThR Z1LZDUV0Vk1Xsm7ufxYCK7JMSCi9ze7aT58y0gI8DduCEfqROFOZn1DWvol3ySSXOM0R 9Jxg== X-Gm-Message-State: AOJu0YwmFpMlyGhRHkwWaXjwOeA01DIe+9gCF4nXl5E2nu1Xk8Rd0Phq utYUubd9ofJCYr7q/tf2LEfkg0Gra1/zXkGEyg== X-Google-Smtp-Source: AGHT+IEUB8y7FlHGgiClQHDg+BF+epMpAgujM4WSfT70NbVVtdlBESpI3lHUW+5X47FNYlnJE0ASnifdqfGqZqrBl44= X-Received: by 2002:a17:90b:4f8c:b0:268:b66b:d9f6 with SMTP id qe12-20020a17090b4f8c00b00268b66bd9f6mr8663581pjb.18.1696637542107; Fri, 06 Oct 2023 17:12:22 -0700 (PDT) List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 References: In-Reply-To: From: Rick Macklem Date: Fri, 6 Oct 2023 17:12:10 -0700 Message-ID: Subject: Re: FreeBSD 13.2 NFS client mount hangs To: J David Cc: FreeBSD FS Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Spamd-Bar: --- X-Spamd-Result: default: False [-3.99 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.99)[-0.986]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20230601]; MIME_GOOD(-0.10)[text/plain]; FROM_HAS_DN(0.00)[]; TAGGED_RCPT(0.00)[]; FREEMAIL_FROM(0.00)[gmail.com]; PREVIOUSLY_DELIVERED(0.00)[freebsd-fs@freebsd.org]; ARC_NA(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::102f:from]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; MLMMJ_DEST(0.00)[freebsd-fs@freebsd.org]; DKIM_TRACE(0.00)[gmail.com:+]; RCPT_COUNT_TWO(0.00)[2]; TO_DN_ALL(0.00)[]; MID_RHS_MATCH_FROMTLD(0.00)[]; TAGGED_FROM(0.00)[]; RCVD_COUNT_ONE(0.00)[1]; FROM_EQ_ENVFROM(0.00)[]; FREEMAIL_TO(0.00)[gmail.com]; FREEMAIL_ENVFROM(0.00)[gmail.com]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_LAST(0.00)[] X-Rspamd-Queue-Id: 4S2Qj33M3Jz4YLZ On Fri, Oct 6, 2023 at 10:48=E2=80=AFAM J David w= rote: > > On Mon, Oct 2, 2023 at 7:08=E2=80=AFPM Rick Macklem wrote: > > > The nfscbd daemon is not running on any of the clients. > > > > > > > If the Linux server still > > > > issues delegations > > > > > > How would I determine that? > > nfsstat -E -c > > and then look at the number under "Delegs". It is a current count of > > delegations, so if it remains 0 over time, no delegations are being iss= ued. > > But if this is done from a client that is not running nfscbd, isn't it > pretty well guaranteed to be zero? Yes. If the NFS server is functioning correctly and there is no nfscbd running, it should remain at 0. > > Checking all the clients I can find, "Deleg" is zero on all of them. > On about half, "DelegRet" is nonzero but small (1-100), but I don't > know what that is or if it's related. It implies that the client is doing a recovery after a NFS server reboot or something like that. Since no delegations are being issued, the DelegRet operations would/should be related to a recovery after a NFS server reboot. > > > I have attached a small patch which should make the NFS client handle > > this error correctly. > > I will look for a way to try this patch, but the clients in this case > are all managed with freebsd-update and don't have enough disk space > to build a kernel locally, so it may be tricky. > > > > > # tcpdump -s 0 -w out.pcap host > > > > Let this run for a while and then pull out.pcap into wireshark and = see what > > > > traffic is going between the NFS client and server. > > > > (Unlike tcpdump, wireshark does know how to decode NFS properly.) > > > > > > If/when the issue happens again, I will attempt to do this and report= back. > > I am also working on getting access to Wireshark. > > In the interim, it did happen again, so the best I can do is put a > little bit of tcpdump output here: https://pastebin.com/UDrphwr5 . This appears to be text. I need the actual pcap file captured by tcpdump. > > I can't vouch for "correct" but it does mostly seem to decode the NFS pac= kets. > > It seems to loop the same couple of actions with long delays (15 > seconds) between retries: > > This sequence: > +0.0000s: Client -> server xid 1205841201 getattr fh 0,7/2 ("Getattr" > in packet body) > +1.4106s: Client -> server xid 1205841202 getattr fh 0,5/2 ("Renew" in > packet body) > +0.0002s: Server -> client xid 1205841202 getattr LNK 12231267145 ids > 1/53 sz 0 ("Renew" in packet body) > +3.8001s: Server -> client xid 1205841201 getattr ERROR: Request > couldn't be completed in time ("Getattr" in packet body) If the server is not replying to a Getattr, then it is broken. That will certainly hang a NFS client. > > Repeats after 15 seconds: > +15.0090s: client -> server 1205841203 getattr fh 0,7/2 ("Getattr" in > packet body) > ... etc > > The "fh 0,7/2" and "fh 0,5/2" seem to be consistent each time. The xid > (transaction/request ID?) increments each time. After 15sec, the client gives up on waiting for the reply from the server, creates a new TCP connection and tries the RPC again. Then it looks like the server does not reply again. If you can give me the pcap file, I will take a look at it, but this suggests a broken NFS server. rick > > Maybe that will provide a lucky flash of insight in the interim. > > Thanks! From nobody Sat Oct 7 00:31:56 2023 X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4S2R7s1QLnz4w9xp for ; Sat, 7 Oct 2023 00:32:09 +0000 (UTC) (envelope-from rick.macklem@gmail.com) Received: from mail-pj1-x1029.google.com (mail-pj1-x1029.google.com [IPv6:2607:f8b0:4864:20::1029]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4S2R7r2WfRz4ZWg for ; Sat, 7 Oct 2023 00:32:08 +0000 (UTC) (envelope-from rick.macklem@gmail.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20230601 header.b=VgHBR5mX; spf=pass (mx1.freebsd.org: domain of rick.macklem@gmail.com designates 2607:f8b0:4864:20::1029 as permitted sender) smtp.mailfrom=rick.macklem@gmail.com; dmarc=pass (policy=none) header.from=gmail.com Received: by mail-pj1-x1029.google.com with SMTP id 98e67ed59e1d1-279294d94acso3019676a91.0 for ; Fri, 06 Oct 2023 17:32:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1696638727; x=1697243527; darn=freebsd.org; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=3UD5HrjRvFPipx8M713ltCWd4/lEv5odr8l686NX73Q=; b=VgHBR5mXniOOtqtr0oVPCuFYFQ+wK7CNOoRxTW9iiFJmCrLB94XIXi1ohxDd3dLTxr nK3YoRR5F0N9rAx/Z25aaE56tnGnjXFoodRzyhKHacs2X3JHDRwnRRx9DRlLPduO3l51 DNd9QGPa9YaNN92dVUAvpQZd5XIn/yKmsbJ4It46bj0zCj2f1OiTxIGna5oRvyHSv3jm u+r3yOP5a5Di8my7qXKdQXXscgxHVo3PN2bzcxhdGPXH1xV/V+8SZhMZtnVFJh3/I1jK XjtZfKjWeMQEkqbIeEYrju5s8EesuU9oJx5j2S8Nt8o86RsOFQeBL2D80+Qj8Fi7xpgX YSyw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1696638727; x=1697243527; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=3UD5HrjRvFPipx8M713ltCWd4/lEv5odr8l686NX73Q=; b=cd6V0rww2wLDPp5N9qADLQZPD4Zf6KSLau3SVfePiTDGClThKlPlcRM0N/RabxNaB5 HYNPurSBFqkOmtpCevJ1PBr24B9gQTh408FWKebfTJ0dvmii5XaZ1j+Db9wn3Fw76RrL sL9vGKqOExMzzJnVt4FeCFAQqzYQcrdZ364Wz/xT8mvVLLIvTJW26y7lWxkhBMPZ+itl 3NItagnPUpOQb272DXV31kgR2ziVtpvJkLmGc4AVA3QUnTRVc/44G6K6G5easoA9ePv5 jR/5yVmJQS/XA1JD6B2yB4sPRsD1QVnt3IlpTpOHyGdSTaBc/k4YeklCw59E2y1chx/M D12g== X-Gm-Message-State: AOJu0Yxlj4hqIbl6fumCJFHU5P479yIDSWW8CQEx4FnHQl8BNmfx5NBF on4FQeuQPbr/iNZYjb+BvrtO25zHC8Ovg5w/gFAuqEI= X-Google-Smtp-Source: AGHT+IF7GPsE4RxQN8ctkwn3jyJmIeey2NLj03NixB+X7bt5dqL4T/48oBxdK0ORLi+jEeB3Vq9xeyrP3A13QRFREd8= X-Received: by 2002:a17:90b:3907:b0:276:5512:13ab with SMTP id ob7-20020a17090b390700b00276551213abmr9021930pjb.10.1696638727087; Fri, 06 Oct 2023 17:32:07 -0700 (PDT) List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 References: In-Reply-To: From: Rick Macklem Date: Fri, 6 Oct 2023 17:31:56 -0700 Message-ID: Subject: Re: FreeBSD 13.2 NFS client mount hangs To: J David Cc: FreeBSD FS Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Spamd-Bar: --- X-Spamd-Result: default: False [-3.99 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.99)[-0.986]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20230601]; MIME_GOOD(-0.10)[text/plain]; FROM_HAS_DN(0.00)[]; TAGGED_RCPT(0.00)[]; FREEMAIL_FROM(0.00)[gmail.com]; PREVIOUSLY_DELIVERED(0.00)[freebsd-fs@freebsd.org]; ARC_NA(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::1029:from]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; MLMMJ_DEST(0.00)[freebsd-fs@freebsd.org]; DKIM_TRACE(0.00)[gmail.com:+]; RCPT_COUNT_TWO(0.00)[2]; TO_DN_ALL(0.00)[]; MID_RHS_MATCH_FROMTLD(0.00)[]; TAGGED_FROM(0.00)[]; RCVD_COUNT_ONE(0.00)[1]; FROM_EQ_ENVFROM(0.00)[]; FREEMAIL_TO(0.00)[gmail.com]; FREEMAIL_ENVFROM(0.00)[gmail.com]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_LAST(0.00)[] X-Rspamd-Queue-Id: 4S2R7r2WfRz4ZWg On Fri, Oct 6, 2023 at 10:48=E2=80=AFAM J David w= rote: > > On Mon, Oct 2, 2023 at 7:08=E2=80=AFPM Rick Macklem wrote: > > > The nfscbd daemon is not running on any of the clients. > > > > > > > If the Linux server still > > > > issues delegations > > > > > > How would I determine that? > > nfsstat -E -c > > and then look at the number under "Delegs". It is a current count of > > delegations, so if it remains 0 over time, no delegations are being iss= ued. > > But if this is done from a client that is not running nfscbd, isn't it > pretty well guaranteed to be zero? > > Checking all the clients I can find, "Deleg" is zero on all of them. > On about half, "DelegRet" is nonzero but small (1-100), but I don't > know what that is or if it's related. > > > I have attached a small patch which should make the NFS client handle > > this error correctly. > > I will look for a way to try this patch, but the clients in this case > are all managed with freebsd-update and don't have enough disk space > to build a kernel locally, so it may be tricky. > > > > > # tcpdump -s 0 -w out.pcap host > > > > Let this run for a while and then pull out.pcap into wireshark and = see what > > > > traffic is going between the NFS client and server. > > > > (Unlike tcpdump, wireshark does know how to decode NFS properly.) > > > > > > If/when the issue happens again, I will attempt to do this and report= back. > > I am also working on getting access to Wireshark. > > In the interim, it did happen again, so the best I can do is put a > little bit of tcpdump output here: https://pastebin.com/UDrphwr5 . Maybe someone who is familiar with tcpdump output can look at this. (I always use wireshark.) It looks to me like the TCP checksum is failing on the client->server reque= st, but maybe I am not reading it correctly. If the checksum is incorrect, then there is something badly broken in your network fabric. rick > > I can't vouch for "correct" but it does mostly seem to decode the NFS pac= kets. > > It seems to loop the same couple of actions with long delays (15 > seconds) between retries: > > This sequence: > +0.0000s: Client -> server xid 1205841201 getattr fh 0,7/2 ("Getattr" > in packet body) > +1.4106s: Client -> server xid 1205841202 getattr fh 0,5/2 ("Renew" in > packet body) > +0.0002s: Server -> client xid 1205841202 getattr LNK 12231267145 ids > 1/53 sz 0 ("Renew" in packet body) > +3.8001s: Server -> client xid 1205841201 getattr ERROR: Request > couldn't be completed in time ("Getattr" in packet body) > > Repeats after 15 seconds: > +15.0090s: client -> server 1205841203 getattr fh 0,7/2 ("Getattr" in > packet body) > ... etc > > The "fh 0,7/2" and "fh 0,5/2" seem to be consistent each time. The xid > (transaction/request ID?) increments each time. > > Maybe that will provide a lucky flash of insight in the interim. > > Thanks! From nobody Sat Oct 7 05:38:57 2023 X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4S2Yy74p0yz4x1T5 for ; Sat, 7 Oct 2023 05:39:11 +0000 (UTC) (envelope-from jdavidlists@gmail.com) Received: from mail-pg1-x536.google.com (mail-pg1-x536.google.com [IPv6:2607:f8b0:4864:20::536]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4S2Yy659bRz4H1s for ; Sat, 7 Oct 2023 05:39:10 +0000 (UTC) (envelope-from jdavidlists@gmail.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20230601 header.b=Fthaq9DG; spf=pass (mx1.freebsd.org: domain of jdavidlists@gmail.com designates 2607:f8b0:4864:20::536 as permitted sender) smtp.mailfrom=jdavidlists@gmail.com; dmarc=pass (policy=none) header.from=gmail.com Received: by mail-pg1-x536.google.com with SMTP id 41be03b00d2f7-584a761b301so2107117a12.3 for ; Fri, 06 Oct 2023 22:39:10 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1696657149; x=1697261949; darn=freebsd.org; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=KmxZnWh/XWh8AqOBkT0RSOG46vDxYxJqWNr7BqptB1s=; b=Fthaq9DGiht2GAQBKUnJIo/Z/qKQJjMVdkF0Pnnk1uGOKKvFXMOoCkC8DVlJIvuBmk W4RxUU9WqzUsvNqLWX8k2dyTOcGXtzQhKwdyeoOKR8oNHf6UwknppBcvtXS0UIyGHLa/ 7EKtf4SiTjNGs/DYOwu8176wsg6+ttMlagIOhy3cAU5gE3auQ7t8Ir0PMF2x+aNaAcBH vRl8K1wZ+i6Jcy5zZiSQGTlDdFty++kSaBNC78+wcQ6tnn765p1efNXZhsJsfghi8h4g fu8bO2nveK3lL9+3W+0gjLuvGVutSGhXJivll/DwVAWO9Gmdhd5Aihi61ZSjjdRFG6zH 556Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1696657149; x=1697261949; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=KmxZnWh/XWh8AqOBkT0RSOG46vDxYxJqWNr7BqptB1s=; b=cNyXD/gKrXcdEpOVxCP1MU6aj/UktJ144vZjAR9IBnM/gBw6tRPwQ8PM0FHKyAqdd/ stSq/xOzKq03z7FOqUNOjvToEVbojLX05rKs0wqGaXuTaNiNx3a6OoM0/b7nExl9TY1s aUXHYstlAUyhGxxkcXwySGu+FygnKsXLDR2AUxTlMdKTLxjkP6bcimE+nbDn8/JbRbuR IL050dcEZkTIltZ9wtoMGl2UaYQmMSLQ2Pyq/BPs0zqcuforkAGr5RtaNqPPLR8FkaUb 8VJMtCvp1zt1GRVYn7N8itdX+YxVOAELygksPHEzTB4/zUJrenUf+/jk7klHH+tASqYi 6ycg== X-Gm-Message-State: AOJu0Yw0DW3H0fX5Am+ELDWyneR7JQRRubl2SnIPLiGd5B/A0+mBQ+iQ xb5/j0aqhk+4xezSzmvgeYsSKmN7W7xilb4hZ04= X-Google-Smtp-Source: AGHT+IFWLz9LtlDVFqq7DkPZUW5+myprsXGlzChAoEqZyEcmqwzmhB5/soxipkr7gkYndNYkaXzUTaXxlUIjh59t/l4= X-Received: by 2002:a05:6a20:549a:b0:159:beec:79ba with SMTP id i26-20020a056a20549a00b00159beec79bamr12895084pzk.2.1696657149264; Fri, 06 Oct 2023 22:39:09 -0700 (PDT) List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 References: In-Reply-To: From: J David Date: Sat, 7 Oct 2023 01:38:57 -0400 Message-ID: Subject: Re: FreeBSD 13.2 NFS client mount hangs To: Rick Macklem Cc: FreeBSD FS Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Spamd-Bar: --- X-Spamd-Result: default: False [-3.99 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.99)[-0.989]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20230601]; MIME_GOOD(-0.10)[text/plain]; FROM_HAS_DN(0.00)[]; TAGGED_RCPT(0.00)[]; FREEMAIL_FROM(0.00)[gmail.com]; PREVIOUSLY_DELIVERED(0.00)[freebsd-fs@freebsd.org]; ARC_NA(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::536:from]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; MLMMJ_DEST(0.00)[freebsd-fs@freebsd.org]; DKIM_TRACE(0.00)[gmail.com:+]; RCPT_COUNT_TWO(0.00)[2]; TO_DN_ALL(0.00)[]; MID_RHS_MATCH_FROMTLD(0.00)[]; TAGGED_FROM(0.00)[]; RCVD_COUNT_ONE(0.00)[1]; FROM_EQ_ENVFROM(0.00)[]; FREEMAIL_TO(0.00)[gmail.com]; FREEMAIL_ENVFROM(0.00)[gmail.com]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_LAST(0.00)[] X-Rspamd-Queue-Id: 4S2Yy659bRz4H1s On Fri, Oct 6, 2023 at 8:32=E2=80=AFPM Rick Macklem wrote: > It looks to me like the TCP checksum is failing on the client->server req= uest, > but maybe I am not reading it correctly. > > If the checksum is incorrect, then there is something badly broken in you= r > network fabric. tcpdump captures outgoing packets before the checksum is calculated. It's always "wrong" because it hasn't been computed yet. Thanks! From nobody Sat Oct 7 15:22:28 2023 X-Original-To: fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4S2pv86nwPz4whdB for ; Sat, 7 Oct 2023 15:22:28 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4S2pv85mq8z4qsG for ; Sat, 7 Oct 2023 15:22:28 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1696692148; a=rsa-sha256; cv=none; b=KkY/HTlR6ln07KKuRNJ6BT00HMQLyk1Ou2848OGPIhTPDGasnLCyG6ihQtGRpYKHrMkvXc VFtuMms1fgrdbYH8ipxgHJPQgjuVZ8bkumMO5XPPyVTs4bZ1t2aS+3FMs32QRQBeyGmtoW TAMdZeGJyc9A5BizmysmCqvTBWaH1ApN7C4aJg2Jnd9iDW/v9jPuDU/m1jXgROEtzoWAS6 wmrlKOSbWHhwuIuMOf38lL4kfts4sPxYAlOMCuxrxZN1jQQqiqE8ooM4tosxWEWE77JRPG JaEHBYSy2JUsrJQbpEH5KGknb0+r/6w2mw+nDkAjDFsBTYkJGQh9y5ydMJkiAQ== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1696692148; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=4cA7XvfAroODhYzmENn9tUQBf54YirBvI7HGFfNiUJo=; b=e0BfsdTB8jGvVYctCFL4I01cB+vrDaHqaMBmRop7keQIllQKCuJfoFuWnroUyGmQUY7hgW tkb8i3lc62s95XwdwLXAMfkHS47E/929HSfdoEoKWj5vT464OuK+GbobSL6UIYfz5tReJI oCvHWoXKUPJZwKetTc+HOo/kRnYO7IFas3JCMnixxFPz1XOKJHRIKxwvIT8LEwticf8x2L 1Pxmm9AtE/iECR/8uEQVLf3eqSwVqPOxQoGGK5i1V8qHdHO3wJtadHMXbG2B1XXp5K2mgM 8X5yqhd9VETPPCUNbGqI/2naVG1AnPFWnM6oDKOkiDyr2Q+JQS3Jrr1/65zEDw== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4S2pv84rSvzc5l for ; Sat, 7 Oct 2023 15:22:28 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 397FMSJL085250 for ; Sat, 7 Oct 2023 15:22:28 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 397FMS9V085216 for fs@FreeBSD.org; Sat, 7 Oct 2023 15:22:28 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 274271] zfs deadlock while/after writing files Date: Sat, 07 Oct 2023 15:22:28 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 13.2-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: grahamperrin@gmail.com X-Bugzilla-Status: Open X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: cc bug_status Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D274271 Graham Perrin changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |grahamperrin@gmail.com Status|New |Open --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Sat Oct 7 15:41:29 2023 X-Original-To: fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4S2qK61rVyz4wlJn for ; Sat, 7 Oct 2023 15:41:30 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4S2qK60bqPz4vtl for ; Sat, 7 Oct 2023 15:41:30 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1696693290; a=rsa-sha256; cv=none; b=RwV7Ouzi9vrSF8V4+RYgkXFm9Sc4dl0KQc795Dc+rX9tfCkREni0NgkyWR088u2t6kdlDn yeWLDdJ8ibHVEuIOLLmYWauX/frG2iZB8GMp2AxThakvKyFasBRQbWEffL7jBJEspj1gG7 N/xeyvrmNPok400bsNkqEfot+NsMPStyaZoPXxhhiL2EtZSWyAJ/YBx77Wqk3zsbMMD3yP KXzn8hM658ZDwPNm+o6G2ijxx9zHJkxYgBhJ0D68q+WBpkIHgE7i7AHN0nbrODRnKEkb45 nsGStYIzw/En7kyo24EKx3tQrEjOot6C9rkAnm+vhX170qDQUcwPS1Sjpm3JQQ== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1696693290; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=IlYF7k1cFS9Ni+04Mx1cMePow3if2kbHhpdu/0ReyK0=; b=u2GSPNNVWb2bmK1EPchS2Pf4vFx3n57Fdn4MrXXGJJRVS/ExQcSZOV4uVHqSmTFC3jMBGp SzNPi+i7oSWcwGT1/E8rWaX8AkdiSrOzs+rkber7DynYSI3c1wYdjQz0X688JaGj0GyrSl 74VrpJei8c4Vne+10hgY5oxdhmeodJgyaeCq2ajfHZ52DW/mMC65jIR+XxZT4zBVUOXSmd GIro5WIrDU5mModcZBfHTAOVSlv7jMBo9TaVM258IpbqxV3moqcjQyVjPN9jGzUTqfQ/Yt gDHF2h7vvz1PZEFCXwomVB1ez+M9odnAPfcBVSAdv+EbWvhRjG+b2Msw9W2aNw== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4S2qK56cD3zcWG for ; Sat, 7 Oct 2023 15:41:29 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 397FfTto010492 for ; Sat, 7 Oct 2023 15:41:29 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 397FfTfK010491 for fs@FreeBSD.org; Sat, 7 Oct 2023 15:41:29 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 274271] zfs deadlock while/after writing files Date: Sat, 07 Oct 2023 15:41:29 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 13.2-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: asomers@FreeBSD.org X-Bugzilla-Status: Open X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: cc Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D274271 Alan Somers changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |asomers@FreeBSD.org --- Comment #1 from Alan Somers --- When ZFS deadlocks, you need to do a "procstat -kka". Hopefully that will reveal where the deadlock lies. --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Sat Oct 7 17:30:23 2023 X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4S2sl15wWxz4x3YY for ; Sat, 7 Oct 2023 17:30:37 +0000 (UTC) (envelope-from jdavidlists@gmail.com) Received: from mail-oo1-xc32.google.com (mail-oo1-xc32.google.com [IPv6:2607:f8b0:4864:20::c32]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4S2sl10xFNz3TyW for ; Sat, 7 Oct 2023 17:30:37 +0000 (UTC) (envelope-from jdavidlists@gmail.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20230601 header.b=Ip4LKPnQ; spf=pass (mx1.freebsd.org: domain of jdavidlists@gmail.com designates 2607:f8b0:4864:20::c32 as permitted sender) smtp.mailfrom=jdavidlists@gmail.com; dmarc=pass (policy=none) header.from=gmail.com Received: by mail-oo1-xc32.google.com with SMTP id 006d021491bc7-57e3c2adbf0so1747105eaf.2 for ; Sat, 07 Oct 2023 10:30:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1696699836; x=1697304636; darn=freebsd.org; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=uDFwnlWu7V6TTo4w/5gR6JSpjOY0zQUETjDQ8WDMP7Q=; b=Ip4LKPnQjgbgZCObTZQBUTH9vGIJCi6vKlfXSaqHWF+t0RBabc8IKxgTwx6X8vnIs2 2E99ZovCMZ9UmLKwgQ4qarPZm9OfR1jRNoV5X8BVJ5J1GhAZ+YD4PfLHgRQN+NI/UhzE pQS7SxNMR6/25j6D6/loeM5BbiS1tdkJFZ493ivw3ZgTsZw1PjSrIeC0yybf0TDEQiFe /BnNpyYSesqZB8bsTqHKd4pTkUj3bjep2Z4ds4bvgl29saCsGaQwxJPwjVC4tFQt/wgF GkJ7DfpdrEPbyjiItKb+Z1uAcHrX/Y0RoWhrFc4ZtZ8jhSjhaowQpiD3n77Xw56j4J+1 tTVw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1696699836; x=1697304636; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=uDFwnlWu7V6TTo4w/5gR6JSpjOY0zQUETjDQ8WDMP7Q=; b=SbY9GyZsOmkF9T+Wf9ckxpA5Arc3BgXc9gSFPrJccPznrMqu4HZHE14tqlAccLZPkQ 3KbXqkKEZrShxp2Kc6gpDUeVwBRICjoyN8gWqyJmWjIy07mHu5xdfqrVrZ+DTjO/bidT 9TuiRDxJxb6sOt3c4LAbJ5GMYGhe9vmhWEY9fN46F9GwwMGXILKN5a2cA3YzXWcaDe99 w935mnzSNPDZrnhkXilySjLmQGLSszYudfVsLI7CH5nzwCZ86JiTP6GyhQBLnKnieyBX Th10wFIZcT0mrEX01pr0LeUfLqxB2nh8oe3Fo3omGjcujGQUJWFhCcMK2gSHDhfuqwHX 2wlg== X-Gm-Message-State: AOJu0YyH0yVdgTS4wUMHVay7RxLlbNKfy9C2ina1VQkNkb9aWDbAoQva +xj5FYeH58s1OmcymtRBiA0KD/tyr/63P3ulvM8= X-Google-Smtp-Source: AGHT+IEZFDL/FRVVsoJvOK+iAOiOzDouPdly3kky4ccc0iOHvVBBYEV/ooy/CiM2YnJY45Nyg/DdAKq8MWiSBpTwIeU= X-Received: by 2002:a05:6358:4311:b0:13c:fd78:bb43 with SMTP id r17-20020a056358431100b0013cfd78bb43mr10962904rwc.27.1696699835509; Sat, 07 Oct 2023 10:30:35 -0700 (PDT) List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 References: In-Reply-To: From: J David Date: Sat, 7 Oct 2023 13:30:23 -0400 Message-ID: Subject: Re: FreeBSD 13.2 NFS client mount hangs To: Rick Macklem Cc: FreeBSD FS Content-Type: multipart/mixed; boundary="000000000000fd1940060723b699" X-Spamd-Bar: --- X-Spamd-Result: default: False [-3.99 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.99)[-0.993]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20230601]; MIME_GOOD(-0.10)[multipart/mixed,text/plain]; ARC_NA(0.00)[]; FROM_HAS_DN(0.00)[]; FREEMAIL_FROM(0.00)[gmail.com]; TAGGED_RCPT(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::c32:from]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; TO_MATCH_ENVRCPT_SOME(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[freebsd-fs@freebsd.org]; TAGGED_FROM(0.00)[]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; FREEMAIL_ENVFROM(0.00)[gmail.com]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; MID_RHS_MATCH_FROMTLD(0.00)[]; HAS_ATTACHMENT(0.00)[]; MLMMJ_DEST(0.00)[freebsd-fs@freebsd.org]; RCPT_COUNT_TWO(0.00)[2]; FREEMAIL_TO(0.00)[gmail.com]; RCVD_COUNT_ONE(0.00)[1]; MIME_TRACE(0.00)[0:+,1:+,2:~]; FROM_EQ_ENVFROM(0.00)[]; RCVD_TLS_LAST(0.00)[] X-Rspamd-Queue-Id: 4S2sl10xFNz3TyW --000000000000fd1940060723b699 Content-Type: text/plain; charset="UTF-8" Here is a small pcap file obtained during a hang. This hang "feels" different from the others. It was a different type of server and the processes affected were different. But the symptoms are the same. (As are the mount flags, etc., except this fs is mounted read-only.) It's not much; only 37 packets matched over about five minutes. I'll try to get one I'm more confident is of the type previously discussed. Nonetheless I figured I'd pass this along in case it is helpful and/or to confirm that this is the format you're looking for. Thanks! --000000000000fd1940060723b699 Content-Type: application/octet-stream; name="nfs-hang.pcap" Content-Disposition: attachment; filename="nfs-hang.pcap" Content-Transfer-Encoding: base64 Content-ID: X-Attachment-Id: f_lngb2ybx0 1MOyoQIABAAAAAAAAAAAAAAABAABAAAAZJAhZfyaCgBCAAAAQgAAAAqEmUs/sM63RlUioQgARQAA NLRaQABABmAxrBQGAawUyA4IAQMg3j8ePBD3dQWAEQH9jUYAAAEBCAqXKWFokPuqGWSQIWUdmwoA QgAAAEIAAADOt0ZVIqEKhJlLP7AIAEUAADQAAEAAQAYUjKwUyA6sFAYBAyAIARD3dQXePx49gBAR ACZfAAABAQgKkPxFr5cpYWhpkCFlEaoNAEIAAABCAAAAzrdGVSKhCoSZSz+wCABFAAA0AABAAEAG FIysFMgOrBQGAQMgCAEQ93UF3j8ePYAREQAmXwAAAQEICpD8Wf+XKWFoaZAhZUKqDQBKAAAASgAA AM63RlUioQqEmUs/sAgARQAAPAAAQABABhSErBTIDqwUBgECmAgBkOmM8wAAAACgAv//JmcAAAIE BbQBAwMGBAIICl3vYzUAAAAAaZAhZYuqDQBCAAAAQgAAAAqEmUs/sM63RlUioQgARQAANAAAQABA BhSMrBQGAawUyA4IAQMg3j8ePRD3dQaAEAH9yQ0AAAEBCAqXKXW5kPxZ/2mQIWWYqg0ASgAAAEoA AAAKhJlLP7DOt0ZVIqEIAEUAADwAAEAAQAYUhKwUBgGsFMgOCAECmHEjgrSQ6Yz0oBL+iDjYAAAC BAW0BAIICpcpdbld72M1AQMDB2mQIWWjqg0AQgAAAEIAAADOt0ZVIqEKhJlLP7AIAEUAADQAAEAA QAYUjKwUyA6sFAYBApgIAZDpjPRxI4K1gBAEAyZfAAABAQgKXe9jNZcpdblpkCFls6oNAL4AAAC+ AAAAzrdGVSKhCoSZSz+wCABFAACwAABAAEAGFBCsFMgOrBQGAQKYCAGQ6Yz0cSOCtYAYEQAm2wAA AQEICl3vYzWXKXW5gAAAeFYLIhoAAAAAAAAAAgABhqMAAAAEAAAAAQAAAAEAAAAYZRJLsAAAAANj MTQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAFUmVuZXcAAAAAAAACAAAAAQAAADWVO+lkHClpXqcA AAAAAAAABEuBFAAAAAAAAAAAAAAAAWmQIWUFqw0AQgAAAEIAAAAKhJlLP7DOt0ZVIqEIAEUAADSq akAAQAZqIawUBgGsFMgOCAECmHEjgrWQ6Y1wgBAB/WO0AAABAQgKlyl1uV3vYzVpkCFlMasNAJ4A AACeAAAACoSZSz+wzrdGVSKhCABFAACQqmtAAEAGacSsFAYBrBTIDggBAphxI4K1kOmNcIAYAf0C CAAAAQEICpcpdbld72M1gAAAWFYLIhoAAAABAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAFUmVuZXcA AAAAAAABAAAANQAAAACVO+lkHClpXqcAAAAAAAAABEuBFAAAAAAAAAAdAAAAHQAAAAFpkCFlIEUO AEIAAABCAAAAzrdGVSKhCoSZSz+wCABFAAA0AABAAEAGFIysFMgOrBQGAQKYCAGQ6Y1wcSODEYAQ EQAmXwAAAQEICl3vY12XKXW5l5AhZWcfAAC+AAAAvgAAAM63RlUioQqEmUs/sAgARQAAsAAAQABA BhQQrBTIDqwUBgECmAgBkOmNcHEjgxGAGBEAJtsAAAEBCApd8BNulyl1uYAAAHhWCyIbAAAAAAAA AAIAAYajAAAABAAAAAEAAAABAAAAGGUSS7AAAAADYzE0AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA BVJlbmV3AAAAAAAAAgAAAAEAAAA1lTvpZBwpaV6nAAAAAAAAAARLgRUAAAAAAAAAAAAAAAGXkCFl 4R8AAJ4AAACeAAAACoSZSz+wzrdGVSKhCABFAACQqmxAAEAGacOsFAYBrBTIDggBAphxI4MRkOmN 7IAYAf2gvAAAAQEICpcqJfBd8BNugAAAWFYLIhsAAAABAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAF UmVuZXcAAAAAAAABAAAANQAAAACVO+lkHClpXqcAAAAAAAAABEuBFQAAAAAAAAAdAAAAHQAAAAGX kCFlkrAAAEIAAABCAAAAzrdGVSKhCoSZSz+wCABFAAA0AABAAEAGFIysFMgOrBQGAQKYCAGQ6Y3s cSODbYAQEQAmXwAAAQEICl3wE5OXKiXwxJAhZVhuAgC+AAAAvgAAAM63RlUioQqEmUs/sAgARQAA sAAAQABABhQQrBTIDqwUBgECmAgBkOmN7HEjg22AGBEAJtsAAAEBCApd8MPNlyol8IAAAHhWCyIc AAAAAAAAAAIAAYajAAAABAAAAAEAAAABAAAAGGUSS7AAAAADYzE0AAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAABVJlbmV3AAAAAAAAAgAAAAEAAAA1lTvpZBwpaV6nAAAAAAAAAARLgRYAAAAAAAAAAAAA AAHEkCFlz24CAJ4AAACeAAAACoSZSz+wzrdGVSKhCABFAACQqm1AAEAGacKsFAYBrBTIDggBAphx I4NtkOmOaIAYAf0/IwAAAQEICpcq1k9d8MPNgAAAWFYLIhwAAAABAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAFUmVuZXcAAAAAAAABAAAANQAAAACVO+lkHClpXqcAAAAAAAAABEuBFgAAAAAAAAAdAAAA HQAAAAHEkCFlmPoCAEIAAABCAAAAzrdGVSKhCoSZSz+wCABFAAA0AABAAEAGFIysFMgOrBQGAQKY CAGQ6Y5ocSODyYAQEQAmXwAAAQEICl3ww/GXKtZP8JAhZVSBBAC+AAAAvgAAAM63RlUioQqEmUs/ sAgARQAAsAAAQABABhQQrBTIDqwUBgECmAgBkOmOaHEjg8mAGBEAJtsAAAEBCApd8XA1lyrWT4AA AHhWCyIdAAAAAAAAAAIAAYajAAAABAAAAAEAAAABAAAAGGUSS7AAAAADYzE0AAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAABVJlbmV3AAAAAAAAAgAAAAEAAAA1lTvpZBwpaV6nAAAAAAAAAARLgRcAAAAA AAAAAAAAAAHwkCFlzIEEAJ4AAACeAAAACoSZSz+wzrdGVSKhCABFAACQqm5AAEAGacGsFAYBrBTI DggBAphxI4PJkOmO5IAYAf3leAAAAQEICpcrgrZd8XA1gAAAWFYLIh0AAAABAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAFUmVuZXcAAAAAAAABAAAANQAAAACVO+lkHClpXqcAAAAAAAAABEuBFwAAAAAA AAAdAAAAHQAAAAHwkCFlXh0FAEIAAABCAAAAzrdGVSKhCoSZSz+wCABFAAA0AABAAEAGFIysFMgO rBQGAQKYCAGQ6Y7kcSOEJYAQEQAmXwAAAQEICl3xcF2XK4K2HZEhZdosAgBCAAAAQgAAAAqEmUs/ sM63RlUioQgARQAANKpvQABABmocrBQGAawUyA4IAQKYcSOEJZDpjuSAEQH9l3cAAAEBCAqXLDHk XfFwXR2RIWXzLAIAQgAAAEIAAADOt0ZVIqEKhJlLP7AIAEUAADQAAEAAQAYUjKwUyA6sFAYBApgI AZDpjuRxI4QmgBARACZfAAABAQgKXfIfZJcsMeQdkSFlX44FAEIAAABCAAAAzrdGVSKhCoSZSz+w CABFAAA0AABAAEAGFIysFMgOrBQGAQKYCAGQ6Y7kcSOEJoAREQAmXwAAAQEICl3yIEKXLDHkHZEh ZYOOBQBKAAAASgAAAM63RlUioQqEmUs/sAgARQAAPAAAQABABhSErBTIDqwUBgECiggBZ4UHjgAA AACgAv//JmcAAAIEBbQBAwMGBAIICkuWJsYAAAAAHZEhZRKPBQBCAAAAQgAAAAqEmUs/sM63RlUi oQgARQAANAAAQABABhSMrBQGAawUyA4IAQKYcSOEJpDpjuWAEAH95rIAAAEBCAqXLDLCXfIgQh2R IWUdjwUASgAAAEoAAAAKhJlLP7DOt0ZVIqEIAEUAADwAAEAAQAYUhKwUBgGsFMgOCAECiovmYOZn hQePoBL+iIB3AAACBAW0BAIICpcsMsJLlibGAQMDBx2RIWUqjwUAQgAAAEIAAADOt0ZVIqEKhJlL P7AIAEUAADQAAEAAQAYUjKwUyA6sFAYBAooIAWeFB4+L5mDngBAEAyZfAAABAQgKS5YmxpcsMsId kSFlT48FAL4AAAC+AAAAzrdGVSKhCoSZSz+wCABFAACwAABAAEAGFBCsFMgOrBQGAQKKCAFnhQeP i+Zg54AYEQAm2wAAAQEICkuWJsaXLDLCgAAAeFYDBZAAAAAAAAAAAgABhqMAAAAEAAAAAQAAAAEA AAAYZRJLsAAAAANjMTQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAFUmVuZXcAAAAAAAACAAAAAQAA ADWVO+lkHClpXqcAAAAAAAAABEuBGAAAAAAAAAAAAAAAAR2RIWXrjwUAQgAAAEIAAAAKhJlLP7DO t0ZVIqEIAEUAADQ350AAQAbcpKwUBgGsFMgOCAECiovmYOdnhQgLgBAB/atTAAABAQgKlywywkuW JsYdkSFl8I8FAJ4AAACeAAAACoSZSz+wzrdGVSKhCABFAACQN+hAAEAG3EesFAYBrBTIDggBAoqL 5mDnZ4UIC4AYAf1mNQAAAQEICpcsMsJLlibGgAAAWFYDBZAAAAABAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAFUmVuZXcAAAAAAAABAAAANQAAAACVO+lkHClpXqcAAAAAAAAABEuBGAAAAAAAAAAdAAAA HQAAAAEdkSFlui4GAEIAAABCAAAAzrdGVSKhCoSZSz+wCABFAAA0AABAAEAGFIysFMgOrBQGAQKK CAFnhQgLi+ZhQ4AQEQAmXwAAAQEICkuWJu+XLDLCSpEhZWzyBgC+AAAAvgAAAM63RlUioQqEmUs/ sAgARQAAsAAAQABABhQQrBTIDqwUBgECiggBZ4UIC4vmYUOAGBEAJtsAAAEBCApLltbplywywoAA AHhWAwWRAAAAAAAAAAIAAYajAAAABAAAAAEAAAABAAAAGGUSS7AAAAADYzE0AAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAABVJlbmV3AAAAAAAAAgAAAAEAAAA1lTvpZBwpaV6nAAAAAAAAAARLgRkAAAAA AAAAAAAAAAFKkSFl7vIGAJ4AAACeAAAACoSZSz+wzrdGVSKhCABFAACQN+lAAEAG3EasFAYBrBTI DggBAoqL5mFDZ4UIh4AYAf0FFQAAAQEICpcs4uRLltbpgAAAWFYDBZEAAAABAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAFUmVuZXcAAAAAAAABAAAANQAAAACVO+lkHClpXqcAAAAAAAAABEuBGQAAAAAA AAAdAAAAHQAAAAFKkSFlkI4HAEIAAABCAAAAzrdGVSKhCoSZSz+wCABFAAA0AABAAEAGFIysFMgO rBQGAQKKCAFnhQiHi+Zhn4AQEQAmXwAAAQEICkuW1xGXLOLkd5EhZWGGCwC+AAAAvgAAAM63RlUi oQqEmUs/sAgARQAAsAAAQABABhQQrBTIDqwUBgECiggBZ4UIh4vmYZ+AGBEAJtsAAAEBCApLl4fd lyzi5IAAAHhWAwWSAAAAAAAAAAIAAYajAAAABAAAAAEAAAABAAAAGGUSS7AAAAADYzE0AAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAABVJlbmV3AAAAAAAAAgAAAAEAAAA1lTvpZBwpaV6nAAAAAAAAAARL gRoAAAAAAAAAAAAAAAF3kSFlBocLAJ4AAACeAAAACoSZSz+wzrdGVSKhCABFAACQN+pAAEAG3EWs FAYBrBTIDggBAoqL5mGfZ4UJA4AYAf2iUQAAAQEICpctk9hLl4fdgAAAWFYDBZIAAAABAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAFUmVuZXcAAAAAAAABAAAANQAAAACVO+lkHClpXqcAAAAAAAAABEuB GgAAAAAAAAAdAAAAHQAAAAF3kSFlOiIMAEIAAABCAAAAzrdGVSKhCoSZSz+wCABFAAA0AABAAEAG FIysFMgOrBQGAQKKCAFnhQkDi+Zh+4AQEQAmXwAAAQEICkuXiAWXLZPY --000000000000fd1940060723b699-- From nobody Sat Oct 7 19:08:14 2023 X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4S2vvv5csnz4xJY4 for ; Sat, 7 Oct 2023 19:08:27 +0000 (UTC) (envelope-from rick.macklem@gmail.com) Received: from mail-oo1-xc31.google.com (mail-oo1-xc31.google.com [IPv6:2607:f8b0:4864:20::c31]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4S2vvt47tHz4GY3 for ; Sat, 7 Oct 2023 19:08:26 +0000 (UTC) (envelope-from rick.macklem@gmail.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20230601 header.b=U0sch3Ri; spf=pass (mx1.freebsd.org: domain of rick.macklem@gmail.com designates 2607:f8b0:4864:20::c31 as permitted sender) smtp.mailfrom=rick.macklem@gmail.com; dmarc=pass (policy=none) header.from=gmail.com Received: by mail-oo1-xc31.google.com with SMTP id 006d021491bc7-57bb6b1f764so2014802eaf.2 for ; Sat, 07 Oct 2023 12:08:26 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1696705704; x=1697310504; darn=freebsd.org; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=lT3O6QD5jAvZfxRP46Es5CUETlaa6QaoS//QnQIUjYs=; b=U0sch3RiHC6Iy/3jjXQI3XIrADB3t4V1HUTCTTA0FKMphXsnW5y+X6GziN0hUO1q+T TRdyuYwvb9R8ogyjDNqFjQ0LdYGrMtHLeDjkCxKs7Wne3j3PYStyHkD3ahn0SbQFsieI LwHXm4Tp45r+BN+QuNzhfqRCabagaXnAhcCjXWlmTc8bvL4Z8+eQmFEwjKrIuv6Zc5E4 9kTIxwlKT2dSxgQd0tVfR8uhGSpZ3xWOLrDCYljVggtPC35OvQ7rYVgepu9yhYJ8iEKQ g5CUyDkcvjcGdAA5Ov07QmGQCYtegYJixGfzqvx5ajBuz4SMK3Po4T6pYkg4Q1kwuJAC 1gHA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1696705704; x=1697310504; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=lT3O6QD5jAvZfxRP46Es5CUETlaa6QaoS//QnQIUjYs=; b=BZZb4TBPeMcMHRYZhuTDCEn7C+CAf/EGww2hnTh9EEizhSwotQoim4H5NJNgblmzWS qNrff822mWNizsWaashwJ1Bxj1YGSRrh/Pw4qiVlq7a+kzh6Q1nT1RRVoo3EpDPSOsCo g0Ghja07iDpf/OLuderR5X1RWgIXjYhczmR0LRC41EkCaka8c36M+BRXhYrUcfDfcljP iUvIeMDc0DwZKGd9mw9Kp28o+kr0EhP4+2m43hbdSO8s1XpzUGs7qF2vYfgSR1pl0YqV X6LJ9cHR2KvqZsIDzID/YAgvrMWrz4wWOXIMu/mqX8pF1KOQmWebYAN4vtOuwdiDvXD5 3fAw== X-Gm-Message-State: AOJu0YztZDmZXpCYR0+IkOZmY2d0gveSvTj/8CVm6n/AfpJhcaxc7PEQ LYLPW6UzLZ8eL0qS3Wc4ebaI/4XU3FAz6/qYnw== X-Google-Smtp-Source: AGHT+IE8MkD9VZ3pas4Qxwge46GZmRm6i9KFOQr5JvCazuE4lsx8o5wuNlHbwEprlDBTQrqWJNDVogUxeOxjKUlin7U= X-Received: by 2002:a05:6358:7299:b0:163:ce28:91ee with SMTP id w25-20020a056358729900b00163ce2891eemr5368399rwf.19.1696705704582; Sat, 07 Oct 2023 12:08:24 -0700 (PDT) List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 References: In-Reply-To: From: Rick Macklem Date: Sat, 7 Oct 2023 12:08:14 -0700 Message-ID: Subject: Re: FreeBSD 13.2 NFS client mount hangs To: J David Cc: FreeBSD FS Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Spamd-Bar: --- X-Spamd-Result: default: False [-3.87 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.87)[-0.874]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36:c]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20230601]; MIME_GOOD(-0.10)[text/plain]; FROM_HAS_DN(0.00)[]; TAGGED_RCPT(0.00)[]; FREEMAIL_FROM(0.00)[gmail.com]; PREVIOUSLY_DELIVERED(0.00)[freebsd-fs@freebsd.org]; ARC_NA(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::c31:from]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; MLMMJ_DEST(0.00)[freebsd-fs@freebsd.org]; DKIM_TRACE(0.00)[gmail.com:+]; RCPT_COUNT_TWO(0.00)[2]; TO_DN_ALL(0.00)[]; MID_RHS_MATCH_FROMTLD(0.00)[]; TAGGED_FROM(0.00)[]; RCVD_COUNT_ONE(0.00)[1]; FROM_EQ_ENVFROM(0.00)[]; FREEMAIL_TO(0.00)[gmail.com]; FREEMAIL_ENVFROM(0.00)[gmail.com]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_LAST(0.00)[] X-Rspamd-Queue-Id: 4S2vvt47tHz4GY3 On Sat, Oct 7, 2023 at 10:30=E2=80=AFAM J David w= rote: > > Here is a small pcap file obtained during a hang. This hang "feels" > different from the others. It was a different type of server and the > processes affected were different. But the symptoms are the same. (As > are the mount flags, etc., except this fs is mounted read-only.) > > It's not much; only 37 packets matched over about five minutes. I'll > try to get one I'm more confident is of the type previously discussed. > > Nonetheless I figured I'd pass this along in case it is helpful and/or > to confirm that this is the format you're looking for. Doesn't tell us much. If you look at it in wireshark (I just download and install it on a windows laptop), you'll see that the very first packet is the NFS server closing down the TCP connection (a FIN,ACK on a connection from port 800). Then the client establishes a new TCP connection and does Renew operations (just a Sequence operation by itself). Since there is no other RPCs than the Renews, I cannot guess why the server closed the connection, nor what the client is up to. The Sequence reply does note that the callback path is down, which is expected, given you are not running the nfscbd(8) daemon. One thing you could try is running the nfscbd(8) daemon. I suggested against doing so to avoid delegations, but it might be that this server does not handle that case well. To do so, add: nfscbd_enable=3D"YES" to the NFS client's /etc/rc.conf. --> It needs to be running before the mount is done, so the NFS client either needs to be rebooted or the NFS mount dismounted, the nfscbd started and then the NFS mount done again. Maybe this server will be happier with the callback path working? rick > > Thanks! From nobody Sat Oct 7 21:52:14 2023 X-Original-To: fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4S2zXt6h5Vz4wXjh for ; Sat, 7 Oct 2023 21:52:14 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4S2zXt4j69z4d6H for ; Sat, 7 Oct 2023 21:52:14 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1696715534; a=rsa-sha256; cv=none; b=Wx4Gm+8TIFXPcl+hdOqS6+6sOsVz2a0jBh29TEdf92OPOi1e0KTyie+fzerSUoFVeqCkf3 laXfNbuienZ+gRJqrdHx1Wc+zjbOM+wxA6iCWzxPXaYA7qm0fmtOotQeARrXYCzIZxHCI0 j3DtL0cv9jO9IwWUq1NNYgPAeOd/jER28wmv+iFFjbFjAtgMN8jccYJFT67HeopbTYyF6i Q4Xky5wstRHeTynvNbiv9J9kBANyEES1BU5/YhE9aYeekQqkrTSn53GdBRkd17GIJ83aLs 6LLvBn9mu+6U/+Z8Rwns/vaXIerdCerSl3AbIdbpiOyfG8YsLMMaL/XDiFarzQ== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1696715534; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=JgXrXEG59ZsShoaUDFhE8bpumZQJEgTxgVml6+HejL4=; b=RfQnMJsOgIPMH9TP1DasJZ3x1TI1vDFfusmqtmmiz5sodMij1EY5Z0jqGSCLk1CUIdg9aq eiYK1rS+k1Q4jeCCrjSMhEJfkgp6VSsb8jAWXzYrOYHMyWw675b1fdCgH5quDLgGOfOR9K bA6cH46GwzZ2Fh86lrzjkV5tvSm0tkyUXJR9coHNxHhk0BGZ9N0oZ//pgZ/OsC/60oTOdF 4ZyPh1Uam/iOcqdeuSBXYm/VnH4jD06QQdzFcUrtl5Mvh6rY1sOUgYSZtvM+IJyYYKYqqN Cx56Q0E+JOtGARz4z+xe9enAx8VzFWknBlMarTjZQBvQ5/zsmPTAd+zt6ncm9g== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4S2zXt3m1ZznjM for ; Sat, 7 Oct 2023 21:52:14 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 397LqEPg098439 for ; Sat, 7 Oct 2023 21:52:14 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 397LqEbZ098438 for fs@FreeBSD.org; Sat, 7 Oct 2023 21:52:14 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 274271] zfs deadlock while/after writing files Date: Sat, 07 Oct 2023 21:52:14 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 13.2-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: pmc@citylink.dinoex.sub.org X-Bugzilla-Status: Open X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D274271 --- Comment #2 from Peter Much --- Thanks, Alan, I really should remember that one. Too bad I was in a hurry, = had to travel a few hours later, no time for proper analysis, just gathered the stalled workproducts and then went for a bit of sleep - actually my whole quarterly portbuild had run through without error and within the calculated timeframe, until the end. I'm not using poudriere, I'm doing it in temporary bhyves, so there were 10'000s of snapshots done in the bhyves, and at times= it went 28 Gig into paging. Somewhere apparently something got lost, but this might be difficult to reproduce at will. I'll see to re-run the whole thing when I get back. --=20 You are receiving this mail because: You are the assignee for the bug.=