From nobody Thu Aug 18 14:49:54 2022 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4M7nqH16T9z4Zgjf for ; Thu, 18 Aug 2022 14:50:03 +0000 (UTC) (envelope-from ler@lerctr.org) Received: from thebighonker.lerctr.org (thebighonker.lerctr.org [IPv6:2602:fcdb:0:10:7ae3:b5ff:fe1b:23b4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "*.lerctr.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4M7nqG3BXZz40P5; Thu, 18 Aug 2022 14:50:02 +0000 (UTC) (envelope-from ler@lerctr.org) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lerctr.org; s=ler2019; h=Content-Transfer-Encoding:Content-Type:Message-ID:Subject:To: From:Date:MIME-Version:Sender:Reply-To:Cc:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: In-Reply-To:References:List-Id:List-Help:List-Unsubscribe:List-Subscribe: List-Post:List-Owner:List-Archive; bh=Y0O6oLaNl/uMSIVCvUV9PsMmtzeQF69MtgmHb5MmawM=; b=btekUu8zs8Mb775ufe/zxlhpys D/H6kDG+TXWcwobsw8tFmN08yGMFPlIIWq6CJSIv/jUGb0gtb8y6D+qqIJBX7ymCKq/C2rSlV4ghY 4Zduk1e5akWo7YvenmsAExVlLdl0910PWy944IaGOM/606z1TwXEOr69wiXmxYkccffaza9EH9R1F xk7CXABR1q1k4IJUGaTmI/vCN+gm6lijkF8XpCmgzZnDR+9lsUBVKANjYMFtdAMLGSVPMAvWdN9sX 33PHC08+2rD9ZqEb678arsK9Ac4RxP3bHIF47kU78d/NvP/wFnl0ud//JOMG0wsygbOmEn5xGklG0 va+2HXUw==; Received-SPF: pass (thebighonker.lerctr.org: domain of lerctr.org designates 2602:fcdb:0:10:7ae3:b5ff:fe1b:23b4 as permitted sender) client-ip=2602:fcdb:0:10:7ae3:b5ff:fe1b:23b4; envelope-from=ler@lerctr.org; helo=webmail.lerctr.org; Received: from thebighonker.lerctr.org ([2602:fcdb:0:10:7ae3:b5ff:fe1b:23b4]:55337 helo=webmail.lerctr.org) by thebighonker.lerctr.org with esmtpsa (TLS1.3) tls TLS_AES_256_GCM_SHA384 (Exim 4.95 (FreeBSD)) (envelope-from ) id 1oOgqI-000G6m-8N; Thu, 18 Aug 2022 09:49:54 -0500 Received: from 76-250-255-117.lightspeed.austtx.sbcglobal.net ([76.250.255.117]) by webmail.lerctr.org with HTTP (HTTP/1.1 POST); Thu, 18 Aug 2022 09:49:54 -0500 List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org MIME-Version: 1.0 Date: Thu, 18 Aug 2022 09:49:54 -0500 From: Larry Rosenman To: Freebsd current , Mark Johnston Subject: Hangs in bacula / NFS? on recent Current Message-ID: <405b3873b709d42feb438b5b954ecdc2@lerctr.org> X-Sender: ler@lerctr.org Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-Rspamd-Queue-Id: 4M7nqG3BXZz40P5 X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=lerctr.org header.s=ler2019 header.b=btekUu8z; dmarc=pass (policy=none) header.from=lerctr.org; spf=pass (mx1.freebsd.org: domain of ler@lerctr.org designates 2602:fcdb:0:10:7ae3:b5ff:fe1b:23b4 as permitted sender) smtp.mailfrom=ler@lerctr.org X-Spamd-Result: default: False [-4.00 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-0.999]; DMARC_POLICY_ALLOW(-0.50)[lerctr.org,none]; R_DKIM_ALLOW(-0.20)[lerctr.org:s=ler2019]; R_SPF_ALLOW(-0.20)[+mx]; MIME_GOOD(-0.10)[text/plain]; RCVD_VIA_SMTP_AUTH(0.00)[]; MLMMJ_DEST(0.00)[freebsd-current@freebsd.org]; MIME_TRACE(0.00)[0:+]; FROM_EQ_ENVFROM(0.00)[]; ASN(0.00)[asn:55103, ipnet:2602:fcdb::/36, country:US]; RCVD_TLS_LAST(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; TO_DN_ALL(0.00)[]; RCVD_COUNT_TWO(0.00)[2]; FREEFALL_USER(0.00)[ler]; ARC_NA(0.00)[]; DKIM_TRACE(0.00)[lerctr.org:+]; FROM_HAS_DN(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; SUBJECT_HAS_QUESTION(0.00)[] X-ThisMailContainsUnwantedMimeParts: N I didn't get all my mail on my bacula backups today (they backup to NFS mounted TrueNAS). Also a df hangs. Here are procstat -kk's for all: ler in 🌐 borg in ~ via C v14.0.5-clang on ☁️ (us-east-1) ❯ ps auxxxwww|grep bacula bacula 2067 0.0 0.0 63188 13652 - Is 11:30 0:17.49 /usr/local/sbin/bacula-sd -u bacula -g bacula -v -c /usr/local/etc/bacula/bacula-sd.conf root 2072 0.0 0.0 59280 31276 - Is 11:30 0:00.31 /usr/local/sbin/bacula-fd -u root -g wheel -v -c /usr/local/etc/bacula/bacula-fd.conf bacula 2075 0.0 0.0 86992 19352 - Is 11:30 0:56.95 /usr/local/sbin/bacula-dir -u bacula -g bacula -v -c /usr/local/etc/bacula/bacula-dir.conf postgres 50241 0.0 0.1 285764 160244 - Is 23:05 0:00.38 postgres: bacula bacula [local] (postgres) postgres 50244 0.0 0.1 298784 74448 - Ds 23:05 0:00.67 postgres: bacula bacula [local] (postgres) ler 66595 0.0 0.0 12888 2600 3 S+ 09:46 0:00.00 grep --color=auto bacula ler in 🌐 borg in ~ via C v14.0.5-clang on ☁️ (us-east-1) ❯ sudo procstat -kk 2067 PID TID COMM TDNAME KSTACK 2067 100742 bacula-sd - mi_switch+0x157 sleepq_switch+0x107 sleepq_catch_signals+0x266 sleepq_wait_sig+0x9 _cv_wait_sig+0x137 kern_select+0x9fe sys_select+0x56 amd64_syscall+0x12e fast_syscall_common+0xf8 2067 101036 bacula-sd - mi_switch+0x157 sleepq_switch+0x107 sleepq_catch_signals+0x266 sleepq_timedwait_sig+0x12 _sleep+0x27d umtxq_sleep+0x242 do_wait+0x26b __umtx_op_wait_uint_private+0x54 sys__umtx_op+0x7e amd64_syscall+0x12e fast_syscall_common+0xf8 2067 101038 bacula-sd - mi_switch+0x157 sleepq_switch+0x107 sleepq_catch_signals+0x266 sleepq_timedwait_sig+0x12 _sleep+0x27d umtxq_sleep+0x242 do_wait+0x26b __umtx_op_wait_uint_private+0x54 sys__umtx_op+0x7e amd64_syscall+0x12e fast_syscall_common+0xf8 2067 124485 bacula-sd - mi_switch+0x157 sleepq_switch+0x107 sleepq_catch_signals+0x266 sleepq_timedwait_sig+0x12 _cv_timedwait_sig_sbt+0x15c kern_poll_kfds+0x457 kern_poll+0x9f sys_poll+0x50 amd64_syscall+0x12e fast_syscall_common+0xf8 ler in 🌐 borg in ~ via C v14.0.5-clang on ☁️ (us-east-1) ❯ sudo procstat -kk 2072 PID TID COMM TDNAME KSTACK 2072 100677 bacula-fd - mi_switch+0x157 sleepq_switch+0x107 sleepq_catch_signals+0x266 sleepq_wait_sig+0x9 _cv_wait_sig+0x137 kern_select+0x9fe sys_select+0x56 amd64_syscall+0x12e fast_syscall_common+0xf8 2072 101039 bacula-fd - mi_switch+0x157 sleepq_switch+0x107 sleepq_catch_signals+0x266 sleepq_timedwait_sig+0x12 _sleep+0x27d umtxq_sleep+0x242 do_wait+0x26b __umtx_op_wait_uint_private+0x54 sys__umtx_op+0x7e amd64_syscall+0x12e fast_syscall_common+0xf8 2072 101040 bacula-fd - mi_switch+0x157 sleepq_switch+0x107 sleepq_catch_signals+0x266 sleepq_timedwait_sig+0x12 _sleep+0x27d umtxq_sleep+0x242 do_wait+0x26b __umtx_op_wait_uint_private+0x54 sys__umtx_op+0x7e amd64_syscall+0x12e fast_syscall_common+0xf8 2072 124490 bacula-fd - mi_switch+0x157 sleepq_switch+0x107 sleepq_catch_signals+0x266 sleepq_timedwait_sig+0x12 _cv_timedwait_sig_sbt+0x15c kern_poll_kfds+0x457 kern_poll+0x9f sys_poll+0x50 amd64_syscall+0x12e fast_syscall_common+0xf8 ler in 🌐 borg in ~ via C v14.0.5-clang on ☁️ (us-east-1) ❯ sudo procstat -kk 2075 PID TID COMM TDNAME KSTACK 2075 101007 bacula-dir - mi_switch+0x157 sleepq_switch+0x107 sleepq_catch_signals+0x266 sleepq_wait_sig+0x9 _sleep+0x29b umtxq_sleep+0x242 do_wait+0x26b __umtx_op_wait+0x53 sys__umtx_op+0x7e amd64_syscall+0x12e fast_syscall_common+0xf8 2075 101041 bacula-dir - mi_switch+0x157 sleepq_switch+0x107 sleepq_catch_signals+0x266 sleepq_timedwait_sig+0x12 _sleep+0x27d umtxq_sleep+0x242 do_wait+0x26b __umtx_op_wait_uint_private+0x54 sys__umtx_op+0x7e amd64_syscall+0x12e fast_syscall_common+0xf8 2075 101045 bacula-dir - mi_switch+0x157 sleepq_switch+0x107 sleepq_catch_signals+0x266 sleepq_wait_sig+0x9 _cv_wait_sig+0x137 kern_select+0x9fe sys_select+0x56 amd64_syscall+0x12e fast_syscall_common+0xf8 2075 101046 bacula-dir - mi_switch+0x157 sleepq_switch+0x107 sleepq_catch_signals+0x266 sleepq_timedwait_sig+0x12 _sleep+0x27d umtxq_sleep+0x242 do_wait+0x26b __umtx_op_wait_uint_private+0x54 sys__umtx_op+0x7e amd64_syscall+0x12e fast_syscall_common+0xf8 2075 101047 bacula-dir - mi_switch+0x157 sleepq_switch+0x107 sleepq_catch_signals+0x266 sleepq_timedwait_sig+0x12 _sleep+0x27d kern_clock_nanosleep+0x1d1 sys_nanosleep+0x3b amd64_syscall+0x12e fast_syscall_common+0xf8 2075 124479 bacula-dir - mi_switch+0x157 sleepq_switch+0x107 sleepq_catch_signals+0x266 sleepq_wait_sig+0x9 _cv_wait_sig+0x137 kern_poll_kfds+0x48c kern_poll+0x9f sys_poll+0x50 amd64_syscall+0x12e fast_syscall_common+0xf8 2075 124480 bacula-dir - mi_switch+0x157 sleepq_switch+0x107 sleepq_catch_signals+0x266 sleepq_timedwait_sig+0x12 _sleep+0x27d kern_clock_nanosleep+0x1d1 sys_nanosleep+0x3b amd64_syscall+0x12e fast_syscall_common+0xf8 2075 124481 bacula-dir - mi_switch+0x157 sleepq_switch+0x107 sleepq_catch_signals+0x266 sleepq_timedwait_sig+0x12 _sleep+0x27d kern_clock_nanosleep+0x1d1 sys_nanosleep+0x3b amd64_syscall+0x12e fast_syscall_common+0xf8 2075 124489 bacula-dir - mi_switch+0x157 sleepq_switch+0x107 sleepq_catch_signals+0x266 sleepq_timedwait_sig+0x12 _cv_timedwait_sig_sbt+0x15c kern_poll_kfds+0x457 kern_poll+0x9f sys_poll+0x50 amd64_syscall+0x12e fast_syscall_common+0xf8 2075 124506 bacula-dir - mi_switch+0x157 sleepq_switch+0x107 sleepq_catch_signals+0x266 sleepq_timedwait_sig+0x12 _sleep+0x27d kern_clock_nanosleep+0x1d1 sys_nanosleep+0x3b amd64_syscall+0x12e fast_syscall_common+0xf8 ler in 🌐 borg in ~ via C v14.0.5-clang on ☁️ (us-east-1) ❯ sudo procstat -kk 66390 PID TID COMM TDNAME KSTACK 66390 101514 df - mi_switch+0x157 sleepq_switch+0x107 sleepq_timedwait+0x4b _sleep+0x28e clnt_reconnect_call+0x809 newnfs_request+0xa95 nfscl_request+0x5a nfsrpc_statfs+0x19d nfs_statfs+0x148 vfs_statfs_sigdefer+0x2e kern_getfsstat+0x1f1 sys_getfsstat+0x22 amd64_syscall+0x12e fast_syscall_common+0xf8 ler in 🌐 borg in ~ via C v14.0.5-clang on ☁️ (us-east-1) ❯ this was built yesterday: ❯ uname -a FreeBSD borg.lerctr.org 14.0-CURRENT FreeBSD 14.0-CURRENT #142 ler/freebsd-main-changes-n257453-175a127a72f: Wed Aug 17 09:23:32 CDT 2022 root@borg.lerctr.org:/usr/obj/usr/src/amd64.amd64/sys/LER-MINIMAL amd64 ler in 🌐 borg in ~ via C v14.0.5-clang on ☁️ (us-east-1) ❯ What else do we need? -- Larry Rosenman http://www.lerctr.org/~ler Phone: +1 214-642-9640 E-Mail: ler@lerctr.org US Mail: 5708 Sabbia Dr, Round Rock, TX 78665-2106