From owner-freebsd-bugs@freebsd.org Wed Apr 14 11:41:29 2021 Return-Path: Delivered-To: freebsd-bugs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 2ACE35D4AB8 for ; Wed, 14 Apr 2021 11:41:29 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mailman.nyi.freebsd.org (unknown [127.0.1.3]) by mx1.freebsd.org (Postfix) with ESMTP id 4FL0tK0XlPz4tWY for ; Wed, 14 Apr 2021 11:41:29 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: by mailman.nyi.freebsd.org (Postfix) id 10B1E5D4C24; Wed, 14 Apr 2021 11:41:29 +0000 (UTC) Delivered-To: bugs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 1075D5D4AB7 for ; Wed, 14 Apr 2021 11:41:29 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4FL0tJ70tzz4t93 for ; Wed, 14 Apr 2021 11:41:28 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id E391B1D172 for ; Wed, 14 Apr 2021 11:41:28 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 13EBfSZI048922 for ; Wed, 14 Apr 2021 11:41:28 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 13EBfS1T048921 for bugs@FreeBSD.org; Wed, 14 Apr 2021 11:41:28 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: bugs@FreeBSD.org Subject: [Bug 255048] ROCKPRO64 freeze during heavy IO on external USB 3 disk Date: Wed, 14 Apr 2021 11:41:29 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 13.0-STABLE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: hlh@restart.be X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: bugs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: cc Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-bugs@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Bug reports List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 14 Apr 2021 11:41:29 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D255048 Henri Hennebert changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |hlh@restart.be --- Comment #1 from Henri Hennebert --- FreeBSD keystone.lab.bel 13.0-STABLE FreeBSD 13.0-STABLE #0 stable/13-n245189-236d1f8c1773 The disk is under zfs During a `git pull` in /usr/ports the system show the da0 disk 100% busy and afer ~2 minutes the system freeze. The system respond to ping I connect on the serial console (CR ~ ^B) to ddb: db> show msgbuf ... swap_pager: cannot allocate bio swap_pager: cannot allocate bio swap_pager: cannot allocate bio swap_pager: cannot allocate bio swap_pager: cannot allocate bio swap_pager: cannot allocate bio db> show allchains chain 1: thread 115444 (pid 87479, git) is sleeping on 0xffff0000ef3dfc08 "aw.aew_c= v" chain 2: thread 115446 (pid 87479, git) is sleeping on 0xffffa00058d653e8 "zio->io_= cv" chain 3: thread 115447 (pid 87479, git) is sleeping on 0xffffa000b26cf3e8 "zio->io_= cv" chain 4: thread 115450 (pid 87479, git) is sleeping on 0xffff0000ef3cbc08 "aw.aew_c= v" chain 5: thread 115452 (pid 87479, git) is sleeping on 0xffffa000091518b8 "zio->io_= cv" chain 6: thread 115453 (pid 87479, git) is sleeping on 0xffff0000ef3e9c08 "aw.aew_c= v" chain 7: thread 115455 (pid 87479, git) is sleeping on 0xffffa00058d028b8 "zio->io_= cv" chain 8: thread 115458 (pid 87479, git) is blocked on sx "zfsvfs->z_hold_mtx[i]" XL= OCK thread 115455 (pid 87479, git) is sleeping on 0xffffa00058d028b8 "zio->io_= cv" chain 9: thread 115459 (pid 87479, git) is sleeping on 0xffffa000971718b8 "zio->io_= cv" chain 10: thread 115460 (pid 87479, git) is sleeping on 0xffff00010420dc08 "aw.aew_c= v" chain 11: thread 115462 (pid 87479, git) is sleeping on 0xffff000104217c08 "aw.aew_c= v" chain 12: thread 115463 (pid 87479, git) is sleeping on 0xffff00010421cc08 "aw.aew_c= v" chain 13: thread 101506 (pid 1214, god.plugin) is sleeping on 0xffff0000ecdb70a8 "aw.aew_cv" chain 14: thread 101508 (pid 1214, god.plugin) is sleeping on 0xffffa000e430da80 "vm= pfw" chain 15: thread 101510 (pid 1214, god.plugin) is sleeping on 0xffff0000ecc5e0a8 "aw.aew_cv" chain 16: thread 101513 (pid 1214, god.plugin) is sleeping on 0xffff0000ecd990a8 "aw.aew_cv" chain 17: thread 101352 (pid 1211, apps.plugin) is sleeping on 0xffffa00058d658b8 "zio->io_cv" chain 18: thread 101448 (pid 1195, httpd) is sleeping on 0xffff0000ecc9f0a8 "aw.aew_= cv" chain 19: thread 101366 (pid 1179, httpd) is sleeping on 0xffffa0008c253d88 "zio->io= _cv" chain 20: thread 101346 (pid 1141, netdata) is sleeping on 0xffff0000e9424cb8 "aw.aew_cv" chain 21: thread 101355 (pid 1121, sshd) is sleeping on 0xffff000000b24d80 "pfault" chain 22: thread 101350 (pid 1083, sendmail) is sleeping on 0xffff000000b24d80 "vmwa= it" chain 23: thread 101349 (pid 1079, inetd) is sleeping on 0xffffa00027577b40 "select" chain 24: thread 101347 (pid 1050, ntpd) is sleeping on 0xffff0000e93fd0a8 "aw.aew_c= v" chain 25: thread 101314 (pid 970, syslogd) is sleeping on 0xffff000000b24d80 "pfault" chain 26: thread 100152 (pid 31, syncer) is sleeping on 0xffff000000d5f480 "syncer" chain 27: thread 100151 (pid 30, vnlru) is blocked on lockmgr DEXCL thread 100151 (pid 30, vnlru) is blocked on lockmgr DEXCL thread 100151 (pid 30, vnlru) is blocked on lockmgr DEXCL thread 100151 (pid 30, vnlru) is blocked on lockmgr DEXCL thread 100151 (pid 30, vnlru) is blocked on lockmgr DEXCL thread 100151 (pid 30, vnlru) is blocked on lockmgr DEXCL thread 100151 (pid 30, vnlru) is blocked on lockmgr DEXCL thread 100151 (pid 30, vnlru) is blocked on lockmgr DEXCL thread 100151 (pid 30, vnlru) is blocked on lockmgr DEXCL thread 100151 (pid 30, vnlru) is blocked on lockmgr DEXCL thread 100151 (pid 30, vnlru) is blocked on lockmgr DEXCL thread 100151 (pid 30, vnlru) is blocked on lockmgr DEXCL thread 100151 (pid 30, vnlru) is blocked on lockmgr DEXCL thread 100151 (pid 30, vnlru) is blocked on lockmgr DEXCL thread 100151 (pid 30, vnlru) is blocked on lockmgr DEXCL thread 100151 (pid 30, vnlru) is blocked on lockmgr DEXCL thread 100151 (pid 30, vnlru) is blocked on lockmgr DEXCL thread 100151 (pid 30, vnlru) is blocked on lockmgr DEXCL thread 100151 (pid 30, vnlru) is blocked on lockmgr DEXCL thread 100151 (pid 30, vnlru) is blocked on lockmgr DEXCL thread 100151 (pid 30, vnlru) is blocked on lockmgr DEXCL ...=20 ...continue for at least 1400 lines. ... db> show proc 30 Process 30 (vnlru) at 0xffffa00006449000: state: NORMAL uid: 0 gids: 0 parent: pid 0 at 0xffff000000cf1828 ABI: null flag: 0x10000204 flag2: 0 reaper: 0xffff000000cf1828 reapsubtree: 30 sigparent: 20 vmspace: 0xffff000000cf22c8 (map 0xffff000000cf22c8) (map.pmap 0xffff000000cf2388) (pmap 0xffff000000cf23e8) threads: 1 100151 D vlruwt 0xffffa00006449000 [vnlru] db> tr 100151 Tracing pid 30 tid 100151 td 0xffffa0000660f580 sched_switch() at mi_switch+0xf0 pc =3D 0xffff0000003dc7c0 lr =3D 0xffff0000003b2544 sp =3D 0xffff0000da1fc760 fp =3D 0xffff0000da1fc7b0 mi_switch() at sleepq_timedwait+0x28 pc =3D 0xffff0000003b2544 lr =3D 0xffff00000040ce64 sp =3D 0xffff0000da1fc7c0 fp =3D 0xffff0000da1fc7f0 sleepq_timedwait() at _sleep+0x194 pc =3D 0xffff00000040ce64 lr =3D 0xffff0000003b1960 sp =3D 0xffff0000da1fc800 fp =3D 0xffff0000da1fc830 _sleep() at vnlru_proc+0xc54 pc =3D 0xffff0000003b1960 lr =3D 0xffff0000004a4c18 sp =3D 0xffff0000da1fc840 fp =3D 0xffff0000da1fc8f0 vnlru_proc() at fork_exit+0x88 pc =3D 0xffff0000004a4c18 lr =3D 0xffff0000003547d0 sp =3D 0xffff0000da1fc900 fp =3D 0xffff0000da1fc950 --More-- fork_exit() at fork_trampoline+0x10 pc =3D 0xffff0000003547d0 lr =3D 0xffff0000007cfef8 sp =3D 0xffff0000da1fc960 fp =3D 0x0000000000000000 db> show mount 0xffff0000e663e100 keystone/ROOT/default on / (zfs) 0xffff0000bbb25100 devfs on /dev (devfs) 0xffff0000e663d600 procfs on /proc (procfs) 0xffff0000e663cb00 fdescfs on /dev/fd (fdescfs) 0xffff0000e9a4e100 keystone/var/spool on /var/spool (zfs) 0xffff0000e9a4d600 keystone/var/log on /var/log (zfs) 0xffff0000bbb24600 keystone/tmp on /tmp (zfs) 0xffff0000e9a48100 keystone/usr/local on /usr/local (zfs) 0xffff0000e663c000 keystone/var/db on /var/db (zfs) 0xffff0000e9a47600 keystone/home on /home (zfs) 0xffff0000bbb23b00 keystone/usr/ports on /usr/ports (zfs) 0xffff0000e9a4c000 keystone/usr/src on /usr/src (zfs) 0xffff0000e9a59100 keystone/var/tmp on /var/tmp (zfs) 0xffff0000e9a68100 keystone/usr/obj on /usr/obj (zfs) 0xffff0000e9a4cb00 keystone/var/imap on /var/imap (zfs) 0xffff0000e9a4b100 keystone/var/audit on /var/audit (zfs) 0xffff0000bbb23000 keystone/var/spool/openldap on /var/spool/openldap (zfs) 0xffff0000e9a58600 keystone/var/spool/httpd on /var/spool/httpd (zfs) 0xffff0000e9a46b00 keystone/var/spool/imap on /var/spool/imap (zfs) 0xffff0000ea23a100 keystone/var/spool/postgres on /var/spool/postgres (zfs) 0xffff0000ea2ef100 keystone/poudriere on /usr/local/poudriere (zfs) 0xffff0000ea239600 keystone/poudriere/data on /usr/local/poudriere/data (zf= s) 0xffff0000e9a57b00 keystone/poudriere/ports on /usr/local/poudriere/ports (= zfs) 0xffff0000e9a4a600 keystone/poudriere/jails on /usr/local/poudriere/jails (= zfs) 0xffff0000ea238b00 keystone/poudriere/ports/default on /usr/local/poudriere/ports/default (zfs) 0xffff0000ea238000 keystone/poudriere/data/packages on /usr/local/poudriere/data/packages (zfs) 0xffff0000ea2ee600 keystone/poudriere/data/cache on /usr/local/poudriere/data/cache (zfs) 0xffff0000ea2edb00 keystone/poudriere/data/logs on /usr/local/poudriere/data/logs (zfs) 0xffff0000e9a57000 keystone/poudriere/data/images on /usr/local/poudriere/data/images (zfs) 0xffff0000e9a66b00 keystone/poudriere/data/.m on /usr/local/poudriere/data/= .m (zfs) 0xffff0000e9a49b00 keystone/poudriere/jails/13aarch64 on /usr/local/poudriere/jails/13aarch64 (zfs) 0xffff0000e9a66000 keystone/poudriere/data/wrkdirs on /usr/local/poudriere/data/wrkdirs (zfs) 0xffff0000e9a49000 keystone/usr/ports/distfiles on /usr/ports/distfiles (zf= s) More info: show mount db> show page vm_cnt.v_free_count: 3095 vm_cnt.v_inactive_count: 80 vm_cnt.v_active_count: 287 vm_cnt.v_laundry_count: 57 vm_cnt.v_wire_count: 979462 vm_cnt.v_free_reserved: 1347 vm_cnt.v_free_min: 6266 vm_cnt.v_free_target: 21023 vm_cnt.v_inactive_target: 31534 db> show pageq pq_free 3095 dom 0 page_cnt 983828 free 3095 pq_act 287 pq_inact 80 pq_laund 57 pq_unsw 0 I encounter the same problem on another ROCKPRO64 under 13.0-RELEASE --=20 You are receiving this mail because: You are the assignee for the bug.=