From owner-freebsd-bugs@freebsd.org Sun Nov 24 06:54:02 2019 Return-Path: Delivered-To: freebsd-bugs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 824631CE376 for ; Sun, 24 Nov 2019 06:54:02 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mailman.nyi.freebsd.org (mailman.nyi.freebsd.org [IPv6:2610:1c1:1:606c::50:13]) by mx1.freebsd.org (Postfix) with ESMTP id 47LLVf2xfWz3CYr for ; Sun, 24 Nov 2019 06:54:02 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: by mailman.nyi.freebsd.org (Postfix) id 64EC31CE374; Sun, 24 Nov 2019 06:54:02 +0000 (UTC) Delivered-To: bugs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 63A2C1CE373 for ; Sun, 24 Nov 2019 06:54:02 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 47LLVf24ygz3CYq for ; Sun, 24 Nov 2019 06:54:02 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 2C976206D6 for ; Sun, 24 Nov 2019 06:54:02 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id xAO6s2p3036020 for ; Sun, 24 Nov 2019 06:54:02 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id xAO6s2ul036019 for bugs@FreeBSD.org; Sun, 24 Nov 2019 06:54:02 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: bugs@FreeBSD.org Subject: [Bug 241980] panic: I/O to pool appears to be hung on vdev Date: Sun, 24 Nov 2019 06:54:01 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 11.3-STABLE X-Bugzilla-Keywords: panic X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: eugen@freebsd.org X-Bugzilla-Status: Open X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: bugs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-bugs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Bug reports List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 24 Nov 2019 06:54:02 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D241980 --- Comment #21 from Eugene Grosbein --- It took me unexpectedly long time to get debugging output as I struggled to force the loader perform one-time (nextboot) loading for patched zfs.ko bei= ng able to perform only one reboot per day for this production machine. It appeared we have documented way to do this for the kernel but not for modul= es. After several attempts, zfs_name=3D"/boot/nextboot/zfs.ko" in the /boot/nextboot.conf did it finally. # TZ=3DUTC sysctl kern.boottime kern.boottime: { sec =3D 1574393119, usec =3D 301171 } Fri Nov 22 03:25:19 = 2019 Local time is UTC+3. After two days of uptime, I got this in the log: Nov 24 06:24:36 col02 kernel: sata SLOW IO: zio io_type 3 timestamp 171750435607373ns, delta 1006734932093ns, last io 172757121535759ns I/O to pool 'sata' appears to be hung on vdev guid 3313589389580178043 at '/dev/da2.eli' active zio 65 I've used this script to convert ZFS timestamps (getnanouptime expressed in nanoseconds) to readable time: #!/bin/sh date -jr $(sysctl -n kern.boottime | awk -vt=3D${1%?????????} -F'[, ]' '{pr= int t+$4}') #EOF First, there was no hardware hung. The system is alive, both ZFS pools still work, dd if=3D/dev/da2.eli works too. Second, ip_type 3 is ZIO_TYPE_FREE fr= om /sys/cddl/contrib/opensolaris/uts/common/sys/fs/zfs.h. # gethrtime 172757121535759 Sun Nov 24 06:24:36 MSK 2019 65 active zio's correspond to L(q) shown by gstat: # gstat -adI1s -f 'da[2-6].*' dT: 1.001s w: 1.000s filter: da[2-6].* L(q) ops/s r/s kBps ms/r w/s kBps ms/w d/s kBps ms/d= =20=20 %busy Name 65 774 37 3924 9.0 540 25707 0.7 193 8532 212.6= =20=20 84.5| da2 65 780 39 4144 14.3 544 26007 0.9 193 7724 241.7= =20=20 91.0| da3 65 771 33 3345 9.3 542 24448 0.7 192 8100 255.5= =20=20 90.7| da4 65 851 38 4004 7.8 554 28840 1.4 255 9679 236.0= =20=20 97.1| da5 65 766 36 4100 4.5 534 22730 0.5 192 8911 208.8= =20=20 75.6| da6 65 693 23 4144 14.8 474 26007 1.2 193 7724 243.0= =20=20 95.0| da3.eli 65 691 20 3345 9.4 476 24448 1.0 192 8100 255.8= =20=20 95.1| da4.eli 65 688 22 3924 9.8 470 25707 1.0 193 8532 213.0= =20=20 89.1| da2.eli 65 753 22 4004 11.6 473 28840 1.7 255 9679 238.3= =20 100.7| da5.eli 65 691 20 4100 5.6 476 22730 0.7 192 8911 210.6= =20=20 80.7| da6.eli --=20 You are receiving this mail because: You are the assignee for the bug.=