From owner-freebsd-bugs@freebsd.org Sun Nov 24 07:44:16 2019 Return-Path: Delivered-To: freebsd-bugs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 3D63F1CF4DE for ; Sun, 24 Nov 2019 07:44:16 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mailman.nyi.freebsd.org (mailman.nyi.freebsd.org [IPv6:2610:1c1:1:606c::50:13]) by mx1.freebsd.org (Postfix) with ESMTP id 47LMcc0xCpz3FpN for ; Sun, 24 Nov 2019 07:44:16 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: by mailman.nyi.freebsd.org (Postfix) id 1FFE01CF4DD; Sun, 24 Nov 2019 07:44:16 +0000 (UTC) Delivered-To: bugs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 1FBE01CF4DC for ; Sun, 24 Nov 2019 07:44:16 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 47LMcb75VYz3FpM for ; Sun, 24 Nov 2019 07:44:15 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id BDB9820FE0 for ; Sun, 24 Nov 2019 07:44:15 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id xAO7iFj1091363 for ; Sun, 24 Nov 2019 07:44:15 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id xAO7iFNv091362 for bugs@FreeBSD.org; Sun, 24 Nov 2019 07:44:15 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: bugs@FreeBSD.org Subject: [Bug 241980] panic: I/O to pool appears to be hung on vdev Date: Sun, 24 Nov 2019 07:44:16 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 11.3-STABLE X-Bugzilla-Keywords: panic X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: eugen@freebsd.org X-Bugzilla-Status: Open X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: bugs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-bugs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Bug reports List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 24 Nov 2019 07:44:16 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D241980 --- Comment #23 from Eugene Grosbein --- As I've suspected, ZFS issues large number of BIO_DELETE/ZIO_TYPE_FREE requ= ests due to massive removal of hourly/daily snapshots for this SSD-only pool. Clearing of SSD cells may be slow. The system has daily cron job started at early morning at 6:05 local time. = It creates many new daily snapshots for ZVOLs and removes old snapshots. First debugging warning I've got at 6:24:36 when ZIO_TYPE_FREE spent 1006/8=3D125= .75 seconds in the avl_tree for active zio's. Then there were more such events (total 50) with increasing "delta" upto 12= 30 seconds at 06:28:26 after which messages stopped. They were not for ZIO_TYPE_FREE(3) zio's only, but for ZIO_TYPE_WRITE(2) too. And for all components of the pool plust for distinct ZIL device /dev/raid/r0s3 that is part of GEOM_RAID's mirror volume based on another pair of SSDs da[01]. I'd like to show all messages appeared in the log for very first event at 06:24:36. sata SLOW IO: zio io_type 3 timestamp 171750435607373ns, delta 100673493209= 3ns, last io 172757121535759ns I/O to pool 'sata' appears to be hung on vdev guid 3313589389580178043 at '/dev/da2.eli' active zio 65 SLOW IO: sata is child of sata sata SLOW IO: zio io_type 2 timestamp 171757174663043ns, delta 999995876423= ns, last io 172756998535496nsI/O to pool 'sata' appears to be hung on vdev guid 17157972224957886076 at '/dev/da3.eli' active zio 65 SLOW IO: sata is child of sata sata SLOW IO: zio io_type 2 timestamp 171757172636758ns, delta 999997902708= ns, last io 172757029559490ns I/O to pool 'sata' appears to be hung on vdev guid 11111943829132819397 at '/dev/da4.eli' active zio 65 SLOW IO: sata is child of sata sata SLOW IO: zio io_type 2 timestamp 171757172636758ns, delta 999997902708= ns, last io 172756997538959ns I/O to pool 'sata' appears to be hung on vdev guid 13912528878380060504 at '/dev/da5.eli' active zio 65 SLOW IO: sata is child of sata sata SLOW IO: zio io_type 2 timestamp 171757172636758ns, delta 999997902708= ns, last io 172756973537187ns I/O to pool 'sata' appears to be hung on vdev guid 18428510808262122301 at '/dev/da6.eli' active zio 65 SLOW IO: sata is child of sata SLOW IO: sata is child of sata sata SLOW IO: zio io_type 2 timestamp 171985896582124ns, delta 771273957342= ns, last io 172756042541811ns I/O to pool 'sata' appears to be hung on vdev guid 15059311870935027496 at '/dev/raid/r0s3' active zio 2 SLOW IO: sata is child of sata --=20 You are receiving this mail because: You are the assignee for the bug.=