From owner-freebsd-fs@freebsd.org Thu Jun 8 22:57:45 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 3297BD85F11 for ; Thu, 8 Jun 2017 22:57:45 +0000 (UTC) (envelope-from delphij@gmail.com) Received: from mail-it0-x232.google.com (mail-it0-x232.google.com [IPv6:2607:f8b0:4001:c0b::232]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id EFBCD75393 for ; Thu, 8 Jun 2017 22:57:44 +0000 (UTC) (envelope-from delphij@gmail.com) Received: by mail-it0-x232.google.com with SMTP id m62so130603957itc.0 for ; Thu, 08 Jun 2017 15:57:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=oQ1MrVVzUcXs7QTJWSDHbZ1tV/tCSz2ut6mN8OMF3QY=; b=IMqJC7qgLHy9vkOOnsPJCJpn9/rMKqfr6yRX47V62PQxSEUIMn6Koqwt49ck4kXJPP Ad6wZzPvwhj4m5oP+oBnlCU5W2I4YQ9VuNNNeTi++BeqBcjKL9WOFLJ/WlXrtYRcRl5c VFzgNCAn6aeHW67AG//d16GgnbZNP23WhWRFiuUI6rbf60RXb6F1/UGiGgVBf8TBerTP DXaqIGdrNvVe/bn5cIIfK0h7OGSNFOvHvi464Ozap3I/fbJ8ecDnWpYEYaqE11LXU42B 81zGJeLTp4o9SDsFPInEcyZeqrCfw6Do67JlVrt1MkdWuJDOL3KOTJuhIQQDQbe06rq5 xK7w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=oQ1MrVVzUcXs7QTJWSDHbZ1tV/tCSz2ut6mN8OMF3QY=; b=ALcjUgQdbU0moQS9eugMDTjHyxBG0dCCadWeflFRQcTPTLJiwiQwrDzlzQNwMImvAS LYYmhodlgvmz0Do0CIULW55M0XpY7dOyMZ73rN8RqPH0BQQuhdKCK08O8cPMPdKQFh5i Hha3l5OXKvSpuQgt9YKKa4emPx9zJcAVAujGNX+TEC+h3ZU56/BhtYOXGDLZH6J2Mdsu bHfn9gsO+MPD1l7wvNdb/UQGBq90JIMnrH+aVAxSik6KyYSZNG1QZXkowN79vANQmNCw IoauKaby+VE++6oIJla04cDag6AhOYX8bv9nfMm8t6TNFlj8/vN4IfqoxBPUz9X5Jum+ XreA== X-Gm-Message-State: AODbwcAlFp4GMqgfgCTGi/hdxYA0/bPVzZ96mH1Ktm+C+KlbL9Wm50GV 3TxuA1OOtFtb0IJ7hrCRikqG0MyMWg== X-Received: by 10.36.65.18 with SMTP id x18mr8245106ita.88.1496962664346; Thu, 08 Jun 2017 15:57:44 -0700 (PDT) MIME-Version: 1.0 Received: by 10.79.4.134 with HTTP; Thu, 8 Jun 2017 15:57:43 -0700 (PDT) In-Reply-To: References: From: Xin LI Date: Thu, 8 Jun 2017 15:57:43 -0700 Message-ID: Subject: Re: ZFS Commands In "D" State To: Tim Gustafson Cc: freebsd-fs Content-Type: text/plain; charset="UTF-8" X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 08 Jun 2017 22:57:45 -0000 procstat -kk 1425 ? (or whatever PID's that is stuck in "D" state) On Thu, Jun 8, 2017 at 2:13 PM, Tim Gustafson wrote: > We have a ZFS server that we've been running for a few months now. > The server is a backup server that receives ZFS sends from its primary > daily. This mechanism has been working for us on several pairs of > servers for years in general, and for several months with this > particular piece of hardware. > > A few days ago, our nightly ZFS send failed. When I looked at the > server, I saw that the "zfs receive" command was in a "D" wait state: > > 1425 - D 0:02.75 /sbin/zfs receive -v -F backup/export > > I rebooted the system, checked that "zpool status" and "zfs list" both > came back correctly (which they did) and then re-started the "zfs > send" on the master server. At first, the "zfs receive" command did > not enter the "D" state, but once the master server started sending > actual data (which I was able to ascertain because I was doing "zfs > send" with the -v option), the receiving process entered the "D" state > again, and another reboot was required. Only about 2MB of data got > sent before this happened. > > I've rebooted several times, always with the same result. I did a > "zpool scrub os" (there's a separate zpool for the OS to live on) and > that completed in a few minutes, but when I did a "zpool scrub > backup", that process immediately went into the "D+" state: > > 895 0 D+ 0:00.04 zpool scrub backup > > We run smartd on this device, and that is showing no disk errors. The > devd process is logging some stuff, but it doesn't appear to be very > helpful: > > Jun 8 13:52:49 backup ZFS: vdev state changed, > pool_guid=2176924632732322522 vdev_guid=11754027336427262018 > Jun 8 13:52:49 backup ZFS: vdev state changed, > pool_guid=2176924632732322522 vdev_guid=11367786800631979308 > Jun 8 13:52:49 backup ZFS: vdev state changed, > pool_guid=2176924632732322522 vdev_guid=18407069648425063426 > Jun 8 13:52:49 backup ZFS: vdev state changed, > pool_guid=2176924632732322522 vdev_guid=9496839124651172990 > Jun 8 13:52:49 backup ZFS: vdev state changed, > pool_guid=2176924632732322522 vdev_guid=332784898986906736 > Jun 8 13:52:50 backup ZFS: vdev state changed, > pool_guid=2176924632732322522 vdev_guid=16384086680948393578 > Jun 8 13:52:50 backup ZFS: vdev state changed, > pool_guid=2176924632732322522 vdev_guid=10762348983543761591 > Jun 8 13:52:50 backup ZFS: vdev state changed, > pool_guid=2176924632732322522 vdev_guid=8585274278710252761 > Jun 8 13:52:50 backup ZFS: vdev state changed, > pool_guid=2176924632732322522 vdev_guid=17456777842286400332 > Jun 8 13:52:50 backup ZFS: vdev state changed, > pool_guid=2176924632732322522 vdev_guid=10533897485373019500 > > No word on which state it changed "from" or "to". Also, the system > only has three vdevs (the OS one, and then two raidz2 vdevs that make > up the "backup" pool, so I'm not sure how it's coming up with more > than 3 vdev GUIDs). > > What's my next step in diagnosing this? > > -- > > Tim Gustafson > BSOE Computing Director > tjg@ucsc.edu > 831-459-5354 > Baskin Engineering, Room 313A > > To request BSOE IT support, please visit https://support.soe.ucsc.edu/ > or send e-mail to help@soe.ucsc.edu. > _______________________________________________ > freebsd-fs@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org"