From owner-freebsd-current@freebsd.org Sat Sep 12 02:10:00 2020 Return-Path: Delivered-To: freebsd-current@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 474183EE93D for ; Sat, 12 Sep 2020 02:10:00 +0000 (UTC) (envelope-from grahamperrin@gmail.com) Received: from mail-wm1-x343.google.com (mail-wm1-x343.google.com [IPv6:2a00:1450:4864:20::343]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4BpGKg1hVSz4DHx; Sat, 12 Sep 2020 02:09:59 +0000 (UTC) (envelope-from grahamperrin@gmail.com) Received: by mail-wm1-x343.google.com with SMTP id q9so5936119wmj.2; Fri, 11 Sep 2020 19:09:58 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:from:to:cc:references:message-id:date :user-agent:mime-version:in-reply-to:content-transfer-encoding :content-language; bh=+NNs9oTFFGUzEUopV2hv/o3TUDrnVVRqDNHwgEnIuow=; b=miEVIr1DPMNQw8RIqGPiGI3N/wpB0OSEZAZKteN2H67uo9ZD8JWqT1h1ajzc+5FKYh ZWXh1Io6CKAFfhTXYRT/ATEHoQvyI282d+3c8/loXlRmBC8YsHxOITPkJ/5KVZXEF2rn f8xaD2u0hBcc4gOPSXCKL+ucNwqJnp9yqOT/6EVqaA4kLOvf5mpoO2ijT2MOTF7phfnK kjfddqDgn7eCJQ7Ha64vCOtZdzDDBR4oUwF09CFArUUjfaywzfubGSINApu64nImtgVf Bu9WaGwqKL7ehiYV99HKEtXhiNp5Hx/VJ41OAcGsftNbBYUTortO/Xd3sTz0rCzD0ZDm eplw== X-Gm-Message-State: AOAM531Pwd6i001ZpNITRjuMTIT/yC8yELdkv/ZjVnXYC5mKkKmYmDZv jkCZNXOx4gxggGapMtzXlhTx3RLxr2PHlvwl X-Google-Smtp-Source: ABdhPJxzNkobGQP4dHAcHn3fHcGUTom5LWlKCOFWfyCtMESu8POEJ//zOCJwUfL5zKEXmuZsRFgzvA== X-Received: by 2002:a7b:ce96:: with SMTP id q22mr4519495wmj.132.1599876596517; Fri, 11 Sep 2020 19:09:56 -0700 (PDT) Received: from [192.168.1.12] (79-66-147-78.dynamic.dsl.as9105.com. [79.66.147.78]) by smtp.gmail.com with ESMTPSA id b2sm7091706wmh.47.2020.09.11.19.09.55 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Fri, 11 Sep 2020 19:09:55 -0700 (PDT) Subject: Re: suspend/resume versus OpenZFS on USB From: Graham Perrin To: FreeBSD Current , Matthew Macy , Ryan Moeller Cc: Hans Petter Selasky References: <6824e9f5-0496-a9ba-9cf2-01a6c8b34a4e@selasky.org> <24c4f60d-d937-7763-9cea-697de75109a6@gmail.com> <7cd8c682-0c4b-c332-c6de-d17dccbd9672@selasky.org> Message-ID: <7ced4903-cf1f-19d1-f2b3-61336992ed11@gmail.com> Date: Sat, 12 Sep 2020 03:09:54 +0100 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:68.0) Gecko/20100101 Thunderbird/68.12.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit Content-Language: en-US X-Rspamd-Queue-Id: 4BpGKg1hVSz4DHx X-Spamd-Bar: --- X-Spamd-Result: default: False [-3.39 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; FREEMAIL_FROM(0.00)[gmail.com]; R_SPF_ALLOW(-0.20)[+ip6:2a00:1450:4000::/36]; RCVD_COUNT_THREE(0.00)[3]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; NEURAL_HAM_SHORT(-0.44)[-0.443]; RECEIVED_SPAMHAUS_PBL(0.00)[79.66.147.78:received]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[gmail.com]; MID_RHS_MATCH_FROM(0.00)[]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-0.96)[-0.956]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[4]; NEURAL_HAM_LONG(-0.99)[-0.987]; MIME_GOOD(-0.10)[text/plain]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[2a00:1450:4864:20::343:from]; RCVD_TLS_ALL(0.00)[]; MAILMAN_DEST(0.00)[freebsd-current] X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 12 Sep 2020 02:10:00 -0000 I'm not sure what was going on with device numbers in my previous report, but this much seems to be easily and consistently reproducible. If the pool is ONLINE and imported at suspend time – and if not actively writing to the device at suspend time – then at resume time: * the pool is SUSPENDED * the device is shown as REMOVED * zpool clear followed by zpool scrub lead to disappearance of permanent metadata errors, without physically touching the device. ---- root@momh167-gjp4-8570p:~ # ls /dev/da* /dev/da0        /dev/da0p1      /dev/da1        /dev/da1p1 root@momh167-gjp4-8570p:~ # zpool status usb   pool: usb  state: ONLINE   scan: scrub repaired 0B in 00:00:08 with 0 errors on Sat Sep 12 02:55:26 2020 config:         NAME        STATE     READ WRITE CKSUM         usb         ONLINE       0     0     0           da0p1     ONLINE       0     0     0 errors: No known data errors root@momh167-gjp4-8570p:~ # date ; uname -v ; zfs version ; acpiconf -s3 Sat Sep 12 02:59:20 BST 2020 FreeBSD 13.0-CURRENT #64 r365364: Sun Sep  6 01:38:18 BST 2020 root@momh167-gjp4-8570p:/usr/obj/usr/src/amd64.amd64/sys/GENERIC-NODEBUG zfs-0.8.0-1 zfs-kmod-0.8.0-1 root@momh167-gjp4-8570p:~ # zpool status -v   pool: copperbowl  state: ONLINE status: Some supported features are not enabled on the pool. The pool can         still be used, but some features are unavailable. action: Enable all features using 'zpool upgrade'. Once this is done,         the pool may no longer be accessible by software that does not support         the features. See zpool-features(5) for details.   scan: scrub repaired 0B in 01:39:31 with 0 errors on Thu Sep  3 01:12:21 2020 config:         NAME          STATE     READ WRITE CKSUM         copperbowl    ONLINE       0     0     0           ada0p4.eli  ONLINE       0     0     0 errors: No known data errors   pool: usb  state: SUSPENDED status: One or more devices are faulted in response to IO failures. action: Make sure the affected devices are connected, then run 'zpool clear'.    see: https://zfsonlinux.org/msg/ZFS-8000-HC   scan: scrub repaired 0B in 00:00:08 with 0 errors on Sat Sep 12 02:55:26 2020 config:         NAME        STATE     READ WRITE CKSUM         usb         UNAVAIL      0     0     0  insufficient replicas           da0p1     REMOVED      0     0     0 errors: List of errors unavailable: pool I/O is currently suspended root@momh167-gjp4-8570p:~ # zpool clear usb && zpool scrub usb root@momh167-gjp4-8570p:~ # date ; zpool status usb Sat Sep 12 03:00:19 BST 2020   pool: usb  state: ONLINE   scan: scrub repaired 0B in 00:00:01 with 0 errors on Sat Sep 12 03:00:11 2020 config:         NAME        STATE     READ WRITE CKSUM         usb         ONLINE       0     0     0           da0p1     ONLINE       0     0     0 errors: No known data errors root@momh167-gjp4-8570p:~ #