From owner-freebsd-fs@freebsd.org Tue Sep 6 20:01:42 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 9EC8DBC7398 for ; Tue, 6 Sep 2016 20:01:42 +0000 (UTC) (envelope-from rm@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:1900:2254:206c::16:87]) by mx1.freebsd.org (Postfix) with ESMTP id 95472754 for ; Tue, 6 Sep 2016 20:01:42 +0000 (UTC) (envelope-from rm@FreeBSD.org) Received: from [127.0.0.1] (localhost [IPv6:::1]) by freefall.freebsd.org (Postfix) with ESMTP id 1F601195A for ; Tue, 6 Sep 2016 20:01:41 +0000 (UTC) (envelope-from rm@FreeBSD.org) To: freebsd-fs@freebsd.org From: Ruslan Makhmatkhanov Subject: ZFS-8000-8A: assistance needed Message-ID: Date: Tue, 6 Sep 2016 23:00:22 +0300 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:45.0) Gecko/20100101 Thunderbird/45.2.0 MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 06 Sep 2016 20:01:42 -0000 Hello, I've got something new here and just not sure where to start on solving that. It's on 10.2-RELEASE-p7 amd64. """ root:~ # zpool status -xv pool: storage_ssd state: ONLINE status: One or more devices has experienced an error resulting in data corruption. Applications may be affected. action: Restore the file in question if possible. Otherwise restore the entire pool from backup. see: http://illumos.org/msg/ZFS-8000-8A scan: scrub repaired 0 in 0h26m with 5 errors on Tue Aug 23 00:40:24 2016 config: NAME STATE READ WRITE CKSUM storage_ssd ONLINE 0 0 59.3K mirror-0 ONLINE 0 0 0 gpt/drive-06 ONLINE 0 0 0 gpt/drive-07 ONLINE 0 0 9 mirror-1 ONLINE 0 0 119K gpt/drive-08 ONLINE 0 0 119K gpt/drive-09 ONLINE 0 0 119K cache mfid5 ONLINE 0 0 0 mfid6 ONLINE 0 0 0 errors: Permanent errors have been detected in the following files: <0x1bd0a>:<0x8> <0x31f23>:<0x8> /storage_ssd/f262f6ebaf5011e39ca7047d7bb28f4a/disk /storage_ssd/7ba3f661fa9811e3bd9d047d7bb28f4a/disk /storage_ssd/2751d305ecba11e3aef0047d7bb28f4a/disk /storage_ssd/6aa805bd22e911e4b470047d7bb28f4a/disk """ The pool looks ok, if I understand correctly, but we have a slowdown in Xen VM's, that are using these disks via iSCSI. So can please anybody explain what exactly that mean? 1. Am I right that we have a hardware failure that lead to data corruption? If so, how to identify failed disk(s) and how it is possible that data is corrupted on zfs mirror? Is there anything I can do to recover except restoring from backup? 2. What first and second damaged "files" are and why they are shown like that? I have this in /var/log/messages, but to me it looks like iSCSI message, that's spring up when accessing damaged files: """ kernel: (1:32:0/28): WRITE command returned errno 122 """ Manual zpool scrub was tried on this pool to not avail. The pool capacity is only 66% full. Thanks for any hints in advance. -- Regards, Ruslan T.O.S. Of Reality