From owner-freebsd-fs@freebsd.org  Mon Oct  2 19:30:07 2017
Return-Path: <owner-freebsd-fs@freebsd.org>
Delivered-To: freebsd-fs@mailman.ysv.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org
 [IPv6:2001:1900:2254:206a::19:1])
 by mailman.ysv.freebsd.org (Postfix) with ESMTP id 0DCFFE25B5D
 for <freebsd-fs@mailman.ysv.freebsd.org>; Mon,  2 Oct 2017 19:30:07 +0000 (UTC)
 (envelope-from SRS0=HbBg=BB=quip.cz=000.fbsd@elsa.codelab.cz)
Received: from elsa.codelab.cz (elsa.codelab.cz [94.124.105.4])
 (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
 (Client did not present a certificate)
 by mx1.freebsd.org (Postfix) with ESMTPS id C4FC574CC5
 for <freebsd-fs@freebsd.org>; Mon,  2 Oct 2017 19:30:06 +0000 (UTC)
 (envelope-from SRS0=HbBg=BB=quip.cz=000.fbsd@elsa.codelab.cz)
Received: from elsa.codelab.cz (localhost [127.0.0.1])
 by elsa.codelab.cz (Postfix) with ESMTP id D2AFA28417;
 Mon,  2 Oct 2017 21:29:57 +0200 (CEST)
Received: from illbsd.quip.test (ip-86-49-16-209.net.upcbroadband.cz
 [86.49.16.209])
 (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits))
 (No client certificate requested)
 by elsa.codelab.cz (Postfix) with ESMTPSA id 2363428411;
 Mon,  2 Oct 2017 21:29:57 +0200 (CEST)
Subject: Re: ZFS stalled after some mirror disks were lost
To: Adam Vande More <amvandemore@gmail.com>, Ben RUBSON <ben.rubson@gmail.com>
Cc: Freebsd fs <freebsd-fs@freebsd.org>
References: <4A0E9EB8-57EA-4E76-9D7E-3E344B2037D2@gmail.com>
 <CA+tpaK2=du8Ghrz9N4sfBk2T4tAC9NEdwOoHV3CQz5DOKcMogw@mail.gmail.com>
From: Miroslav Lachman <000.fbsd@quip.cz>
Message-ID: <59D293B4.2020702@quip.cz>
Date: Mon, 2 Oct 2017 21:29:56 +0200
User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:42.0) Gecko/20100101
 Firefox/42.0 SeaMonkey/2.39
MIME-Version: 1.0
In-Reply-To: <CA+tpaK2=du8Ghrz9N4sfBk2T4tAC9NEdwOoHV3CQz5DOKcMogw@mail.gmail.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-BeenThere: freebsd-fs@freebsd.org
X-Mailman-Version: 2.1.23
Precedence: list
List-Id: Filesystems <freebsd-fs.freebsd.org>
List-Unsubscribe: <https://lists.freebsd.org/mailman/options/freebsd-fs>,
 <mailto:freebsd-fs-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-fs/>
List-Post: <mailto:freebsd-fs@freebsd.org>
List-Help: <mailto:freebsd-fs-request@freebsd.org?subject=help>
List-Subscribe: <https://lists.freebsd.org/mailman/listinfo/freebsd-fs>,
 <mailto:freebsd-fs-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Mon, 02 Oct 2017 19:30:07 -0000

Adam Vande More wrote on 10/02/2017 20:44:
> On Mon, Oct 2, 2017 at 1:12 PM, Ben RUBSON <ben.rubson@gmail.com> wrote:
>
>> Hi,
>>
>> On a FreeBSD 11 server, the following online/healthy zpool :
>>
>> home
>>    mirror-0
>>      label/local1
>>      label/local2
>>      label/iscsi1
>>      label/iscsi2
>>    mirror-1
>>      label/local3
>>      label/local4
>>      label/iscsi3
>>      label/iscsi4
>> cache
>>    label/local5
>>    label/local6
>>
>> A sustained read throughput of 180 MB/s, 45 MB/s on each iscsi disk
>> according to "zpool iostat", nothing on local disks (strange but I
>> noticed that IOs always prefer iscsi disks to local disks).
>> No write IOs.
>>
>> Let's disconnect all iSCSI disks :
>> iscsictl -Ra
>>
>> Expected behavior :
>> IO activity flawlessly continue on local disks.
>>
>
> Perhaps I'm misunderstanding your setup, but my expected behavior would be
> exactly what you see.  I think you'd need something more along the lines of:
>
> home
>    mirror
>      label/local1
>      label/iscsi1
>    mirror
>      label/local2
>      label/iscsi2
> etc...

The OP has four way mirror. It is supposed to work even if 3 devices are 
missing. Just 1 device should be enough.

Miroslav Lachman