From owner-freebsd-fs@freebsd.org Tue Oct 3 07:25:35 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 949ADE33582 for ; Tue, 3 Oct 2017 07:25:35 +0000 (UTC) (envelope-from steven@multiplay.co.uk) Received: from mail-wr0-x22d.google.com (mail-wr0-x22d.google.com [IPv6:2a00:1450:400c:c0c::22d]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 2D265683AA for ; Tue, 3 Oct 2017 07:25:34 +0000 (UTC) (envelope-from steven@multiplay.co.uk) Received: by mail-wr0-x22d.google.com with SMTP id t76so5418607wrc.3 for ; Tue, 03 Oct 2017 00:25:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=multiplay-co-uk.20150623.gappssmtp.com; s=20150623; h=from:subject:to:cc:references:message-id:date:user-agent :mime-version:in-reply-to:content-language; bh=vzQ7cqW5WJnSjZqzK8x0JWjMZpLEFsb2v9BmtqZxRsw=; b=laW3PEn68R74ibCuUpOptWME7wCjoeHWjgAw4/wmkfaoUxXJvuBgy7vrTNbQ2DwoBo lYseNsr2lHTYjjvgCP0TtPafYaOTFOJKk85kfLZKRIkPCmEpzdqI/u31/IRaobxNK3sV sXcnpM6p9be5H04ems6TacKtA4hj5dSOMH2bEudNZKCGUkGCX+FxEVqGsre5RfPjjZbQ ScDjACkUn4J+qr9I3kcSkedRS+8bLKmafLHDi5QRyEAs+AKJoWVeS6dRTbSN0i3sv2Ty DlOB/nfnYUIzx8FMpTN+9mL+3WB9qyZFeNvvG6bdEf9NpaFTx2qXP8xDOY+Xh5bPtnp/ w2qw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:subject:to:cc:references:message-id:date :user-agent:mime-version:in-reply-to:content-language; bh=vzQ7cqW5WJnSjZqzK8x0JWjMZpLEFsb2v9BmtqZxRsw=; b=MGlzNJI0hd0q+48qddiez2RDeDQaJo5pfloDeX8f4ybsxRSlmqN5MhKxMvtZLM+Afa 3Znqz/8AfCQU50Vv5HCaZnUPc5VT5dCxlaPKQ68NkVAaoravzLnQ+Dji7F4sfLMRZwmh Vl3OJdF6H+5L4UlF2hiCKfR/nF3OROFYYKNWXT2iotSBUDy0OzmXvn2Uz6vGjbj6DQ1r jbNrq65soUxyYjiqrFv87EVgFfY6cpCGhDvKMbxjmLEwjAnu3ndqCG/5DJ2VHb6sk61u eUwwonRa/20fqJdmXgVXHfsRnY4IL5z2/3kFtrO35GlraF0huYLCv9GfUfBxXFv8SpFT zPpw== X-Gm-Message-State: AMCzsaV7/Oxo4BC/o9/JNGvz9heLYwnvvjs4ae2ej2eiTPDZSLxdv6JX ppWeKXt1UMf+IV6exi/s4OJTajtwziA= X-Google-Smtp-Source: AOwi7QD0RWuQH4aeAS9YDpUoE0W5E1iZnprxYy7Ofc05LPRkGylehReBH22AxuQLhYYidihQOYhvLg== X-Received: by 10.223.187.201 with SMTP id z9mr10835472wrg.195.1507015533111; Tue, 03 Oct 2017 00:25:33 -0700 (PDT) Received: from [10.10.1.111] ([185.97.61.1]) by smtp.gmail.com with ESMTPSA id 69sm13766172wmm.22.2017.10.03.00.25.31 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 03 Oct 2017 00:25:32 -0700 (PDT) From: Steven Hartland X-Google-Original-From: Steven Hartland Subject: Re: ZFS stalled after some mirror disks were lost To: Andriy Gapon , Ben RUBSON Cc: Freebsd fs References: <4A0E9EB8-57EA-4E76-9D7E-3E344B2037D2@gmail.com> Message-ID: <69fbca90-9a18-ad5d-a2f7-ad527d79f8ba@freebsd.org> Date: Tue, 3 Oct 2017 08:25:34 +0100 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:52.0) Gecko/20100101 Thunderbird/52.3.0 MIME-Version: 1.0 In-Reply-To: Content-Language: en-US Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit X-Content-Filtered-By: Mailman/MimeDel 2.1.23 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 03 Oct 2017 07:25:35 -0000 On 03/10/2017 07:12, Andriy Gapon wrote: > On 02/10/2017 21:12, Ben RUBSON wrote: >> A sustained read throughput of 180 MB/s, 45 MB/s on each iscsi disk >> according to "zpool iostat", nothing on local disks (strange but I >> noticed that IOs always prefer iscsi disks to local disks). > Are your local disks SSD or HDD? > Could it be that iSCSI disks appear to be faster than the local disks to the > smart ZFS mirror code? > > Steve, what do you think? Yes that quite possible, the mirror balancing uses the queue depth + rotating bias to determine the load of the disk so if your iSCSI host is processing well and / or is reporting non-rotating vs rotating for the local disks it could well be the mirror is preferring reads from the the less loaded iSCSI devices.     Regards     Steve