From owner-freebsd-fs@freebsd.org Tue Oct 3 14:40:21 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id B07C0E3ED48; Tue, 3 Oct 2017 14:40:21 +0000 (UTC) (envelope-from ben.rubson@gmail.com) Received: from mail-wr0-x22c.google.com (mail-wr0-x22c.google.com [IPv6:2a00:1450:400c:c0c::22c]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 493DD77004; Tue, 3 Oct 2017 14:40:21 +0000 (UTC) (envelope-from ben.rubson@gmail.com) Received: by mail-wr0-x22c.google.com with SMTP id u5so6323484wrc.5; Tue, 03 Oct 2017 07:40:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:subject:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to; bh=453TtCA1ft5VW+g9Wy9s0Ho4p5EcUtomwJSbflpk6qY=; b=Um2NeeZyImQIjVmbPcMmkvcEwdmShbpBahrTRuULwQrCVmyDXw9FwQRzUOppwpWGdA Y9R4Or9u62aLxdB9IYAFb5mC8JNXo3F6EN5LnA4lQW7rM3qweIKxdOcJhyrD9AwSV+yi RCeZVZMCHFLSrVPWE34tP3b0pBUjn4LCmwPZ/ypmCY9zY7nT0Z4Y/1NOHAG4pu552o6M PnzfEYX8t/r9lf8dDl2ogfrf3n6Ngu54jtBNyIxftjSM2pKUHejPNVrUEp4mo/JfvnZL eqf3gDP2q8gEw3Frk19E4phnhpN552wRITTxHLAU91H81Sfs0gFYLLwNxN+GbXAn/J7I k6RQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:subject:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to; bh=453TtCA1ft5VW+g9Wy9s0Ho4p5EcUtomwJSbflpk6qY=; b=l4QYKyKiw6SiRwveaWFb66uhBn5sw9DtUEVueYDAEBZIMQznrocdraN3fVpqzHRmsH KEEoY/STt3P2TqJ+OjU7M3pBaoocJ5oLFq9C1IFp9gaUztn1tH3SvzyrJav/ciYRGb3c dwWRpOql9M3nJPGSgaC7AJYk8TWUAhqO8U6nhB2gC4trXfrfybH/+3K2cQrqhE8Cs6YX 2IzY/vrXPnGysjwc6PJ0lY0bHWtqQUitl48+OKlX3snL9Mp3DDwtICnuYIoY03KbxU/f 7/CQWE4QNRb3rFOTgiKDMl3Y5/Zd6B4pfNmNMSsCcaePEFK5V8gvRuh7WoVQ8fOWRDfg BTYw== X-Gm-Message-State: AMCzsaVk8zt5rRlCBX0o7uHZHsVTmje/rwPdC9yUCw7PtF2syjfZZdGP WKw3Ae3t/oAVIwlJBX6Lahc6OKUr X-Google-Smtp-Source: AOwi7QAks5RkA2w/emplIiLA0la1EH3lSiJrqnKQ4Tz6cJPJ2t0zpkwg2BaSYYcVoCMvOHXdnCeuUw== X-Received: by 10.223.151.210 with SMTP id t18mr245916wrb.261.1507041619416; Tue, 03 Oct 2017 07:40:19 -0700 (PDT) Received: from bens-mac.home (LFbn-MAR-1-445-220.w2-15.abo.wanadoo.fr. [2.15.38.220]) by smtp.gmail.com with ESMTPSA id v8sm29638wrg.80.2017.10.03.07.40.18 (version=TLS1 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Tue, 03 Oct 2017 07:40:18 -0700 (PDT) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 9.3 \(3124\)) Subject: ZFS prefers iSCSI disks over local ones ? From: Ben RUBSON In-Reply-To: <69fbca90-9a18-ad5d-a2f7-ad527d79f8ba@freebsd.org> Date: Tue, 3 Oct 2017 16:40:17 +0200 Cc: Andriy Gapon Content-Transfer-Encoding: quoted-printable Message-Id: <9342D2A7-CE29-445B-9C40-7B6A9C960D59@gmail.com> References: <4A0E9EB8-57EA-4E76-9D7E-3E344B2037D2@gmail.com> <69fbca90-9a18-ad5d-a2f7-ad527d79f8ba@freebsd.org> To: Freebsd fs , FreeBSD-scsi , Steven Hartland X-Mailer: Apple Mail (2.3124) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 03 Oct 2017 14:40:21 -0000 Hi, I start a new thread to avoid confusion in the main one. (ZFS stalled after some mirror disks were lost) > On 03 Oct 2017, at 09:39, Steven Hartland wrote: >=20 >> On 03/10/2017 08:31, Ben RUBSON wrote: >>=20 >>> On 03 Oct 2017, at 09:25, Steven Hartland wrote: >>>=20 >>>> On 03/10/2017 07:12, Andriy Gapon wrote: >>>>=20 >>>>> On 02/10/2017 21:12, Ben RUBSON wrote: >>>>>=20 >>>>> Hi, >>>>>=20 >>>>> On a FreeBSD 11 server, the following online/healthy zpool : >>>>>=20 >>>>> home >>>>> mirror-0 >>>>> label/local1 >>>>> label/local2 >>>>> label/iscsi1 >>>>> label/iscsi2 >>>>> mirror-1 >>>>> label/local3 >>>>> label/local4 >>>>> label/iscsi3 >>>>> label/iscsi4 >>>>> cache >>>>> label/local5 >>>>> label/local6 >>>>>=20 >>>>> A sustained read throughput of 180 MB/s, 45 MB/s on each iscsi = disk >>>>> according to "zpool iostat", nothing on local disks (strange but I >>>>> noticed that IOs always prefer iscsi disks to local disks). >>>>=20 >>>> Are your local disks SSD or HDD? >>>> Could it be that iSCSI disks appear to be faster than the local = disks >>>> to the smart ZFS mirror code? >>>>=20 >>>> Steve, what do you think? >>>=20 >>> Yes that quite possible, the mirror balancing uses the queue depth + >>> rotating bias to determine the load of the disk so if your iSCSI = host >>> is processing well and / or is reporting non-rotating vs rotating = for >>> the local disks it could well be the mirror is preferring reads from >>> the the less loaded iSCSI devices. >>=20 >> Note that local & iscsi disks are _exactly_ the same HDD (same model = number, >> same SAS adapter...). So iSCSI ones should be a little bit slower due = to >> network latency (even if it's very low in my case). >=20 > The output from gstat -dp on a loaded machine would be interesting to = see too. So here is the gstat -dp : L(q) ops/s r/s kBps ms/r w/s kBps ms/w d/s kBps ms/d %busy Name 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0| da0 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0| da1 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0| da2 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0| da3 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0| da4 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0| da5 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0| da6 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0| da7 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0| da8 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0| da9 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0| da10 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0| da11 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0| da12 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0| da13 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0| da14 1 370 370 47326 0.7 0 0 0.0 0 0 0.0 23.2| da15 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0| da16 0 357 357 45698 1.4 0 0 0.0 0 0 0.0 39.3| da17 0 348 348 44572 0.7 0 0 0.0 0 0 0.0 22.5| da18 0 432 432 55339 0.7 0 0 0.0 0 0 0.0 27.5| da19 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0| da20 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0| da21 The 4 active drives are the iSCSI targets of the above quoted pool. A local disk : Geom name: da7 Providers: 1. Name: da7 Mediasize: 4000787030016 (3.6T) Sectorsize: 512 Mode: r0w0e0 descr: HGSTxxx lunid: 5000xxx ident: NHGDxxx rotationrate: 7200 fwsectors: 63 fwheads: 255 A iSCSI disk : Geom name: da19 Providers: 1. Name: da19 Mediasize: 3999688294912 (3.6T) Sectorsize: 512 Mode: r1w1e2 descr: FREEBSD CTLDISK lunname: FREEBSD MYDEVID 12 lunid: FREEBSD MYDEVID 12 ident: iscsi4 rotationrate: 0 fwsectors: 63 fwheads: 255 Sounds like then the faulty thing is the rotationrate set to 0 ? Thx, Ben