From owner-freebsd-fs@freebsd.org Tue Oct 3 15:40:25 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 183B8E40A14; Tue, 3 Oct 2017 15:40:25 +0000 (UTC) (envelope-from ben.rubson@gmail.com) Received: from mail-wr0-x233.google.com (mail-wr0-x233.google.com [IPv6:2a00:1450:400c:c0c::233]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id B396A7E6C6; Tue, 3 Oct 2017 15:40:24 +0000 (UTC) (envelope-from ben.rubson@gmail.com) Received: by mail-wr0-x233.google.com with SMTP id 54so6649055wrz.10; Tue, 03 Oct 2017 08:40:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:subject:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to; bh=FSE6usK7Y8BeHJ1vB2YCw29sv4py6eOBtrt4Z1muMPs=; b=tjQLJ7HEyqC3Ne8lBwZ2X/zmKkev5ktDsQ40Kwl3ZNfeTjxQi2SU7NJfDa+klfKiv0 OIwDz24K0+ETcObhfbvUezspResLUnEhCxUQ0Nl8c0Y5lpzqEofkw54jxb1CYEQqP5K1 uREb58aCoKkdWuWh0IgigAU3TR0zecjqMOHU5YEpWpEBvBYxngHWXHqWd+r3sxhvJuVB XsTbajoojVSg6hjj9SGxIQTOCK74NSkTq4xZ48HDIzK07NzCokTP6rwOmKLPqjSaMnv9 QUyyjELdG+jOeB8qnPwuRLqhlQlu6W58moiNMnSozx3KIA8BU93JnXyqYggAXzXMVmJp Ujpg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:subject:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to; bh=FSE6usK7Y8BeHJ1vB2YCw29sv4py6eOBtrt4Z1muMPs=; b=dKM7qifoWMImcXBSTTRxFdnXjCiIXUJQe2sZq5cFrZ+9ifB1JnfRL3D8BAW/zbMpNb vlhUVERHtSJ4dYUFb8ebPH5G8C4nJYSqu9+uDxSTY4fHlHG56CN0Hw92eoHI63ua9LuN 5rfzeK31dJPJbx0Zp/JdrbiC2FgVrCrAR65VhxPYPpJ48jFFdDV5RNfxIqi7ZS4qtWz2 0Hkm9vyMySbvWpwBmEbn5G2z8JZPtvsMpvWxn3sfcRhCCZMOHdHcTSoSp9m3NzKsJK3v PeiyiG6ZXj3Abk/fiFBZs/LbCKRRo+gMWAfP45jvO2RDCTlcY3OrusOMzLdVEPL2qFtI Og4g== X-Gm-Message-State: AMCzsaW0653F+xyassHVQ2C+pnrmd/DWcs/p3eOtx4QOgtEBueNrBkKV eF5VmHXoBLyvtP7FzxacaaaxY4gnh/g= X-Google-Smtp-Source: AOwi7QBu1cG2mB/ax3xgwxZYYa3fd6Cy6w9FPD5NqP+PSqnqx0tuHYrM4BOnL49KSqFFr1UujujjWQ== X-Received: by 10.223.171.73 with SMTP id r9mr3244715wrc.118.1507045222910; Tue, 03 Oct 2017 08:40:22 -0700 (PDT) Received: from bens-mac.home (LFbn-MAR-1-445-220.w2-15.abo.wanadoo.fr. [2.15.38.220]) by smtp.gmail.com with ESMTPSA id d18sm7277435wra.89.2017.10.03.08.40.22 (version=TLS1 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Tue, 03 Oct 2017 08:40:22 -0700 (PDT) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 9.3 \(3124\)) Subject: Re: ZFS prefers iSCSI disks over local ones ? From: Ben RUBSON In-Reply-To: <20171003151850.GA65538@in-addr.com> Date: Tue, 3 Oct 2017 17:40:21 +0200 Cc: Steven Hartland , FreeBSD-scsi , Freebsd fs , Andriy Gapon Content-Transfer-Encoding: quoted-printable Message-Id: References: <4A0E9EB8-57EA-4E76-9D7E-3E344B2037D2@gmail.com> <69fbca90-9a18-ad5d-a2f7-ad527d79f8ba@freebsd.org> <9342D2A7-CE29-445B-9C40-7B6A9C960D59@gmail.com> <20171003151850.GA65538@in-addr.com> To: Gary Palmer X-Mailer: Apple Mail (2.3124) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 03 Oct 2017 15:40:25 -0000 > On 03 Oct 2017, at 17:18, Gary Palmer wrote: >=20 > On Tue, Oct 03, 2017 at 05:03:18PM +0200, Ben RUBSON wrote: >>> On 03 Oct 2017, at 16:58, Steven Hartland = wrote: >>>=20 >>> On 03/10/2017 15:40, Ben RUBSON wrote: >>>> Hi, >>>>=20 >>>> I start a new thread to avoid confusion in the main one. >>>> (ZFS stalled after some mirror disks were lost) >>>>=20 >>>>=20 >>>>> On 03 Oct 2017, at 09:39, Steven Hartland wrote: >>>>>=20 >>>>>=20 >>>>>> On 03/10/2017 08:31, Ben RUBSON wrote: >>>>>>=20 >>>>>>=20 >>>>>>> On 03 Oct 2017, at 09:25, Steven Hartland wrote: >>>>>>>=20 >>>>>>>=20 >>>>>>>> On 03/10/2017 07:12, Andriy Gapon wrote: >>>>>>>>=20 >>>>>>>>=20 >>>>>>>>> On 02/10/2017 21:12, Ben RUBSON wrote: >>>>>>>>>=20 >>>>>>>>> Hi, >>>>>>>>>=20 >>>>>>>>> On a FreeBSD 11 server, the following online/healthy zpool : >>>>>>>>>=20 >>>>>>>>> home >>>>>>>>> mirror-0 >>>>>>>>> label/local1 >>>>>>>>> label/local2 >>>>>>>>> label/iscsi1 >>>>>>>>> label/iscsi2 >>>>>>>>> mirror-1 >>>>>>>>> label/local3 >>>>>>>>> label/local4 >>>>>>>>> label/iscsi3 >>>>>>>>> label/iscsi4 >>>>>>>>> cache >>>>>>>>> label/local5 >>>>>>>>> label/local6 >>>>>>>>>=20 >>>>>>>>> A sustained read throughput of 180 MB/s, 45 MB/s on each iscsi = disk >>>>>>>>> according to "zpool iostat", nothing on local disks (strange = but I >>>>>>>>> noticed that IOs always prefer iscsi disks to local disks). >>>>>>>>>=20 >>>>>>>> Are your local disks SSD or HDD? >>>>>>>> Could it be that iSCSI disks appear to be faster than the local = disks >>>>>>>> to the smart ZFS mirror code? >>>>>>>>=20 >>>>>>>> Steve, what do you think? >>>>>>>>=20 >>>>>>> Yes that quite possible, the mirror balancing uses the queue = depth + >>>>>>> rotating bias to determine the load of the disk so if your iSCSI = host >>>>>>> is processing well and / or is reporting non-rotating vs = rotating for >>>>>>> the local disks it could well be the mirror is preferring reads = from >>>>>>> the the less loaded iSCSI devices. >>>>>>>=20 >>>>>> Note that local & iscsi disks are _exactly_ the same HDD (same = model number, >>>>>> same SAS adapter...). So iSCSI ones should be a little bit slower = due to >>>>>> network latency (even if it's very low in my case). >>>>>>=20 >>>>> The output from gstat -dp on a loaded machine would be interesting = to see too. >>>>>=20 >>>> So here is the gstat -dp : >>>>=20 >>>> L(q) ops/s r/s kBps ms/r w/s kBps ms/w d/s kBps ms/d %busy Name >>>> 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0| da0 >>>> 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0| da1 >>>> 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0| da2 >>>> 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0| da3 >>>> 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0| da4 >>>> 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0| da5 >>>> 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0| da6 >>>> 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0| da7 >>>> 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0| da8 >>>> 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0| da9 >>>> 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0| da10 >>>> 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0| da11 >>>> 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0| da12 >>>> 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0| da13 >>>> 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0| da14 >>>> 1 370 370 47326 0.7 0 0 0.0 0 0 0.0 23.2| da15 >>>> 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0| da16 >>>> 0 357 357 45698 1.4 0 0 0.0 0 0 0.0 39.3| da17 >>>> 0 348 348 44572 0.7 0 0 0.0 0 0 0.0 22.5| da18 >>>> 0 432 432 55339 0.7 0 0 0.0 0 0 0.0 27.5| da19 >>>> 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0| da20 >>>> 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0| da21 >>>>=20 >>>> The 4 active drives are the iSCSI targets of the above quoted pool. >>>>=20 >>>> A local disk : >>>>=20 >>>> Geom name: da7 >>>> Providers: >>>> 1. Name: da7 >>>> Mediasize: 4000787030016 (3.6T) >>>> Sectorsize: 512 >>>> Mode: r0w0e0 >>>> descr: HGSTxxx >>>> lunid: 5000xxx >>>> ident: NHGDxxx >>>> rotationrate: 7200 >>>> fwsectors: 63 >>>> fwheads: 255 >>>>=20 >>>> A iSCSI disk : >>>>=20 >>>> Geom name: da19 >>>> Providers: >>>> 1. Name: da19 >>>> Mediasize: 3999688294912 (3.6T) >>>> Sectorsize: 512 >>>> Mode: r1w1e2 >>>> descr: FREEBSD CTLDISK >>>> lunname: FREEBSD MYDEVID 12 >>>> lunid: FREEBSD MYDEVID 12 >>>> ident: iscsi4 >>>> rotationrate: 0 >>>> fwsectors: 63 >>>> fwheads: 255 >>>>=20 >>>> Sounds like then the faulty thing is the rotationrate set to 0 ? >>>=20 >>> Absolutely >>=20 >> Good catch then, thank you ! >>=20 >>> and from the looks you're not stressing the iSCSI disks so they get = high queuing depths hence the preference. >>> As load increased I would expect the local disks to start seeing = activity. >>=20 >> Yes this is also what I see. >>=20 >> Any way however to set rotationrate to 7200 (or to a slightly greater = value) as well for iSCSI drives ? >> I looked through ctl.conf(5) and iscsi.conf(5) but did not found = anything related. >>=20 >> Many thanks ! >=20 > Use the "option" setting in ctl.conf to change the rpm value = (documented > in the OPTIONS section of ctladm(8)). Thank you also Gary, and sorry as your mail went to spam :/ Ben