From owner-freebsd-fs@freebsd.org Tue Oct 3 07:31:53 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id E19E9E33743; Tue, 3 Oct 2017 07:31:53 +0000 (UTC) (envelope-from ben.rubson@gmail.com) Received: from mail-wr0-x22d.google.com (mail-wr0-x22d.google.com [IPv6:2a00:1450:400c:c0c::22d]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 7245F68665; Tue, 3 Oct 2017 07:31:53 +0000 (UTC) (envelope-from ben.rubson@gmail.com) Received: by mail-wr0-x22d.google.com with SMTP id l39so5638168wrl.12; Tue, 03 Oct 2017 00:31:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:subject:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to; bh=UgM3Amcn+bco2urIcwSNTrv7H6g7cAH4FC9jCmWj2iQ=; b=Hcpc0NPqR9cJf4a7Tozx0Gyd3JCDuU2zWrMyytXgc4gjUJibL0K67ko8kaX0sWTqI7 wnxwz+lRvWb5L7414g6vwtI/IJjd7E3ClqTyXwkBVBPMuoJHxh5z2gVwQUiv7hs6V6jM Auyu7r5Xf/kkMFK4MJRtkUpDT9j3DSfNPdJe/tBVKFPnd8H3bxEqXiTjgc9RNSsvCkT/ 7eeVpZKbK6ha9o6OrVG2uOWnP4cWPNpLJBYCPbTqJvUhqvfWwvduXcH98pgjMCUkg6Ht gpefHIy+K0uKCWKcNTBM7h85vCb4kYyE3Sw1tVoLAMpbe/3LtOA/jWj/Rp+fQAABgYEF KKSQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:subject:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to; bh=UgM3Amcn+bco2urIcwSNTrv7H6g7cAH4FC9jCmWj2iQ=; b=bgsHn1ezV+DGjycls9hErTBwaoOm9P4odfZEF51/oG7PLSKRBM3+QvrGe5Jo1LA9l3 1XwBiQ1ESWWoiu631RMRXq3lVbPwgBJfndCa9mHscemA+iic0DHEtK0vrdmbUvsmXwmt AMuimk1/ZDSjltZxAp+Wc5CDKxJoRJTxCrf6/iroW1CKl8/VmR3riRN18iSHiGK1KDOt eNhLAiapcT04L2153fCNSFORmA0MZYQUulAQli3e4axtPw6Wgyq/sZJCyR9El99GPm5b CUdsNzB+nmpOHY0H3EhGUnJIzMZaQgI4YPLMvxnyHO1WL7aSe/9B2P8JC2i6ipkolLYc 56Pw== X-Gm-Message-State: AMCzsaW+1lzJJz1i2FTeL+hMCNtsKpzegOs+07RNMR99V54idlVxMEao ePF5Sn5oVRi3WVw+CODWecpoIa5n X-Google-Smtp-Source: AOwi7QCTm0qGWPIDdfXs7QTgC5IN6WbC6rNxM2Natyy7Msx1YA8M1dP7b3RIL1jv7NUZwxSfRwMmVg== X-Received: by 10.223.178.144 with SMTP id g16mr11264078wrd.76.1507015911918; Tue, 03 Oct 2017 00:31:51 -0700 (PDT) Received: from bens-mac.home (LFbn-MAR-1-445-220.w2-15.abo.wanadoo.fr. [2.15.38.220]) by smtp.gmail.com with ESMTPSA id l37sm12954776wrl.47.2017.10.03.00.31.51 (version=TLS1 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Tue, 03 Oct 2017 00:31:51 -0700 (PDT) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 9.3 \(3124\)) Subject: Re: ZFS stalled after some mirror disks were lost From: Ben RUBSON In-Reply-To: <69fbca90-9a18-ad5d-a2f7-ad527d79f8ba@freebsd.org> Date: Tue, 3 Oct 2017 09:31:50 +0200 Cc: Freebsd fs , FreeBSD-scsi Content-Transfer-Encoding: quoted-printable Message-Id: <1990B359-FC8D-4D6A-992B-7F77A07D83A6@gmail.com> References: <4A0E9EB8-57EA-4E76-9D7E-3E344B2037D2@gmail.com> <69fbca90-9a18-ad5d-a2f7-ad527d79f8ba@freebsd.org> To: Steven Hartland , Andriy Gapon X-Mailer: Apple Mail (2.3124) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 03 Oct 2017 07:31:54 -0000 > On 03 Oct 2017, at 09:25, Steven Hartland = wrote: >=20 > On 03/10/2017 07:12, Andriy Gapon wrote: >> On 02/10/2017 21:12, Ben RUBSON wrote: >>=20 >>> A sustained read throughput of 180 MB/s, 45 MB/s on each iscsi disk >>> according to "zpool iostat", nothing on local disks (strange but I >>> noticed that IOs always prefer iscsi disks to local disks). >>>=20 >> Are your local disks SSD or HDD? >> Could it be that iSCSI disks appear to be faster than the local disks = to the >> smart ZFS mirror code? >>=20 >> Steve, what do you think? >>=20 > Yes that quite possible, the mirror balancing uses the queue depth + = rotating bias to determine the load of the disk so if your iSCSI host is = processing well and / or is reporting non-rotating vs rotating for the = local disks it could well be the mirror is preferring reads from the the = less loaded iSCSI devices. Note that local & iscsi disks are _exactly_ the same (same model number, = same SAS adapter...). So iSCSI ones should be a little bit slower due to network latency (even = if it's very low in my case). Once production back, after having analysed the main issue of this = thread, I should then try to find whether or not iSCSI disks are seen as rotating disks. Thanks for the hint ! Ben