From owner-freebsd-fs@FreeBSD.ORG Wed Feb 5 18:45:35 2014 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 4A4EF389 for ; Wed, 5 Feb 2014 18:45:35 +0000 (UTC) Received: from mail-pa0-x236.google.com (mail-pa0-x236.google.com [IPv6:2607:f8b0:400e:c03::236]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 1CBBB1A73 for ; Wed, 5 Feb 2014 18:45:35 +0000 (UTC) Received: by mail-pa0-f54.google.com with SMTP id fa1so691421pad.41 for ; Wed, 05 Feb 2014 10:45:34 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=content-type:mime-version:subject:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to; bh=Ab/ybE796nf7UDISi3Li1IsFF96b5FDjaVb40L8KU94=; b=xPNz+1EMzuf9xstPM5wzclbIwXrWKKCei8eb3J0YP1JtrtkIGd7/SXnr1ng2ddNwXj d+VYUU1PqX4BGVevhlvEZZB/zaGaS5LKvQxmBj6fEIuVrHW24pgWStdu5AFOievUUo0S JmGMDAymN0/wDn+EqBg1SZDqd3cvovxcWzYAYojEjxgUlIPar1Bz6I73RZo6vkonvq5E zjvLVMz5agUFo362aXKElrMBy2jrFAUmuBXmLGgPuhL+7jBk3Cxk7bn3ByaqainHMRM+ eOqp0uzTkTspvdY+jcY/HcQLy9kDHfZ9SgfFlxVi2dhZp/BYgWSwGyFYbZJWmXlT8RNN /WVw== X-Received: by 10.68.190.169 with SMTP id gr9mr4326336pbc.30.1391625934591; Wed, 05 Feb 2014 10:45:34 -0800 (PST) Received: from briankrusicw.logan.tv ([64.17.255.138]) by mx.google.com with ESMTPSA id ja8sm38472960pbd.3.2014.02.05.10.45.33 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Wed, 05 Feb 2014 10:45:34 -0800 (PST) Content-Type: text/plain; charset=windows-1252 Mime-Version: 1.0 (Mac OS X Mail 7.1 \(1827\)) Subject: Re: practical maximum number of drives From: aurfalien In-Reply-To: <52F24DEA.9090905@physics.umn.edu> Date: Wed, 5 Feb 2014 10:45:33 -0800 Content-Transfer-Encoding: quoted-printable Message-Id: <94A20D8E-292D-47B4-8D82-61A131B3010D@gmail.com> References: <52F1BDA4.6090504@physics.umn.edu> <7D20F45E-24BC-4595-833E-4276B4CDC2E3@gmail.com> <52F24DEA.9090905@physics.umn.edu> To: Graham Allan X-Mailer: Apple Mail (2.1827) Cc: FreeBSD FS X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 05 Feb 2014 18:45:35 -0000 Ah great info many thanks. And pplz, ignore my reply to Daniel as I got the posts confused. I = recently switched to Sanka :) - aurf On Feb 5, 2014, at 6:42 AM, Graham Allan wrote: >=20 >=20 > On 2/4/2014 11:36 PM, aurfalien wrote: >> Hi Graham, >>=20 >> When you say behaved better with 1 HBA, what were the issues that >> made you go that route? >=20 > It worked fine in general with 3 HBAs for a while but OTOH 2 of the = drive chassis were being very lightly used (and note I was being quite = conservative and keeping each chassis as an independent zfs pool). >=20 > Actual problems occurred once while I was away but our notes show we = got some kind of repeated i/o deadlock. As well as all drive i/o = stopping, we also couldn't use the sg_ses utilities to query the = enclosures. This reoccurred several times after restarts throughout the = day, and eventually "we" (again i wasn't here) removed the extra HBAs = and daisy-chained all the chassis together. An inspired hunch, I guess. = No issues since then. >=20 > Coincidentally a few days later I saw a message on this list from Xin = Li "Re: kern/177536: [zfs] zfs livelock (deadlock) with high = write-to-disk load": >=20 > One problem we found in field that is not easy to reproduce is that > there is a lost interrupt issue in FreeBSD core. This was fixed in > r253184 (post-9.1-RELEASE and before 9.2, the fix will be part of the > upcoming FreeBSD 9.2-RELEASE): >=20 > = http://svnweb.freebsd.org/base/stable/9/sys/kern/kern_intr.c?r1=3D249402&r= 2=3D253184&view=3Dpatch >=20 > The symptom of this issue is that you basically see a lot of processes > blocking on zio->zio_cv, while there is no disk activity. However, > the information you have provided can neither prove or deny my guess. > I post the information here so people are aware of this issue if they > search these terms. >=20 > Something else suggested to me that multiple mps adapters would make = this worse but I'm not quite sure what. This issue wouldn't exist after = 9.1 anyway. >=20 >> Also, curious that you have that many drives on 1 PCI card, is it PCI >> 3 etc=85 and is saturation an issue? >=20 > Pretty sure it's PCIe 2.x but we haven't seen any saturation issues. = That was of course the motivation for using separate HBAs in the initial = design but it was more of a hypothetical concern than a real one - at = least given our use pattern at present. This is more backing storage, = the more intensive i/o usually goes to a hadoop filesystem. >=20 > Graham