From owner-freebsd-fs@FreeBSD.ORG Fri Feb 7 08:35:31 2014 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 746FAC6E for ; Fri, 7 Feb 2014 08:35:31 +0000 (UTC) Received: from mail-qa0-x230.google.com (mail-qa0-x230.google.com [IPv6:2607:f8b0:400d:c00::230]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 3142E112B for ; Fri, 7 Feb 2014 08:35:31 +0000 (UTC) Received: by mail-qa0-f48.google.com with SMTP id f11so4722418qae.35 for ; Fri, 07 Feb 2014 00:35:30 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=hu1lba1KLi+JahO4auC+I33FZkECS+nl1GY3gMljjC0=; b=FZBbsJ27teen3QXbKFJZizFpIS8goIPaph8FQ7eIWSg7nnU+Urtrh07iDouVimhztY 9VkvisC+jLM5c0YqeGHJEy4cQa8BCIMuGL3Ivkzxb1aj8s/xuKOe8KDY4S51wUkanxs6 gLIOLV2HNPwVkl8uzqEkb82U+owUHCARKCXA3Tuq3vJa7MUANIeGbqvPIdlzHEfhb9K2 +PBZZJJxcKHDq9HBVsL+7v27JcUo9GeO8uyDtcuhIgpk2DdAlHqbkRUc8HXzjl/pjGDp PwIRPlQ8UpEacDbtfkjjJrkP2lFOcHXgn4lIz+I2nrPo6aL6xO5IwrjxEIrDk4fc/a/V WQ/Q== MIME-Version: 1.0 X-Received: by 10.224.40.130 with SMTP id k2mr19445068qae.91.1391762130282; Fri, 07 Feb 2014 00:35:30 -0800 (PST) Received: by 10.96.37.227 with HTTP; Fri, 7 Feb 2014 00:35:30 -0800 (PST) In-Reply-To: <94A20D8E-292D-47B4-8D82-61A131B3010D@gmail.com> References: <52F1BDA4.6090504@physics.umn.edu> <7D20F45E-24BC-4595-833E-4276B4CDC2E3@gmail.com> <52F24DEA.9090905@physics.umn.edu> <94A20D8E-292D-47B4-8D82-61A131B3010D@gmail.com> Date: Fri, 7 Feb 2014 08:35:30 +0000 Message-ID: Subject: Re: practical maximum number of drives From: krad To: aurfalien Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.17 Cc: FreeBSD FS X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 07 Feb 2014 08:35:31 -0000 im confused by all this, do you need massive storage, lots or redundancy or just plain speed? If its redundancy, you kind of messed that up by going of one controller. On 5 February 2014 18:45, aurfalien wrote: > Ah great info many thanks. > > And pplz, ignore my reply to Daniel as I got the posts confused. I > recently switched to Sanka :) > > - aurf > > On Feb 5, 2014, at 6:42 AM, Graham Allan wrote: > > > > > > > On 2/4/2014 11:36 PM, aurfalien wrote: > >> Hi Graham, > >> > >> When you say behaved better with 1 HBA, what were the issues that > >> made you go that route? > > > > It worked fine in general with 3 HBAs for a while but OTOH 2 of the > drive chassis were being very lightly used (and note I was being quite > conservative and keeping each chassis as an independent zfs pool). > > > > Actual problems occurred once while I was away but our notes show we got > some kind of repeated i/o deadlock. As well as all drive i/o stopping, we > also couldn't use the sg_ses utilities to query the enclosures. This > reoccurred several times after restarts throughout the day, and eventually > "we" (again i wasn't here) removed the extra HBAs and daisy-chained all the > chassis together. An inspired hunch, I guess. No issues since then. > > > > Coincidentally a few days later I saw a message on this list from Xin Li > "Re: kern/177536: [zfs] zfs livelock (deadlock) with high write-to-disk > load": > > > > One problem we found in field that is not easy to reproduce is that > > there is a lost interrupt issue in FreeBSD core. This was fixed in > > r253184 (post-9.1-RELEASE and before 9.2, the fix will be part of the > > upcoming FreeBSD 9.2-RELEASE): > > > > > http://svnweb.freebsd.org/base/stable/9/sys/kern/kern_intr.c?r1=249402&r2=253184&view=patch > > > > The symptom of this issue is that you basically see a lot of processes > > blocking on zio->zio_cv, while there is no disk activity. However, > > the information you have provided can neither prove or deny my guess. > > I post the information here so people are aware of this issue if they > > search these terms. > > > > Something else suggested to me that multiple mps adapters would make > this worse but I'm not quite sure what. This issue wouldn't exist after 9.1 > anyway. > > > >> Also, curious that you have that many drives on 1 PCI card, is it PCI > >> 3 etc... and is saturation an issue? > > > > Pretty sure it's PCIe 2.x but we haven't seen any saturation issues. > That was of course the motivation for using separate HBAs in the initial > design but it was more of a hypothetical concern than a real one - at least > given our use pattern at present. This is more backing storage, the more > intensive i/o usually goes to a hadoop filesystem. > > > > Graham > > _______________________________________________ > freebsd-fs@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" >