From owner-freebsd-stable@FreeBSD.ORG Mon Jan 23 19:21:01 2012 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 8CA82106564A for ; Mon, 23 Jan 2012 19:21:01 +0000 (UTC) (envelope-from prvs=1369c89d38=killing@multiplay.co.uk) Received: from mail1.multiplay.co.uk (mail1.multiplay.co.uk [85.236.96.23]) by mx1.freebsd.org (Postfix) with ESMTP id 18C568FC15 for ; Mon, 23 Jan 2012 19:21:00 +0000 (UTC) X-Spam-Processed: mail1.multiplay.co.uk, Mon, 23 Jan 2012 19:20:54 +0000 X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on mail1.multiplay.co.uk X-Spam-Level: X-Spam-Status: No, score=-5.0 required=6.0 tests=USER_IN_WHITELIST shortcircuit=ham autolearn=disabled version=3.2.5 Received: from r2d2 ([188.220.16.49]) by mail1.multiplay.co.uk (mail1.multiplay.co.uk [85.236.96.23]) (MDaemon PRO v10.0.4) with ESMTP id md50017676792.msg for ; Mon, 23 Jan 2012 19:20:54 +0000 X-MDRemoteIP: 188.220.16.49 X-Return-Path: prvs=1369c89d38=killing@multiplay.co.uk X-Envelope-From: killing@multiplay.co.uk X-MDaemon-Deliver-To: freebsd-stable@freebsd.org Message-ID: From: "Steven Hartland" To: "Chuck Swiger" References: <520B9285BCC0498286196195933D67E9@multiplay.co.uk> Date: Mon, 23 Jan 2012 19:21:04 -0000 MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset="iso-8859-1"; reply-type=original Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.5931 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.6157 Cc: freebsd-stable@freebsd.org Subject: Re: ZFS: i/o error - all block copies unavailable on large disk number machines X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 23 Jan 2012 19:21:01 -0000 ----- Original Message ----- From: "Chuck Swiger" > On Jan 23, 2012, at 9:04 AM, Steven Hartland wrote: >> After some digging we discovered that this was likely due >> to the fact that the BIOS only enumerates the first 12 disks >> and this machine has more than that in the root zpool which >> was a striped raidz2 volume. This in turn means that the >> bootcode can't complete and hence the machine can't boot. > > As far as I can tell, ZFS best practices guides recommend no > more than nine drives in a group/pool. Putting more than that > into a pool, much less something you are trying to boot from, > seems like a fine experiment to make but is not something which > I would rely upon... Not something I've seem made clear, but quite possibly. Even with 9 disks you could easily get this if the BIOS doesn't see all of said disks, be that initially or due to disks added to the machine. For reference the original install was done on a zpool with 6 disks in a raidz2 config but then 6 additional disks where added to expand capacity. It was only when the new kernel was installed that data required to boot was then written to disks in the seconds raidz2 which is inaccessible to the boot code even though in perfect working order on a booted system. So something to document, watch out for and potentially safe guard against? It maybe something specific to machines with legacy BIOS hence not an issue with Sun kit? What made it more interesting is the boot code could see the directory structure but clearly not all of the required data. Would it be possible for the boot code to provide a more coherent message in this case? Regards Steve ================================================ This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it. In the event of misdirection, illegible or incomplete transmission please telephone +44 845 868 1337 or return the E.mail to postmaster@multiplay.co.uk.