From owner-freebsd-hackers@FreeBSD.ORG Sat Oct 11 20:54:26 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D53D310656A4 for ; Sat, 11 Oct 2008 20:54:26 +0000 (UTC) (envelope-from fjwcash@gmail.com) Received: from rv-out-0506.google.com (rv-out-0506.google.com [209.85.198.225]) by mx1.freebsd.org (Postfix) with ESMTP id A9F878FC16 for ; Sat, 11 Oct 2008 20:54:26 +0000 (UTC) (envelope-from fjwcash@gmail.com) Received: by rv-out-0506.google.com with SMTP id b25so960966rvf.43 for ; Sat, 11 Oct 2008 13:54:26 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:to :subject:in-reply-to:mime-version:content-type :content-transfer-encoding:content-disposition:references; bh=GFWAbjKG8AhMCovDIqXlP/LECdgNNTBh0W45msQTr9E=; b=ZXSdHzIzLCwWUrOBn2vvb7zVVpZ/ItMLvyz5hQeMQqGEA1JKbbQFhOivY3AkIhftWD KLmQPvlwlliYoqwegmnFTJc99uvSXuC/tzjBdkLcleIKSZMDKBpP80PVtziz7sqHSG4H y7Zc1Gk03HwUNG+UTJGUkq4NYSbnCYgRu6AEU= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:to:subject:in-reply-to:mime-version :content-type:content-transfer-encoding:content-disposition :references; b=C24N8KEnq3Kb1rM/ftUu/HP/0sirVIa9ahDVuf45CDtSjWfUo2Nril0Aqqn3J1MMNC bdzFySEmdf1BL4ses3HWpKV2uv81j/Q3jM6JTlYVfCCntNFdVKpWXbkW9EdGrvMeNC9M pGY3iuODbZEKWmFmEIffJoUuRyT6+SxFKRMMo= Received: by 10.141.205.10 with SMTP id h10mr2355933rvq.54.1223756979977; Sat, 11 Oct 2008 13:29:39 -0700 (PDT) Received: by 10.141.168.12 with HTTP; Sat, 11 Oct 2008 13:29:39 -0700 (PDT) Message-ID: Date: Sat, 11 Oct 2008 20:29:39 +0000 From: "Freddie Cash" To: freebsd-hackers@freebsd.org In-Reply-To: MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit Content-Disposition: inline References: Subject: Re: ZFS boot X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 11 Oct 2008 20:54:27 -0000 On 10/11/08, Danny Braniss wrote: >> > I'm asking, because I want to deploy some zfs fileservers soon, and so >> > far the solution is either PXE boot, or keep one disk UFS (or boot off a >> > USB) For the servers we're deploying FreeBSD+ZFS on, mainly large backup systems with 24 drives, we're putting / onto either CompactFlash (using IDE adapters) or USB sticks (using internal connectors), using gmirror to provide fail-over for /. That way, we can boot off UFS, have full access to single-user mode and /rescue, and use every bit of each disk for ZFS. Works quite nicely. >> > Today's /(root+usr) is somewhere between .5 to 1Gb(kernel+debug+src), >> > and is readonly, so having 1 disk UFS seems to be a pitty. / by itself (no /usr, /home, /tmp, or /var) is under 300 MB on our systems (FreeBSD 7-STABLE from August, amd64). Definitely not worth dedicating an entire 500 GB drive to, or even a single slice or partition to. By putting / onto separate media (like CF, USB, whatever), you can dedicate all your harddrive space to ZFS. > /OT > Initially, I was not thrilled with ZFS, but once you cross the Once you start using ZFS features, especially snapshots, it's really hard to move to non-pooled-storage setups. Even LVM on Linux becomes hard to work with. There's just no easier way to work with multi-TB storage setups using 10+ drives. Even for smaller systems with only 3 drives, it's so much nicer working with pooled storage systems like ZFS. My home server uses a 2 GB USB stick for / with 3x 120 GB drives for ZFS, with separate filesystems for /usr, /usr/ports, /usr/src, /usr/obj, /usr/local, /home, /var, and /tmp. No fussing around with partition sizes ahead of time is probably the single greatest feature, with instant/unlimited snapshots a very close second. >> I think (hope?) you can use the "remaining" (e.g. non-UFS/non-gmirror) >> part of the 2nd disk for ZFS as well, otherwise the space would go >> to waste. The "Root on ZFS configuration" FreeBSD ZFS Wiki page >> seems to imply you can. I did this for awhile. 3x 120 GB drives configured as: 10 GB slice for / 2 GB slice for swap 108 GB slice to ZFS The first slice was configured as a 3-way gmirror, and the last slice was configured as a raidz pool. But performance wasn't that great. Moved / to a USB stick, and dedicated the entire drives to the zpool, and things have been a lot smoother. -- Freddie Cash fjwcash@gmail.com