From owner-freebsd-questions@FreeBSD.ORG Thu Aug 7 16:24:31 2014 Return-Path: Delivered-To: freebsd-questions@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 4653F7A4 for ; Thu, 7 Aug 2014 16:24:31 +0000 (UTC) Received: from na01-bn1-obe.outbound.protection.outlook.com (dns-bn1lp0143.outbound.protection.outlook.com [207.46.163.143]) (using TLSv1 with cipher ECDHE-RSA-AES256-SHA (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "MSIT Machine Auth CA 2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id E51AC2435 for ; Thu, 7 Aug 2014 16:24:30 +0000 (UTC) Received: from [IPv6:2601:2:4780:2fd:e806:9aa:f68a:db81] (2601:2:4780:2fd:e806:9aa:f68a:db81) by CY1PR0301MB0841.namprd03.prod.outlook.com (25.160.163.147) with Microsoft SMTP Server (TLS) id 15.0.995.14; Thu, 7 Aug 2014 16:24:28 +0000 Message-ID: <53E3A836.9080904@my.hennepintech.edu> Date: Thu, 7 Aug 2014 11:24:22 -0500 From: Andrew Berg User-Agent: Mozilla/5.0 (Windows NT 6.2; WOW64; rv:24.0) Gecko/20100101 Thunderbird/24.6.0 MIME-Version: 1.0 To: Scott Bennett Subject: Re: some ZFS questions References: <201408070816.s778G9ug015988@sdf.org> In-Reply-To: <201408070816.s778G9ug015988@sdf.org> Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 7bit X-Originating-IP: [2601:2:4780:2fd:e806:9aa:f68a:db81] X-ClientProxiedBy: BY2PR03CA061.namprd03.prod.outlook.com (10.141.249.34) To CY1PR0301MB0841.namprd03.prod.outlook.com (25.160.163.147) X-Microsoft-Antispam: BCL:0;PCL:0;RULEID: X-Forefront-PRVS: 029651C7A1 X-Forefront-Antispam-Report: SFV:NSPM; SFS:(6009001)(24454002)(189002)(199002)(51704005)(65956001)(74502001)(83506001)(77096002)(31966008)(20776003)(64126003)(54356999)(65806001)(19580405001)(83322001)(80316001)(59896001)(77982001)(65816999)(85306004)(110136001)(88552001)(47776003)(87266999)(19580395003)(80022001)(76176999)(86362001)(101416001)(64706001)(95666004)(23676002)(74662001)(107046002)(50986999)(79102001)(46102001)(42186005)(50466002)(92726001)(87976001)(76482001)(4396001)(21056001)(85852003)(102836001)(89122001)(83072002)(81542001)(106356001)(33656002)(92566001)(75432001)(105586002)(99396002)(81342001)(89472002)(3826002); DIR:OUT; SFP:; SCL:1; SRVR:CY1PR0301MB0841; H:[IPv6:2601:2:4780:2fd:e806:9aa:f68a:db81]; FPR:; MLV:nov; PTR:InfoNoRecords; MX:1; LANG:en; X-OriginatorOrg: my.hennepintech.edu Cc: freebsd-questions@freebsd.org X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 07 Aug 2014 16:24:31 -0000 On 2014.08.07 03:16, Scott Bennett wrote: > On Wed, 6 Aug 2014 03:49:37 -0500 Andrew Berg > wrote: >>On 2014.08.06 02:32, Scott Bennett wrote: >>> I have a number of questions that I need answered before I go about >>> setting up any raidz pools. They are in no particular order. >>> >>> 1) What is the recommended method of using geli encryption with >>> ZFS? >> >>> Does one first create .eli devices and then specify those >>> .eli devices in the zpool(8) command as the devices to include >>> in the pool? >>This. > > Oh. Well, that's doable, if not terribly convenient, but it brings up > another question. After a reboot, for example, what does ZFS do while the > array of .eli devices is being attached one by one? Does it see the first > one attached without the others in sight and decide it has a failed pool? Once you bring the .eli devices back online, zpool will see them and your pool will be back online. Before then, it won't really do anything but tell you the disks are not available and therefore, neither is your pool. The status of the pool is 'unavailable', not 'faulted'. >>mercilessly thrash disks; standard reads and writes are given higher priority >>in the scheduler than resilver and scrub operations. > > If two pools use different partitions on a drive and both pools are > rebuilding those partitions at the same time, then how could ZFS *not* > be hammering the drive? A good reason not to setup your pools like that. >>> 3) If a raidz2 or raidz3 loses more than one component, does one >>> simply replace and rebuild all of them at once? Or is it necessary >>> to rebuild them serially? In some particular order? >>AFAIK, replacement of several disks can't be done in a single command, but I >>don't think you need to wait for a resilver to finish on one before you can >>replace another. > > That looks good. What happens if a "zpool replace failingdrive newdrive" > is running when the failingdrive actually fails completely? Assuming you don't trigger some race condition (which would be rare if you're using decent controllers), nothing special. A disk doesn't need to be present and functioning to be replaced. >>> 5) When I upgrade to amd64, the usage would continue to be low- >>> intensity as defined above. Will the 4 GB be enough? I will not >>> be using the "deduplication" feature at all. >>It will be enough unless you are managing tens of TB of data. I recommend >>setting an ARC limit of 3GB or so. There is a patch that makes the ARC handle > > 3 GB for ARC plus whatever is needed for FreeBSD would leave much room > for applications to run. Maybe I won't be able to use ZFS if it requires > so vastly more page-fixed memory than UFS. :-( 3GB is the hard limit here. If applications need more, they'll get it. The only reason to set a limit at all is that the ARC currently has issues giving up memory gracefully. As I said, there's a patch in discussion to fix it. > One thing I ran across was the following from the zpool(8) man page. > > "For pools to be portable, you must give the zpool command whole > disks, not just slices, so that ZFS can label the disks with portable > EFI labels. Otherwise, disk drivers on platforms of different endian- > ness will not recognize the disks." Well, that is kind of confusing since slices != partitions and partitions aren't mentioned. Using slices is also something someone would generally not do with GPT. I'll look at that part of the man page and maybe bring it up on the doc and fs MLs. > If I have one raidzN comprising .eli partitions and another raidzN comprising > a set of unencrypted partitions on those same drives, will I be able to > export both raidzN pools from a 9-STABLE system and then import them > into, say, a 10-STABLE system on a different Intel amd64 machine? By your > answer to question 1), it would seem that I need to have two raidzN pools, > although there might be a number of benefits to having both encrypted and > unencrypted file systems allocated inside a single pool were that an option. Having any physical disk be a part of more than one pool is not recommended (except perhaps for cache and log devices where failure is not a big deal). Not only can it cause thrashing as you mentioned above, but one disk dying makes both pools degraded. Lose two disks, and you lose both pools. If you need only some things encrypted, perhaps something that works above the FS layer such as PEFS would be a better option for you.