From owner-freebsd-fs@freebsd.org Thu Jan 26 04:20:18 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 60B3DCC1E1C for ; Thu, 26 Jan 2017 04:20:18 +0000 (UTC) (envelope-from octavianh@gmail.com) Received: from mail-qt0-x22e.google.com (mail-qt0-x22e.google.com [IPv6:2607:f8b0:400d:c0d::22e]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 1B8B1B7F for ; Thu, 26 Jan 2017 04:20:18 +0000 (UTC) (envelope-from octavianh@gmail.com) Received: by mail-qt0-x22e.google.com with SMTP id v23so55276702qtb.0 for ; Wed, 25 Jan 2017 20:20:18 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=qzllWipZFDhdKHl2RlOJ93QU/d1l4dJW4GKchmcyB/A=; b=ApM0bBqidv0wbangiTXZEq9nISBFVdTZfvEoPUPh3c/I8NZfGT/bKlan4gsfmTUtmi /UNgSlcQqtZVsUruoa384Sjh8q6xXVc0XSUwsji2p1NMTWAePWVySPtvm2rCo/L83UAk FCGPUi6NvNcljJOXKM0Wk5uYZAuNS62twwLi6VFOk0SEtLAoHYDFYk4tMdXNk2M8X8Sx 7TAxZN+HOUP2J4RpDNjoRpd+Ok6DpbP6QjAds2sxK8hCehdLwjmkeRClKL4IoV6+WcbN lXlPQV52lw87rXnqa5Jl2SOVvMJRDhrJNPOLR9QNd1UYvtufetZ3apWKo47w6+onplZW 8R9A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=qzllWipZFDhdKHl2RlOJ93QU/d1l4dJW4GKchmcyB/A=; b=jTG5KBnHyvSUo3FfLAaCmkGHz8bYl6AwYU6Bf74oAt2tDHBnfqa5ppaR9dJ+Tm0QGt pGTN8jogtsL3cAQkqX03to8ZrrGuxWNZNczucVi0xEf1v0mNqkwZzbKLaGb1pgcLd4DS HUClJvCtW+mUPQek4Lk8mlMppHS9h0Iy0Xp8NTMluQ08wtiPNC+C+E4cawXcSuv/stjJ fHA5xcMZWreTbfhCquz4vT1jf4KoeApRlSgmOOCsVDuHOmnfXcFnlTnipleoi3SoiKp+ /dQ9JsAuB3XK6L+rSLBO9fyGWU4iFO6e/v4h/XWePmDCZzEP8d6a+YDvMzoO+pKXlyv9 zqJQ== X-Gm-Message-State: AIkVDXJcHGgt09PUkVWNQ26x5ttFRvJ7iL9qYWYHQSqx+zr6b6KZD9Y36KJy/yyvX7MYlxa7wBtt9ChQy7N2+g== X-Received: by 10.237.34.250 with SMTP id q55mr763574qtc.127.1485404417317; Wed, 25 Jan 2017 20:20:17 -0800 (PST) MIME-Version: 1.0 Received: by 10.140.82.49 with HTTP; Wed, 25 Jan 2017 20:20:16 -0800 (PST) In-Reply-To: <9522d5cccd704b8fbe6cfe00d3bbd51a@SERVER.ad.usd-group.com> References: <9522d5cccd704b8fbe6cfe00d3bbd51a@SERVER.ad.usd-group.com> From: Octavian Hornoiu Date: Wed, 25 Jan 2017 20:20:16 -0800 Message-ID: Subject: Re: Question on gmirror and zfs fs behavior in unusual setup To: Matt Churchyard Cc: freebsd-fs Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.23 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 26 Jan 2017 04:20:18 -0000 On Mon, Jan 11, 2016 at 4:07 AM, Matt Churchyard wrote: > >I currently have several storage servers. For historical reasons they > have 6x 1TB Western Digital Black SATA drives in each server. Configuration > is >as follows: > > >GPT disk config with boot sector > >/dev/ada0p1 freebsd-boot 64k > >/dev/ada0p2 freebsd-swap 1G > >/dev/ada0p3 freebsd-ufs 30G > >/dev/ada0p4 freebsd-zfs rest of drive > > >The drive names are ada0 through ada5. > > >The six drives all have the same partition scheme. > >- They are all bootable > >- Each swap has a label from swap0 through swap5 which all mount on boot > >- The UFS partitions are all in mirror/rootfs mirrored using gmirror in a > 6 way mirror (The goal of the boot and mirror redundancy is any drive can > >die and I can still boot off any other drive like nothing happened. This > partition contains the entire OS. > >- The zfs partitions are in RAIDZ-2 configuration and are redundant > automatically. They contain the network accessible storage data. > > >My dilemma is this. I am upgrading to 5 TB Western Digital Black drives. > I have replaced drive ada5 as a test. I used the -a 4k command while > >partitioning to make sure sector alignment is correct. There are two major > >changes: > > >- ada5p3 is now 100 G > >- ada5p4 is now much larger due to the size of the drive > > >My understanding is that zfs will automatically change the total volume > size once all drives are upgraded to the new 5 TB drives. Please correct > >me if I'm wrong! The resilver went without a hitch. > > You may have to run "zpool online -e pool" once all the disk have been > replaced, but yes it should be fairly easy to get ZFS to pick up the new > space. > > The only other issue you may see is that if you built the original pool > with 512b sectors (ashift 9) you may find "zpool status" start complaining > that you are configured for 512b sectors when your disks are 4k (I haven't > checked but considering the size I expect those 5TB disks are 4k). If that > happens you either have to live with the warning or rebuild the pool. > > >My concern is with gmirror. Will gmirror grow to fit the new 100 G size > automatically once the last drive is replaced? I got no errors using insert > >with the 100 G partition into the mix with the other 5 30 G partitions. It > synchronized fine. The volume shows as complete and all providers are > >healthy. > > A quick test suggests you'll need to run "gmirror resize provider" once > all the disks are replaced to get gmirror to update the size stored in the > metadata - > > # gmirror list > Geom name: test > State: COMPLETE > Components: 2 > ... > Providers: > 1. Name: mirror/test > Mediasize: 104857088 (100M) > Sectorsize: 512 > Mode: r0w0e0 > Consumers: > 1. Name: md0 > Mediasize: 209715200 (200M) > ... > > # gmirror resize test > # gmirror list > ... > Providers: > 1. Name: mirror/test > Mediasize: 209714688 (200M) > Sectorsize: 512 > Mode: r0w0e0 > ... > > You will then need to expand the filesystem to fill the space using > growfs. Never done this but it should be a fairly straight forward process > from what I can see, although it seems resizing while mounted only works on > 10.0+ > > >Anyone with knowledge of gmirror and zfs replication able to confirm that > they'll grow automatically once all 6 drives are replaced or do I have >to > sync them at existing size and do some growfs trick later? > > >Thanks! > Thanks Matt! This advice was really great, it all worked as expected! Unfortunately, like you suspected the pool is whining about sectors. #zpool status pool: data state: ONLINE status: One or more devices are configured to use a non-native block size. Expect reduced performance. action: Replace affected devices with devices that support the configured block size, or migrate data to a properly configured pool. scan: resilvered 861G in 2h59m with 0 errors on Sat Jan 21 01:33:02 2017 config: NAME STATE READ WRITE CKSUM data ONLINE 0 0 0 raidz2-0 ONLINE 0 0 0 gpt/data0 ONLINE 0 0 0 block size: 512B configured, 4096B native gpt/data1 ONLINE 0 0 0 block size: 512B configured, 4096B native gpt/data2 ONLINE 0 0 0 block size: 512B configured, 4096B native gpt/data3 ONLINE 0 0 0 block size: 512B configured, 4096B native gpt/data4 ONLINE 0 0 0 block size: 512B configured, 4096B native gpt/data5 ONLINE 0 0 0 block size: 512B configured, 4096B native Is my best bet to do the following: 1) create a new pool on another server 2) transfer data to other pool 3) re-create existing pool 4) transfer data back Or are there options for changing block sizes in-place coming up at some point in the feature set I can wait for? Thanks! Octavian