Date: Fri, 19 May 2023 18:37:34 +0200 From: Jan Bramkamp <crest@rlwinm.de> To: freebsd-fs@freebsd.org Subject: Re: Expanding storage in a ZFS pool using draid Message-ID: <9aa8b4d7-cd0c-59f0-6a56-3b37e19a5022@rlwinm.de> In-Reply-To: <CAOjFWZ7Fcnu=MfKY57gWz1g%2BFiaTFK-6-am5Wy0afDx6X2p_xg@mail.gmail.com> References: <CAOjFWZ7Fcnu=MfKY57gWz1g%2BFiaTFK-6-am5Wy0afDx6X2p_xg@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On 03.05.23 18:08, Freddie Cash wrote: > I might be missing something, or not understanding how draid works > "behind the scenes". > > With a ZFS pool using multiple raidz vdevs, it's possible to increase > the available storage in a pool by replacing each drive in the raidz > vdev. Once the last drive is replaced, either the extra storage space > appears automatically, or you run "zpool online -e <poolname> <disk>" > for each disk. > > For example, if you create a pool with 2 raidz vdevs using 6x 1 TB > drives per vdev you'll end up with ~ 10 TB of space available to the > pool. Later, you can replace all 6 drives in one raidz vdev with 2 TB > drives, and get an extra 5 TB of free space in the pool. Later, you > can replace the 6 drives in the other raidz vdev with 2 TB drives, and > get another 5 TB of free space in the pool. > > We've been doing this for years, and it works great. > > When draid became available, we configured our new storage pools using > that instead of multiple raidz vdevs. One of the pools uses 44x 2 TB > drives, configured in a draid pool using: > mnparity: 2 > draid_ndata: 4 > draid_ngroups: 7 > draid_nspares: 2 > > IIUC, this means the drives are configured in 7 groups of 6, using 4 > drives for data and 2 for parity in each group, with 2 drives > configured as spares. > > The pool works great, but we're running out of space. So, we replaced > the first 6 drives in the pool with 4 TB drives, expecting to get an > extra 4*4=16 TB of free space in the pool. However, to our great > surprise, that is not the case! Total storage capacity of the pool > has not changed. Even after running "zpool online -e" against each of > the 4 TB drives. > > Do we need to replace EVERY drive in the draid vdev in order to get > extra free space in the pool? Or is there some other command that > needs to be run to tell ZFS to use the extra storage space available? > Or ... ? > > Usually, we just replace drives in groups of 6, going from 1 TB to 2 > TB to 4 TB as needed. Having to buy 44 (or 88 in our other > draid-using storage server) and replace them all at once is going to > be a massive (and expensive) undertaking! That might be enough to > rethink how we use draid going forward. :( This is not going to work with dRAID. All devices need to be the same size (it uses the size of the smallest device on all devices.).
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?9aa8b4d7-cd0c-59f0-6a56-3b37e19a5022>