Date: Sat, 5 Aug 2017 10:49:39 -0700 From: Freddie Cash <fjwcash@gmail.com> To: "Eugene M. Zheganin" <emz@norma.perm.ru> Cc: FreeBSD Stable <freebsd-stable@freebsd.org>, FreeBSD Filesystems <freebsd-fs@freebsd.org> Subject: Re: a strange and terrible saga of the cursed iSCSI ZFS SAN Message-ID: <CAOjFWZ6m_nbyG4r0LYw4bGb2BXQoo-zwi_31coBxBpZZ-BFtTw@mail.gmail.com> In-Reply-To: <CAOjFWZ6wW7GVD8UzZnRo3O-d0-e4d=xRu=yRz5RfgqfqZbeJ4w@mail.gmail.com> References: <1bd10b1e-0583-6f44-297e-3147f6daddc5@norma.perm.ru> <CAOjFWZ6wW7GVD8UzZnRo3O-d0-e4d=xRu=yRz5RfgqfqZbeJ4w@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On Aug 5, 2017 10:09 AM, "Eugene M. Zheganin" <emz@norma.perm.ru> wrote: And I want to also ask - what happens when the system's memory isn't enough for deduplication - does it crash, or does the problem of mounting the pool appear, like some articles mention ? Can't really help with the other issues, but can speak to this one. "Insufficient" RAM for dedupe won't be an issue during normal operation, reading and writing to the pool. Performance will slow down over time as the DDT increases in size taking up ARC/L2ARC space, possibly even spilling over into raw disk reads. Where the issues crop up is during snapshot deletion and filesystem destruction. Those operations can run the system out of usable kmem and lock things up. (Deletions cause a lot of churn in the DDT.) On reboot, the pool import process will continue the deletion process, which will run the system out of kmen locking it up. Rinse and repeat for a week until the deletion process completes. L2ARC helps, but not nearly as much as adding more RAM! Device replacement will also be an issue as the resilver process will be extremely slow (at least, it is for us with a very full, very fragmented pool; equally full pools without d we dedupe resilver very quickly). And don't try to replace two drives in separate vdevs unless you want to increase the resilver time by a huge factor. This is from experience with a 24-drive system with only 48 GB of RAM, and a 90-disk system with 128 GB of RAM. Both with dedupe enabled. Both running at about 90% full, very fragmented as they create and delete snapshots of all filesystems on a daily basis. They've been running file about 5 years now, possibly longer. Cheers, Freddie
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAOjFWZ6m_nbyG4r0LYw4bGb2BXQoo-zwi_31coBxBpZZ-BFtTw>