Date: Wed, 8 May 2019 09:53:49 -0700 From: Freddie Cash <fjwcash@gmail.com> To: FreeBSD Stable <freebsd-stable@freebsd.org> Subject: Re: ZFS... Message-ID: <CAOjFWZ4hb=Sixj4B6TScu6wWmcvXHs3_f7P38svwZKGP4ri3UQ@mail.gmail.com> In-Reply-To: <d9086e22-fa73-1b01-d455-c32e0be70783@denninger.net> References: <30506b3d-64fb-b327-94ae-d9da522f3a48@sorbs.net> <70fac2fe3f23f85dd442d93ffea368e1@ultra-secure.de> <70C87D93-D1F9-458E-9723-19F9777E6F12@sorbs.net> <CAGMYy3tYqvrKgk2c==WTwrH03uTN1xQifPRNxXccMsRE1spaRA@mail.gmail.com> <5ED8BADE-7B2C-4B73-93BC-70739911C5E3@sorbs.net> <d0118f7e-7cfc-8bf1-308c-823bce088039@denninger.net> <2e4941bf-999a-7f16-f4fe-1a520f2187c0@sorbs.net> <20190430102024.E84286@mulder.mintsol.com> <41FA461B-40AE-4D34-B280-214B5C5868B5@punkt.de> <20190506080804.Y87441@mulder.mintsol.com> <08E46EBF-154F-4670-B411-482DCE6F395D@sorbs.net> <33D7EFC4-5C15-4FE0-970B-E6034EF80BEF@gromit.dlib.vt.edu> <A535026E-F9F6-4BBA-8287-87EFD02CF207@sorbs.net> <26B407D8-3EED-47CA-81F6-A706CF424567@gromit.dlib.vt.edu> <42ba468a-2f87-453c-0c54-32edc98e83b8@sorbs.net> <4A485B46-1C3F-4EE0-8193-ADEB88F322E8@gromit.dlib.vt.edu> <14ed4197-7af7-f049-2834-1ae6aa3b2ae3@sorbs.net> <453BCBAC-A992-4E7D-B2F8-959B5C33510E@gromit.dlib.vt.edu> <92330c95-7348-c5a2-9c13-f4cbc99bc649@sorbs.net> <d9086e22-fa73-1b01-d455-c32e0be70783@denninger.net>
next in thread | previous in thread | raw e-mail | index | archive | help
On Wed, May 8, 2019 at 9:31 AM Karl Denninger <karl@denninger.net> wrote: > I have a system here with about the same amount of net storage on it as > you did. It runs scrubs regularly; none of them take more than 8 hours > on *any* of the pools. The SSD-based pool is of course *much* faster > but even the many-way RaidZ2 on spinning rust is an ~8 hour deal; it > kicks off automatically at 2:00 AM when the time comes but is complete > before noon. I run them on 14 day intervals. > Damn, I wish our scrubs took 8 hours. :) Storage pool 1: 90 drives in 6-disk raidz2 vdevs (mix of 2 TB and 4 TB SATA). 45 hours to scrub. Storage pool 2: 90 drives in 6-disk raidz2 vdevs (mix of 2 TB and 4 TB SATA). 33 hours to scrub. Storage pool 3: 24 drives in 6-disk raidz2 vdevs (mix of 2 TB and 4 TB SATA). 134 hours to scrub. Storage pool 4: 24 drives in 6-disk raidz2 vdevs (mix of 1 TB, 2 TB, 4 TB SATA). Dedupe enabled. 256 hours to scrub. Storage pool 5: 90 drives in 6-disk raidz2 vdevs (mix of 2 TB and 4 TB SATA). Dedupe enabled. Takes about 6 weeks to resilver a drive, and it's constantly resilvering drives these days as it's the oldest pool, and all the drives are dying. :D Pools 1, 3, and 4 are in DC1. Pools 2 and 5 are in DC2 across town. Pool 1 sends snapshots to pool 2. Pools 3 and 4 send snapshots to pool 5. These pools are highly fragmented. :) > If you have pool(s) that are taking *two weeks* to run a scrub IMHO > either something is badly wrong or you need to rethink organization of > the pool structure -- that is, IMHO you likely either have a severe > performance problem with one or more members or an architectural problem > you *really* need to determine and fix. If a scrub takes two weeks > *then a resilver could conceivably take that long as well* and that's > *extremely* bad as the window for getting screwed is at its worst when a > resilver is being run. > Thankfully, ours are strictly storage for backups of other systems, so as long as the nightly backups complete successfully before 6 am, we're not worried about performance. :) And we do have plans to replace pools 2 and 5 to remove dedupe from the equation. There's not a lot we can do about the fragmentation issue, as these servers all run rsync backups from 200-odd other servers, and remove the oldest snapshot every night. So, while a 2-week scrub may be horrible, it all depends on the use-case. If these were direct storage systems for in-production servers, then I'd be worried. But as redundant backup systems (3 copies of everything, in 3 separate locations around the city), I'm not too worried. Yet. :D -- Freddie Cash fjwcash@gmail.com
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAOjFWZ4hb=Sixj4B6TScu6wWmcvXHs3_f7P38svwZKGP4ri3UQ>