Date: Thu, 9 May 2019 08:43:11 -0500 From: Karl Denninger <karl@denninger.net> To: freebsd-stable@freebsd.org Subject: Re: ZFS... Message-ID: <8fc532d3-19cb-2612-d83d-602eabe18bff@denninger.net> In-Reply-To: <20190509002809.GA81574@neutralgood.org> References: <08E46EBF-154F-4670-B411-482DCE6F395D@sorbs.net> <33D7EFC4-5C15-4FE0-970B-E6034EF80BEF@gromit.dlib.vt.edu> <A535026E-F9F6-4BBA-8287-87EFD02CF207@sorbs.net> <26B407D8-3EED-47CA-81F6-A706CF424567@gromit.dlib.vt.edu> <42ba468a-2f87-453c-0c54-32edc98e83b8@sorbs.net> <4A485B46-1C3F-4EE0-8193-ADEB88F322E8@gromit.dlib.vt.edu> <14ed4197-7af7-f049-2834-1ae6aa3b2ae3@sorbs.net> <453BCBAC-A992-4E7D-B2F8-959B5C33510E@gromit.dlib.vt.edu> <92330c95-7348-c5a2-9c13-f4cbc99bc649@sorbs.net> <d9086e22-fa73-1b01-d455-c32e0be70783@denninger.net> <20190509002809.GA81574@neutralgood.org>
index | next in thread | previous in thread | raw e-mail
[-- Attachment #1 --]
On 5/8/2019 19:28, Kevin P. Neal wrote:
> On Wed, May 08, 2019 at 11:28:57AM -0500, Karl Denninger wrote:
>> If you have pool(s) that are taking *two weeks* to run a scrub IMHO
>> either something is badly wrong or you need to rethink organization of
>> the pool structure -- that is, IMHO you likely either have a severe
>> performance problem with one or more members or an architectural problem
>> you *really* need to determine and fix. If a scrub takes two weeks
>> *then a resilver could conceivably take that long as well* and that's
>> *extremely* bad as the window for getting screwed is at its worst when a
>> resilver is being run.
> Wouldn't having multiple vdevs mitigate the issue for resilvers (but not
> scrubs)? My understanding, please correct me if I'm wrong, is that a
> resilver only reads the surviving drives in that specific vdev.
Yes.
In addition while "most-modern" revisions have material improvements
(very much so) in scrub times "out of the box" a bit of tuning makes for
very material differences in older revisions. Specifically maxinflight
can be a big deal given a reasonable amount of RAM (e.g. 16 or 32Gb) as
are async_write_min_active (raise it to "2"; you may get a bit more with
"3", but not a lot)
I have a scrub running right now and this is what it looks like:
Disks da2 da3 da4 da5 da8 da9 da10
KB/t 10.40 11.03 103 108 122 98.11 98.48
tps 46 45 1254 1205 1062 1324 1319
MB/s 0.46 0.48 127 127 127 127 127
%busy 0 0 48 62 97 28 31
Here's the current stat on that pool:
pool: zs
state: ONLINE
scan: scrub in progress since Thu May 9 03:10:00 2019
11.9T scanned at 643M/s, 11.0T issued at 593M/s, 12.8T total
0 repaired, 85.58% done, 0 days 00:54:29 to go
config:
NAME STATE READ WRITE CKSUM
zs ONLINE 0 0 0
raidz2-0 ONLINE 0 0 0
gpt/rust1.eli ONLINE 0 0 0
gpt/rust2.eli ONLINE 0 0 0
gpt/rust3.eli ONLINE 0 0 0
gpt/rust4.eli ONLINE 0 0 0
gpt/rust5.eli ONLINE 0 0 0
errors: No known data errors
Indeed it will be done in about an hour; this is an "automatic" kicked
off out of periodic. It's comprised of 4Tb disks and is about 70%
occupied. When I get somewhere around another 5-10% I'll swap in 6Tb
drives for the 4Tb ones and swap in 8Tb "primary" backup disks for the
existing 6Tb ones.
This particular machine has a spinning rust pool (which is this one) and
another that's comprised of 240Gb Intel 730 SSDs (fairly old as SSDs go
but much faster than spinning rust and they have power protection which
IMHO is utterly mandatory for SSDs in any environment where you actually
care about the data being there after a forced, unexpected plug-pull.)
This machine is UPS-backed with apcupsd monitoring it so *in theory* it
should never have an unsolicited power failure without notice but "crap
happens"; a few years ago there was an undetected fault in one of the
batteries (the UPS didn't know about it despite it being programmed to
do automated self-tests and hadn't reported the fault), power glitched
and blammo -- down it went, no warning.
My current "consider those" SSDs for similar replacement or size
upgrades would likely be the Micron units -- not the fastest out there
but plenty fast, reasonably priced, available in several different
versions depending on write endurance and power-protected.
--
Karl Denninger
karl@denninger.net <mailto:karl@denninger.net>
/The Market Ticker/
/[S/MIME encrypted email preferred]/
[-- Attachment #2 --]
0 *H
010
`He 0 *H
00 H^Ōc!5
H0
*H
01 0 UUS10U Florida10U Niceville10U
Cuda Systems LLC10U Cuda Systems CA1!0 U Cuda Systems LLC 2017 CA0
170817164217Z
270815164217Z0{1 0 UUS10U Florida10U
Cuda Systems LLC10U Cuda Systems CA1%0#U Cuda Systems LLC 2017 Int CA0"0
*H
0
h-5B>[;olӴ0~͎O9}9Ye*$ g !ukvʶLzN`jL>MD'7U 45CB+kY`bd~b*c3Ny-78ju]9H euέsӬDؽmgwER?&UURj'}9nWD i`XcbGz \gG=u%\Oi13ߝ4
K44pYQr]Ie/r0+eEޝݖ0C15Mݚ@JSZ(zȏ NTa(25DD5.l<g[[Z arQQ%Buȴ~~`IohRbʳڟu2MS8EdFUClCMaѳ !}ș+2k/bųE,n当ꖛ\(8 WV8 d]b yXw ܊:I39
00 U ]^§Q\ӎ0U #0T039N0b01 0 UUS10U Florida10U Niceville10U
Cuda Systems LLC10U Cuda Systems CA1!0 U Cuda Systems LLC 2017 CA @Ui0U 0 0U 0
*H
:P U!>vJ n io -#ן]Wyu jǑR̀Q
nƇ!G ѦF g\ yLx gw=OPycehf[ }ܷ['4ڝ\[p 6\o.B&JF" ZC{;*o*mcCcLY߾`
t*S!(` ]DHP5A~/NPp6=mhk밣'doA$86hm5ӚS@ j ެEgl
)0JG`%k35PaC?σ
׳HEt}!P㏏%*BxbQwaKG$6 h¦Mve; [o-Iی&
I,Tc ߎ#t wPA@l0P+KXBպT zGv;NcI3&J ĬUPNa?/%W6G۟N000 k#Xd\ =0
*H
0{1 0 UUS10U Florida10U
Cuda Systems LLC10U Cuda Systems CA1%0#U Cuda Systems LLC 2017 Int CA0
170817212120Z
220816212120Z0W1 0 UUS10U Florida10U
Cuda Systems LLC10U karl@denninger.net0"0
*H
0
T[I-ΆϏ dn;Å@שy.us~_ZG%<MYd\gvfnsa1'6Egyjs"C [{~ _K Pn+<*pv#Q+H/7[-vqDV^U>f%GX)H.|l`M(Cr >е͇6#odc"YljҦln8@5SA0&ۖ"OGj?UDWZ5 dDB 7k-)9Izs-JAv
J 6L$Ն1SmY.Lqw*SH;EF'DĦH]MOgQQ|Mٙג2Z9y@ y]}6ٽeY9Y2xˆ$T=e CǺǵbn֛{j |@LLt1[Dk5:$= ` M 00<+00.0,+0 http://ocsp.cudasystems.net:88880 U 0 0 `HB0U 0 U %0++03 `HB
&$OpenSSL Generated Client Certificate0 U %՞V =;bzQ0U #0]^§Q\ӎϡ01 0 UUS10U Florida10U Niceville10U
Cuda Systems LLC10U Cuda Systems CA1!0 U Cuda Systems LLC 2017 CA H^Ōc!5
H0 U 0karl@denninger.net0
*H
۠A0-j%--$% g2#ޡ1^>{K+uGE v1ş7Af&b&O; .;A5* U)ND2bF|\= ]<sˋL!wrw٧> YMÄ3\mWR hSv!_zvl? 3_ xU%\^#O*Gk̍YI_&Fꊛ@&1n } ͬ:{hTP3B.;bU8:Z=^Gw8 !k-@xE@i,+'Iᐚ:f hztX7/(hY` O.1}a`%RW^akǂpCAufgDix UTЩ/7}%=jnVZvcF<M=
2^GKH5魉
_O4ެByʈySkw=5@h.0z>
W1000{1 0 UUS10U Florida10U
Cuda Systems LLC10U Cuda Systems CA1%0#U Cuda Systems LLC 2017 Int CA k#Xd\ =0
`He E0 *H
1 *H
0 *H
1
190509134311Z0O *H
1B@f{#MbS 4Ѵ{Z2YAG:n 4( p6"~,R?0l *H
1_0]0 `He*0 `He0
*H
0*H
0
*H
@0+0
*H
(0 +7100{1 0 UUS10U Florida10U
Cuda Systems LLC10U Cuda Systems CA1%0#U Cuda Systems LLC 2017 Int CA k#Xd\ =0 *H
10{1 0 UUS10U Florida10U
Cuda Systems LLC10U Cuda Systems CA1%0#U Cuda Systems LLC 2017 Int CA k#Xd\ =0
*H
E42kLF ',
ا.D\D ӌ1Jz-?6 }ST${&0%r-=6pD: "ݴ씄R&Ǚˊl;N-E;#_No%r_Cǁ"CyS,g.ΨNQy-2I\n.s| |o`D=+soŅyaD48+gwAu7~SFh'[gOT4 ҸCjkҥU*S@ߎҲ|Vr[%,֏s1Z>A-.t;Vv5XMʊ\Ķl O7r(H=b:]Ju4Ii
_>;6wӀ!
R$/槣m6z0ͦQO{iݼ TfZ9 S{t/Ls ֊rЃ&`J
{/>(}./kY;2'Mob S' O:
home |
help
Want to link to this message? Use this
URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?8fc532d3-19cb-2612-d83d-602eabe18bff>