Date: Sat, 3 Aug 2013 18:50:52 -0500 From: Kevin Day <toasty@dragondata.com> To: Karl Denninger <karl@denninger.net> Cc: freebsd-fs@freebsd.org Subject: Re: ZFS option vfs.zfs.cache_flush_disable=1 Message-ID: <9F85D42E-D431-455E-8DFC-CEA7AC53F4BE@dragondata.com> In-Reply-To: <51FD867B.8070803@denninger.net> References: <51FD867B.8070803@denninger.net>
next in thread | previous in thread | raw e-mail | index | archive | help
[-- Attachment #1 --]
On Aug 3, 2013, at 5:38 PM, Karl Denninger <karl@denninger.net> wrote:
> Hi folks;
>
> I'm trying to run down a rather odd bit of ugly interaction between an
> ARECA 1680-IX adapter (in JBOD mode) and ZFS.
>
> The adapter has both a fair bit of cache memory on it and a BBU unit.
> If I turn off write-back caching ZFS performance goes straight into the
> toilet. With it on, however, it appears that cache flushes are being
> honored and, what's worse, when they come down the pipe they force all
> further I/O to the adapter to cease until the ENTIRE cache is flushed.
>
> This can and occasionally does lead to degenerate cases where very
> severe performance problems ensue.
Hey, Karl!
We had similar problems with the 3ware 9650SE cards. Cache flush operations would just kill performance to the entire array, even if ZFS was only being used on one volume on that card.
We've got vfs.zfs.cache_flush_disable=1 set on piles of servers, and so far we've only had one bit of weird ZFS corruption that I'm not even sure was related to that setting. While this isn't answering your question, I can say that having that flag enabled isn't causing a ton of corruption for us even with random hard reboots happening sometimes.
I will say though that the LSI 2008 based cards are substantially faster and less problematic for ZFS than anything else we've used. The "drives getting reordered" thing seems to only be happening on a few systems with a specific SATA/SAS backplane that doesn't appear to deterministically initialize things. gpart labels make that far less of a problem though, if you can use them. Failing that, it is possible to use "camcontrol inquiry daXX" to see the serial number of the drives, which you can then label the fronts of the drives with. Not as pretty, but being able to have a naked JBOD interface without controllers treating every drive as it's own 1-drive RAID0 volume is so much easier that I'll put up with a lot to have it.
-- Kevin
[-- Attachment #2 --]
0 *H
01 0 + 0 *H
/00Šq_Mtq40
*H
0{1 0 UGB10U Greater Manchester10U Salford10U
Comodo CA Limited1!0 U AAA Certificate Services0
040101000000Z
281231235959Z01 0 UUS1 0 UUT10USalt Lake City1 0 U
The USERTRUST Network1!0 U http://www.usertrust.com1604U-UTN-USERFirst-Client Authentication and Email0"0
*H
0
9}A;bF7`u9eJGHjM5BI/ |1Nd.)բdąQ5yNh{ zɤ2 O0 nFxoY^/m/묡j.g5y iF͠ v:z'[=s "HaLi.1 ,CZ qYں
gT:
we tbh ~GeMW(t40b0, '0#0 U #0
#>)00 U g}ĝ&p KPH|=n}0U 0U 00 U %0++0U
00U 0{U t0r08642http://crl.comodoca.com/AAACertificateServices.crl06420http://crl.comodo.net/AAACertificateServices.crl0 `HB0
*H
<~ v9<O ૄ]Te;m |7,%T_!7OT klE`-QLf<J?VvÂOl atG@We"'gOWdZٍ /i)J /LQFĊ7N 1hǞċ~2hD*Q`Mt:C29V:RAC3'9N&9≸])&A곛wuʵeJc>D^s 00mOj3""2zq0
*H
01 0 UUS1 0 UUT10USalt Lake City1 0 U
The USERTRUST Network1!0 U http://www.usertrust.com1604U-UTN-USERFirst-Client Authentication and Email0
110428000000Z
200530104838Z01 0 UGB10UGreater Manchester10USalford10U
COMODO CA Limited1907U0COMODO Client Authentication and Secure Email CA0"0
*H
0
[KW^/@ȣSX_fe2N2}UxLUB'qi2@'Vbqi c^`ʢAj HmeC*.+c8w߱ڂ2jgo \5Tq
7
PSl Y1 LR@[HhJ$ :q_㬿;%qh=XF<hmz!W42~JRrd&N`ohQcB}"cөΞD\[5 K0G0 U #0g}ĝ&p KPH|=n}0 U zN t[xcd'/[y{0U 0U 0 0U
00U 0XU Q0O0MKIGhttp://crl.usertrust.com/UTN-USERFirst-ClientAuthenticationandEmail.crl0t+h0f0=+01http://crt.usertrust.com/UTNAddTrustClient_CA.crt0%+0http://ocsp.usertrust.com0
*H
־xWUm3DRB
JAIZҭsn>&|L0(B<%>
u=9fѡMo(l tZڱuz/y VtCr`9 G:eH<=%`I?C
3_н`j;:< I3B)93i.EMiڀ =]|Gm]W0KID~y83:]&XaU!ՙC@B0Ұun0,0 7Ca`)̀0
*H
01 0 UGB10UGreater Manchester10USalford10U
COMODO CA Limited1907U0COMODO Client Authentication and Secure Email CA0
130616000000Z
140616235959Z0&1$0" *H
toasty@dragondata.com0"0
*H
0
-`ghX/9+ˈe`?l/ E^~T׀/q+5SzSrIuert`:"k_s{&
0T
kˢbW ieK;2 LX8w.YpI!3|M19 Y2vGrA ҽ5yX^0
@JbfBќ`
T ZO$mx1}LN0Rw 00 U #0zN t[xcd'/[y{0 U p =#P}L:wqX0U 0 U 0 0 U %0+ +10 `HB 0FU ?0=0; +10+0)+ https://secure.comodo.net/CPS0WU P0N0LJHFhttp://crl.comodoca.com/COMODOClientAuthenticationandSecureEmailCA.crl0+|0z0R+0Fhttp://crt.comodoca.com/COMODOClientAuthenticationandSecureEmailCA.crt0$+0http://ocsp.comodoca.com0 U 0toasty@dragondata.com0
*H
i qk4Dlse-8 uy:1:ߩ( w}tRƼHyT 4Y=GeKr:/ܜ1n*tښ6ڇڿ3D.~fm5Q(Yœ!@'=vn5i뷹 K^#MF0=~<h GgһKR}KrBai1cvqm3RNSeEx\G4
=FwɺL4rI U10001 0 UGB10UGreater Manchester10USalford10U
COMODO CA Limited1907U0COMODO Client Authentication and Secure Email CA 7Ca`)̀0 + 0 *H
1 *H
0 *H
1
130803235053Z0# *H
1
4$s%XɮQ-t#0 +71001 0 UGB10UGreater Manchester10USalford10U
COMODO CA Limited1907U0COMODO Client Authentication and Secure Email CA 7Ca`)̀0 *H
101 0 UGB10UGreater Manchester10USalford10U
COMODO CA Limited1907U0COMODO Client Authentication and Secure Email CA 7Ca`)̀0
*H
# 75a I'@p J%Z1Dkl?-VYd}v\?silZa8S&{id
D x' qb]fҹU䂥AvG yR! FyI)<jl.:¿&;x[O rR?=ʆ8:_#] ݣ^u]=-6-[MYY yȔTtCf^9L7 ͉6xH
5 !m ,
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?9F85D42E-D431-455E-8DFC-CEA7AC53F4BE>