Date: Mon, 31 Mar 2014 11:53:21 +0800 From: Marcelo Araujo <araujobsdport@gmail.com> To: Rick Macklem <rmacklem@uoguelph.ca> Cc: FreeBSD Filesystems <freebsd-fs@freebsd.org>, Alexander Motin <mav@freebsd.org> Subject: Re: RFC: How to fix the NFS/iSCSI vs TSO problem Message-ID: <CAOfEmZhUtUhX_OOGV6R4ogTJPTL0cEPGDv3WgPM2M3hiPs9mxQ@mail.gmail.com> In-Reply-To: <1377879526.2465097.1396046676367.JavaMail.root@uoguelph.ca> References: <CAOfEmZjxxWtYO9BAg1i_k5k-eD8jR%2BmuVPZGauOdOsxdRd%2B=JA@mail.gmail.com> <1377879526.2465097.1396046676367.JavaMail.root@uoguelph.ca>
index | next in thread | previous in thread | raw e-mail
[-- Attachment #1 --] Hello Rick, We have made couple more benchmarks here with additional options, such like '64 threads' and readahead=8. Now, we add nfsstat and netstat -m into the table. Here attached is the full benchmark, and I can say, this patch really improved the read speed. I understand your concern about add more one sysctl, however maybe we can do something like ZFS does, if it detect the system is AMD and have more than X of RAM it enables some options by default, or a kind of warning can be displayed show the new sysctl option. Of, course other people opinion will be very welcome. Best Regards, 2014-03-29 6:44 GMT+08:00 Rick Macklem <rmacklem@uoguelph.ca>: > Marcelo Araujo wrote: > > 2014-03-28 5:37 GMT+08:00 Rick Macklem <rmacklem@uoguelph.ca>: > > > > > Christopher Forgeron wrote: > > > > I'm quite sure the problem is on 9.2-RELEASE, not 9.1-RELEASE or > > > > earlier, > > > > as a 9.2-STABLE from last year I have doesn't exhibit the > > > > problem. > > > > New > > > > code in if.c at line 660 looks to be what is starting this, which > > > > makes me > > > > wonder how TSO was being handled before 9.2. > > > > > > > > I also like Rick's NFS patch for cluster size. I notice an > > > > improvement, but > > > > don't have solid numbers yet. I'm still stress testing it as we > > > > speak. > > > > > > > Unfortunately, this causes problems for small i386 systems, so I > > > am reluctant to commit it to head. Maybe a variant that is only > > > enabled for amd64 systems with lots of memory would be ok? > > > > > > > > Rick, > > > > Maybe you can create a SYSCTL to enable/disable it by the end user > > will be > > more reasonable. Also, of course, it is so far safe if only 64Bits > > CPU can > > enable this SYSCTL. Any other option seems not OK, will be hard to > > judge > > what is lots of memory and what is not, it will depends what is > > running > > onto the system. > > > I guess adding it so it can be optionally enabled via a sysctl isn't > a bad idea. I think the largest risk here is "how do you tell people > what the risk of enabling this is"? > > There are already a bunch of sysctls related to NFS that few people > know how to use. (I recall that Alexander has argued that folk don't want > worry about these tunables and I tend to agree.) > > If I do a variant of the patch that uses m_getjcl(..M_WAITOK..), then > at least the "breakage" is thread(s) sleeping on "btallo", which is > fairly easy to check for, althouggh rather obscure. > (Btw, I've never reproduced this for a patch that changes the code to > always use MJUMPAGESIZE mbuf clusters. > I can only reproduce it intermittently when the patch mixes allocation of > MCLBYTES clusters and MJUMPAGESIZE clusters.) > > I've been poking at it to try and figure out how to get > m_getjcl(..M_NOWAIT..) > to return NULL instead of looping when it runs out of boundary tags (to > see if that can result in a stable implementation of the patch), but > haven't had much luck yet. > > Bottom line: > I just don't like committing a patch that can break the system in such an > obscure way, even if it is enabled via a sysctl. > > Others have an opinion on this? > > Thanks, rick > > > The SYSCTL will be great, and in case you don't have time to do it, I > > can > > give you a hand. > > > > I'm gonna do more benchmarks today and will send another report, but > > in our > > product here, I'm inclined to use this patch, because 10~20% speed up > > in > > read for me is a lot. :-) > > > > Thank you so much and best regards, > > -- > > Marcelo Araujo > > araujo@FreeBSD.org > > _______________________________________________ > > freebsd-net@freebsd.org mailing list > > http://lists.freebsd.org/mailman/listinfo/freebsd-net > > To unsubscribe, send any mail to > > "freebsd-net-unsubscribe@freebsd.org" > > > -- Marcelo Araujo araujo@FreeBSD.org [-- Attachment #2 --] %PDF-1.5 % 1 0 obj <</Type/Catalog/Pages 2 0 R/Lang(zh-TW) /StructTreeRoot 109 0 R/MarkInfo<</Marked true>>>> endobj 2 0 obj <</Type/Pages/Count 42/Kids[ 3 0 R 13 0 R 17 0 R 19 0 R 21 0 R 23 0 R 25 0 R 27 0 R 29 0 R 31 0 R 33 0 R 35 0 R 37 0 R 39 0 R 41 0 R 43 0 R 45 0 R 47 0 R 51 0 R 55 0 R 59 0 R 61 0 R 63 0 R 65 0 R 67 0 R 69 0 R 71 0 R 73 0 R 75 0 R 77 0 R 79 0 R 81 0 R 83 0 R 85 0 R 87 0 R 89 0 R 91 0 R 93 0 R 95 0 R 97 0 R 102 0 R 107 0 R] >> endobj 3 0 obj <</Type/Page/Parent 2 0 R/Resources<</Font<</F1 5 0 R/F2 7 0 R/F3 9 0 R>>/XObject<</Image11 11 0 R>>/ProcSet[/PDF/Text/ImageB/ImageC/ImageI] >>/MediaBox[ 0 0 612 792] /Contents 4 0 R/Group<</Type/Group/S/Transparency/CS/DeviceRGB>>/Tabs/S/StructParents 0>> endobj 4 0 obj <</Filter/FlateDecode/Length 1703>> stream xYmo6nby|44):݆؇b(TGybv)9ll$||t/i6=>{有/v;⊙pmX%[Wͳ鄽z{Xg%-[Te^M'` yatL`ʲLir:Mɇ_WWȐ.ȸC3JuRMgJ$,K֫zYxlHGM z [So<W1&F]GfE *Y*U)S||Z8w]݅dSERS#H!*;|tq ,0YzUr800ibв4ܘMKzkb':r]yhUMNO!kۇGZ}wr'wZ_HB!ɐjkςZH)*tS:xd.m=%7T.B9c +reZ@Z7-Cn_Ls.u|K]z fu !7m@\m/e)(/_ 2տKrɉҧ8oD7ȿhR_2dvcf$.ɝ6qw@Wi݊Ia%> -wuPo1#32Lg9g7]n:So>c.X}ؼ"&}P2Ňm0pPʭ6.éW$sz"ϱrCX/,֗Zʾ,-Ed>3,XaK@ܲ% fV*9ܕkzj^8NVCj88榯kF`A# BP}WPpnQF5vP5Nvc,]ڒrӮ1H^α`cKn=چq$@^w\KkՎs ڵo1Z8m,g)% qRHCYW:~<ZGuU /Capd7hX>%rI%=v-|*uBI遬$U d9;]W'8Jb8,Ru#Ͼ28`Gn`ߐȃYRJARYH*-C6 K15NƀWXgܜ*yiL_#Ф ɇZʓ8;}Ƴ>*f 2xDr$V2.^V}}=ܸGwXRYM]iZ_YKw'_S#ԫEr>渙GXF~Cv'T"8^$_<nMZk2OٳZSpNDLzڗUbc9ǏԂ[!N .KF7 ,?_v=qcNkyj08:YQ.ﺋ8:5vB1{Ț%7MI :\4{(а,)C>Ma~>4?/7)> &xb zXuIv-l0nQoBqsx(Q'2P eQp1TR #GT˘c*(, P(;\fxTlUf,!L\fMw%8%c 2*fU: K_3qneypmsJw>n*{omWFY endstream endobj 5 0 obj <</Type/Font/Subtype/TrueType/Name/F1/BaseFont/ABCDEE+Trebuchet#20MS,Bold/Encoding/WinAnsiEncoding/FontDescriptor 6 0 R/FirstChar 32/LastChar 121/Widths 3113 0 R>> endobj 6 0 obj <</Type/FontDescriptor/FontName/ABCDEE+Trebuchet#20MS,Bold/Flags 32/ItalicAngle 0/Ascent 939/Descent -205/CapHeight 737/AvgWidth 474/MaxWidth 1349/FontWeight 700/XHeight 250/StemV 47/FontBBox[ -219 -205 1130 737] /FontFile2 3114 0 R>> endobj 7 0 obj <</Type/Font/Subtype/TrueType/Name/F2/BaseFont/Arial/Encoding/WinAnsiEncoding/FontDescriptor 8 0 R/FirstChar 32/LastChar 122/Widths 3115 0 R>> endobj 8 0 obj <</Type/FontDescriptor/FontName/Arial/Flags 32/ItalicAngle 0/Ascent 905/Descent -210/CapHeight 728/AvgWidth 441/MaxWidth 2710/FontWeight 400/XHeight 250/Leading 33/StemV 44/FontBBox[ -665 -210 2046 728] >> endobj 9 0 obj <</Type/Font/Subtype/TrueType/Name/F3/BaseFont/Arial,Bold/Encoding/WinAnsiEncoding/FontDescriptor 10 0 R/FirstChar 32/LastChar 122/Widths 3116 0 R>> endobj 10 0 obj <</Type/FontDescriptor/FontName/Arial,Bold/Flags 32/ItalicAngle 0/Ascent 905/Descent -210/CapHeight 728/AvgWidth 479/MaxWidth 2628/FontWeight 700/XHeight 250/Leading 33/StemV 47/FontBBox[ -628 -210 2000 728] >> endobj 11 0 obj <</Type/XObject/Subtype/Image/Width 622/Height 180/ColorSpace/DeviceRGB/BitsPerComponent 8/Interpolate false/SMask 12 0 R/Filter/FlateDecode/Length 7053>> stream x}Uyq~DT*T FQ(87'UVQ#bHq~2DieF[%:64If?eu9}/g{<Ü:k}8T*B٭kt-w8꽛/C'B;zfXrDrc&%*N`RDB'0)xII"ʁ$P L `RHr&o0)I$T9p7$8ILJ U &%*N`RDB'0)xII"ʁMt]6fl]v9W\g}i-/j'q1G:N퓧]egɕvze+[*#$b~ӶF%Nbڍ]7ȴ|eG>dp81q*_o}@3}u_-j-/I6|}*c=V̖8#Âl=5_shVkkۻ>ڻj&6Џz?嫟rMnrR_eDSt]քx]he2Kw4Ef[Q(`ŗZZm7X\*䬃exAbﺦ㆖rQȤ+Xmnc&Cm}OпAhךݦIՕЬ:u?^&%Kj_cfWm,`I 0U4 m4l,W0T5Ͻ65{5ZP4)ek֤奣DI_)ӭߴM#m|}G|I ;v%+VWaiIfQHڟn0)I$=bRgU"Txb] ;hG2uXR)ϪZ̢Jih2HXv~T؍--뮝{P:2K5ROڱ5+\QmdRc6pݬz9_!4D;&*qvՎkXj=8Rd6& N֖p.I;xhF u7ؤ/ky(=hRhҮ$2B]KlH[1\OftkՆ.T?֤zƚJѷH֫{LcR0<նζW`0)I$oRU9՜?suU*QSTo.]^5PC2}8[kRL)OERaU^dڍusLT_|դh+T~dnq_W5M'Lt2^k܉_/囧 '"d.'Zj1VjImi`8&%wۍjK"&1"jf>-v6d,wO+"IqY5Ӻ[]w.oL_qG`Ako$ӏ$]Z_c`[Ү^2ݴLrNT75sv7ߤv!5>acḟLHӥIU+=c #y+b@&g3AUJ]&UQ %N>7#SkT3xI-:'vǤ y$ku.B3WOƆVjʎ 9ÚԽNI.m]ֽE;Cz%^Z?ګß{.$ݸ"fFuRJS5w|cR;]CPk@Q"\M*m-zϢڹvo;\_5,hǿ>F9&^;QdQu@>$6*b~Wεal#Jn"S'ZJw5t~ucR9R6͢5IOy2ofỏ}sIkMjg\r]!{J?>ȤV0퓉;C'IHӳ&/b_%:3fvF&{5}'v]SdI!_w}>Ti`RϤv"'\s3PvAp^/t uRפ6<?vB\6쮶Er?_Bmd$I͕9ELI5*~uMy;&3{34xB9gw\[FU*mKG3~vq|ڌu.W},tb:i[xXa@X:ԡQ8*êrtP~ᕆ%{.kc& hXvip5s<v2`RHz֤]1EEF3N?3i|=P·ˡRzP`{'bm<-瘿RS7zewuqT&M*I7ZyuPtv?~O2.gTofcdU7R2vpZkq?ړ<[ͻ3]LN$yVs0)I$9UN-[ĭAsNuPy(M*SBY2KgKqUt0ˍV;ӕJ0]~}]FxII"ʁ$P L `RHr&o0)I$T9p7$8ILJ U &%*N`RDB'0)xII"ʁ$P L `RHr&o0)I$T9p7$8ILJ U &%*N`RDB'0)xII"ʁ$P L `RHr&o0)I$T9p7$8ILJ U &%*N`RDB'0)xII"ʁ$P L `RHr&o0)I$T9p7$8ILJ U &%*N`RDB'0)xII"ʁ$P L `RHr&o0)I$T9p7$8ILJ U &%*N`RDB'0)xII"ʁ$P L `RHr&o0)I$T9p7$8ILJ U &%*N`RDB'0)xII"ʁ$P L `RHr&o0)I$T9p7$8ILJ U &%*N`RDB'0)xII"ʁ$P L `RHr&o0)I$T9p7$8ILJ U &%*N`RDB'0)xII"ʁ$P L `RHr&o0)I$T9p7$8ILJ U &%*N`RDB'0)xII"ʁ$P L `RHr&o0)I$T9p7$8IWݨdYas1BZrHͤKVn_i" DHGj&p rY[0)I$T9}i%ܿޅL-$D>0)&*L IAB!0))orPLJʛD>0){U>^qPLJʛnT{[gB!0))oQlryoG8w̨}V}L IW/cGA2gPLJLIPLJLIPLJLIPLJLIPLJLIPLJLIPLJLIPLJLIPLJLIPLJLIPLJLIPLJLIPLJLIPLJLI(I_ly.dcj:7 &%卫I?zMJ-?K5i%L6j0j=M۞&uc-7o%]vYç^+LJ`LΫϨ{[6t)9u[gxnzď[6>I(I?>lvYy{ؤ=ʹ<m6S' csZ.}~m4Ɍ`RRw4#ԉaJX|mm4ɜӚ\=L=JjRԬ˳-1g_~ՔfZj;b`gIIyo-[yN{o&=h٪O zQlh=bsnj¤{\|nMtsN:8[nC.ٰy}]kȹW=6/5E^57Vvs{kH[L]\֪헿~jz%:WпiֈQeR" YsfczkNn\U/iZOjYd۰g6R1}mj}߮&~,iTŧmZ>fF;6?ܛZЃ-kӏy/]=~ؿz=z)fcn+ϭIJ2^WcƱl|lRpKOi֮15HG?@ʪY)MzJM&,,Bo1fɀB>9l憉a8Hw±a>.87_?J8cRkȤ!JmƤ`[[?4$ɭ֧f7f4N7-:x]l01'xjΌryJD&3W:[3&`Jm,~zӺJU IsI['LZ*ɧMj_;kjj./iFvF5&/`{kMg+Ujqҥ&d֭clY>|7(Jf1ZsmUmCjǤksTk:@g({jܭ5ǤOIƵ_d#5v\x^{^6cҴ4$zjD9xѕHcRs뜙4,\ `_vL~<?jdzء+fοq)~43&cJskƤ=D'GR0WH36i|sFtMᛚ0tp|CLC%M0,ib8*WtL)Zxl.rj \'au<hǎ?[C;{:)!}$]ʤr9+h殩֨ЃaoҬިqW<FRgd^pA2l{3ý\'3JskƤ\{mcRiEXqULYCNZfMµ_BMѿlS*Ұ1`'ujcنh\i{0cR Tx5D5юO*>#&~SMrژ{~gݿ+Yi[c=᪥iHuMfɏBYI5hǀ-Z,qz>Iܚ1t\<vIm+4V<mRW52&VmWj`5ӚănsLjlH`ܕX|cvVH9\;>숁sgsܻKlr+:ʎ69&(R*.M*/-]ԍ6jS^'pUИ4Ť{\|n]T#Jt6Ր;$)4T5d7iUtLj&!va֤ZVj({J21)ɿh'.XlmcR nTE7ζolO++aϙ9ÿ7b=ft>.MzwoW7CI)͍ϣ=HLjW93W˲2_QȤ`<ж- wedIK2k'k֙Qe㷘42-_lȤķgm/~JCN7;:t iՉ^?ϯY3F&eLg֥̕I%vcj|ǑLW>{ҥIN86̒F&(OTOIn}q/(dRm84>[4Տ&P1j3I;l6&%}0v].F&5K~3myc"4o_%)fōf')5WݧvLƤNJs7\&#| 'w_58a`(ߤe}.8w{5 Ի[pg͙Yפ6SZbr'[3_QԤ0Y aCԿ m8.o50W)UMZ=4*k=HcǟVry6&%}*=mo=ԓj5ԤI?ye NRoKڭGvӯO8u@5rJJ&|{FFJ+7tԋ+g)IWoX)dH}C!QWM`ԏbo8N:iKM:rZƧI7zz/ &[VئY= mC䬰Z1+̭VU3kE,ğ,,j0`Rw&8alM2ڬo_R۲ASk*\9W:x$(isOR𰪺ՔaCmE9LifWYLAL -0i7PLJLIPLJLIPLJLIPLJLIPLJLIPLJLIPLJLIPLJLIPLJLIPLJLIPLJLIPLJLIPLJLIPLJLIPLJLIPLJʛnK%>gPLJʛnTz[k7\%B!0))oW$S}%+VB!0))orPLJʛD>0)&*L IAB!0))orHӤW̘*BLh|ѽDHGj&Oo+5ä$$RI;[fcRFҩr˸>=B/DB'0)xII"ʁ$P L `RHr&o0)I$T9p"T˦nkI*NP!esi endstream endobj 12 0 obj <</Type/XObject/Subtype/Image/Width 622/Height 180/ColorSpace/DeviceGray/Matte[ 0 0 0] /BitsPerComponent 8/Interpolate false/Filter/FlateDecode/Length 973>> stream x/lg,((00((4((((((0xwv+m<5ݾ'E#t;맏dɵg:z=Zt<tf該Q9j#Gm䨍6rFQ9j#Gm䨍6rFQ9j#Gm䨍6rFQ9j#Gm䨍6rFQ9j#Gm䨍6rFQ9j#Gm䨍6rFQ9j#Gm䨍6rFQ9j#Gm䨍6rFQ9j#Gm䨍6rFQ9j#Gm䨍6rFQ9j#Gm䨍6rFQ9j#Gm䨍6rFQ9j#Gm䨍6rFQ9j#Gm䨍6rFQ9j#Gm䨍$tf該]: ]zl6r:L'wO^[+;sgQ):kzH.k該]r~Gs=+eu>~WWӢ~zlzPʫzWE+g};kϥo$cmZ[֛vyRnWet{Qjm^F˪XN}vLmt&ۗEU[XۓvFjcCnhl]Azlgm'u^-2ZnvPҏ5COyy:o_6ju=;h*Smӽv1t.>>~ǧOW{sF].wgSe;8"Gm䨍|C7b該6rFQ9j#Gm䨍6rFQ9j#Gm䨍6rFQ9j#Gm䨍6rFQ9j#Gm䨍6rFQ9j#Gm䨍F=> endstream endobj 13 0 obj <</Type/Page/Parent 2 0 R/Resources<</Font<</F1 5 0 R/F2 7 0 R/F4 15 0 R>>/ProcSet[/PDF/Text/ImageB/ImageC/ImageI] >>/MediaBox[ 0 0 612 792] /Contents 14 0 R/Group<</Type/Group/S/Transparency/CS/DeviceRGB>>/Tabs/S/StructParents 1>> endobj 14 0 obj <</Filter/FlateDecode/Length 2691>> stream x\mo8 _ 8K*$^o8J_v#%ۑ)fkwm53g1_>spz~0;BSH2#`^^PyO@/+0Z=@2j,;~أޫoiz@{G7{ #K#қr;v%{/V%oBԲyilmn?İJLD"p bv"V)k\3bP3/`bUaM(ee@2}VBB iAYڿ*{H4XߺN(3-TI*4H d]eyS*~ŭk7䶯B!/iK*q)< $T Х)UYv@ea qŌ]pOt8I!sz% Hgv~V&bHkBLh[. : qO< ܬQ'0s*8hFNb?b:iSCp5!P~?=ͭ,IqfI )$,t e"Zu3L*LL{`( `r17j+x8I!%Rj,A]%dM%J)DfŵH" "K[!#6H9Bnx!/Fñpt6H'ف'K㩚 04(@0Tda >hχ J59sLhM SN 39!s;!v #s dii'! nk$f;*^HpE˓o7InpLT>Ȣ!7]=WdGeb`YvNòs!6?v?Z KUW+~j枭{$"'T*i& xߤҌHV1?(qPӚE+rp!!iX `"Y0J0T<e lHiqv"ӅN-G :8W4Iqt"܂ Ug?4. D<J$Tu.Չ˧+LwkzDA0ӺEpfSp,Mo2,-[d1*롮"*9\-Rpw9+;b)y6K$Dv/LHMsҟUn,b쪎XE+pB jM/#`-I X.b+[\Eƚ&-Axker4RƼXVVf1XI,2_yi%h<[ZpCt.L<
