From owner-freebsd-current@freebsd.org Fri Sep 7 17:27:59 2018 Return-Path: Delivered-To: freebsd-current@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id DF99310072B3 for ; Fri, 7 Sep 2018 17:27:58 +0000 (UTC) (envelope-from cy.schubert@cschubert.com) Received: from smtp-out-no.shaw.ca (smtp-out-no.shaw.ca [64.59.134.9]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "Client", Issuer "CA" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 4600971C64; Fri, 7 Sep 2018 17:27:57 +0000 (UTC) (envelope-from cy.schubert@cschubert.com) Received: from spqr.komquats.com ([70.67.125.17]) by shaw.ca with ESMTPA id yKXqfjbt7WppDyKXsfNbxS; Fri, 07 Sep 2018 11:27:50 -0600 X-Authority-Analysis: v=2.3 cv=YIcrNiOx c=1 sm=1 tr=0 a=VFtTW3WuZNDh6VkGe7fA3g==:117 a=VFtTW3WuZNDh6VkGe7fA3g==:17 a=JBFolyDoGHsA:10 a=YxBL1-UpAAAA:8 a=6I5d2MoRAAAA:8 a=5R4wDWXbYSEQHXC5hsYA:9 a=1GIGVPUOJR0cIUBz:21 a=kV9XivK--J73BiRn:21 a=wPNLvfGTeEIA:10 a=rkHIVglFI-3vNNMI_WsA:9 a=wijcN7gyilmvY5eC:21 a=j0QCUOnUCc-NYdUK:21 a=LJZpL5zafY3ct8uS:21 a=_W_S_7VecoQA:10 a=Ia-lj3WSrqcvXOmTRaiG:22 a=IjZwj45LgO3ly-622nXo:22 Received: from [25.168.101.95] (S0106788a207e2972.gv.shawcable.net [70.66.154.233]) by spqr.komquats.com (Postfix) with ESMTPSA id 38609A00; Fri, 7 Sep 2018 10:28:43 -0700 (PDT) MIME-Version: 1.0 From: Cy Schubert Subject: RE: ZFS perfomance regression in FreeBSD 12 APLHA3->ALPHA4 Date: Fri, 7 Sep 2018 10:27:49 -0700 To: Mark Johnston , Jakob Alvermark CC: Subbsd , "allanjude@freebsd.org" , freebsd-current Current Message-Id: <20180907172843.38609A00@spqr.komquats.com> X-CMAE-Envelope: MS4wfFqcFk16wTWoFwWvQg7vmz4QafBqmq03nGAOpOgrK7wn2lt73V5Xtmb5cM5EqQxhxMD71GnV9fieClDTONVRx5Lzg1gzchVSf61uBxgNogkAOduOxJSD hn2Id2a8vjkctU0O4U/ZBAoyaSfNYMtSiOrbsVIod74hKR2xf7zXpUMeexFfbhT0dsnTz5kA7LaEU9Ec9Jq/wwUkOde9LYzVfMq9ZB5BJCRESHuDMWUDUQRC wWGZBKfxPsAtmImraySEfR4ThyFHDFCdcTJDC/7LTVuRRIVrCJy/zA7+OidkfPAm Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.27 X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 07 Sep 2018 17:27:59 -0000 I'd be interested in seeing systat -z output. --- Sent using a tiny phone keyboard. Apologies for any typos and autocorrect. Also, this old phone only supports top post. Apologies. Cy Schubert or The need of the many outweighs the greed of the few. --- -----Original Message----- From: Mark Johnston Sent: 07/09/2018 09:09 To: Jakob Alvermark Cc: Subbsd; allanjude@freebsd.org; freebsd-current Current Subject: Re: ZFS perfomance regression in FreeBSD 12 APLHA3->ALPHA4 On Fri, Sep 07, 2018 at 03:40:52PM +0200, Jakob Alvermark wrote: > On 9/6/18 2:28 AM, Mark Johnston wrote: > > On Wed, Sep 05, 2018 at 11:15:03PM +0300, Subbsd wrote: > >> On Wed, Sep 5, 2018 at 5:58 PM Allan Jude wrot= e: > >>> On 2018-09-05 10:04, Subbsd wrote: > >>>> Hi, > >>>> > >>>> I'm seeing a huge loss in performance ZFS after upgrading FreeBSD 12 > >>>> to latest revision (r338466 the moment) and related to ARC. > >>>> > >>>> I can not say which revision was before except that the newver.sh > >>>> pointed to ALPHA3. > >>>> > >>>> Problems are observed if you try to limit ARC. In my case: > >>>> > >>>> vfs.zfs.arc_max=3D"128M" > >>>> > >>>> I know that this is very small. However, for two years with this the= re > >>>> were no problems. > >>>> > >>>> When i send SIGINFO to process which is currently working with ZFS, = i > >>>> see "arc_reclaim_waiters_cv": > >>>> > >>>> e.g when i type: > >>>> > >>>> /bin/csh > >>>> > >>>> I have time (~5 seconds) to press several times 'ctrl+t' before csh = is executed: > >>>> > >>>> load: 0.70 cmd: csh 5935 [arc_reclaim_waiters_cv] 1.41r 0.00u 0.00s= 0% 3512k > >>>> load: 0.70 cmd: csh 5935 [zio->io_cv] 1.69r 0.00u 0.00s 0% 3512k > >>>> load: 0.70 cmd: csh 5935 [arc_reclaim_waiters_cv] 1.98r 0.00u 0.01s= 0% 3512k > >>>> load: 0.73 cmd: csh 5935 [arc_reclaim_waiters_cv] 2.19r 0.00u 0.01s= 0% 4156k > >>>> > >>>> same story with find or any other commans: > >>>> > >>>> load: 0.34 cmd: find 5993 [zio->io_cv] 0.99r 0.00u 0.00s 0% 2676k > >>>> load: 0.34 cmd: find 5993 [arc_reclaim_waiters_cv] 1.13r 0.00u 0.00= s 0% 2676k > >>>> load: 0.34 cmd: find 5993 [arc_reclaim_waiters_cv] 1.25r 0.00u 0.00= s 0% 2680k > >>>> load: 0.34 cmd: find 5993 [arc_reclaim_waiters_cv] 1.38r 0.00u 0.00= s 0% 2684k > >>>> load: 0.34 cmd: find 5993 [arc_reclaim_waiters_cv] 1.51r 0.00u 0.00= s 0% 2704k > >>>> load: 0.34 cmd: find 5993 [arc_reclaim_waiters_cv] 1.64r 0.00u 0.00= s 0% 2716k > >>>> load: 0.34 cmd: find 5993 [arc_reclaim_waiters_cv] 1.78r 0.00u 0.00= s 0% 2760k > >>>> > >>>> this problem goes away after increasing vfs.zfs.arc_max > >>>> _______________________________________________ > >>>> freebsd-current@freebsd.org mailing list > >>>> https://lists.freebsd.org/mailman/listinfo/freebsd-current > >>>> To unsubscribe, send any mail to "freebsd-current-unsubscribe@freebs= d.org" > >>>> > >>> Previously, ZFS was not actually able to evict enough dnodes to keep > >>> your arc_max under 128MB, it would have been much higher based on the > >>> number of open files you had. A recent improvement from upstream ZFS > >>> (r337653 and r337660) was pulled in that fixed this, so setting an > >>> arc_max of 128MB is much more effective now, and that is causing the > >>> side effect of "actually doing what you asked it to do", in this case= , > >>> what you are asking is a bit silly. If you have a working set that is > >>> greater than 128MB, and you ask ZFS to use less than that, it'll have= to > >>> constantly try to reclaim memory to keep under that very low bar. > >>> > >> Thanks for comments. Mark was right when he pointed to r338416 ( > >> https://svnweb.freebsd.org/base/head/sys/cddl/contrib/opensolaris/uts/= common/fs/zfs/arc.c?r1=3D338416&r2=3D338415&pathrev=3D338416 > >> ). Commenting aggsum_value returns normal speed regardless of the rest > >> of the new code from upstream. > >> I would like to repeat that the speed with these two lines is not just > >> slow, but _INCREDIBLY_ slow! Probably, this should be written in the > >> relevant documentation for FreeBSD 12+ >=20 > Hi, >=20 > I am experiencing the same slowness when there is a bit of load on the=20 > system (buildworld for example) which I haven't seen before. Is it a regression following a recent kernel update? > I have vfs.zfs.arc_max=3D2G. >=20 > Top is reporting >=20 > ARC: 607M Total, 140M MFU, 245M MRU, 1060K Anon, 4592K Header, 217M Other > =A0=A0=A0=A0 105M Compressed, 281M Uncompressed, 2.67:1 Ratio >=20 > Should I test the patch? I would be interested in the results, assuming it is indeed a regression. _______________________________________________ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscribe@freebsd.org"