From owner-freebsd-current@freebsd.org Fri Sep 11 05:08:24 2020 Return-Path: Delivered-To: freebsd-current@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 02F3B3EDABD for ; Fri, 11 Sep 2020 05:08:24 +0000 (UTC) (envelope-from bdragon@FreeBSD.org) Received: from smtp.freebsd.org (smtp.freebsd.org [96.47.72.83]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4BnkKz6LXQz3ZmS for ; Fri, 11 Sep 2020 05:08:23 +0000 (UTC) (envelope-from bdragon@FreeBSD.org) Received: from auth1-smtp.messagingengine.com (auth1-smtp.messagingengine.com [66.111.4.227]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) (Authenticated sender: bdragon/mail) by smtp.freebsd.org (Postfix) with ESMTPSA id B6D9D207D0 for ; Fri, 11 Sep 2020 05:08:23 +0000 (UTC) (envelope-from bdragon@FreeBSD.org) Received: from compute4.internal (compute4.nyi.internal [10.202.2.44]) by mailauth.nyi.internal (Postfix) with ESMTP id A183927C0058 for ; Fri, 11 Sep 2020 01:08:23 -0400 (EDT) Received: from imap1 ([10.202.2.51]) by compute4.internal (MEProxy); Fri, 11 Sep 2020 01:08:23 -0400 X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeduiedrudehkedgleefucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne gfrhhlucfvnfffucdluddtmdenucfjughrpefofgggkfgjfhffhffvufgtgfesthhqredt reerjeenucfhrhhomhepfdeurhgrnhguohhnuceuvghrghhrvghnfdcuoegsughrrghgoh hnsefhrhgvvgeuufffrdhorhhgqeenucggtffrrghtthgvrhhnpeeileejjeevgefgieef tdefvdefgffgheffveeiffevvddvhfelleduvdevhedvheenucevlhhushhtvghrufhiii gvpedtnecurfgrrhgrmhepmhgrihhlfhhrohhmpegsughrrghgohhnodhmvghsmhhtphgr uhhthhhpvghrshhonhgrlhhithihqddutdegvdefheekieegqddukedutdekheduqdgsug hrrghgohhnpeephfhrvggvuefuffdrohhrghesihhmrghprdgttg X-ME-Proxy: Received: by mailuser.nyi.internal (Postfix, from userid 501) id 42C24C200A5; Fri, 11 Sep 2020 01:08:23 -0400 (EDT) X-Mailer: MessagingEngine.com Webmail Interface User-Agent: Cyrus-JMAP/3.3.0-259-g88fbbfa-fm-20200903.003-g88fbbfa3 Mime-Version: 1.0 Message-Id: In-Reply-To: <474222f1-1e48-4453-a94e-4f1f5f4a4c39@www.fastmail.com> References: <7d54dc30-b8b1-a127-ec39-9fb759c8a55d@gmail.com> <474222f1-1e48-4453-a94e-4f1f5f4a4c39@www.fastmail.com> Date: Fri, 11 Sep 2020 00:08:01 -0500 From: "Brandon Bergren" To: "FreeBSD Current" Subject: Re: time sysctl kstat.zfs.misc.dbufs | wc (was: OpenZFS and L2ARC) Content-Type: text/plain;charset=utf-8 Content-Transfer-Encoding: quoted-printable X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 11 Sep 2020 05:08:24 -0000 On Thu, Sep 10, 2020, at 11:50 PM, Brandon Bergren wrote: > On Thu, Sep 10, 2020, at 11:17 PM, Graham Perrin wrote: > > On 09/09/2020 07:46, Stefan Esser wrote: > > > =E2=80=A6 an annoyance that I had noticed before but now have > > > tracked down: > > > > > > $ time sysctl kstat.zfs.misc.dbufs | wc > > > =C2=A0=C2=A0 55327 2047031 16333472 > > > > > > real=C2=A0=C2=A0=C2=A0 0m16,446s > > > user=C2=A0=C2=A0=C2=A0 0m0,055s > > > sys=C2=A0=C2=A0=C2=A0 0m16,397s > > > > > > =E2=80=A6 > >=20 > >=20 >=20 > That's nothing: >=20 > root@talos:~/devel/poudriere # /usr/bin/time sysctl kstat.zfs.misc.dbu= fs | wc > 603.59 real 0.03 user 603.39 sys > 63677 2355981 18646506 >=20 > It literally takes ten minutes on my Talos II. FWIW: Tracing command sysctl pid 25337 tid 104535 td 0xc00800010362b600 (CPU 5= 9) 0xc00800015424cba0: at intr_event_handle+0x130 0xc00800015424cc40: at powerpc_dispatch_intr+0x8c 0xc00800015424ccc0: at xive_dispatch+0x94 0xc00800015424cd50: at PIC_DISPATCH+0x78 0xc00800015424cd90: at powerpc_interrupt+0xb8 0xc00800015424ce20: kernel trap 0xea0 by memset+0x10: srr1=3D0x900000000= 0009032 r1=3D0xc00800015424d0d0 cr=3D0x42400004 xer=3D0 ctr=3D0xded = r2=3D0xc000000003a57000 frame=3D0xc00800015424ce50 0xc00800015424d0d0: at dbuf_stats_hash_table_data+0x1e4 0xc00800015424d180: at kstat_sysctl_raw+0x1e8 0xc00800015424d250: at sysctl_root_handler_locked+0x104 0xc00800015424d2c0: at sysctl_root+0x294 0xc00800015424d3b0: at userland_sysctl+0x174 0xc00800015424d4c0: at sys___sysctl+0x8c 0xc00800015424d5b0: at syscallenter+0x184 0xc00800015424d600: at syscall+0x60 0xc00800015424d640: at trap+0x440 0xc00800015424d750: at powerpc_interrupt+0x110 0xc00800015424d7e0: user SC trap by 0x8102de5d0: srr1=3D0x900000000200f0= 32 r1=3D0xfffffbfffbfd0 cr=3D0x44000382 xer=3D0 ctr=3D0x8102de5= c0 r2=3D0x810306bf0 frame=3D0xc00800015424d810 db> show frame 0xc00800015424ce50 trap frame 0xc00800015424ce50 r0: 0xc000000002566044 (-4611686018388172732) r1: 0xc00800015424d0d0 (-4609434212907036464) r2: 0xc000000003a57000 (-4611686018366214144) r3: 0xc000000ca00cf000 (-4611685964202577920) r4: 0 (0) r5: 0x1000 (4096) r6: 0xc000000ca00cf212 (-4611685964202577390) r7: 0x155a0b7 (22388919) r8: 0x1ffffff (33554431) r9: 0 (0) r10: 0xc0000000035859fe (-4611686018371266050) r11: 0 (0) r12: 0xc0000000026145dc (-4611686018387458596) r13: 0xc00800010362b600 (-4609434214261934592) r14: 0x1003d230 (268685872) r15: 0x1003d230 (268685872) r16: 0x1003d230 (268685872) r17: 0x1003d230 (268685872) r18: 0x810317b08 (34631416584) r19: 0x155a0b7 (22388919) r20: 0 (0) r21: 0xc000000003b61600 (-4611686018365123072) r22: 0xc0000000035902b0 (-4611686018371222864) r23: 0xc0000000035859fe (-4611686018371266050) r24: 0xc000000003b21488 (-4611686018365385592) r25: 0xc0000000039ad5b0 (-4611686018366909008) r26: 0xc00000000261480c (-4611686018387458036) r27: 0xc00000000361d05a (-4611686018370645926) r28: 0xc000000ca00cf000 (-4611685964202577920) r29: 0x1000 (4096) r30: 0xc000000003b61600 (-4611686018365123072) r31: 0xc00800015424d0d0 (-4609434212907036464) lr: 0xc0000000026146a0 cr: 0x42400004 xer: 0 ctr: 0xded (3565) srr0: 0xc0000000030a5cc0 srr1: 0x9000000000009032 exc: 0xea0 dar: 0xc0080001f2036b23 dsisr: 0x2000000 Every time I've looked in on it, it appears to be zeroing a page of memo= ry. I believe there is something going wrong with the buffer management here. --=20 Brandon Bergren bdragon@FreeBSD.org