From owner-freebsd-hackers@freebsd.org Sun Jul 5 08:06:07 2020 Return-Path: Delivered-To: freebsd-hackers@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id A73C5364B55 for ; Sun, 5 Jul 2020 08:06:07 +0000 (UTC) (envelope-from gbe@freebsd.org) Received: from smtp.freebsd.org (smtp.freebsd.org [96.47.72.83]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4B01VR3tFLz4dFg; Sun, 5 Jul 2020 08:06:07 +0000 (UTC) (envelope-from gbe@freebsd.org) Received: from localhost (p548612f6.dip0.t-ipconnect.de [84.134.18.246]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) (Authenticated sender: gbe) by smtp.freebsd.org (Postfix) with ESMTPSA id 0E2A813EF7; Sun, 5 Jul 2020 08:06:06 +0000 (UTC) (envelope-from gbe@freebsd.org) Date: Sun, 5 Jul 2020 10:06:05 +0200 From: Gordon Bergling To: Andriy Gapon Cc: Allan Jude , freebsd-hackers@freebsd.org Subject: Re: Constant load of 1 on a recent 12-STABLE Message-ID: <20200705080605.GA53520@lion.0xfce3.net> References: <20200603101607.GA80381@lion.0xfce3.net> <20200603202929.GA65032@lion.0xfce3.net> <8b1498ea-e343-506e-79c7-c25b594808f0@freebsd.org> <20200604123720.GA63595@lion.0xfce3.net> <20200623072332.GA54653@lion.0xfce3.net> <0a10a722-3eec-456b-d771-97a49af89ad1@FreeBSD.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <0a10a722-3eec-456b-d771-97a49af89ad1@FreeBSD.org> X-Url: X-Operating-System: FreeBSD 12.1-STABLE amd64 X-Host-Uptime: 9:49AM up 1 day, 22:39, 6 users, load averages: 4.89, 4.94, 4.91 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 05 Jul 2020 08:06:07 -0000 On Thu, Jul 02, 2020 at 10:40:50AM +0300, Andriy Gapon wrote: > On 23/06/2020 10:23, Gordon Bergling wrote: > > What is maybe also related to the problem is the following top output, > > collected by 'top -HS'. The three zfskern threads generate little load, but that > > constant. The annoying part of the problem is, that the load of 1 is reported and > > that the hostsystem schedules the thread over all 4 physical cores and keeps the > > clockrate at the highest frequency. > > > > PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU COMMAND > > 11 root 155 ki31 0B 64K CPU2 2 17:28 97.39% idle{idle: cpu2} > > 11 root 155 ki31 0B 64K CPU3 3 17:29 96.78% idle{idle: cpu3} > > 11 root 155 ki31 0B 64K CPU1 1 17:29 96.40% idle{idle: cpu1} > > 11 root 155 ki31 0B 64K RUN 0 17:25 96.13% idle{idle: cpu0} > > 8 root -8 - 0B 1040K mmp->m 2 0:44 4.32% zfskern{mmp_thread_enter} > > 8 root -8 - 0B 1040K mmp->m 1 0:44 4.28% zfskern{mmp_thread_enter} > > 8 root -8 - 0B 1040K mmp->m 3 0:44 4.25% zfskern{mmp_thread_enter} > > > > Does anyone has some insight where zfskern is spawned? I maybe can bisect then the resulting change > > which leads to this behaviour. > > That's strange. Those mmp threads shouldn't really be consuming much CPU. > Normally they should just sleep all the time. > Do you use the new ZFS multi-modifier protection feature? I guess not. > Do you have 'multihost' property set on any of your pools? > > The change you were looking for is > r361383 MFC r354804,r354806,r354807,r354885: 10499 Multi-modifier protection (MMP) I didn't use the ZFS multi-modifier protection feature and didn't have the 'multihost' propery set on any of my 3 ZFS pools. Thanks for identifing the particular change. I have tried OpenZFS from ports, which seems to improve the situation, but I went back to the base version since only the zroot pool was found and the two others pools weren't detected. --Gordon > -- > Andriy Gapon