From owner-freebsd-hackers@freebsd.org Thu Jun 4 12:16:09 2020 Return-Path: Delivered-To: freebsd-hackers@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 96D8C2F05D9 for ; Thu, 4 Jun 2020 12:16:09 +0000 (UTC) (envelope-from gbergling@gmail.com) Received: from mail-wm1-x343.google.com (mail-wm1-x343.google.com [IPv6:2a00:1450:4864:20::343]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 49d4WD6l8Dz4bgm; Thu, 4 Jun 2020 12:16:08 +0000 (UTC) (envelope-from gbergling@gmail.com) Received: by mail-wm1-x343.google.com with SMTP id j198so6724244wmj.0; Thu, 04 Jun 2020 05:16:08 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:from:to:subject:message-id :references:mime-version:content-disposition:in-reply-to; bh=cRiIWlwXSmeGD5K5/YR899nPoATB2rEWsVjeJSYr/m0=; b=fVUfXu3q6khAjobO31J++bR0DdyABCx6B5jv8Drf5KftwxBQFPtyDJK+R26WOO1gTR f3TYCNiYdpUnyYls8LFJppmGUmfNLgkj5ov9nNoEZhUEIXxhKRfJ4N9McUcuk9bh3wof mcFGHLZtXpRhDZ1+7xBF5UX7CfUVIBnK6X5azMxiJUWDijIH1WKvxOkC2ObfNZO2thGv shN/roxMC1r5BqKZvI7cBzBfbDp2iWWhzgoULCned3T+Knp5JvZdCHgVwm7g/ptRz4h0 dlEl2GFk0Z0hGOen1P7pIdKujxIqu1JBqU6jys65wlg1JLUtDqaYkJyMogARghF8mmw3 51iQ== X-Gm-Message-State: AOAM5331kd4a+HUH7AbT0u17L3kMEFpOOxqVePzrGyzD7Bc5IQeMmpYi b2M49UkMLP5KfsxrUB4BeU9ApWAC X-Google-Smtp-Source: ABdhPJxFxyH0oFeEnx0HfeMh/omezBm8zpkV98VPMufMOSG7nSktbb+LZe2N97jWOxyZtmdX2fB1WQ== X-Received: by 2002:a7b:c311:: with SMTP id k17mr3699394wmj.148.1591272967170; Thu, 04 Jun 2020 05:16:07 -0700 (PDT) Received: from lion.0xfce3.net (p4fd3af72.dip0.t-ipconnect.de. [79.211.175.114]) by smtp.gmail.com with ESMTPSA id l18sm6670464wmj.22.2020.06.04.05.16.05 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 04 Jun 2020 05:16:06 -0700 (PDT) Sender: Gordon Bergling Date: Thu, 4 Jun 2020 14:16:04 +0200 From: Gordon Bergling To: Daniel Ebdrup Jensen , freebsd-hackers@freebsd.org Subject: Re: Constant load of 1 on a recent 12-STABLE Message-ID: <20200604121604.GA23413@lion.0xfce3.net> References: <20200603101607.GA80381@lion.0xfce3.net> <20200603202929.GA65032@lion.0xfce3.net> <20200603204511.6qmsub2gqc44jkjw@nerd-thinkpad.local> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20200603204511.6qmsub2gqc44jkjw@nerd-thinkpad.local> X-Url: X-Operating-System: FreeBSD 12.1-STABLE amd64 X-Host-Uptime: 2:11PM up 1 day, 6:53, 3 users, load averages: 4.49, 4.81, 4.64 X-Rspamd-Queue-Id: 49d4WD6l8Dz4bgm X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US] X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 04 Jun 2020 12:16:09 -0000 Hi Daniel, On Wed, Jun 03, 2020 at 10:45:11PM +0200, Daniel Ebdrup Jensen wrote: > On Wed, Jun 03, 2020 at 10:29:29PM +0200, Gordon Bergling via freebsd-hackers wrote: > >On Wed, Jun 03, 2020 at 03:13:47PM -0400, Allan Jude wrote: > >> On 2020-06-03 06:16, Gordon Bergling via freebsd-hackers wrote: > >> > since a while I am seeing a constant load of 1.00 on 12-STABLE, > >> > but all CPUs are shown as 100% idle in top. > >> > > >> > Has anyone an idea what could caused this? > >> > > >> > The load seems to be somewhat real, since the buildtimes on this > >> > machine for -CURRENT increased from about 2 hours to 3 hours. > >> > > >> > This a virtualized system running on Hyper-V, if that matters. > >> > > >> > Any hints are more then appreciated. > >> > >> Try running 'top -SP' and see if that shows a specific CPU being busy, > >> or a specific process using CPU time > > > >Below is the output of 'top -SP'. The only relevant process / thread that is > >relatively constant consumes CPU time seams to be 'zfskern'. > > > >----------------------------------------------------------------------------- > >last pid: 68549; load averages: 1.10, 1.19, 1.16 up 0+14:59:45 22:17:24 > >67 processes: 2 running, 64 sleeping, 1 waiting > >CPU 0: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle > >CPU 1: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle > >CPU 2: 0.0% user, 0.0% nice, 0.4% system, 0.0% interrupt, 99.6% idle > >CPU 3: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle > >Mem: 108M Active, 4160M Inact, 33M Laundry, 3196M Wired, 444M Free > >ARC: 1858M Total, 855M MFU, 138M MRU, 96K Anon, 24M Header, 840M Other > > 461M Compressed, 1039M Uncompressed, 2.25:1 Ratio > >Swap: 2048M Total, 2048M Free > > > > PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND > > 11 root 4 155 ki31 0B 64K RUN 0 47.3H 386.10% idle > > 8 root 65 -8 - 0B 1040K t->zth 0 115:39 12.61% zfskern > >------------------------------------------------------------------------------- > > > >The only key performance indicator that is relatively high IMHO, for a > >non-busy system, are the context switches, that vmstat has reported. > > > >------------------------------------------------------------------------------- > >procs memory page disks faults cpu > >r b w avm fre flt re pi po fr sr da0 da1 in sy cs us sy id > >0 0 0 514G 444M 7877 2 7 0 9595 171 0 0 0 4347 43322 17 2 81 > >0 0 0 514G 444M 1 0 0 0 0 44 0 0 0 121 40876 0 0 100 > >0 0 0 514G 444M 0 0 0 0 0 40 0 0 0 133 42520 0 0 100 > >0 0 0 514G 444M 0 0 0 0 0 40 0 0 0 120 43830 0 0 100 > >0 0 0 514G 444M 0 0 0 0 0 40 0 0 0 132 42917 0 0 100 > >-------------------------------------------------------------------------------- > > > >Any other ideas what could generate that load? > > > I seem to recall bde@ (may he rest in peace) mentioning that the ULE scheduler > had some weirdness around sometimes generating a higher load number (one of my > systems would regularily idle at 0.60, but doesn't do it on 12.1 so I gave up > trying to debug it) for no apparent reason, and it maybe being linked to how > WCPU and CPU don't differ on the ULE scheduler? > > Have you tried setting the kern.eventtimer.periodic sysctl to 1? > > Yours, > Daniel Ebdrup Jensen thanks for the hint regarding the kern.eventtimer.periodic sysctl, but it doesn't changed anything. I had running with enabled for about 8 hours. I try now to collect more information like Allan has suggested. Best regards, Gordon -- Gordon Bergling Mobile: +49 170 23 10 948 Web: https://www.gordons-perspective.com/ Mail: gbergling@gmail.com Think before you print!