From owner-freebsd-hackers@freebsd.org Thu Jun 4 03:26:30 2020 Return-Path: Delivered-To: freebsd-hackers@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 2747F3280D5 for ; Thu, 4 Jun 2020 03:26:30 +0000 (UTC) (envelope-from ian@freebsd.org) Received: from outbound5a.ore.mailhop.org (outbound5a.ore.mailhop.org [44.233.67.66]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 49crm5355Mz4Wcy for ; Thu, 4 Jun 2020 03:26:29 +0000 (UTC) (envelope-from ian@freebsd.org) ARC-Seal: i=1; a=rsa-sha256; t=1591216809; cv=none; d=outbound.mailhop.org; s=arc-outbound20181012; b=vGmhXP7ZRKPb0EhLawG/oGJAm456qIba3AXU853gp24/jwMRaMKkqNCpfBoAi0fIdYtJ5tA4vW7t3 O9bY2xfrL47jQf89D0P1NVXk0n4ASneJwyTvXcWCNhPJ3+isSjUqasi7z0fufmkq1DpTpY2g/wnizg 6Fkotw6ejyUkXr/1haDXiUihv42DRQAmoeFNVWeY87aS/pvTtu6BynqbjIKXhh9Qh2s4lqh9ugQiB4 BRugbA/MRQo/MMA+jN1CQ1EUQznyzqZAYbOF6jAPNjBXqU4VeW9pSZ2Yo1GnkIy6rs4zVpyAm+FB2s sqnDk6ZVCZeKAQz1LIAWz3QyP9kECCQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=outbound.mailhop.org; s=arc-outbound20181012; h=content-transfer-encoding:mime-version:content-type:references:in-reply-to: date:cc:to:from:subject:message-id:dkim-signature:from; bh=b6JRTNFpAfZkMi17Qu05FJlSCNZ8tt/r8r9r9WpQNnI=; b=DYLqoQ4j9FAWEV3zxMEb6O8eoNO9DYkBFTTEEGfS65UG2/kcW2YSW5Mxk7Q6yixp/76Yx432z6dGt IOtf0wcSoFR1jMJAXnGaiWFIOyvIDGhHDdst4mvVHfSSS3sChT149aY4Mf67H8MBG4By5sNIOJxRQo 79gi0++7c7PXfZAFd0h/hUaT5hJ7PFcg3u1hUbTqiTV+eJEv9RjlnyaSWRqFri6SD4D3kUJOD11UPM BqbSco3a6bSs0pqrA1xJ5XtCFAIRGBixzkNAWXC9AbJvCQrn/aUOhhWRCf9NEIIpuzBMyvvR6npwWY TeBCBPIV+VDPvfIg2S7JOFXYCpYU3QQ== ARC-Authentication-Results: i=1; outbound3.ore.mailhop.org; spf=softfail smtp.mailfrom=freebsd.org smtp.remote-ip=67.177.211.60; dmarc=none header.from=freebsd.org; arc=none header.oldest-pass=0; DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=outbound.mailhop.org; s=dkim-high; h=content-transfer-encoding:mime-version:content-type:references:in-reply-to: date:cc:to:from:subject:message-id:from; bh=b6JRTNFpAfZkMi17Qu05FJlSCNZ8tt/r8r9r9WpQNnI=; b=ZtI9KmsscKmIopsJdesJQ/AMYRMu3Ft75L0jE7Pxy6wKVkjLNorkk7HJd5RUJs6kkWygxbFg5YVZJ kQbMkZx+VjcL6JxMT94EpPeXcV3Q1OZsqM5X8PbWQuyxPUz+PlWl/tjH2Pqm9UfmwLpErp0up+yzJx f/dZLXfIN6Tzx2GNnVwU6px+N0eMCtvleDIy1CwMd23OvjLu/zgKvZsKWfpCbVMu5VrKL1hozzbjKJ bfyKD1Qhv7zyjy912qN1SUF4CFB7YotTwn9SM/ZM/sKoH9NxYS9xL0heSptox4HrmZX5karXz6wb4+ HuD5HkJGek7xTFXYrc51R9Wh5yHie8w== X-MHO-RoutePath: aGlwcGll X-MHO-User: 6928357d-a5da-11ea-b10c-b5956a7dd1a1 X-Report-Abuse-To: https://support.duocircle.com/support/solutions/articles/5000540958-duocircle-standard-smtp-abuse-information X-Originating-IP: 67.177.211.60 X-Mail-Handler: DuoCircle Outbound SMTP Received: from ilsoft.org (c-67-177-211-60.hsd1.co.comcast.net [67.177.211.60]) by outbound3.ore.mailhop.org (Halon) with ESMTPSA id 6928357d-a5da-11ea-b10c-b5956a7dd1a1; Wed, 03 Jun 2020 20:40:08 +0000 (UTC) Received: from rev (rev [172.22.42.240]) by ilsoft.org (8.15.2/8.15.2) with ESMTP id 053Ke7uT069856; Wed, 3 Jun 2020 14:40:07 -0600 (MDT) (envelope-from ian@freebsd.org) Message-ID: Subject: Re: Constant load of 1 on a recent 12-STABLE From: Ian Lepore To: Gordon Bergling , freebsd-hackers@freebsd.org Cc: Allan Jude Date: Wed, 03 Jun 2020 14:40:07 -0600 In-Reply-To: <20200603202929.GA65032@lion.0xfce3.net> References: <20200603101607.GA80381@lion.0xfce3.net> <20200603202929.GA65032@lion.0xfce3.net> Content-Type: text/plain; charset="ASCII" X-Mailer: Evolution 3.28.5 FreeBSD GNOME Team Mime-Version: 1.0 Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: 49crm5355Mz4Wcy X-Spamd-Bar: / Authentication-Results: mx1.freebsd.org; none X-Spamd-Result: default: False [0.00 / 15.00]; ASN(0.00)[asn:16509, ipnet:44.224.0.0/11, country:US]; local_wl_from(0.00)[freebsd.org] X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 04 Jun 2020 03:26:30 -0000 On Wed, 2020-06-03 at 22:29 +0200, Gordon Bergling via freebsd-hackers wrote: > [...] > > The only key performance indicator that is relatively high IMHO, for a > non-busy system, are the context switches, that vmstat has reported. > > ------------------------------------------------------------------------------- > procs memory page disks faults cpu > r b w avm fre flt re pi po fr sr da0 da1 in sy cs us sy id > 0 0 0 514G 444M 7877 2 7 0 9595 171 0 0 0 4347 43322 17 2 81 > 0 0 0 514G 444M 1 0 0 0 0 44 0 0 0 121 40876 0 0 100 > 0 0 0 514G 444M 0 0 0 0 0 40 0 0 0 133 42520 0 0 100 > 0 0 0 514G 444M 0 0 0 0 0 40 0 0 0 120 43830 0 0 100 > 0 0 0 514G 444M 0 0 0 0 0 40 0 0 0 132 42917 0 0 100 > -------------------------------------------------------------------------------- > > Any other ideas what could generate that load? > > Best regards, > > Interrupts/second consistently zero is the strange part of that vmstat output. That makes me think the problem has something to do with timekeeping and is a problem with the statistics used by top rather than having some "stuck process" actually consuming time. Are there any other signs of timekeeping trouble on the machine, like ntpd repeatedly stepping the clock? What's the output for sysctl kern.timecounter sysctl kern.eventtimer -- Ian