From owner-freebsd-hackers@freebsd.org Tue Oct 27 01:34:37 2015 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 07FB5A1C23A for ; Tue, 27 Oct 2015 01:34:37 +0000 (UTC) (envelope-from jiashiun@gmail.com) Received: from mail-vk0-x22d.google.com (mail-vk0-x22d.google.com [IPv6:2607:f8b0:400c:c05::22d]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id B01421B45; Tue, 27 Oct 2015 01:34:36 +0000 (UTC) (envelope-from jiashiun@gmail.com) Received: by vkex70 with SMTP id x70so110619351vke.3; Mon, 26 Oct 2015 18:34:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc:content-type; bh=1aSPuXp9tsOpFyn0iYM2Il/2PBwZBz5lo/huToee8Yk=; b=GRBRW+FSkgPSd+bjuw9jl6lWf0LBiQkQ4tn6woOestRPIz1O6dh2A4RzoqEswDOKJl MMwYPTXhuvr+briYzc/VnGhSnSRwr7+tEISKvHCgv8X3Y63OtS/DMnGT6fsaSQsGTmSn elJfCBX/vvZVTbILXOIE722jH74+8ksM5Y1BRsDzOB91Bqd6hsKWFWJj41aEaBTZSQFw 0LMVkJADF5/G7CNC7GBe4KV9Tg2VOmXhmaTvdA34Wh5Z5cxhhM5Nd4oXHAlHV+CU2bVb 9j48hkoIxSe1grvCHhW1DM3YBbUOcI2H1rzbGPK1uLYbC8LHtg+SgtWuKZVyNx4UraEB KWxg== X-Received: by 10.31.148.206 with SMTP id w197mr27568871vkd.100.1445909675599; Mon, 26 Oct 2015 18:34:35 -0700 (PDT) MIME-Version: 1.0 Received: by 10.103.66.66 with HTTP; Mon, 26 Oct 2015 18:34:06 -0700 (PDT) In-Reply-To: <562E4D3F.8080204@freebsd.org> References: <562E4D3F.8080204@freebsd.org> From: Jia-Shiun Li Date: Tue, 27 Oct 2015 09:34:06 +0800 Message-ID: Subject: Re: vmtotal consumes significant portion of cpu cycles To: Allan Jude Cc: freebsd-hackers@freebsd.org Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 27 Oct 2015 01:34:37 -0000 On Mon, Oct 26, 2015 at 11:56 PM, Allan Jude wrote: > On 2015-10-26 09:28, Jia-Shiun Li wrote: > > Hi all, > > > > I noticed that 'sysctl -vm 1' consumes about 5% cpu time on a machine > with > > 2x 6-core Xeon E5v3 and 64GB memory. That's a lot for a monitoring tool. > > > > After digging a while I found that it is vmtotal() in kernel that > consumes > > major cycles. When memory usage is high the cost of vmtotal() rises too. > It > > is reproducible with sysctl when memory utilization is high: > > > > % time repeat 100 sysctl vm.vmtotal > /dev/null > > 0.055u 8.102s 0:08.19 99.5% 31+175k 0+0io 0pf+0w > > > > % top > > last pid: 40272; load averages: 0.32, 4.74, 8.01 up 3+01:19:54 > > 17:23:59 > > 58 processes: 1 running, 57 sleeping > > CPU: 0.1% user, 0.0% nice, 1.6% system, 0.1% interrupt, 98.3% idle > > Mem: 4509M Active, 52G Inact, 2819M Wired, 1572M Buf, 2930M Free > > Swap: 3598M Total, 3598M Free > > > > PID USERNAME THR PRI NICE SIZE RES STATE C TIME CPU > > COMMAND > > 46841 root 30 20 0 9248M 7930M kqread 9 20.8H 11.88% > bhyve > > 49914 jsli 1 23 0 19320K 3884K select 5 134:08 4.79% > systat > > > > > > In FreeBSD source tree systat and vmstat are major user. Other tools like > > bsnmpd may use it too via sysctl. > > > > I don't have idea yet how this can be improved. Shall I create a bug to > > keep track of it? > > > > > > -Jia-Shiun > > _______________________________________________ > > freebsd-hackers@freebsd.org mailing list > > https://lists.freebsd.org/mailman/listinfo/freebsd-hackers > > To unsubscribe, send any mail to " > freebsd-hackers-unsubscribe@freebsd.org" > > > > In the first 2 references you say 'sysctl', do you mean 'systat' in all > instances? > > Allan, sorry the first one is typo and should be 'systat -vm', or 'systat -vmstat 1'. And the steps to reproduce the unexpected loading can be minimized to getting sysctl variable vm.vmtotal, which systat uses to get vm and process stats. -Jia-shiun.