From owner-freebsd-hackers@freebsd.org Mon Oct 26 13:28:46 2015 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 4BF53867F for ; Mon, 26 Oct 2015 13:28:46 +0000 (UTC) (envelope-from jiashiun@gmail.com) Received: from mail-vk0-x22f.google.com (mail-vk0-x22f.google.com [IPv6:2607:f8b0:400c:c05::22f]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 0B10A1AA7 for ; Mon, 26 Oct 2015 13:28:46 +0000 (UTC) (envelope-from jiashiun@gmail.com) Received: by vkfw189 with SMTP id w189so98138432vkf.2 for ; Mon, 26 Oct 2015 06:28:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:from:date:message-id:subject:to:content-type; bh=DDR1UszZgSiuG+A8rZ7VzNA23D97hDW4rqIiEqcq0mE=; b=ZLUY/qYE3ikVDO1+2xdAOZh5Ns7NwTZECw4pOq1xSjwW5EeTDKm/JPjPHBIe5ABPAI Wa5Fuwf/ZlStVPQ+L1a5nQYl6gXJnoutg9IAOc/YIsXUy75sN3/9b6foTla4rkprVuGU MkTImz9N0Wpl267rX7HkT0zKE1xxgpWLeyXN9xWbb715YAHaZ+SsVEQRjPSEGIlCO2GW nGnrr7m1DIrNe+qqz8XJY759MfkytCTwDFV8fVzEh8SneUxgxy/ZOpFX1GomcZoXNhSx hSPM7tV4Un0Gk6RAjv61o71gQA4u9qHkJWVnWwdQdC3wSfF9H+dPs7SC/lB7kVNCeZUT okNw== X-Received: by 10.31.135.12 with SMTP id j12mr21014103vkd.73.1445866124942; Mon, 26 Oct 2015 06:28:44 -0700 (PDT) MIME-Version: 1.0 Received: by 10.103.66.66 with HTTP; Mon, 26 Oct 2015 06:28:15 -0700 (PDT) From: Jia-Shiun Li Date: Mon, 26 Oct 2015 21:28:15 +0800 Message-ID: Subject: vmtotal consumes significant portion of cpu cycles To: freebsd-hackers@freebsd.org Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 26 Oct 2015 13:28:46 -0000 Hi all, I noticed that 'sysctl -vm 1' consumes about 5% cpu time on a machine with 2x 6-core Xeon E5v3 and 64GB memory. That's a lot for a monitoring tool. After digging a while I found that it is vmtotal() in kernel that consumes major cycles. When memory usage is high the cost of vmtotal() rises too. It is reproducible with sysctl when memory utilization is high: % time repeat 100 sysctl vm.vmtotal > /dev/null 0.055u 8.102s 0:08.19 99.5% 31+175k 0+0io 0pf+0w % top last pid: 40272; load averages: 0.32, 4.74, 8.01 up 3+01:19:54 17:23:59 58 processes: 1 running, 57 sleeping CPU: 0.1% user, 0.0% nice, 1.6% system, 0.1% interrupt, 98.3% idle Mem: 4509M Active, 52G Inact, 2819M Wired, 1572M Buf, 2930M Free Swap: 3598M Total, 3598M Free PID USERNAME THR PRI NICE SIZE RES STATE C TIME CPU COMMAND 46841 root 30 20 0 9248M 7930M kqread 9 20.8H 11.88% bhyve 49914 jsli 1 23 0 19320K 3884K select 5 134:08 4.79% systat In FreeBSD source tree systat and vmstat are major user. Other tools like bsnmpd may use it too via sysctl. I don't have idea yet how this can be improved. Shall I create a bug to keep track of it? -Jia-Shiun