From owner-freebsd-virtualization@freebsd.org Thu Nov 26 04:21:28 2015 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 8B5F7A3796B for ; Thu, 26 Nov 2015 04:21:28 +0000 (UTC) (envelope-from dmarquess@gmail.com) Received: from mail-ig0-x232.google.com (mail-ig0-x232.google.com [IPv6:2607:f8b0:4001:c05::232]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 5B72C1C18 for ; Thu, 26 Nov 2015 04:21:28 +0000 (UTC) (envelope-from dmarquess@gmail.com) Received: by igcph11 with SMTP id ph11so3758814igc.1 for ; Wed, 25 Nov 2015 20:21:27 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:date:message-id:subject:from:to:content-type; bh=f+T+Rn43/h2FCPNkalPeu2itCEYRIJEngvEwiWy6sfE=; b=z5MToQwyld1lKQN1Tb+3Nlx3GCBnBbAjavYOwyCZicEKv/iOg6a+xU/79WnhNrtyru WSSxHziL/U3PAjCAVNxrabYWAfbLumrgrTWXUPZDlAWUjcDgFimHIfnx++D0Qrc2wpPP PXP0TXwEyOQx9X0pbI9CyqKnNRHnpEMbLnm9LwFbdyBYpS8r9GLJjtSrtESWDaiOEjaq fweE5tYiELVn5CHmpmgL5EtozAuOWXaMdPCbkFyd00IOVWsKhnuRm0gcO9362Q5XnSLD HsqzkzUNNBA+AAYXM9ha+BViUEWGw01tUOTE7SKQoBYNMASssI+BZZf4rMJdKuEWDSgw FtMA== MIME-Version: 1.0 X-Received: by 10.50.4.105 with SMTP id j9mr1074398igj.76.1448511687783; Wed, 25 Nov 2015 20:21:27 -0800 (PST) Received: by 10.107.16.22 with HTTP; Wed, 25 Nov 2015 20:21:27 -0800 (PST) Date: Wed, 25 Nov 2015 22:21:27 -0600 Message-ID: Subject: Debugging high(ish) CPU usage on an idle VM From: Dustin Marquess To: freebsd-virtualization@freebsd.org Content-Type: text/plain; charset=UTF-8 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 26 Nov 2015 04:21:28 -0000 I'm running an 11.0-CURRENT (from exactly a week ago) on a 32-core (16-module) AMD Bulldozer system. Guest is NetBSD 7.99.21 HEAD, with timecounter hard-forced to ACPI-safe instead of hpet (HPET seemed unstable, although HPET is being used in the host FreeBSD). I start bhyve using a simple script: #!/bin/sh RAM=8192M VM=netbsd VMROOT=/usr/local/vm CPUS=8 grub-bhyve -m ${VMROOT}/${VM}/device.map -d ${VMROOT}/${VM} -M ${RAM} ${VM} exec bhyve -A -H -P -w -u -s 0:0,amd_hostbridge -s 1:0,lpc -s 2:0,virtio-net,tap0 -s 3:0,virtio-blk,/dev/zvol/pool/vm/${VM},nocache -s 4:0,virtio-rnd -l com1,stdio -c ${CPUS} -m ${RAM} ${VM} While the guest is entirely idle: 37975 root 1 20 0 13084K 3016K wait 31 0:00 0.00% sh load averages: 0.00, 0.00, 0.00; up 1+22:00:40 03:34:40 18 processes: 16 sleeping, 2 on CPU CPU states: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle Memory: 923M Act, 6996K Wired, 12M Exec, 900M File, 5911M Free Swap: 8195M Total, 8195M Free Bhyve itself seems to go bounce around between ~6% CPU usage to 18%: last pid: 37995; load averages: 0.22, 0.38, 0.41 up 5+20:05:17 21:26:35 24 processes: 1 running, 23 sleeping CPU: 0.0% user, 0.0% nice, 0.5% system, 0.1% interrupt, 99.4% idle Mem: 2572M Active, 5735M Inact, 19G Wired, 61M Buf, 35G Free ARC: 14G Total, 3656M MFU, 10G MRU, 16K Anon, 128M Header, 566M Other Swap: 64G Total, 64G Free PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND 18577 root 18 20 0 9247M 5048M kqread 15 427:55 18.65% bhyve Any ideas to help troubleshoot where the CPU usage is going, DTrace? Thanks! -Dustin