From owner-freebsd-virtualization@FreeBSD.ORG Fri Apr 25 19:05:13 2014 Return-Path: Delivered-To: freebsd-virtualization@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id F38D074B for ; Fri, 25 Apr 2014 19:05:12 +0000 (UTC) Received: from mail-lb0-f178.google.com (mail-lb0-f178.google.com [209.85.217.178]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 7B9B41A63 for ; Fri, 25 Apr 2014 19:05:11 +0000 (UTC) Received: by mail-lb0-f178.google.com with SMTP id s7so3433487lbd.9 for ; Fri, 25 Apr 2014 12:05:04 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:from:to:subject:date:message-id:user-agent :mime-version:content-transfer-encoding:content-type; bh=oEmQpoTNxMM1URru2VRADzFrh7bxvBCxKAgnsp4xZQ4=; b=XH1e7IOOXYH6nfbT4djN/W19yGOF1s8P5p5SPJXj6NICmQ4liOtw8sZIhXbRE/J8xZ WYw/aQqHFPOJU0alr8ZgIBg4nviSX7kdxKN1kRhMMizmR7QtxUk9AyiBMs1PwrSeUrTh xDpFCIju/u4aXfA1KoRbeVJ5pESSvb38LsPwfVlHWTJdFxYqjk5u7CaOl+A3QZCxf1Hw aM0idEAGM0IzrDn3kPV+0gu0Vui8M5bWAvuM5HmTZ1nT08TLZIhPquc4g3Du5YvwKDUe HbQxAY2JivxF2Zii9wzEnJRcrY0Y9ST7PoN+T9KuKETvIk4HlRsDdW6ig4YqP7awXb+X vPmA== X-Gm-Message-State: ALoCoQm78DgjAJ6M17gvV9hp9VMI/CcCCFlVpPGekyerj1ieeXf8WYc6EUg31w03X/lA4CtkSyHa X-Received: by 10.112.46.225 with SMTP id y1mr6466537lbm.12.1398452704607; Fri, 25 Apr 2014 12:05:04 -0700 (PDT) Received: from kde4.my.domain (nat-server-217.15.27.35.static.futures.ru. [217.15.27.35]) by mx.google.com with ESMTPSA id q8sm9028680lbr.3.2014.04.25.12.05.01 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Fri, 25 Apr 2014 12:05:03 -0700 (PDT) From: Oleg Ginzburg To: freebsd-virtualization@freebsd.org Subject: bhyve behavior under cpuset_setaffinity Date: Fri, 25 Apr 2014 23:05:06 +0400 Message-ID: <11465971.5E2sZFzEpK@kde4.my.domain> User-Agent: KMail/4.12.4 (FreeBSD/11.0-CURRENT; KDE/4.12.4; amd64; ; ) MIME-Version: 1.0 Content-Transfer-Encoding: 7Bit Content-Type: text/plain; charset="us-ascii" X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 25 Apr 2014 19:05:13 -0000 Hi, I planned to look at the bhyve through PMC and found bad behavior if you apply cpuset mask for the bhyve process: guest process on the host begins to consume up to 40-50% without any load within guest. For example by executing follow command on bhyve pid: % cpuset -l 2 -p 3476 % top PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND 3082 root 3 75 0 2075M 5016K CPU2 2 0:01 10.06% bhyve % ktrace -p 3476 % ktrace -C % kdump at this point shows only those entries: ... 3476 vcpu 0 RET ioctl 0 3476 vcpu 0 CALL ioctl(0x3,0xc0787601,0x7fffff9fce40) 3476 vcpu 0 RET ioctl 0 3476 vcpu 0 CALL ioctl(0x3,0xc0787601,0x7fffff9fce40) 3476 vcpu 0 RET ioctl 0 3476 vcpu 0 CALL ioctl(0x3,0xc0787601,0x7fffff9fce40) 3476 vcpu 0 RET ioctl 0 3476 vcpu 0 CALL ioctl(0x3,0xc0787601,0x7fffff9fce40) 3476 vcpu 0 RET ioctl 0 3476 vcpu 0 CALL ioctl(0x3,0xc0787601,0x7fffff9fce40) 3476 vcpu 0 RET ioctl 0 3476 vcpu 0 CALL ioctl(0x3,0xc0787601,0x7fffff9fce40) 3476 vcpu 0 RET ioctl 0 3476 vcpu 0 CALL ioctl(0x3,0xc0787601,0x7fffff9fce40) 3476 vcpu 1 RET ioctl 0 3476 vcpu 1 CALL ioctl(0x3,0xc0787601,0x7fffff7fbe40) 3476 vcpu 1 RET ioctl 0 3476 vcpu 1 CALL ioctl(0x3,0xc0787601,0x7fffff7fbe40) 3476 vcpu 0 RET ioctl 0 3476 vcpu 0 CALL ioctl(0x3,0xc0787601,0x7fffff9fce40) 3476 vcpu 0 RET ioctl 0 3476 vcpu 0 CALL ioctl(0x3,0xc0787601,0x7fffff9fce40) 3476 vcpu 0 RET ioctl 0 3476 vcpu 0 CALL ioctl(0x3,0xc0787601,0x7fffff9fce40) 3476 vcpu 0 RET ioctl 0 3476 vcpu 0 CALL ioctl(0x3,0xc0787601,0x7fffff9fce40) 3476 vcpu 1 RET ioctl 0 3476 vcpu 1 CALL ioctl(0x3,0xc0787601,0x7fffff7fbe40) 3476 vcpu 1 RET ioctl 0 3476 vcpu 1 CALL ioctl(0x3,0xc0787601,0x7fffff7fbe40) 3476 vcpu 0 RET ioctl 0 .. All process in the guest is very slow and any action adjust the load on the CPU>= 100% Without cpuset_setaffinity in the same conditions ktrace output is similar: .. 22506 vcpu 1 CALL ioctl(0x3,0xc0787601,0x7fffff7fbe40) 22506 vcpu 0 RET ioctl 0 22506 vcpu 0 CALL ioctl(0x3,0xc0787601,0x7fffff9fce40) 22506 vcpu 0 RET ioctl 0 22506 vcpu 0 CALL ioctl(0x3,0xc0787601,0x7fffff9fce40) 22506 vcpu 0 RET ioctl 0 22506 vcpu 0 CALL ioctl(0x3,0xc0787601,0x7fffff9fce40) 22506 vcpu 0 RET ioctl 0 22506 vcpu 0 CALL ioctl(0x3,0xc0787601,0x7fffff9fce40) 22506 vcpu 1 RET ioctl 0 22506 vcpu 0 RET ioctl 0 22506 vcpu 1 CALL ioctl(0x3,0xc0787601,0x7fffff7fbe40) 22506 vcpu 0 CALL ioctl(0x3,0xc0787601,0x7fffff9fce40) 22506 vcpu 1 RET ioctl 0 22506 vcpu 1 CALL ioctl(0x3,0xc0787601,0x7fffff7fbe40) 22506 vcpu 1 RET ioctl 0 22506 vcpu 1 CALL ioctl(0x3,0xc0787601,0x7fffff7fbe40) 22506 vcpu 1 RET ioctl 0 22506 vcpu 1 CALL ioctl(0x3,0xc0787601,0x7fffff7fbe40) 22506 vcpu 1 RET ioctl 0 22506 vcpu 1 CALL ioctl(0x3,0xc0787601,0x7fffff7fbe40) .. but the cpu usage on the core is close to 5-10% wherein guest processes are very responsive.