From owner-freebsd-virtualization@FreeBSD.ORG Sun Apr 27 10:46:02 2014 Return-Path: Delivered-To: freebsd-virtualization@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 731A97EC for ; Sun, 27 Apr 2014 10:46:02 +0000 (UTC) Received: from mail-la0-x235.google.com (mail-la0-x235.google.com [IPv6:2a00:1450:4010:c03::235]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id E8F867C7 for ; Sun, 27 Apr 2014 10:46:01 +0000 (UTC) Received: by mail-la0-f53.google.com with SMTP id ec20so4199886lab.40 for ; Sun, 27 Apr 2014 03:45:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=sender:date:from:to:subject:message-id:mime-version:content-type :content-disposition:user-agent; bh=Bo5R78XCsmRM/abulXQe5pzmmSSnhI+zOl5GEdyq7TM=; b=OGII31cynY1hpYfyKq5q8hfS2hS5peC1MpBTwPGf5GcHHWpkpCid+/K3rRbFDqLGUR i1aqSg5/O4M/6hK9Gskja8J1a3NWAOVUXjBKx2JrZ4H+exInjpSxyDw8kk05jek5whj9 JAvSjw5gqAkhe5K7CCTpQw9eIL5BpYYYG5dFPDXZ8m/CJVt/SgsuD9SOBA3qw8ZruI0L 32UgQ5weC09pMBBGAZ7wPv5Sf4InqN/Muvlyy9q2Swhacl6lZn3BlhAETdcKB88mFE2D kIHx3894Pl7HdKlWH2cGwjyOeRlO5xQ99hfbukd2FkGxLtubkg/zyJeCmj5M/zvXf6sF iTXQ== X-Received: by 10.152.170.137 with SMTP id am9mr14255307lac.15.1398595559824; Sun, 27 Apr 2014 03:45:59 -0700 (PDT) Received: from kloomba ([95.104.135.172]) by mx.google.com with ESMTPSA id bm3sm14871756lbb.12.2014.04.27.03.45.58 for (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sun, 27 Apr 2014 03:45:58 -0700 (PDT) Sender: Roman Bogorodskiy Date: Sun, 27 Apr 2014 14:45:13 +0400 From: Roman Bogorodskiy To: freebsd-virtualization@FreeBSD.org Subject: [PATCH] Flexible vcpu pinning configuration Message-ID: <20140427104511.GA7804@kloomba> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="Pd0ReVV5GZGQvF3a" Content-Disposition: inline User-Agent: Mutt/1.5.23 (2014-03-12) X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 27 Apr 2014 10:46:02 -0000 --Pd0ReVV5GZGQvF3a Content-Type: multipart/mixed; boundary="6c2NcOVqGQ03X4Wi" Content-Disposition: inline --6c2NcOVqGQ03X4Wi Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable I've created an initial version of the patch which allows more flexible vcpu pinning configuration. Current schema is: bhyve -p N=20 pins vcpu i to hostcpu N + i. The propsed extension is: bhyve -p N:M .... -p 0:1 -p 3:5 which pins vcpu N to host pcpu M. Option needs to be specified individually for each vcpu. So it works like that for me: sudo /usr/sbin/bhyve -p 0:0 -p 1:3 -c 2 ... # sudo cpuset -g -t 100262 tid 100262 mask: 0 # sudo cpuset -g -t 100264 tid 100264 mask: 3 PS I used cpumat_t* array to store these values instead of int, because if the idea is OK, I'll extend it to support ranges like e.g. cpuset(1) supports, e.g.: "1:2-5". The questions are: - Is it OK to chance '-p' arg syntax or it's better to introduce a new one? - Is the syntax OK (currently: 'vcpu:pcpu', later 'vcpu:pcpuN-pcpuM,pcpuX")? Roman Bogorodskiy --6c2NcOVqGQ03X4Wi Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="bhyve_vcpupin.diff" Content-Transfer-Encoding: quoted-printable Index: bhyverun.c =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D --- bhyverun.c (revision 264921) +++ bhyverun.c (working copy) @@ -83,7 +83,6 @@ =20 int guest_ncpus; =20 -static int pincpu =3D -1; static int guest_vmexit_on_hlt, guest_vmexit_on_pause, disable_x2apic; static int virtio_msix =3D 1; =20 @@ -119,6 +118,8 @@ int mt_vcpu;=09 } mt_vmm_info[VM_MAXCPU]; =20 +cpuset_t *vcpumap[VM_MAXCPU] =3D { NULL }; + static void usage(int code) { @@ -146,6 +147,56 @@ exit(code); } =20 +static int +pincpu_parse(const char *cpupin) +{ + char *token, *string; + size_t i =3D 0; + int vcpu =3D -1, pcpu =3D -1, ret =3D -1; + cpuset_t *mask; + + if ((string =3D strdup(cpupin)) =3D=3D NULL) { + fprintf(stderr, "strdup failed: %d\n", errno); + return -1; + } + + while ((token =3D strsep(&string, ":")) !=3D NULL) { + switch (i) { + case 0: + vcpu =3D atoi(token); + break; + case 1: + pcpu =3D atoi(token); + break; + default: + fprintf(stderr, "invalid format: %s\n", token); + goto cleanup; + } + i++; + } + + if (vcpu =3D=3D -1 || pcpu =3D=3D -1) { + fprintf(stderr, "invalid format: %s\n", cpupin); + goto cleanup; + } + + if ((mask =3D malloc(sizeof(cpuset_t))) =3D=3D NULL) { + fprintf(stderr, "malloc failed: %d\n", errno); + goto cleanup; + } + CPU_ZERO(mask); + CPU_SET(pcpu, mask); + + vcpumap[vcpu] =3D mask; + + ret =3D 0; + +cleanup: + free(string); + + return ret; +} + void * paddr_guest2host(struct vmctx *ctx, uintptr_t gaddr, size_t len) { @@ -479,15 +530,13 @@ static void vm_loop(struct vmctx *ctx, int vcpu, uint64_t rip) { - cpuset_t mask; int error, rc, prevcpu; enum vm_exitcode exitcode; =20 - if (pincpu >=3D 0) { - CPU_ZERO(&mask); - CPU_SET(pincpu + vcpu, &mask); + if (vcpumap[vcpu] !=3D NULL) { error =3D pthread_setaffinity_np(pthread_self(), - sizeof(mask), &mask); + sizeof(cpuset_t), + vcpumap[vcpu]); assert(error =3D=3D 0); } =20 @@ -622,7 +671,10 @@ bvmcons =3D 1; break; case 'p': - pincpu =3D atoi(optarg); + if (pincpu_parse(optarg) < 0) { + errx(EX_USAGE, "invalid cpu pin " + "configuration '%s'", optarg); + } break; case 'c': guest_ncpus =3D atoi(optarg); --6c2NcOVqGQ03X4Wi-- --Pd0ReVV5GZGQvF3a Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.13 (FreeBSD) iQEcBAEBAgAGBQJTXN+3AAoJEMltX/4IwiJqNHUH/3AdX0YTgLg45PFDyrwV04Gr VfQIwTEJ1FneeY2FCJ27cTYuJRGI6T/uYuZO5y0r1XPKXOc8tvbF4kT7dOr/nx80 3Ipoh0m+sgQE92zfuKpn6FTSzNkEv8Qo3y3a1CDueX+f+8SfzhwqPNpRTBzCekNW XIHmqQHyan6XrQ8lvWx1cs+502ZSsQXd5+s5DqM527u1pHUJ8IVrFzidFVSPgwO+ medP6M4GEpTvjBTCp75LhgL4q3o9hxu1t9f0MMateQha+H7h8s33e378dBuip+Nw 212/bg3VU2W4cVZ528DfQz9KlnjPZrzTTLZhBTfq5X21Bz1dSuaydnDoPjoUWfA= =1fNw -----END PGP SIGNATURE----- --Pd0ReVV5GZGQvF3a--