From owner-freebsd-virtualization@FreeBSD.ORG Wed Jun 10 06:24:14 2015 Return-Path: Delivered-To: freebsd-virtualization@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 5DFCC4B8; Wed, 10 Jun 2015 06:24:14 +0000 (UTC) (envelope-from neelnatu@gmail.com) Received: from mail-wi0-x229.google.com (mail-wi0-x229.google.com [IPv6:2a00:1450:400c:c05::229]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id EAFDC17DB; Wed, 10 Jun 2015 06:24:13 +0000 (UTC) (envelope-from neelnatu@gmail.com) Received: by wibdq8 with SMTP id dq8so36175231wib.1; Tue, 09 Jun 2015 23:24:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=VAxef04DUL6H7PbP0vCgleh3Kw4O0ER7nkAo5aokfI0=; b=CD0TYE2H6bAUM8/M4cQhpBTG0Gc0g+E0vpbhQzEPpmQteeA8aSpktxWE2z7fAqANAq Bf6FsbNap0aJHY3VWVKktKkKPZnql4IVrBFQCP4JnUJY9aJrHSWcDsF3mhyrsq7hfiEl uTM+LCeRnVMrlLT2MTTCfsc/6Gb6F1F0BqHb6cn0RQ+NVNWAMKBIoLrv3xFbuPYkpQ7l B4SYFMNeE9OXF0x1AsHcCZAKnkE6jd1I59wGD99JyEZhWzjj4n9MsNZCBTRDCWt7uGoL BQBU3pdw0zZJjAy3UcNHMdOIa5JFwktyf5Nt6xH8UimBaJW6dPtxCeu5MNE5LVtkA6JV Re1Q== MIME-Version: 1.0 X-Received: by 10.180.94.168 with SMTP id dd8mr15319233wib.76.1433917452523; Tue, 09 Jun 2015 23:24:12 -0700 (PDT) Received: by 10.27.52.18 with HTTP; Tue, 9 Jun 2015 23:24:12 -0700 (PDT) In-Reply-To: <556FEA92.4050704@FreeBSD.org> References: <556D9005.4020802@FreeBSD.org> <556DDDA9.6090005@FreeBSD.org> <556ED071.5030009@FreeBSD.org> <556FEA92.4050704@FreeBSD.org> Date: Tue, 9 Jun 2015 23:24:12 -0700 Message-ID: Subject: Re: bhyve: corrupting zfs pools? From: Neel Natu To: Andriy Gapon Cc: "freebsd-virtualization@freebsd.org" Content-Type: text/plain; charset=UTF-8 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 10 Jun 2015 06:24:14 -0000 Hi Andriy, On Wed, Jun 3, 2015 at 11:05 PM, Andriy Gapon wrote: > On 04/06/2015 07:37, Neel Natu wrote: >> Ok, there are some differences in our systems. The interesting ones >> are number of ASIDs (64 versus 65536), flush-by-asid capability, >> vmcb-clean capability and the number of cores. >> >> I was able to mimic all of these on my Opteron but still wasn't able >> to reproduce the issue. I am going to get a Sempron tomorrow which >> belongs to the same processor family as the Athlon II so hoping that >> it is easier to repro. >> >> BTW does this happen consistently on your system? > > Yes, it happened every time I tried that scenario. > > Thank you very much! If there is anything else that I can do on my side to > help the investigation just let me know. > I tried to repro on a Sempron 3850 APU but still no luck. Since you can repro it readily I'll follow up with you so we can narrow this down on your end. best Neel > -- > Andriy Gapon From owner-freebsd-virtualization@FreeBSD.ORG Wed Jun 10 07:19:29 2015 Return-Path: Delivered-To: freebsd-virtualization@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id CE860CBE for ; Wed, 10 Jun 2015 07:19:29 +0000 (UTC) (envelope-from hafenbrack@freenet.de) Received: from mout2.freenet.de (mout2.freenet.de [IPv6:2001:748:100:40::2:4]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "*.freenet.de", Issuer "TeleSec ServerPass DE-2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 8B6E513A5 for ; Wed, 10 Jun 2015 07:19:29 +0000 (UTC) (envelope-from hafenbrack@freenet.de) Received: from [195.4.92.140] (helo=mjail0.freenet.de) by mout2.freenet.de with esmtpa (ID hafenbrack@freenet.de) (port 25) (Exim 4.82 #2) id 1Z2aII-0004aA-4Q for freebsd-virtualization@freebsd.org; Wed, 10 Jun 2015 09:19:26 +0200 Received: from localhost ([::1]:34382 helo=mjail0.freenet.de) by mjail0.freenet.de with esmtpa (ID hafenbrack@freenet.de) (Exim 4.82 #2) id 1Z2aIH-0003AX-IV for freebsd-virtualization@freebsd.org; Wed, 10 Jun 2015 09:19:25 +0200 Received: from mx4.freenet.de ([195.4.92.14]:56932) by mjail0.freenet.de with esmtpa (ID hafenbrack@freenet.de) (Exim 4.82 #2) id 1Z2aEm-00070I-8y for freebsd-virtualization@freebsd.org; Wed, 10 Jun 2015 09:15:48 +0200 Received: from dslb-188-106-027-177.188.106.pools.vodafone-ip.de ([188.106.27.177]:50003 helo=[192.168.2.103]) by mx4.freenet.de with esmtpsa (ID hafenbrack@freenet.de) (TLSv1.2:DHE-RSA-AES128-SHA:128) (port 465) (Exim 4.82 #2) id 1Z2aEm-0005dn-J8 for freebsd-virtualization@freebsd.org; Wed, 10 Jun 2015 09:15:48 +0200 Message-ID: <5577E427.4010406@freenet.de> Date: Wed, 10 Jun 2015 09:15:51 +0200 From: Gerd Hafenbrack User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:31.0) Gecko/20100101 Thunderbird/31.7.0 MIME-Version: 1.0 To: freebsd-virtualization@freebsd.org Subject: Re: bhyve: corrupting zfs pools? References: <556D9005.4020802@FreeBSD.org> <556DDDA9.6090005@FreeBSD.org> <556ED071.5030009@FreeBSD.org> <556FEA92.4050704@FreeBSD.org> In-Reply-To: Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-Originated-At: 188.106.27.177!50003 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 10 Jun 2015 07:19:29 -0000 On 2015-06-10 08:24, Neel Natu wrote: > ... > On Wed, Jun 3, 2015 at 11:05 PM, Andriy Gapon wrote: >> >On 04/06/2015 07:37, Neel Natu wrote: >>> >>... I am going to get a Sempron tomorrow which >>> >>belongs to the same processor family as the Athlon II ... >> > ... > I tried to repro on a Sempron 3850 APU but still no luck. ... The Sempron 3850 APU is a Kabini (2013) and several generations younger than an Athlon II, which is a K10 architecture (2009). From owner-freebsd-virtualization@FreeBSD.ORG Wed Jun 10 20:14:49 2015 Return-Path: Delivered-To: freebsd-virtualization@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 107465FF for ; Wed, 10 Jun 2015 20:14:49 +0000 (UTC) (envelope-from stefan.andritoiu@gmail.com) Received: from mail-oi0-x241.google.com (mail-oi0-x241.google.com [IPv6:2607:f8b0:4003:c06::241]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id CC41F13E6 for ; Wed, 10 Jun 2015 20:14:48 +0000 (UTC) (envelope-from stefan.andritoiu@gmail.com) Received: by oiax69 with SMTP id x69so4485472oia.1 for ; Wed, 10 Jun 2015 13:14:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:date:message-id:subject:from:to:content-type; bh=jlxV0A7Y7cO2zwg4jE5bhg1yZpOJE4PEw3t7WWgwkqk=; b=YBDFwmVKJM/pGYNDh3GXowIp1yY0ZPvkfjgU+YuGAd+hbB08fb3zR6xxG+g0iaBbTj KrxBRak89ujlJp4ougv/vuXxKu75+FzaXM/Lnk03lrXz3dqakrB/DsPbjaxCXoSfy5pi EWe2VbYVnChAk6C10LEjvyOIsrwsIstCYwnLgLzqbdfU6zY3pgXhdpWg6qSRYyJkLhDL lvSroL+wZ3KkUYbb5sbCNLHgx54O4mG+plCzaz/lnWc1zdWPmJgBGc06Cyd9Rj/iY41s i9kVW626yG1zlw22hXlG02CMmI6a9dPIpTooGIQNLUAVFB1Ab2p7JUSZDTOniuUl2kv4 mRAQ== MIME-Version: 1.0 X-Received: by 10.202.50.198 with SMTP id y189mr4135537oiy.21.1433967288018; Wed, 10 Jun 2015 13:14:48 -0700 (PDT) Received: by 10.60.82.168 with HTTP; Wed, 10 Jun 2015 13:14:47 -0700 (PDT) Date: Wed, 10 Jun 2015 23:14:47 +0300 Message-ID: Subject: Gang scheduling implementation in the ULE scheduler From: Stefan Andritoiu To: freebsd-virtualization@freebsd.org Content-Type: text/plain; charset=UTF-8 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 10 Jun 2015 20:14:49 -0000 Hello, I am currently working on a gang scheduling implementation for the bhyve VCPU-threads on FreeBSD 10.1. I have added a new field "int gang" to the thread structure to specify the gang it is part of (0 for no gang), and have modified the bhyve code to initialize this field when a VCPU is created. I will post these modifications in another message. When I start a Virtual Machine, during the guest's boot, IPIs are sent and received correctly between CPUs, but after a few seconds I get: spin lock 0xffffffff8164c290 (smp rendezvous) held by 0xfffff8000296c000 (tid 100009) too long panic: spin lock held too long If I limit the number of IPIs that are sent, I do not have this problem. Which leads me to believe that (because of the constant context-switch when the guest boots), the high number of IPIs sent starve the system. Does anyone know what is happening? And maybe know of a possible solution? Thank you, Stefan ====================================================================================== I have added here the modifications to the sched_ule.c file and a brief explanation of it: In struct tdq, I have added two new field: - int scheduled_gang; /* Set to a non-zero value if the respective CPU is required to schedule a thread belonging to a gang. The value of scheduled_gang also being the ID of the gang that we want scheduled. For now I have considered only one running guest, so the value is 0 or 1 */ - int gang_leader; /* Set if the respective CPU is the one who has initialized gang scheduling. Zero otherwise. Not relevant to the final code and will be removed. Just for debugging purposes. */ Created a new function "static void schedule_gang(void * arg)" that will be called by each processor when it receives an IPI from the gang leader: - sets scheduled_gang = 1 - informs the system that it needs to reschedule. Not yet implemented In function "struct thread* tdq_choose (struct tdq * tdq)": if (tdq->scheduled_gang) - checks to see if a thread belonging to a gang must be scheduled. If so, calls functions that check the runqs and return a gang thread. I have yet to implement these functions. In function "sched_choose()": if (td->gang) - checks if the chosen thread is part of a gang. If so it signals all other CPUs to run function "schedule_gang(void * gang)". if (tdq->scheduled_gang) - if scheduled_gang is set it means that the scheduler is called after the the code in schedule_gang() has ran, and bypasses sending IPIs to the other CPUs. If not for this checkup, a CPU would receive a IPI; set scheduled_gang=1; the scheduler would be called and would choose a thread to run; that thread would be part of a gang; an IPI would be sent to all other CPUs. A constant back-and-forth of IPIs between the CPUs would be created. The CPU that initializes gang scheduling, does not receive an IPI, and does not even call the "schedule_gang(void * gang)" function. It continues in scheduling the gang-thread it selected, the one that started the gang scheduling process. =================================================================== --- sched_ule.c (revision 24) +++ sched_ule.c (revision 26) @@ -247,6 +247,9 @@ struct runq tdq_timeshare; /* timeshare run queue. */ struct runq tdq_idle; /* Queue of IDLE threads. */ char tdq_name[TDQ_NAME_LEN]; + + int gang_leader; + int scheduled_gang; #ifdef KTR char tdq_loadname[TDQ_LOADNAME_LEN]; #endif @@ -1308,6 +1311,20 @@ struct thread *td; TDQ_LOCK_ASSERT(tdq, MA_OWNED); + + /* Pick gang thread to run */ + if (tdq->scheduled_gang){ + /* basically the normal choosing of threads but with regards to scheduled_gang + tdq = runq_choose_gang(&tdq->realtime); + if (td != NULL) + return (td); + + td = runq_choose_from_gang(&tdq->tdq_timeshare, tdq->tdq_ridx); + if (td != NULL) + return (td); + */ + } + td = runq_choose(&tdq->tdq_realtime); if (td != NULL) return (td); @@ -2295,6 +2312,22 @@ return (load); } +static void +schedule_gang(void * arg){ + struct tdq *tdq; + struct tdq *from_tdq = arg; + tdq = TDQ_SELF(); + + if(tdq == from_tdq){ + /* Just for testing IPI. Code is never reached, and should never be*/ + tdq->scheduled_gang = 1; +// printf("[schedule_gang] received IPI from himself\n"); + } + else{ + tdq->scheduled_gang = 1; +// printf("[schedule_gang] received on cpu: %s \n", tdq->tdq_name); + } +} /* * Choose the highest priority thread to run. The thread is removed from * the run-queue while running however the load remains. For SMP we set @@ -2305,11 +2338,26 @@ { struct thread *td; struct tdq *tdq; + cpuset_t map; tdq = TDQ_SELF(); TDQ_LOCK_ASSERT(tdq, MA_OWNED); td = tdq_choose(tdq); if (td) { + if(tdq->scheduled_gang){ + /* Scheduler called after IPI + jump over rendezvous*/ + tdq->scheduled_gang = 0; + } + else{ + if(td->gang){ + map = all_cpus; + CPU_CLR(curcpu, &map); + + smp_rendezvous_cpus(map, NULL, schedule_gang, NULL, tdq); + } + } + tdq_runq_rem(tdq, td); tdq->tdq_lowpri = td->td_priority; return (td); From owner-freebsd-virtualization@FreeBSD.ORG Wed Jun 10 20:41:02 2015 Return-Path: Delivered-To: freebsd-virtualization@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 94324458 for ; Wed, 10 Jun 2015 20:41:02 +0000 (UTC) (envelope-from grehan@freebsd.org) Received: from alto.onthenet.com.au (alto2.onthenet.com.au [203.13.68.14]) by mx1.freebsd.org (Postfix) with ESMTP id 512AC1AF4 for ; Wed, 10 Jun 2015 20:41:01 +0000 (UTC) (envelope-from grehan@freebsd.org) Received: from iredmail.onthenet.com.au (iredmail.onthenet.com.au [203.13.68.150]) by alto.onthenet.com.au (Postfix) with ESMTP id 4629312263 for ; Thu, 11 Jun 2015 06:40:59 +1000 (EST) Received: from localhost (iredmail.onthenet.com.au [127.0.0.1]) by iredmail.onthenet.com.au (Postfix) with ESMTP id 3E256280F76 for ; Thu, 11 Jun 2015 06:40:59 +1000 (AEST) X-Amavis-Modified: Mail body modified (using disclaimer) - iredmail.onthenet.com.au Received: from iredmail.onthenet.com.au ([127.0.0.1]) by localhost (iredmail.onthenet.com.au [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id BRRqPaOXALZS for ; Thu, 11 Jun 2015 06:40:59 +1000 (AEST) Received: from Peters-MacBook-Pro.local (c-67-180-92-13.hsd1.ca.comcast.net [67.180.92.13]) by iredmail.onthenet.com.au (Postfix) with ESMTPSA id 18F35280F5D; Thu, 11 Jun 2015 06:40:56 +1000 (AEST) Message-ID: <5578A0D7.1080505@freebsd.org> Date: Wed, 10 Jun 2015 13:40:55 -0700 From: Peter Grehan User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:31.0) Gecko/20100101 Thunderbird/31.7.0 MIME-Version: 1.0 To: Stefan Andritoiu CC: freebsd-virtualization@freebsd.org Subject: Re: Gang scheduling implementation in the ULE scheduler References: In-Reply-To: Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 10 Jun 2015 20:41:02 -0000 Hi Stefan, > When I start a Virtual Machine, during the guest's boot, IPIs are sent > and received correctly between CPUs, but after a few seconds I get: > spin lock 0xffffffff8164c290 (smp rendezvous) held by > 0xfffff8000296c000 (tid 100009) too long > panic: spin lock held too long Is this on a lightly loaded machine ? I've not looked at your diff in detail but it's possibly a code path where a spinlock wasn't unlocked, leaving another CPU spinning. later, Peter. From owner-freebsd-virtualization@FreeBSD.ORG Wed Jun 10 20:54:25 2015 Return-Path: Delivered-To: freebsd-virtualization@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 34D26A33; Wed, 10 Jun 2015 20:54:25 +0000 (UTC) (envelope-from stefan.andritoiu@gmail.com) Received: from mail-ob0-x242.google.com (mail-ob0-x242.google.com [IPv6:2607:f8b0:4003:c01::242]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id F130E1E7A; Wed, 10 Jun 2015 20:54:24 +0000 (UTC) (envelope-from stefan.andritoiu@gmail.com) Received: by obcwm4 with SMTP id wm4so6145857obc.3; Wed, 10 Jun 2015 13:54:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=T08klaNeBVKiGJnMwoeA+MxMQUXCEEVovrF8+b80o5o=; b=pmBl0WYB8TFoNe/LrPD35MnQw3xw+f2eDSn1KRAZdzrWUq6oGnLH1Z/Nb68vRLREhW B66npZ1kA+ceJczYZgzyUTWDacuUBgZhdlYzYQ0BFnWnt78bu8fcD2hG/+yDjxnXA6C8 Lw6O9Zcgsy9CPaTv52I8+O4oZ3XzLz89mYzMTmCpng7Y5MzIgw8rZHgqQju60qPLv+dL YA5EJHFGMTHqwBok5cb+Mg3tuZIFslmgWCxqL/YzNPjrbC8sC2SnmXFDvmPFGP7ltwqi /nPnPcxcPwhfkAFM/uQpgrtfitvlluYoE48XypIkDPHj4ogVeiovEFR9K/auMEl2qfRm XNXg== MIME-Version: 1.0 X-Received: by 10.202.50.198 with SMTP id y189mr4257205oiy.21.1433969664389; Wed, 10 Jun 2015 13:54:24 -0700 (PDT) Received: by 10.60.82.168 with HTTP; Wed, 10 Jun 2015 13:54:24 -0700 (PDT) In-Reply-To: <5578A0D7.1080505@freebsd.org> References: <5578A0D7.1080505@freebsd.org> Date: Wed, 10 Jun 2015 23:54:24 +0300 Message-ID: Subject: Re: Gang scheduling implementation in the ULE scheduler From: Stefan Andritoiu To: Peter Grehan , freebsd-virtualization@freebsd.org Content-Type: text/plain; charset=UTF-8 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 10 Jun 2015 20:54:25 -0000 Hi Peter, Yes, I think it is lightly loaded. I am trying to run a FreeBSD guest with 2 VCPUs on a 4 CPU host, with no other program running. But I am having the same problem even when I start a 1 VCPU guest. later, Stefan On Wed, Jun 10, 2015 at 11:40 PM, Peter Grehan wrote: > Hi Stefan, > >> When I start a Virtual Machine, during the guest's boot, IPIs are sent >> and received correctly between CPUs, but after a few seconds I get: >> spin lock 0xffffffff8164c290 (smp rendezvous) held by >> 0xfffff8000296c000 (tid 100009) too long >> panic: spin lock held too long > > > Is this on a lightly loaded machine ? I've not looked at your diff in > detail but it's possibly a code path where a spinlock wasn't unlocked, > leaving another CPU spinning. > > later, > > Peter. > From owner-freebsd-virtualization@FreeBSD.ORG Thu Jun 11 00:02:25 2015 Return-Path: Delivered-To: freebsd-virtualization@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 83009669 for ; Thu, 11 Jun 2015 00:02:25 +0000 (UTC) (envelope-from neelnatu@gmail.com) Received: from mail-wi0-x231.google.com (mail-wi0-x231.google.com [IPv6:2a00:1450:400c:c05::231]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 0D3191E01 for ; Thu, 11 Jun 2015 00:02:25 +0000 (UTC) (envelope-from neelnatu@gmail.com) Received: by wiga1 with SMTP id a1so62023733wig.0 for ; Wed, 10 Jun 2015 17:02:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=XIB9Jd04stfrBWJ2oxJq3BbAuIbDyA+sSDwwQb4+YME=; b=uzRq7g03LPI+r9cRn48FpJXmb07hYmIPEicK8A/kyUQZGqSfMvlQXgdCIG2Xxo0a+q LguAQgAacmhSHqs91PTZLLRfyzt18SEs4RYhOjSWG+0kmNIBlIamMdbf3sVLVcEvqzNo afU6kqlSwBSm9yaYv8j55qpW9FAxEFbaz9YHRrV72Jg0945FMUQtDHiz0X78QF9rpe/s mBXxGPLXeBvwCRxEqUCW8EXjzrV9AHwBTY7ffMnVaxR0dFh+n1Lx+ESh8HL3Q7hcgRfb vnkrwKla/W/KxuWyE0GzLkloRWMGS45rVbaHrSzg9uBG1LH9RS/baf3olK5eGEqQQKw1 7p1Q== MIME-Version: 1.0 X-Received: by 10.180.88.99 with SMTP id bf3mr23966701wib.75.1433980942741; Wed, 10 Jun 2015 17:02:22 -0700 (PDT) Received: by 10.27.52.18 with HTTP; Wed, 10 Jun 2015 17:02:22 -0700 (PDT) In-Reply-To: References: Date: Wed, 10 Jun 2015 17:02:22 -0700 Message-ID: Subject: Re: Gang scheduling implementation in the ULE scheduler From: Neel Natu To: Stefan Andritoiu Cc: "freebsd-virtualization@freebsd.org" Content-Type: text/plain; charset=UTF-8 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 11 Jun 2015 00:02:25 -0000 Hi Stefan, On Wed, Jun 10, 2015 at 1:14 PM, Stefan Andritoiu wrote: > Hello, > > I am currently working on a gang scheduling implementation for the > bhyve VCPU-threads on FreeBSD 10.1. > I have added a new field "int gang" to the thread structure to specify > the gang it is part of (0 for no gang), and have modified the bhyve > code to initialize this field when a VCPU is created. I will post > these modifications in another message. > > When I start a Virtual Machine, during the guest's boot, IPIs are sent > and received correctly between CPUs, but after a few seconds I get: > spin lock 0xffffffff8164c290 (smp rendezvous) held by > 0xfffff8000296c000 (tid 100009) too long > panic: spin lock held too long > > If I limit the number of IPIs that are sent, I do not have this > problem. Which leads me to believe that (because of the constant > context-switch when the guest boots), the high number of IPIs sent > starve the system. > > Does anyone know what is happening? And maybe know of a possible solution? > In your patch 'smp_rendezvous()' is being called with the TDQ locked. There are a few code paths in ULE where it will want to lock two TDQs at the same time (see tdq_lock_pair()). This has the potential to cause a deadlock if the 2nd TDQ in tdq_lock_pair() is the one that was locked before calling 'smp_rendezvous()'. To verify this theory can you set the following sysctls and repeat the test? $ sysctl kern.sched.steal_idle=0 $ sysctl kern.sched.rebalance=0 best Neel > Thank you, > Stefan > > > ====================================================================================== > I have added here the modifications to the sched_ule.c file and a > brief explanation of it: > > In struct tdq, I have added two new field: > - int scheduled_gang; > /* Set to a non-zero value if the respective CPU is required to > schedule a thread belonging to a gang. The value of scheduled_gang > also being the ID of the gang that we want scheduled. For now I have > considered only one running guest, so the value is 0 or 1 */ > - int gang_leader; > /* Set if the respective CPU is the one who has initialized gang > scheduling. Zero otherwise. Not relevant to the final code and will be > removed. Just for debugging purposes. */ > > Created a new function "static void schedule_gang(void * arg)" that > will be called by each processor when it receives an IPI from the gang > leader: > - sets scheduled_gang = 1 > - informs the system that it needs to reschedule. Not yet implemented > > In function "struct thread* tdq_choose (struct tdq * tdq)": > if (tdq->scheduled_gang) - checks to see if a thread belonging to > a gang must be scheduled. If so, calls functions that check the runqs > and return a gang thread. I have yet to implement these functions. > > In function "sched_choose()": > if (td->gang) - checks if the chosen thread is part of a gang. If > so it signals all other CPUs to run function "schedule_gang(void * > gang)". > if (tdq->scheduled_gang) - if scheduled_gang is set it means that > the scheduler is called after the the code in schedule_gang() has ran, > and bypasses sending IPIs to the other CPUs. If not for this checkup, > a CPU would receive a IPI; set scheduled_gang=1; the scheduler would > be called and would choose a thread to run; that thread would be part > of a gang; an IPI would be sent to all other CPUs. A constant > back-and-forth of IPIs between the CPUs would be created. > > The CPU that initializes gang scheduling, does not receive an IPI, and > does not even call the "schedule_gang(void * gang)" function. It > continues in scheduling the gang-thread it selected, the one that > started the gang scheduling process. > > > =================================================================== > --- sched_ule.c (revision 24) > +++ sched_ule.c (revision 26) > @@ -247,6 +247,9 @@ > struct runq tdq_timeshare; /* timeshare run queue. */ > struct runq tdq_idle; /* Queue of IDLE threads. */ > char tdq_name[TDQ_NAME_LEN]; > + > + int gang_leader; > + int scheduled_gang; > #ifdef KTR > char tdq_loadname[TDQ_LOADNAME_LEN]; > #endif > @@ -1308,6 +1311,20 @@ > struct thread *td; > > TDQ_LOCK_ASSERT(tdq, MA_OWNED); > + > + /* Pick gang thread to run */ > + if (tdq->scheduled_gang){ > + /* basically the normal choosing of threads but with regards to scheduled_gang > + tdq = runq_choose_gang(&tdq->realtime); > + if (td != NULL) > + return (td); > + > + td = runq_choose_from_gang(&tdq->tdq_timeshare, tdq->tdq_ridx); > + if (td != NULL) > + return (td); > + */ > + } > + > td = runq_choose(&tdq->tdq_realtime); > if (td != NULL) > return (td); > @@ -2295,6 +2312,22 @@ > return (load); > } > > +static void > +schedule_gang(void * arg){ > + struct tdq *tdq; > + struct tdq *from_tdq = arg; > + tdq = TDQ_SELF(); > + > + if(tdq == from_tdq){ > + /* Just for testing IPI. Code is never reached, and should never be*/ > + tdq->scheduled_gang = 1; > +// printf("[schedule_gang] received IPI from himself\n"); > + } > + else{ > + tdq->scheduled_gang = 1; > +// printf("[schedule_gang] received on cpu: %s \n", tdq->tdq_name); > + } > +} > /* > * Choose the highest priority thread to run. The thread is removed from > * the run-queue while running however the load remains. For SMP we set > @@ -2305,11 +2338,26 @@ > { > struct thread *td; > struct tdq *tdq; > + cpuset_t map; > > tdq = TDQ_SELF(); > TDQ_LOCK_ASSERT(tdq, MA_OWNED); > td = tdq_choose(tdq); > if (td) { > + if(tdq->scheduled_gang){ > + /* Scheduler called after IPI > + jump over rendezvous*/ > + tdq->scheduled_gang = 0; > + } > + else{ > + if(td->gang){ > + map = all_cpus; > + CPU_CLR(curcpu, &map); > + > + smp_rendezvous_cpus(map, NULL, schedule_gang, NULL, tdq); > + } > + } > + > tdq_runq_rem(tdq, td); > tdq->tdq_lowpri = td->td_priority; > return (td); > _______________________________________________ > freebsd-virtualization@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization > To unsubscribe, send any mail to "freebsd-virtualization-unsubscribe@freebsd.org" From owner-freebsd-virtualization@FreeBSD.ORG Thu Jun 11 00:56:57 2015 Return-Path: Delivered-To: freebsd-virtualization@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id CFC8722D for ; Thu, 11 Jun 2015 00:56:57 +0000 (UTC) (envelope-from crodr001@gmail.com) Received: from mail-yh0-x233.google.com (mail-yh0-x233.google.com [IPv6:2607:f8b0:4002:c01::233]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 907DA1ABC for ; Thu, 11 Jun 2015 00:56:57 +0000 (UTC) (envelope-from crodr001@gmail.com) Received: by yhid80 with SMTP id d80so27345338yhi.1 for ; Wed, 10 Jun 2015 17:56:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:date:message-id:subject:from:to:content-type; bh=rvZa6QdXTD+B3I9xE+cnNzIF4z+PLm/06bE3nwJADHk=; b=PStJim73Uh9Gpma83k9kKT5h8lPiKXT14xl6/6+jLm4aiq3yNeJc2LytGEeCIyfRXy w280LGMKCTDzquuVX6mqgAOmPXAtCJ1CNKY1TrvcEICF2I1ReldyJ633whk4dtz8IZsm 95tgZ7CmrB/sayMwpYzzMS/NwQplvMYN/JyLzlYukaYvu2A5fYzGhyhuvbFeH+WamtBs aw8sjT1iiR/4Qv8dBdo8TSZnE+UlkfrdjWBZBcr5+qqieFLaQn5KJsPyPiegTRUtghjK jdMf0nmn6VBqlYR6lWD4bnoibQ2erl2JQ4AnlCPb2WvQGy53TgE6HwVMuKwZcve87bri XfWA== MIME-Version: 1.0 X-Received: by 10.129.53.23 with SMTP id c23mr7929667ywa.50.1433984216754; Wed, 10 Jun 2015 17:56:56 -0700 (PDT) Sender: crodr001@gmail.com Received: by 10.13.233.70 with HTTP; Wed, 10 Jun 2015 17:56:56 -0700 (PDT) Date: Wed, 10 Jun 2015 17:56:56 -0700 X-Google-Sender-Auth: YCpOjGsZlgFvGIuIEmPdbCd8ynU Message-ID: Subject: xhyve: bhyve for OS X? From: Craig Rodrigues To: "freebsd-virtualization@freebsd.org" Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 11 Jun 2015 00:56:57 -0000 Hi, Has anyone here tried xhyve (bhyve port for OS X)? http://www.pagetable.com/?p=831 -- Craig From owner-freebsd-virtualization@FreeBSD.ORG Thu Jun 11 01:16:15 2015 Return-Path: Delivered-To: freebsd-virtualization@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 8D97E507; Thu, 11 Jun 2015 01:16:15 +0000 (UTC) (envelope-from ben.perrault@gmail.com) Received: from mail-qk0-x234.google.com (mail-qk0-x234.google.com [IPv6:2607:f8b0:400d:c09::234]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 47B291F2B; Thu, 11 Jun 2015 01:16:15 +0000 (UTC) (envelope-from ben.perrault@gmail.com) Received: by qkhq76 with SMTP id q76so33654965qkh.2; Wed, 10 Jun 2015 18:16:14 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=content-type:mime-version:subject:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to; bh=tHNWXdt/ctyqwuFgkIzOH8I+PwdFg7ya/T2WCh4LbYM=; b=BQrxEAstpOHv2Eqe1QcqpF6OeYdkBX4ZVkSIHNO6M4iMhcy2HfNhfnWsrSYTPMX36j YmTdCjNpMBLjIyO3cPcak7cIONBRLHhDdZI+MDnPFt6A2CGpAdglrircsIiD/Nsv+ZuK cB85uVUMoBgsef+msYpTlT1kXwUDVKeU9kZkG19YwYO6dsPj5iTa8f8w4tPtYp86uJ0t kyBgOVfjbYU0PYdAD9sK7DbkbFXlNKtSMfS6jk2znNQQ3zEgtVSJ4w9WW1190TQJDZOB Q4QTP6CXvKctqKWulBSWSIkXBBxyqIC1TA67TqtPKEzBI7y2ynlvDHKSZse5RzeXSR2v Xr7A== X-Received: by 10.55.17.226 with SMTP id 95mr12809043qkr.63.1433985374291; Wed, 10 Jun 2015 18:16:14 -0700 (PDT) Received: from [10.0.1.11] ([173.228.31.205]) by mx.google.com with ESMTPSA id i7sm4954229qge.32.2015.06.10.18.16.12 (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Wed, 10 Jun 2015 18:16:13 -0700 (PDT) Content-Type: text/plain; charset=utf-8 Mime-Version: 1.0 (Mac OS X Mail 8.2 \(2098\)) Subject: Re: xhyve: bhyve for OS X? From: Benjamin Perrault In-Reply-To: Date: Wed, 10 Jun 2015 18:16:10 -0700 Content-Transfer-Encoding: quoted-printable Message-Id: <30481DF2-5305-48D3-842A-13A8A1E1FFF2@gmail.com> References: To: "freebsd-virtualization@freebsd.org" X-Mailer: Apple Mail (2.2098) X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 11 Jun 2015 01:16:15 -0000 It=E2=80=99s a bit limited. IMO, it is more a mutation of bhyve then a = direct port. It can only run Linux at the moment (or at least, I can=E2=80= =99t get freebsd to run under it, for example - and that seems to = conform to the github documentation ).=20 SVM, PCI passthrough, VMX host & EPT where removed. It is single process = model ( because of the OS X Hypervisor.framework ) - and thus there is = no support for bhyvectl, bhyveload and grub2-bhyve.=20 (more info here: https://github.com/mist64/xhyve )=20 Performance is substantially worse then bhyve on the equivalent hardware = under FreeBSD from my testing. I=E2=80=99ve found that VMware Fusion or = Parallels are consistently quicker across the board at the moment for = anything I=E2=80=99ve tried with Ubuntu/Centos. With that said, it does = run in userspace which should make it fairly secure, especially combined = with OS X=E2=80=99s sandboxing.=20 While it=E2=80=99s an interesting project to watch, it definitely has a = ways to go. best, -bp > On Jun 10, 2015, at 5:56 PM, Craig Rodrigues = wrote: >=20 > Hi, >=20 > Has anyone here tried xhyve (bhyve port for OS X)? >=20 > http://www.pagetable.com/?p=3D831 >=20 > -- > Craig > _______________________________________________ > freebsd-virtualization@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization > To unsubscribe, send any mail to = "freebsd-virtualization-unsubscribe@freebsd.org" From owner-freebsd-virtualization@FreeBSD.ORG Thu Jun 11 09:32:22 2015 Return-Path: Delivered-To: freebsd-virtualization@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 85CDAE16 for ; Thu, 11 Jun 2015 09:32:22 +0000 (UTC) (envelope-from prvs=597ee80ab=roger.pau@citrix.com) Received: from SMTP.CITRIX.COM (smtp.citrix.com [66.165.176.89]) (using TLSv1 with cipher RC4-SHA (128/128 bits)) (Client CN "mail.citrix.com", Issuer "Cybertrust Public SureServer SV CA" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 2CEF71C1D for ; Thu, 11 Jun 2015 09:32:21 +0000 (UTC) (envelope-from prvs=597ee80ab=roger.pau@citrix.com) X-IronPort-AV: E=Sophos;i="5.13,594,1427760000"; d="scan'208";a="270967870" Message-ID: <5579555D.5060002@citrix.com> Date: Thu, 11 Jun 2015 11:31:09 +0200 From: =?windows-1252?Q?Roger_Pau_Monn=E9?= User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:31.0) Gecko/20100101 Thunderbird/31.7.0 MIME-Version: 1.0 To: Stefan Andritoiu , Subject: Re: Lock Holder Preemption on bhyve References: In-Reply-To: Content-Type: text/plain; charset="windows-1252" Content-Transfer-Encoding: 7bit X-DLP: MIA1 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 11 Jun 2015 09:32:22 -0000 Hello, El 06/06/15 a les 0.59, Stefan Andritoiu ha escrit: > Hello everyone, > > My name is Stefan Andritoiu and I'm currently studying Computer > Science in my 4th year at the University POLITEHNICA of Bucharest. > I'm fairly new to the FreeBSD operating system, having only a > background in Linux. For the past few months I've been investigating > the problem of Lock Holder Preemption on bhyve, how other hypervisors > deal with this problem and a possible implementation of these > solutions on bhyve. > I am currently working on implementing Gang Scheduling on FreeBSD to > test if it is a viable solution. I also plan to continue my work, by > implementing and testing other techniques of avoiding overhead caused > by lock holder preemption, and comparing their results. FWIW, Xen doesn't do gang-scheduling, so the lock-holder preemption problem is solved inside of the guest by using pvspinlocks/pvticketlocks. Not to detriment the work you are doing on gang-scheduling, but having some like this would help FreeBSD when running in all virtualized environments regardless of whether the underlying hypervisor does gang-scheduling or not. Some more information about it: https://blog.xenproject.org/2012/05/11/benchmarking-the-new-pv-ticketlock-implementation/ http://www-archive.xenproject.org/files/xensummitboston08/LHP.pdf https://lwn.net/Articles/556141/ Roger. From owner-freebsd-virtualization@FreeBSD.ORG Thu Jun 11 11:24:31 2015 Return-Path: Delivered-To: freebsd-virtualization@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 99093522 for ; Thu, 11 Jun 2015 11:24:31 +0000 (UTC) (envelope-from stefan.andritoiu@gmail.com) Received: from mail-ob0-x244.google.com (mail-ob0-x244.google.com [IPv6:2607:f8b0:4003:c01::244]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 5AFC61ADD for ; Thu, 11 Jun 2015 11:24:31 +0000 (UTC) (envelope-from stefan.andritoiu@gmail.com) Received: by obbnt9 with SMTP id nt9so283251obb.1 for ; Thu, 11 Jun 2015 04:24:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=f8UPe8cghFrQ3foqwuhoL0gVQqhUyJBNvhM/QhObdGk=; b=CBpMkkOD75Nzq9lznGtzc1JLBwr2pAA00oJObYPk0hMfORH3lnYWCiRMZgoojYEDiX QBiDtowh3gZ+1ISv8iDmBHl8C0J2hZNHGzOeNREGDGF02g8MO/zqFvKjPZBXzPF6h+On 75hjXLmkbtjrdxHNiRkkcqEd6M74/A5n6hFJhaiyIE7SWZLzcjGglAjJU09uYW9YUyuS MXCu32gWyoz81XCW0Uz4YCPMQVVkKsxFa+t1Bhu18jVcODRC9CMlkx8XEzDNvivv8A5H /4S7hlTFDogGF1WaA5G3kd7PJSeaGxtG6oTgAI0E4iy8zXNxRRsfB92PK90kUVVwLaCP vxrg== MIME-Version: 1.0 X-Received: by 10.182.97.2 with SMTP id dw2mr7361680obb.85.1434021870084; Thu, 11 Jun 2015 04:24:30 -0700 (PDT) Received: by 10.60.82.168 with HTTP; Thu, 11 Jun 2015 04:24:30 -0700 (PDT) In-Reply-To: References: Date: Thu, 11 Jun 2015 14:24:30 +0300 Message-ID: Subject: Re: Gang scheduling implementation in the ULE scheduler From: Stefan Andritoiu To: Neel Natu Cc: "freebsd-virtualization@freebsd.org" Content-Type: text/plain; charset=UTF-8 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 11 Jun 2015 11:24:31 -0000 On Thu, Jun 11, 2015 at 3:02 AM, Neel Natu wrote: > Hi Stefan, > > On Wed, Jun 10, 2015 at 1:14 PM, Stefan Andritoiu > wrote: >> Hello, >> >> I am currently working on a gang scheduling implementation for the >> bhyve VCPU-threads on FreeBSD 10.1. >> I have added a new field "int gang" to the thread structure to specify >> the gang it is part of (0 for no gang), and have modified the bhyve >> code to initialize this field when a VCPU is created. I will post >> these modifications in another message. >> >> When I start a Virtual Machine, during the guest's boot, IPIs are sent >> and received correctly between CPUs, but after a few seconds I get: >> spin lock 0xffffffff8164c290 (smp rendezvous) held by >> 0xfffff8000296c000 (tid 100009) too long >> panic: spin lock held too long >> >> If I limit the number of IPIs that are sent, I do not have this >> problem. Which leads me to believe that (because of the constant >> context-switch when the guest boots), the high number of IPIs sent >> starve the system. >> >> Does anyone know what is happening? And maybe know of a possible solution? >> > > In your patch 'smp_rendezvous()' is being called with the TDQ locked. > > There are a few code paths in ULE where it will want to lock two TDQs > at the same time (see tdq_lock_pair()). This has the potential to > cause a deadlock if the 2nd TDQ in tdq_lock_pair() is the one that was > locked before calling 'smp_rendezvous()'. > > To verify this theory can you set the following sysctls and repeat the test? > $ sysctl kern.sched.steal_idle=0 > $ sysctl kern.sched.rebalance=0 > > best > Neel > Hi Neel I do not seem have a kern.sched.rebalance variable. I do have kern.sched.balance> I have tested with both sysctl kern.sched.steal_idle=0 sysctl kern.sched.balance=0 and sysctl kern.sched.steal_idle=0 sysctl kern.sched.balance=1 Unfortunately, in both cases the result is the same as before: panic: spin lock held too long best Stefan >> Thank you, >> Stefan >> >> >> ====================================================================================== >> I have added here the modifications to the sched_ule.c file and a >> brief explanation of it: >> >> In struct tdq, I have added two new field: >> - int scheduled_gang; >> /* Set to a non-zero value if the respective CPU is required to >> schedule a thread belonging to a gang. The value of scheduled_gang >> also being the ID of the gang that we want scheduled. For now I have >> considered only one running guest, so the value is 0 or 1 */ >> - int gang_leader; >> /* Set if the respective CPU is the one who has initialized gang >> scheduling. Zero otherwise. Not relevant to the final code and will be >> removed. Just for debugging purposes. */ >> >> Created a new function "static void schedule_gang(void * arg)" that >> will be called by each processor when it receives an IPI from the gang >> leader: >> - sets scheduled_gang = 1 >> - informs the system that it needs to reschedule. Not yet implemented >> >> In function "struct thread* tdq_choose (struct tdq * tdq)": >> if (tdq->scheduled_gang) - checks to see if a thread belonging to >> a gang must be scheduled. If so, calls functions that check the runqs >> and return a gang thread. I have yet to implement these functions. >> >> In function "sched_choose()": >> if (td->gang) - checks if the chosen thread is part of a gang. If >> so it signals all other CPUs to run function "schedule_gang(void * >> gang)". >> if (tdq->scheduled_gang) - if scheduled_gang is set it means that >> the scheduler is called after the the code in schedule_gang() has ran, >> and bypasses sending IPIs to the other CPUs. If not for this checkup, >> a CPU would receive a IPI; set scheduled_gang=1; the scheduler would >> be called and would choose a thread to run; that thread would be part >> of a gang; an IPI would be sent to all other CPUs. A constant >> back-and-forth of IPIs between the CPUs would be created. >> >> The CPU that initializes gang scheduling, does not receive an IPI, and >> does not even call the "schedule_gang(void * gang)" function. It >> continues in scheduling the gang-thread it selected, the one that >> started the gang scheduling process. >> >> >> =================================================================== >> --- sched_ule.c (revision 24) >> +++ sched_ule.c (revision 26) >> @@ -247,6 +247,9 @@ >> struct runq tdq_timeshare; /* timeshare run queue. */ >> struct runq tdq_idle; /* Queue of IDLE threads. */ >> char tdq_name[TDQ_NAME_LEN]; >> + >> + int gang_leader; >> + int scheduled_gang; >> #ifdef KTR >> char tdq_loadname[TDQ_LOADNAME_LEN]; >> #endif >> @@ -1308,6 +1311,20 @@ >> struct thread *td; >> >> TDQ_LOCK_ASSERT(tdq, MA_OWNED); >> + >> + /* Pick gang thread to run */ >> + if (tdq->scheduled_gang){ >> + /* basically the normal choosing of threads but with regards to scheduled_gang >> + tdq = runq_choose_gang(&tdq->realtime); >> + if (td != NULL) >> + return (td); >> + >> + td = runq_choose_from_gang(&tdq->tdq_timeshare, tdq->tdq_ridx); >> + if (td != NULL) >> + return (td); >> + */ >> + } >> + >> td = runq_choose(&tdq->tdq_realtime); >> if (td != NULL) >> return (td); >> @@ -2295,6 +2312,22 @@ >> return (load); >> } >> >> +static void >> +schedule_gang(void * arg){ >> + struct tdq *tdq; >> + struct tdq *from_tdq = arg; >> + tdq = TDQ_SELF(); >> + >> + if(tdq == from_tdq){ >> + /* Just for testing IPI. Code is never reached, and should never be*/ >> + tdq->scheduled_gang = 1; >> +// printf("[schedule_gang] received IPI from himself\n"); >> + } >> + else{ >> + tdq->scheduled_gang = 1; >> +// printf("[schedule_gang] received on cpu: %s \n", tdq->tdq_name); >> + } >> +} >> /* >> * Choose the highest priority thread to run. The thread is removed from >> * the run-queue while running however the load remains. For SMP we set >> @@ -2305,11 +2338,26 @@ >> { >> struct thread *td; >> struct tdq *tdq; >> + cpuset_t map; >> >> tdq = TDQ_SELF(); >> TDQ_LOCK_ASSERT(tdq, MA_OWNED); >> td = tdq_choose(tdq); >> if (td) { >> + if(tdq->scheduled_gang){ >> + /* Scheduler called after IPI >> + jump over rendezvous*/ >> + tdq->scheduled_gang = 0; >> + } >> + else{ >> + if(td->gang){ >> + map = all_cpus; >> + CPU_CLR(curcpu, &map); >> + >> + smp_rendezvous_cpus(map, NULL, schedule_gang, NULL, tdq); >> + } >> + } >> + >> tdq_runq_rem(tdq, td); >> tdq->tdq_lowpri = td->td_priority; >> return (td); >> _______________________________________________ >> freebsd-virtualization@freebsd.org mailing list >> http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization >> To unsubscribe, send any mail to "freebsd-virtualization-unsubscribe@freebsd.org" From owner-freebsd-virtualization@FreeBSD.ORG Thu Jun 11 11:49:08 2015 Return-Path: Delivered-To: freebsd-virtualization@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 0C7C5ACE for ; Thu, 11 Jun 2015 11:49:08 +0000 (UTC) (envelope-from stefan.andritoiu@gmail.com) Received: from mail-ob0-x244.google.com (mail-ob0-x244.google.com [IPv6:2607:f8b0:4003:c01::244]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id C6C151FB5 for ; Thu, 11 Jun 2015 11:49:07 +0000 (UTC) (envelope-from stefan.andritoiu@gmail.com) Received: by obcwm4 with SMTP id wm4so418848obc.3 for ; Thu, 11 Jun 2015 04:49:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type:content-transfer-encoding; bh=CjJaGwNQv6r9NETt1pUlK7GC41Nd0LlkJsgNojaK5qk=; b=S5N7+nbOcql+lj3MawruCyJVvZJ7ekWia8E2XKlgvGUC8DU1iv0/8A7JrDJmfeUMDC V9tb2Kd/xDJC7A7wL0ufEZagI6TIQNH2eIzRIcJ/F91BwMYxTBr85oswFwE4yqUFPzrG Q2Aa+tlSEz9zTgygfpxhb3JLfM7lGN1DA2XW/H/fa5j1ogyAt/7wfVDTIWpdMdFiMtO+ R1VJy9fOWZoGviHra5iOqL7OSx3jqWjvqyYp5Uiqt0nLdpiCkuV9ChhHKLyZBhgFuzfk GeyaXkAetUhE2tZtxsMETM18w8GgACIcHUfK3LJHsEDnIEA9kdyDJ/0q/59mjMzKA5lf 2Gsg== MIME-Version: 1.0 X-Received: by 10.60.155.97 with SMTP id vv1mr7513585oeb.15.1434023347200; Thu, 11 Jun 2015 04:49:07 -0700 (PDT) Received: by 10.60.82.168 with HTTP; Thu, 11 Jun 2015 04:49:07 -0700 (PDT) In-Reply-To: <5579555D.5060002@citrix.com> References: <5579555D.5060002@citrix.com> Date: Thu, 11 Jun 2015 14:49:07 +0300 Message-ID: Subject: Re: Lock Holder Preemption on bhyve From: Stefan Andritoiu To: =?UTF-8?Q?Roger_Pau_Monn=C3=A9?= Cc: freebsd-virtualization@freebsd.org Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 11 Jun 2015 11:49:08 -0000 Hi Roger Thank you for the links, I have been looking for more information on Paravirtualized ticket spinlocks for a later project. I know how Xen handles LHP, but it does so through what are called "invasive methods", that modify the guest. For now, my work focuses on a non-invasive methods in which the guest is not aware that it is a guest. In hopes to offer a solution to LHP, regardless of the guest OS. Thank you, Stefan On Thu, Jun 11, 2015 at 12:31 PM, Roger Pau Monn=C3=A9 wrote: > Hello, > > El 06/06/15 a les 0.59, Stefan Andritoiu ha escrit: >> Hello everyone, >> >> My name is Stefan Andritoiu and I'm currently studying Computer >> Science in my 4th year at the University POLITEHNICA of Bucharest. >> I'm fairly new to the FreeBSD operating system, having only a >> background in Linux. For the past few months I've been investigating >> the problem of Lock Holder Preemption on bhyve, how other hypervisors >> deal with this problem and a possible implementation of these >> solutions on bhyve. >> I am currently working on implementing Gang Scheduling on FreeBSD to >> test if it is a viable solution. I also plan to continue my work, by >> implementing and testing other techniques of avoiding overhead caused >> by lock holder preemption, and comparing their results. > > FWIW, Xen doesn't do gang-scheduling, so the lock-holder preemption > problem is solved inside of the guest by using > pvspinlocks/pvticketlocks. Not to detriment the work you are doing on > gang-scheduling, but having some like this would help FreeBSD when > running in all virtualized environments regardless of whether the > underlying hypervisor does gang-scheduling or not. Some more information > about it: > > https://blog.xenproject.org/2012/05/11/benchmarking-the-new-pv-ticketlock= -implementation/ > http://www-archive.xenproject.org/files/xensummitboston08/LHP.pdf > https://lwn.net/Articles/556141/ > > Roger. From owner-freebsd-virtualization@FreeBSD.ORG Thu Jun 11 13:50:53 2015 Return-Path: Delivered-To: freebsd-virtualization@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 1DE76CA2 for ; Thu, 11 Jun 2015 13:50:53 +0000 (UTC) (envelope-from stefan.andritoiu@gmail.com) Received: from mail-oi0-x241.google.com (mail-oi0-x241.google.com [IPv6:2607:f8b0:4003:c06::241]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id DB5871125 for ; Thu, 11 Jun 2015 13:50:52 +0000 (UTC) (envelope-from stefan.andritoiu@gmail.com) Received: by oiav1 with SMTP id v1so950832oia.0 for ; Thu, 11 Jun 2015 06:50:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:date:message-id:subject:from:to:content-type; bh=K08rc8CoKE6nBbW8Pn35RNvefXs1fgceDjc/xlIuAMY=; b=WIuBA4pUZxlg1idy+i3i/hWscu81CDgRkvxKSsqrSCCeBseqC5TRbEPgHrhWR52Wc1 H0VoLkQ2XWh6dtgrncb0o/0F5tOvi7+RIbArsKz/chV2g99yEr7fFik+UW9lAa41MG66 h4QXAIAcLH2QY5x+Nxur4M8VM8LOsuEEDzPlkOOhf8j+kKRFKyZt7+KFlweLy48KGstQ aOLrpzyPpwO9xGPTEWRoXkTPjDguVgJRy+JAcfe8EJn0YSSy1f+XkFM8nSWfyDKBqYIb fN81ZaGcziW1F8EqqJpJteDP9uMDF7i6O0ouIlYs/vm0Sp2rJMxcfS/IVM1upLnZf5t8 G0Yw== MIME-Version: 1.0 X-Received: by 10.60.16.133 with SMTP id g5mr7851991oed.66.1434030652055; Thu, 11 Jun 2015 06:50:52 -0700 (PDT) Received: by 10.60.82.168 with HTTP; Thu, 11 Jun 2015 06:50:52 -0700 (PDT) Date: Thu, 11 Jun 2015 16:50:52 +0300 Message-ID: Subject: What is the sequence of context switches when an IPI is received? From: Stefan Andritoiu To: freebsd-virtualization@freebsd.org Content-Type: text/plain; charset=UTF-8 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 11 Jun 2015 13:50:53 -0000 Hello, >From the FreeBSD Handbook: "FreeBSD deals with interrupt handlers by giving them their own thread context". >From my understanding when a IPI is received the thread that will run it is placed on the real-time runq, and the scheduler will be invoked to schedule it. So the sequence should be: currently running thread -> scheduler thread -> interrupt handler -> scheduler thread -> previously interrupted thread (if no thread priority change took place inside the interrupt handler) Is this correct? Thank you, Stefan From owner-freebsd-virtualization@FreeBSD.ORG Thu Jun 11 14:21:33 2015 Return-Path: Delivered-To: freebsd-virtualization@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id A038831E for ; Thu, 11 Jun 2015 14:21:33 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from citadel.icyb.net.ua (citadel.icyb.net.ua [212.40.38.140]) by mx1.freebsd.org (Postfix) with ESMTP id E52FB1BEA for ; Thu, 11 Jun 2015 14:21:32 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from porto.starpoint.kiev.ua (porto-e.starpoint.kiev.ua [212.40.38.100]) by citadel.icyb.net.ua (8.8.8p3/ICyb-2.3exp) with ESMTP id RAA05968; Thu, 11 Jun 2015 17:13:41 +0300 (EEST) (envelope-from avg@FreeBSD.org) Received: from localhost ([127.0.0.1]) by porto.starpoint.kiev.ua with esmtp (Exim 4.34 (FreeBSD)) id 1Z33Ej-0002bY-1n; Thu, 11 Jun 2015 17:13:41 +0300 Message-ID: <5579975C.7020802@FreeBSD.org> Date: Thu, 11 Jun 2015 17:12:44 +0300 From: Andriy Gapon User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:31.0) Gecko/20100101 Thunderbird/31.7.0 MIME-Version: 1.0 To: Neel Natu , "freebsd-virtualization@freebsd.org" Subject: Re: bhyve: corrupting zfs pools? References: <556D9005.4020802@FreeBSD.org> <556DDDA9.6090005@FreeBSD.org> <556ED071.5030009@FreeBSD.org> <556FEA92.4050704@FreeBSD.org> In-Reply-To: Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 11 Jun 2015 14:21:33 -0000 On 10/06/2015 09:24, Neel Natu wrote: > I tried to repro on a Sempron 3850 APU but still no luck. > > Since you can repro it readily I'll follow up with you so we can > narrow this down on your end. After narrowing down the problem by following suggestions from Neel it became obvious that the problem was caused by my local modifications to ZFS on the host system. Sorry for the noise and thank you very much for the help. -- Andriy Gapon From owner-freebsd-virtualization@FreeBSD.ORG Fri Jun 12 07:51:21 2015 Return-Path: Delivered-To: freebsd-virtualization@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id AC7D5813 for ; Fri, 12 Jun 2015 07:51:21 +0000 (UTC) (envelope-from stefan.andritoiu@gmail.com) Received: from mail-ob0-x22d.google.com (mail-ob0-x22d.google.com [IPv6:2607:f8b0:4003:c01::22d]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 763201B83 for ; Fri, 12 Jun 2015 07:51:21 +0000 (UTC) (envelope-from stefan.andritoiu@gmail.com) Received: by obcej4 with SMTP id ej4so18520711obc.0 for ; Fri, 12 Jun 2015 00:51:20 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:date:message-id:subject:from:to:content-type; bh=UFnGnixUEBqVQryd3v5YCLimJw4K9CFbJmau5WSH2wA=; b=ZRA6UXVHVlLQO659CEk7yew9xdnqcpzc7EwAAogK+GMWRnkiC0yPU9Qyoghwd3E9dh 4Mr/nZnYvluksy16x6rhP0/R50A0EDgFWj6si6aJR5Evd6g5QNodlEYxkqx1VgLZsKkE wCz8VIUSEe/ZDnjseIBQiF7hLKoWaxmmyWYZZxKE63SJnuB5V8+5SMRdiAOcL4js1OAf PRNsTsNv4Fmf4r3ZAOWaQ7xjzQveA9rqXKk5y/dxQP3z8wBuThAwS1e+dBi0jdj2uMaB 7RFKqOuar64S/FGacjPNBtzhZCzy9wq9bGTjsnVstBIZJ3XzYRlTeLUTe/7aAuONQO8s HXXA== MIME-Version: 1.0 X-Received: by 10.202.106.147 with SMTP id f141mr1159911oic.65.1434095480806; Fri, 12 Jun 2015 00:51:20 -0700 (PDT) Received: by 10.60.82.168 with HTTP; Fri, 12 Jun 2015 00:51:20 -0700 (PDT) Date: Fri, 12 Jun 2015 10:51:20 +0300 Message-ID: Subject: Are the sched_choose() or tdq_choose() functions called after returning from an interrupt? From: Stefan Andritoiu To: freebsd-virtualization@freebsd.org Content-Type: text/plain; charset=UTF-8 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 12 Jun 2015 07:51:21 -0000 Hello, When returning from an interrupt, does it switch directly the thread that was interrupted? Or is the scheduler called to choose a thread to run (most probable the thread that was interrupted)? More specifically, are the sched_choose() or tdq_choose() functions called after returning from an IPI? Does an interrupt have it's own thread, or does it run in the context of the interrupted thread as in Linux? Thank you, Stefan From owner-freebsd-virtualization@FreeBSD.ORG Sat Jun 13 10:33:43 2015 Return-Path: Delivered-To: freebsd-virtualization@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 3DEBC46C for ; Sat, 13 Jun 2015 10:33:43 +0000 (UTC) (envelope-from stefan.andritoiu@gmail.com) Received: from mail-oi0-x230.google.com (mail-oi0-x230.google.com [IPv6:2607:f8b0:4003:c06::230]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 06B4869F for ; Sat, 13 Jun 2015 10:33:43 +0000 (UTC) (envelope-from stefan.andritoiu@gmail.com) Received: by oial131 with SMTP id l131so996334oia.3 for ; Sat, 13 Jun 2015 03:33:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:date:message-id:subject:from:to:content-type; bh=xUxMFkRNKdX70O+GtqxwaSvqphKC6LUt5dWH3oahgxM=; b=e9GAB4KfFG9ICh/0GHHD14C+UF0EzkneM4nHf7STYo0KjP9wt6niunuJXvunpf9dwN kYEK6gmQtNCF9DNSWApTKrAW8oJRoqoIp1HKzJ3WzyjeftpWZPCV+Rq5M1+EWaQGr5V9 zcSU4AqJrhMnL+uTNh6KIXL/PXUfKTfJ+NsU21or1pwI98VLLzr/B/FU6LlwDBbHWoHs ZEda/XIZndKuoSqu118HjpVIevvQQMw3ri7X/71ecwLoWhZz6ZnFdz9Gp61L2OjRjAe5 atMebMRfZDnsE3dYt3Wa5Y3d5evzGfPzZdWj9lwAqShPZbOcDZVzHQ8mFAhG+LlBarWG PN+A== MIME-Version: 1.0 X-Received: by 10.60.79.97 with SMTP id i1mr2319649oex.44.1434191622298; Sat, 13 Jun 2015 03:33:42 -0700 (PDT) Received: by 10.60.82.168 with HTTP; Sat, 13 Jun 2015 03:33:42 -0700 (PDT) Date: Sat, 13 Jun 2015 13:33:42 +0300 Message-ID: Subject: How to bind a thread to a CPU? From: Stefan Andritoiu To: freebsd-virtualization@freebsd.org Content-Type: text/plain; charset=UTF-8 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 13 Jun 2015 10:33:43 -0000 Hi, How can I pin a thread to run only on a specific CPU? Is it enough just to set the "struct cpuset *td_cpuset" of the thread? Thank you, Stefan From owner-freebsd-virtualization@FreeBSD.ORG Sat Jun 13 18:00:20 2015 Return-Path: Delivered-To: freebsd-virtualization@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 81673A4E for ; Sat, 13 Jun 2015 18:00:20 +0000 (UTC) (envelope-from neelnatu@gmail.com) Received: from mail-wi0-x235.google.com (mail-wi0-x235.google.com [IPv6:2a00:1450:400c:c05::235]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 1BECBA59 for ; Sat, 13 Jun 2015 18:00:20 +0000 (UTC) (envelope-from neelnatu@gmail.com) Received: by wibdq8 with SMTP id dq8so41454098wib.1 for ; Sat, 13 Jun 2015 11:00:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=YHWUhI5/koumKo/ImWAzzQD8RGihpJtZKF+ExalK9hg=; b=w7Qm74CQWXrYlW+ZTo8M1VEENmCmvChkamqtBILUdx+/afFuEkoAraFoD643IkKr6T eoETQ/gfjqJOKeBSbGw7+4hNh61v86KTxkqeCnKfuRQRtGIBmejrcj83BJY/MGpRKb0+ 2UsVP0y1jcX27R0qRimKZu5krhknQXIbER1nTSnHA6YoTyoYyu0fh2uLdTXMsZtviI71 i7tp4h05mOfA36/jGRWKiiq+gw0Qu0N7XqnCsongB8N2gtXIkrtkqDnmM/kRX96HBza5 rAgvHxYFbd0HYzvWKGDNwFt1TCSNbX7muvPjWqWU9TWwycpOOx6sRG9KF7Iu5/1QKwmf fBRA== MIME-Version: 1.0 X-Received: by 10.194.2.137 with SMTP id 9mr2268435wju.75.1434218418592; Sat, 13 Jun 2015 11:00:18 -0700 (PDT) Received: by 10.27.52.18 with HTTP; Sat, 13 Jun 2015 11:00:18 -0700 (PDT) In-Reply-To: References: Date: Sat, 13 Jun 2015 11:00:18 -0700 Message-ID: Subject: Re: How to bind a thread to a CPU? From: Neel Natu To: Stefan Andritoiu Cc: "freebsd-virtualization@freebsd.org" Content-Type: text/plain; charset=UTF-8 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 13 Jun 2015 18:00:20 -0000 Hi Stefan, On Sat, Jun 13, 2015 at 3:33 AM, Stefan Andritoiu wrote: > Hi, > > How can I pin a thread to run only on a specific CPU? > Is it enough just to set the "struct cpuset *td_cpuset" of the thread? > sched_bind() is the proper way to do this. best Neel > Thank you, > Stefan > _______________________________________________ > freebsd-virtualization@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization > To unsubscribe, send any mail to "freebsd-virtualization-unsubscribe@freebsd.org" From owner-freebsd-virtualization@FreeBSD.ORG Sat Jun 13 19:12:11 2015 Return-Path: Delivered-To: freebsd-virtualization@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 3C0A7C06 for ; Sat, 13 Jun 2015 19:12:11 +0000 (UTC) (envelope-from stefan.andritoiu@gmail.com) Received: from mail-ob0-x22d.google.com (mail-ob0-x22d.google.com [IPv6:2607:f8b0:4003:c01::22d]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 02001E14 for ; Sat, 13 Jun 2015 19:12:11 +0000 (UTC) (envelope-from stefan.andritoiu@gmail.com) Received: by obcej4 with SMTP id ej4so41494146obc.0 for ; Sat, 13 Jun 2015 12:12:10 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=UNeLP/xKGL7PeWfOfJkuOyaB5cyPsUrQ2ybMVynbxBo=; b=bTbxsW8tMBQwjKUyhyvLEpqrmuIwrsRm0ELNOtsSfGHzaIgR17aUrwKau+yqkE5aYw CWgJ21Whj95jJLFws6rMnDgNAI3XO8724Kfd3wtRpxcwnoybd6Cc+1wQasZGXSeLGSqm QCcmHT2xmIaKG7pUapg4gj55+PWYdQ1uktoG9sHzOywDqv+2HbqwsVHsCuScKFfwRQfB JwatQ9mg2FfIVMkK/bZ09YZDVPsUB7NuR5sjswHdFZz/vVnodTv3CjMwmIpeBgkE0fEl JlYAR5JUIVcnq2xf7G1cpMqFOXIj8TQCw5G0NlGcEEEIdjHQ7QmQO4NoVheFrUYMvgPT zzag== MIME-Version: 1.0 X-Received: by 10.60.16.133 with SMTP id g5mr17299808oed.66.1434222730193; Sat, 13 Jun 2015 12:12:10 -0700 (PDT) Received: by 10.60.82.168 with HTTP; Sat, 13 Jun 2015 12:12:10 -0700 (PDT) In-Reply-To: References: Date: Sat, 13 Jun 2015 22:12:10 +0300 Message-ID: Subject: Re: How to bind a thread to a CPU? From: Stefan Andritoiu To: Neel Natu Cc: "freebsd-virtualization@freebsd.org" Content-Type: text/plain; charset=UTF-8 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 13 Jun 2015 19:12:11 -0000 Hi Neel, On Sat, Jun 13, 2015 at 9:00 PM, Neel Natu wrote: > Hi Stefan, > > On Sat, Jun 13, 2015 at 3:33 AM, Stefan Andritoiu > wrote: >> Hi, >> >> How can I pin a thread to run only on a specific CPU? >> Is it enough just to set the "struct cpuset *td_cpuset" of the thread? >> > > sched_bind() is the proper way to do this. But in sched_bind() I notice: KASSERT(td == curthread, ("sched_bind: can only bind curthread")); How can I bind a thread that is not curthread? > > best > Neel > >> Thank you, >> Stefan >> _______________________________________________ >> freebsd-virtualization@freebsd.org mailing list >> http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization >> To unsubscribe, send any mail to "freebsd-virtualization-unsubscribe@freebsd.org"