From owner-freebsd-current@FreeBSD.ORG Mon May 16 20:46:05 2011 Return-Path: Delivered-To: freebsd-current@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id E29DF1065670; Mon, 16 May 2011 20:46:05 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from cyrus.watson.org (cyrus.watson.org [65.122.17.42]) by mx1.freebsd.org (Postfix) with ESMTP id B8AFF8FC12; Mon, 16 May 2011 20:46:05 +0000 (UTC) Received: from bigwig.baldwin.cx (66.111.2.69.static.nyinternet.net [66.111.2.69]) by cyrus.watson.org (Postfix) with ESMTPSA id 6E39446B3B; Mon, 16 May 2011 16:46:05 -0400 (EDT) Received: from jhbbsd.localnet (unknown [209.249.190.124]) by bigwig.baldwin.cx (Postfix) with ESMTPSA id 0BBCA8A052; Mon, 16 May 2011 16:46:05 -0400 (EDT) From: John Baldwin To: Max Laier Date: Mon, 16 May 2011 16:46:03 -0400 User-Agent: KMail/1.13.5 (FreeBSD/8.2-CBSD-20110325; KDE/4.5.5; amd64; ; ) References: <4DCD357D.6000109@FreeBSD.org> <201105161421.27665.jhb@freebsd.org> <201105161630.44577.max@love2party.net> In-Reply-To: <201105161630.44577.max@love2party.net> MIME-Version: 1.0 Content-Type: Text/Plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit Message-Id: <201105161646.03338.jhb@freebsd.org> X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.2.6 (bigwig.baldwin.cx); Mon, 16 May 2011 16:46:05 -0400 (EDT) Cc: FreeBSD current , Peter Grehan , Andriy Gapon , neel@freebsd.org Subject: Re: proposed smp_rendezvous change X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 16 May 2011 20:46:06 -0000 On Monday, May 16, 2011 4:30:44 pm Max Laier wrote: > On Monday 16 May 2011 14:21:27 John Baldwin wrote: > > Yes, we need to fix that. Humm, it doesn't preempt when you do a > > critical_exit() though? Or do you use a hand-rolled critical exit that > > doesn't do a deferred preemption? > > Right now I just did a manual td_critnest++/--, but I guess ... Ah, ok, so you would "lose" a preemption. That's not really ideal. > > Actually, I'm curious how the spin unlock inside the IPI could yield the > > CPU. Oh, is rmlock doing a wakeup inside the IPI handler? I guess that is > > ok as long as the critical_exit() just defers the preemption to the end of > > the IPI handler. > > ... the earliest point where it is safe to preempt is after doing the > > atomic_add_int(&smp_rv_waiters[2], 1); > > so that we can start other IPIs again. However, since we don't accept new > IPIs until we signal EOI in the MD code (on amd64), this might still not be a > good place to do the yield?!? Hmm, yeah, you would want to do the EOI before you yield. However, we could actually move the EOI up before calling the MI code so long as we leave interrupts disabled for the duration of the handler (which we do). > The spin unlock boils down to a critical_exit() and unless we did a > critical_enter() at some point during the redenvouz setup, we will yield() if > we owepreempt. I'm not quite sure how that can happen, but it seems like > there is a path that allows the scheduler to set it from a foreign CPU. No, it is only set on curthread by curthread. This is actually my main question. I've no idea how this could happen unless the rmlock code is actually triggering a wakeup or sched_add() in its rendezvous handler. I don't see anything in rm_cleanIPI() that would do that however. I wonder if your original issue was really fixed just by the first patch you had which fixed the race in smp_rendezvous()? -- John Baldwin