From owner-svn-src-head@FreeBSD.ORG Thu May 3 22:24:26 2012 Return-Path: Delivered-To: svn-src-head@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 772C6106564A; Thu, 3 May 2012 22:24:26 +0000 (UTC) (envelope-from asmrookie@gmail.com) Received: from mail-lb0-f182.google.com (mail-lb0-f182.google.com [209.85.217.182]) by mx1.freebsd.org (Postfix) with ESMTP id 84D848FC14; Thu, 3 May 2012 22:24:25 +0000 (UTC) Received: by lbon10 with SMTP id n10so2195187lbo.13 for ; Thu, 03 May 2012 15:24:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type :content-transfer-encoding; bh=Ll9aLERW+fYxBAo7n3vXDfzhaye9ydx8Nb6rB0hMh9s=; b=bXL65MQguDkolXotkLAgkli/g4tsmC6F/nCqttppRE74e0ljo7Rmu9T566rm2/lNsu WrWQGBXX85aUf/8Vnxxj9EzGIU/kRUT0rDU+aokYfwYGIEqVG3JAGqRvjzmWhRYqU1Ec 1JAI4dup7ePWKdMbvKSROEvX9QEGCXex9K93N4d/81gtA0zS9tPCw5GXPC+7bV2fYgDD nq7+hV1ukRFiQjGqe3dNVshLiHwmKdHzg67CK0RN2oqOMNM1Wu3FGPuOFZCwmybXA8Pf MJa3c34jWp4wCFTcAhMFTm78P1LvyssRQUHxFt3LgMERlU+aBwTVl/sUcldQt1i6W7xQ AfRg== MIME-Version: 1.0 Received: by 10.112.37.132 with SMTP id y4mr1857393lbj.8.1336083864102; Thu, 03 May 2012 15:24:24 -0700 (PDT) Sender: asmrookie@gmail.com Received: by 10.112.27.65 with HTTP; Thu, 3 May 2012 15:24:23 -0700 (PDT) In-Reply-To: References: <201205031038.q43Ac2eZ032779@svn.freebsd.org> <20120503114913.GJ2358@deviant.kiev.zoral.com.ua> <20120503132715.GN2358@deviant.kiev.zoral.com.ua> <20120503212834.GP2358@deviant.kiev.zoral.com.ua> Date: Thu, 3 May 2012 23:24:23 +0100 X-Google-Sender-Auth: P3Ax9-DfnhXpngnoCSfSm4I0tVM Message-ID: From: Attilio Rao To: Konstantin Belousov Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Cc: svn-src-head@freebsd.org, svn-src-all@freebsd.org, src-committers@freebsd.org, christian.esken@trivago.com Subject: Re: svn commit: r234952 - in head/sys: kern sys X-BeenThere: svn-src-head@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: SVN commit messages for the src tree for head/-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 03 May 2012 22:24:26 -0000 2012/5/3 Attilio Rao : > 2012/5/3 Konstantin Belousov : >> On Thu, May 03, 2012 at 10:06:53PM +0100, Attilio Rao wrote: >>> 2012/5/3 Konstantin Belousov : >>> > On Thu, May 03, 2012 at 02:14:20PM +0100, Attilio Rao wrote: >>> >> 2012/5/3, Konstantin Belousov : >>> >> > On Thu, May 03, 2012 at 12:02:08PM +0100, Attilio Rao wrote: >>> >> >> 2012/5/3, Konstantin Belousov : >>> >> >> > Author: kib >>> >> >> > Date: Thu May =C2=A03 10:38:02 2012 >>> >> >> > New Revision: 234952 >>> >> >> > URL: http://svn.freebsd.org/changeset/base/234952 >>> >> >> > >>> >> >> > Log: >>> >> >> > =C2=A0 When callout_reset_on() cannot immediately migrate a cal= lout since it >>> >> >> > =C2=A0 is running on other cpu, the CALLOUT_PENDING flag is tem= porarily >>> >> >> > =C2=A0 cleared. Then, callout_stop() on this, in fact active, c= allout fails >>> >> >> > =C2=A0 because CALLOUT_PENDING is not set, and callout_stop() r= eturns 0. >>> >> >> > >>> >> >> > =C2=A0 Now, in sleepq_check_timeout(), the failed callout_stop(= ) causes the >>> >> >> > =C2=A0 sleepq code to execute mi_switch() without even setting = the wmesg, >>> >> >> > =C2=A0 since the switch-out is supposed to be transient. In fac= t, the thread >>> >> >> > =C2=A0 is put off the CPU for full timeout interval, instead of= being put on >>> >> >> > =C2=A0 runq immediately. =C2=A0Until timeout fires, the process= is unkillable for >>> >> >> > =C2=A0 obvious reasons. >>> >> >> > >>> >> >> > =C2=A0 Fix this by marking the migrating callouts with CALLOUT_= DFRMIGRATION >>> >> >> > =C2=A0 flag. The flag is cleared by callout_stop_safe() when th= e function >>> >> >> > =C2=A0 detects a migration, besides returning the success. The = softclock() >>> >> >> > =C2=A0 rechecks the flag for migrating callout and cancels its = execution if >>> >> >> > =C2=A0 the flag was cleared meantime. >>> >> >> >>> >> >> Can you please clarify why you cannot simply drop the deferred >>> >> >> migration in the case !CALLOUT_PENDING in callout_stop_safe()? >>> >> > >>> >> > I probably can, I think I went with the route of committed patch >>> >> > because it is slightly less work. Also, the comment in the while() >>> >> > loop suggested me to rely on softclock. >>> >> >>> >> I don't think this is more work at all, the attached patch >>> >> (pre-r234952, untested) should address it properly in few than 10 >>> >> lines: >>> >> http://www.freebsd.org/~attilio/callout_cancel_mig_stop.patch >>> >> >>> >> without the need to add further flags and re-using existing mechanis= ms. >>> > >>> > (cc->cc_curr !=3D c) is not the case which caused the issue. It might= be >>> > needed to treatened this way, but the reported case is opposite. >>> >>> Yes, of course, because the migration handover happens in the same >>> critical context of cc->cc_curr =3D=3D c, but now I wonder if this fix = is >>> really right. >>> >>> It seems to me that in the case you describe callout_stop() must >>> return 0 and the migration must not be cancelled because the callout >>> is not stopped. It is not stopped not because of the deferred >>> migration but because cc->cc_curr =3D=3D c. It seems a perfectly valid >>> situation to me. >> Yes, and my patch makes the callout to be indeed stopped right after >> migration is finished. Did you looked at the patch itself ? >> >> What is the valid situation ? callout_stop returning 0 but not stopping >> a pending callout ? I have to disagree. > > The function callout_stop() cancels a callout if it is currently pending. > =C2=A0 =C2=A0 If the callout is pending, then callout_stop() will return = a non-zero > =C2=A0 =C2=A0 value. =C2=A0If the callout is not set, has already been se= rviced or is cur=E2=80=90 > =C2=A0 =C2=A0 rently being serviced, then zero will be returned. =C2=A0If= the callout has an > =C2=A0 =C2=A0 associated mutex, then that mutex must be held when this fu= nction is > =C2=A0 =C2=A0 called. > > [ From the callout manpage ] > > If the "callout is currently being serviced" means cc->cc_curr =3D=3D c > and it must return 0. Elaborating some more, I see a discrepancy here in the callout interface, which is also present pre-your patch and pre-migration delay. Basically, immagine a callout rearmed during its callback (pretty typical) and a callout_stop() running just after the callout has been rearmed and it is *still* in the callback. What we find is that CALLOUT_PENDING is on and that cc_curr =3D=3D cc. I don't think that the callout should stop successfully in this case. However, because of how _callout_stop_safe() is written, CALLOUT_PENDING check has precedence and wins, returning 1 and removing the CALLOUT_PENDING flag, but please note that the callback is still running (even if only for little time). I think this generally works ok because most of the callout callbacks rearm the callout as last thing in their operation. But I think this is highly fragile and we cannot really rely on this feature. You are seeing a problem in the deferred migration case because it does the other way around, it prefers the check over cc_curr =3D=3D cc to the "pending" (callout migration in this case). I think this is only a problem, also, with callouts which don't have a lock associated with them, like the sleepqueue case, because otherwise the interlock would leave the state consistent. I think we need to think carefully about a pattern for this case that deals with all the races, I need to give this more thinking, but definitively it seems to me we need a patch at the callout policy level. Attilio --=20 Peace can only be achieved by understanding - A. Einstein