Date: Thu, 3 May 2012 23:24:23 +0100 From: Attilio Rao <attilio@freebsd.org> To: Konstantin Belousov <kostikbel@gmail.com> Cc: svn-src-head@freebsd.org, svn-src-all@freebsd.org, src-committers@freebsd.org, christian.esken@trivago.com Subject: Re: svn commit: r234952 - in head/sys: kern sys Message-ID: <CAJ-FndAyhdaBoS008-wj4GknCkTNnC4qqDVpO-qqTnbqibO6_w@mail.gmail.com> In-Reply-To: <CAJ-FndCme4cEqog4hq-_e1-V==TxCLE_N%2Bnk=mSoPUdLKg6Otg@mail.gmail.com> References: <201205031038.q43Ac2eZ032779@svn.freebsd.org> <CAJ-FndBF7%2BfLwivrMdH=rMHsEVQ5jojk3noceN8wxmaj2ZaTvg@mail.gmail.com> <20120503114913.GJ2358@deviant.kiev.zoral.com.ua> <CAJ-FndCa8xBmKJNnm6kfVpAyCtWtJmoti75XvCBiOR7ppREqng@mail.gmail.com> <20120503132715.GN2358@deviant.kiev.zoral.com.ua> <CAJ-FndAhRPyMsFwCEh0w6=VmRy4faL4L0rC4kG4de8rNkX-DQw@mail.gmail.com> <20120503212834.GP2358@deviant.kiev.zoral.com.ua> <CAJ-FndCme4cEqog4hq-_e1-V==TxCLE_N%2Bnk=mSoPUdLKg6Otg@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
2012/5/3 Attilio Rao <attilio@freebsd.org>: > 2012/5/3 Konstantin Belousov <kostikbel@gmail.com>: >> On Thu, May 03, 2012 at 10:06:53PM +0100, Attilio Rao wrote: >>> 2012/5/3 Konstantin Belousov <kostikbel@gmail.com>: >>> > On Thu, May 03, 2012 at 02:14:20PM +0100, Attilio Rao wrote: >>> >> 2012/5/3, Konstantin Belousov <kostikbel@gmail.com>: >>> >> > On Thu, May 03, 2012 at 12:02:08PM +0100, Attilio Rao wrote: >>> >> >> 2012/5/3, Konstantin Belousov <kib@freebsd.org>: >>> >> >> > Author: kib >>> >> >> > Date: Thu May =C2=A03 10:38:02 2012 >>> >> >> > New Revision: 234952 >>> >> >> > URL: http://svn.freebsd.org/changeset/base/234952 >>> >> >> > >>> >> >> > Log: >>> >> >> > =C2=A0 When callout_reset_on() cannot immediately migrate a cal= lout since it >>> >> >> > =C2=A0 is running on other cpu, the CALLOUT_PENDING flag is tem= porarily >>> >> >> > =C2=A0 cleared. Then, callout_stop() on this, in fact active, c= allout fails >>> >> >> > =C2=A0 because CALLOUT_PENDING is not set, and callout_stop() r= eturns 0. >>> >> >> > >>> >> >> > =C2=A0 Now, in sleepq_check_timeout(), the failed callout_stop(= ) causes the >>> >> >> > =C2=A0 sleepq code to execute mi_switch() without even setting = the wmesg, >>> >> >> > =C2=A0 since the switch-out is supposed to be transient. In fac= t, the thread >>> >> >> > =C2=A0 is put off the CPU for full timeout interval, instead of= being put on >>> >> >> > =C2=A0 runq immediately. =C2=A0Until timeout fires, the process= is unkillable for >>> >> >> > =C2=A0 obvious reasons. >>> >> >> > >>> >> >> > =C2=A0 Fix this by marking the migrating callouts with CALLOUT_= DFRMIGRATION >>> >> >> > =C2=A0 flag. The flag is cleared by callout_stop_safe() when th= e function >>> >> >> > =C2=A0 detects a migration, besides returning the success. The = softclock() >>> >> >> > =C2=A0 rechecks the flag for migrating callout and cancels its = execution if >>> >> >> > =C2=A0 the flag was cleared meantime. >>> >> >> >>> >> >> Can you please clarify why you cannot simply drop the deferred >>> >> >> migration in the case !CALLOUT_PENDING in callout_stop_safe()? >>> >> > >>> >> > I probably can, I think I went with the route of committed patch >>> >> > because it is slightly less work. Also, the comment in the while() >>> >> > loop suggested me to rely on softclock. >>> >> >>> >> I don't think this is more work at all, the attached patch >>> >> (pre-r234952, untested) should address it properly in few than 10 >>> >> lines: >>> >> http://www.freebsd.org/~attilio/callout_cancel_mig_stop.patch >>> >> >>> >> without the need to add further flags and re-using existing mechanis= ms. >>> > >>> > (cc->cc_curr !=3D c) is not the case which caused the issue. It might= be >>> > needed to treatened this way, but the reported case is opposite. >>> >>> Yes, of course, because the migration handover happens in the same >>> critical context of cc->cc_curr =3D=3D c, but now I wonder if this fix = is >>> really right. >>> >>> It seems to me that in the case you describe callout_stop() must >>> return 0 and the migration must not be cancelled because the callout >>> is not stopped. It is not stopped not because of the deferred >>> migration but because cc->cc_curr =3D=3D c. It seems a perfectly valid >>> situation to me. >> Yes, and my patch makes the callout to be indeed stopped right after >> migration is finished. Did you looked at the patch itself ? >> >> What is the valid situation ? callout_stop returning 0 but not stopping >> a pending callout ? I have to disagree. > > The function callout_stop() cancels a callout if it is currently pending. > =C2=A0 =C2=A0 If the callout is pending, then callout_stop() will return = a non-zero > =C2=A0 =C2=A0 value. =C2=A0If the callout is not set, has already been se= rviced or is cur=E2=80=90 > =C2=A0 =C2=A0 rently being serviced, then zero will be returned. =C2=A0If= the callout has an > =C2=A0 =C2=A0 associated mutex, then that mutex must be held when this fu= nction is > =C2=A0 =C2=A0 called. > > [ From the callout manpage ] > > If the "callout is currently being serviced" means cc->cc_curr =3D=3D c > and it must return 0. Elaborating some more, I see a discrepancy here in the callout interface, which is also present pre-your patch and pre-migration delay. Basically, immagine a callout rearmed during its callback (pretty typical) and a callout_stop() running just after the callout has been rearmed and it is *still* in the callback. What we find is that CALLOUT_PENDING is on and that cc_curr =3D=3D cc. I don't think that the callout should stop successfully in this case. However, because of how _callout_stop_safe() is written, CALLOUT_PENDING check has precedence and wins, returning 1 and removing the CALLOUT_PENDING flag, but please note that the callback is still running (even if only for little time). I think this generally works ok because most of the callout callbacks rearm the callout as last thing in their operation. But I think this is highly fragile and we cannot really rely on this feature. You are seeing a problem in the deferred migration case because it does the other way around, it prefers the check over cc_curr =3D=3D cc to the "pending" (callout migration in this case). I think this is only a problem, also, with callouts which don't have a lock associated with them, like the sleepqueue case, because otherwise the interlock would leave the state consistent. I think we need to think carefully about a pattern for this case that deals with all the races, I need to give this more thinking, but definitively it seems to me we need a patch at the callout policy level. Attilio --=20 Peace can only be achieved by understanding - A. Einstein
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAJ-FndAyhdaBoS008-wj4GknCkTNnC4qqDVpO-qqTnbqibO6_w>