Date: Tue, 11 Dec 2018 01:19:57 -0700 From: Warner Losh <imp@bsdimp.com> To: Scott Long <scottl@samsco.org> Cc: Konstantin Belousov <kostikbel@gmail.com>, John Baldwin <jhb@freebsd.org>, Kevin Bowling <kevin.bowling@kev009.com>, Mateusz Guzik <mjguzik@gmail.com>, Ian Lepore <ian@freebsd.org>, svn-src-head@freebsd.org, svn-src-all@freebsd.org, src-committers <src-committers@freebsd.org>, Scott Long <scottl@freebsd.org> Subject: Re: svn commit: r341682 - head/sys/sys Message-ID: <CANCZdfoHWZ7w8jS-73EWEzATKRQ2LaE2cy2uhADf1rcjzuZrvA@mail.gmail.com> In-Reply-To: <E2DBB096-4B52-4F6D-BC38-9AB84E3C008D@samsco.org> References: <201812071205.wB7C5BvA038350@repo.freebsd.org> <1544206201.1860.288.camel@freebsd.org> <CAGudoHGUqiByb37wcbhMD3eWmxFrX5=BKMiF3bk6Ptr2WWggAg@mail.gmail.com> <CAK7dMtBj26Fa-eS3WcKezjnrweDJ9RgZvBpbiRK-9S-A4Svb8Q@mail.gmail.com> <CANCZdfoXuKxuOztCU7Xoavak5b6KYB9bBQonssWUABgQabBBUQ@mail.gmail.com> <45f85061-2633-852c-3cc0-41f64d51e4f0@FreeBSD.org> <20181210234754.GD60291@kib.kiev.ua> <E2DBB096-4B52-4F6D-BC38-9AB84E3C008D@samsco.org>
next in thread | previous in thread | raw e-mail | index | archive | help
On Mon, Dec 10, 2018 at 9:57 PM Scott Long <scottl@samsco.org> wrote: > > > > On Dec 10, 2018, at 4:47 PM, Konstantin Belousov <kostikbel@gmail.com> > wrote: > > > > On Mon, Dec 10, 2018 at 02:15:20PM -0800, John Baldwin wrote: > >> On 12/8/18 7:43 PM, Warner Losh wrote: > >>> > >>> > >>> On Sat, Dec 8, 2018, 8:36 PM Kevin Bowling <kevin.bowling@kev009.com > <mailto:kevin.bowling@kev009.com> wrote: > >>> > >>> On Sat, Dec 8, 2018 at 12:09 AM Mateusz Guzik <mjguzik@gmail.com > <mailto:mjguzik@gmail.com>> wrote: > >>> > >>>> > >>>> Fully satisfying solution would be that all architectures get 64-bit > >>>> ops, even if in the worst case they end up taking a lock. Then > >>>> subsystems would not have to ifdef on anything. However, there > >>>> was some opposition to this proposal and I don't think this is > >>>> important enough to push. > >>> > >>> Mateusz, > >>> > >>> Who is opposing this particular polyfill solution? Scott Long > brought > >>> up a situation in driver development where this would be useful as > >>> well. The polyfills lower the cognitive load and #ifdef soup whic= h > >>> are the right call here regardless of performance on toy ports. > >>> > >>> > >>> I don't recall seeing the opposition either. It would have to be a > global lock for all 64bit atomics.... but I think it would only be 2 > atomics on those architectures. > >> > >> It would have to be a spin lock, so in the case of unrl you would be > trading > >> an operation on one of N regular mutexes for a single spin lock that w= as > >> also contested by other things. This would be pretty crappy. For > drivers > >> that aren't actually used on platforms without 32-bit atomics we can > simply > >> not build them in sys/modules/Makefile or not put them in GENERIC. Fo= r > >> something in the core kernel like unrl I think we will have to do what > >> Mateusz has done here. > > > > It is worse. All atomics that acess the same location must use the same > > lock. Otherwise, you could observe torn writes and out of thin air > > values. Since you cannot know in advance which locations are acceses > > by the locked variant, all freebsd atomics ops have to be switched to > > locked variant on the architecture. > > 64bit atomics on I486 already suffer the risk of torn reads; the > implementation > merely does a CLI to protect against local preemption (though you could > still > get unlucky with an NMI). I suppose you could argue that SMP isn=E2=80= =99t really > viable on I486 and therefore this fact is irrelevant, but it does > illustrate > precedence for having API completeness in a platform. > We haven't ever supported SMP on i486, to my knowledge. Certainly by the 5.x time frame with SMPng it wasn't there. The 64-bit ops that are there are mostly to smoothly support some of the (now older) embedded boards. I haven't looked at the SMP work smp did for 4.x, but IIRC, it wasn't even supported there. > Really, this isn=E2=80=99t that hard. Part of the existing contract of u= sing > atomics is > that you carefully evaluate all uses of the variable and decide when to u= se > an atomic instruction. Arguing that we can=E2=80=99t make this process a= utomatic > and foolproof for 64bit quantities, especially for a subset of subset of > platforms/architectures, and therefore we should be even more of a > difficult > landmine, is not=E2=80=A6. I don=E2=80=99t know what to say=E2=80=A6 sens= ical? > I think it's fine to say that 64-bit atomics need to be efficient on the supported platforms (more on that below). > 64bit operations are a reality for MI code in a modern OS, and I=E2=80=99= m tired of > having to tip-toe around them due to incomplete MD implementations. The > instructions have been available on Intel CPUs for 25 years! My > very strong preference is to have a complete and functional implementatio= n > of atomic.h for any architecture that is hooked up to the build. We can > then > tackle the details of optimization and edge case refinement, just like we > do > with every other API and service that we work on. It doesn=E2=80=99t hav= e to be > perfect to be useful, and at this point we=E2=80=99re providing neither p= erfection > nor > utility, just =E2=80=9Cbuts=E2=80=9D and =E2=80=9Cwhat ifs=E2=80=9D. > I think you miss the point of discussion, at least on my part. I'm looking at the MIPS side and asking the question whatever 32-bit SMP support we may have had in the past should die. We only ever supported it on one not-so-common board that's aged out of relevance (JZ4780 is the only one I found that needs 32-bit MIPS SMP, and it's a 4 year old embedded board that's not that relevant today and there's no successor products in the market or as far as I can tell planned). We also have kernels that run in 32-bit mode on 64-bit hardware, but those provide little value and w already transitioned our largest 64-bit mips platform away from that support, so we can deorbit as well, I think. They give little value to the project. some house keeping here is likely in order. If that just leaves an odd PPC thing, then I think it's perfectly fine to start conversations there as well about trimming that support with the powerpc guys. Going forward, I=E2=80=99m going to start using 64bit atomics where they=E2= =80=99re prudent, > instead of avoiding them due to this niche 32bit argument. If that means > more and more of what I do no longer compiles on a mips or a ppc32, then > that=E2=80=99s a sacrifice that is fine with me. It still creates extra > development work, > and having a uniformly available implementation would be much nicer. > To be clear, I'm not making the argument since I could dig up one place in mips land that uses it, we should hold things back. I think the cost / benefit ratio for mips at least is the same as we have for some of the older arm stuff: we can implement it by disabling interrupts and declaring by fiat we won't support what few SMP chips that might be out there on that platform. FreeBSD/arm decided years ago we would never support the armv4 and armv5 SMP designs that were in the market place, for example. I think it's fine that we do that today with at least the mips32 stuff. And if people want to keep the mips32 UP stuff going, it's with the same CLI ; do it; POPF trick we do on intel to keep i486 going. My argument is that we need to find all the places that this would hurt, and we should make a calm, dispassionate decision about that pain. So far, the pain brought up on the 'hard to support end' suggests a minor trimming of what we support is in order rather than a holding back of other gains that can be made. I do plan on making this argument in the thread that's going on in mips@ right now, and hope to have it all squared away by the end of the week what the plans will be. We need to take a look at the trailing edge more aggressively. I'm looking at super-old SCSI cards, PC Card, armv5 and a few others that we should trim in 13. We've already circulated these ideas, and apart from very specific objections met with general support. Maybe we should seriously look at adding i486 to the list, as well as taking a hard look at 32-bit platforms post 13 and planning for that transition starting today. Brooks also has in motion this on the older nic card side of things too, so we'll see a big trimming there in the coming months as well. One thing the project has been bad about has been collecting data on what's actually in use and using that to more aggressively drive deprecation and removal of old drivers and features from the system that are no longer relevant. So we get stuck in arguments of someone has it, might be using it and since there's not organized pushback that articulates the real cost this has to the project, we accumulate a lot of drivers whose only changes have been API changes (and we're not even sure those were done right). The 64-bit atomics friction and grumpiness is but one symptom of this larger, deeper problem. And unless we address that, these narrow issues that are easy to talk to death will continue to plague our forward progress. Warner
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CANCZdfoHWZ7w8jS-73EWEzATKRQ2LaE2cy2uhADf1rcjzuZrvA>