Date: Tue, 8 Apr 2014 09:42:16 -0600 From: Warner Losh <imp@bsdimp.com> To: Jordan Hubbard <jkh@ixsystems.com> Cc: freebsd-arch <freebsd-arch@freebsd.org> Subject: Re: Compiler toolchain roadmap Message-ID: <94AE42FF-9F01-4A92-AAB9-86D9705AE27C@bsdimp.com> In-Reply-To: <482357EF-5596-45FD-8D2C-3742BB4872E8@ixsystems.com> References: <201404021607.s32G7mhw051355@svn.freebsd.org> <20140404115256.GA85137@ivaldir.etoilebsd.net> <F2A33EA8-14F2-4D62-9021-9023A1751E48@FreeBSD.org> <8D6AF193-A5A3-4A28-A230-97A543395ACA@ixsystems.com> <2E0EC8CB-B3EE-4DB8-A33D-58FD2107F14D@FreeBSD.org> <6A02504F-5543-4F91-92F6-7B4FB9A34DC4@ixsystems.com> <152D73EE-DF9E-4757-B547-F1F22B12C824@FreeBSD.org> <B06E1588-8828-485F-A407-3F19231F8EA5@ixsystems.com> <8E3BD3C1-A441-48C5-97BC-45EF67513096@FreeBSD.org> <6418BE83-BE78-473B-9311-C849507FA885@ixsystems.com> <CAJ-Vmom-19LujsTQ%2Bv4XozE%2BiEH18LMEQitBLC-At=DmsgkB%2BQ@mail.gmail.com> <EB9CE8A8-E897-4DE1-A8BC-80C6CC23E612@ixsystems.com> <9E11A6D4-9D18-422D-9514-4714AADDAEF4@gmail.com> <8E22F8FA-CF71-4A47-BDE8-F3CE6158E1C9@ixsystems.com> <EFEFB531-6279-41C1-B0BF-A0EA1F722E24@bsdimp.com> <482357EF-5596-45FD-8D2C-3742BB4872E8@ixsystems.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On Apr 8, 2014, at 1:03 AM, Jordan Hubbard <jkh@ixsystems.com> wrote: >=20 > On Apr 7, 2014, at 8:03 PM, Warner Losh <imp@bsdimp.com> wrote: >=20 >> This is a desire, not a hard requirement. As such, there may be = missing bits if you chose to go this route. At least that=92s been the = notion in the past when using llvm features has come up. The notion for = this path has always been =91It is possible, but only a subset of the = functionality may be available.=94 >>=20 >> So if one were to add blocks, it would need a knob WITHOUT_BLOCKS = that would disable all functionality tied to them. For many = applications, this is a reasonable subset (based on my guesses at how = intrusive this would be to the system). And it might even be = automatically selected based on compiler support, but that=92s another = can of worms. >=20 > I=92m glad to hear that building with foreign compilers is more of a = desire than a hard constraint. I=92m still not sure what the scenario = would look like where libdispatch has been built WITHOUT_BLOCKS and is = now essentially API incompatible with the libdispatch compiled = WITH_BLOCKS (since you wouldn=92t get functions like dispatch_async() = but would get the dispatch_async_f() function which takes a function = pointer/context variable instead), but I guess that=92s a problem for = the hypothetical wacky-compiler user to figure out? Yes.=20 > You seemed to imply that when you said FreeBSD was already using a = number of custom compiler extensions, I just wasn=92t sure that this = extended as far as =93you can build FreeBSD, for the most part, with = your weird-ass compiler but you can also expect it to be a different = FreeBSD than what=92s in base for x86_64 (for example) because various = things will be switched off, API incompatible, or simply not compiled at = all.=94 The extensions right now are fairly modest. The biggest one being the = extended format checker for the kernel printf %b. You can expect it to = be a bit different than the amd64 FreeBSD right now if it is a different = architecture due to differing levels of pmap maturity and optimization, = for example. While most of the experience is the same, the further you = get away from the solidly Tier 1 platforms the more roll your own = FreeBSD becomes and the more handholding it requires. This doesn=92t = make that experience bad, mind you, just that more care is needed for = tier 2 and tier 3 platforms (whatever those terms really mean :). > If that=92s the case, then woohoo, when can I import libdispatch into = base? :) [I=92m being semi-facetious here, I realize there are a few = more steps to go]. If it is either completely disabled when the compiler doesn=92t support = blocks, or degrades gracefully automatically to avoiding blocks, I have = no problems with it. >> So to take this a step further=85 There=92s many levels of = integration here=85 First, there=92s the kernel, which is most often = the bit of code people want/need the special compiler for. Next, there=92s= having the feature available in user land. Finally, there=92d be a = wide-scale integration of this feature. I see very few programs in base = benefiting from libdispatch, honestly, but that doesn=92t mean the set = is empty. Do you have a longer write up on what you=92d like to do here? >=20 > Sure, I can write something up as part of my =93any objections to = doing this?=94 step, but just to summarize briefly for this = conversation: >=20 > 1. Libdispatch does not need to touch the kernel at all, nor does the = kernel require blocks support. The pthread_workqueue() stuff that = Stacey Son did all those years ago would be a nice optimization (this = lets libdispatch balance its thread pool loads across multiple = applications, not just within a single application) but it=92s not = strictly necessary, and it can also be added later without any consumer = of libdispatch (henceforth referred to by its other name of GCD, because = that=92s easier to type!) needing to know or care. >=20 > 2. Once we have GCD in base, I can then start looking at adding = libnotify and its associated notifyd daemon, which is a generic system = notification mechanism. >=20 > 3. Once libnotify is in base, I=92ll then add some basic notifications = to Libc, like timezone changes, hostname changes, and so on, = invalidating some of its internal caches in the face of said changes = (there are some fairly expensive stat() calls you can eliminate once you = have the far lighter-weight shared memory flag checking that notifyd = provides, which also helps save power by not walking around the = filesystem so much). The notifyd service can also translate filesystem = notifications into system notifications, which means services like samba = and netatalk can detect (cheaply) when something has changed out from = under them. I see no reason in principle that this couldn=92t be done. There might = be bumps along the way, but there always are=85 I=92d love to see this = stuff in FreeBSD, but I=92m also not in a position to devote time to it. > Beyond that, we probably need to talk about another daemon, which on = OS X is called configd, which deals with larger scope configuration = changes like network devices coming and going, default routes changing, = and so on. This does require some kernel up-call mechanisms, but it = will =93publish=94 its results in terms of notify(3) notifications so = applications can subscribe to those events the same way. If we can hack = usbd into doing the same for USB events, that=92s great, otherwise I = would imagine configd dealing with that sort of hardware notification = service as well. usbd? Where have you been. We=92ve not had a usbd in the base system = since before you started at Apple :). What you call configd is basically = handled by devd at the moment. It would be nice if devd grew = notification support, since right now it just has a tiny character = device interface with a workable, but imperfect, protocol. >> I see a continuum of answer here: If you want to modify the kernel = extensive to use blocks, then that=92s going to be a much bigger problem = than having a few daemons and a library in the tree that require them = which is a bigger problem than having a few daemons using it, but = optionally, and a library which is a bigger problem than a library in = the tree which is a bigger problem than the status quo. >=20 > I.. think=85 I follow that sentence enough to hope I=92ve answered it. = No blocks in the kernel. Nothing more than a few daemons and a library = and, at some point, a way of figuring out when the kernel has made = certain changes to the system configuration, if and where said changes = can=92t be intercepted and dealt with at the libc layer or otherwise = noticed by a user-land daemon which can post the notifications for any = subscribers of them. Yea, it was a tortured way of stating the continuum... > As to the subscribers, they won=92t have to use blocks unless they = choose to consume one or more of notify(3)=92s block APIs. The notify = mechanism was written with both old-style and new-style clients in mind. = If your daemon wants a signal when the notification it=92s subscribing = to changes, fine. If it wants some data on a file descriptor which it=92s= poll(3)/kqueue(3)ing, fine. See = https://developer.apple.com/library/mac/documentation/Darwin/Reference/Man= Pages/man3/notify.3.html and, of course, ignore the mach-specific = delivery methods as N/A. That=92s really cool stuff. I=92d love to see it. >> And 8.1 for libdispatch? Is it really so portable it would work with = our old, crappy gcc pre blocks update? >=20 > Sorry, misunderstanding. I was referring to the original slashdot / = phoronix article which stated that GCD would be coming to FreeBSD in = 8.1. Apparently, according to FreeBSD=92s own wiki = (https://wiki.freebsd.org/GCD) it works, too. Yea, I=92m not sure what happened with that. But the FreeBSD wiki is = editable by any developer at any time, or any friend of a developer that = creates an account, so with controls like that, there=92s bound to be = some inaccurate information there :) Warner
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?94AE42FF-9F01-4A92-AAB9-86D9705AE27C>