From owner-freebsd-arch@FreeBSD.ORG Tue Apr 8 15:42:25 2014 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id AB13E9A8 for ; Tue, 8 Apr 2014 15:42:25 +0000 (UTC) Received: from mail-pb0-f43.google.com (mail-pb0-f43.google.com [209.85.160.43]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 7196410CF for ; Tue, 8 Apr 2014 15:42:25 +0000 (UTC) Received: by mail-pb0-f43.google.com with SMTP id um1so1209057pbc.2 for ; Tue, 08 Apr 2014 08:42:19 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:sender:content-type:mime-version:subject:from :in-reply-to:date:cc:content-transfer-encoding:message-id:references :to; bh=XDoKCl2QJS1groSbN8A2CA7qVzl3hh7baCFz8d6wuEs=; b=KXUIJ0sU+Zu9BCqv/72+CApbBKZdH7JJQWLs2OmQq50Fzqipo4hW//1waM37c4aIjv r3DlxgNrJu8q6cgEHaaZTTgRhFJo8t/QsgGiZK2VghM8nkGKzGKzhsG4e2btpp/I11Y1 2LISrSYlwuuX5aydDVglDkdwAvnxN1uSVQ/cI5N6ePbmiTwYGyE48fzXjZexHGxm0/HD jB147YEekkwAS7o0qB9m6mndx/S4zuCjuUoLsXccBAS7Ll43ApyrosPlAIMTOtf9uXSU R+47MFVFSkRp/V9I5g74U9fjww0T1MqMRQmrq7Q4HDPiIg1dm9ooKj0rxao2jZEzLYaw FMiA== X-Gm-Message-State: ALoCoQmjUIoBDeZ4N8wPpV0QcvZH6QvfIl4hJKiqk0AhH5bceL+/CXMjhDmxJJ0LHH43eM80jH2t X-Received: by 10.68.198.97 with SMTP id jb1mr5511052pbc.104.1396971739522; Tue, 08 Apr 2014 08:42:19 -0700 (PDT) Received: from [10.64.24.116] (dc1-prod.netflix.com. [69.53.236.251]) by mx.google.com with ESMTPSA id pq3sm5363311pbb.57.2014.04.08.08.42.18 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Tue, 08 Apr 2014 08:42:18 -0700 (PDT) Sender: Warner Losh Content-Type: text/plain; charset=windows-1252 Mime-Version: 1.0 (Mac OS X Mail 7.2 \(1874\)) Subject: Re: Compiler toolchain roadmap From: Warner Losh In-Reply-To: <482357EF-5596-45FD-8D2C-3742BB4872E8@ixsystems.com> Date: Tue, 8 Apr 2014 09:42:16 -0600 Content-Transfer-Encoding: quoted-printable Message-Id: <94AE42FF-9F01-4A92-AAB9-86D9705AE27C@bsdimp.com> References: <201404021607.s32G7mhw051355@svn.freebsd.org> <20140404115256.GA85137@ivaldir.etoilebsd.net> <8D6AF193-A5A3-4A28-A230-97A543395ACA@ixsystems.com> <2E0EC8CB-B3EE-4DB8-A33D-58FD2107F14D@FreeBSD.org> <6A02504F-5543-4F91-92F6-7B4FB9A34DC4@ixsystems.com> <152D73EE-DF9E-4757-B547-F1F22B12C824@FreeBSD.org> <8E3BD3C1-A441-48C5-97BC-45EF67513096@FreeBSD.org> <6418BE83-BE78-473B-9311-C849507FA885@ixsystems.com> <9E11A6D4-9D18-422D-9514-4714AADDAEF4@gmail.com> <8E22F8FA-CF71-4A47-BDE8-F3CE6158E1C9@ixsystems.com> <482357EF-5596-45FD-8D2C-3742BB4872E8@ixsystems.com> To: Jordan Hubbard X-Mailer: Apple Mail (2.1874) Cc: freebsd-arch X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 08 Apr 2014 15:42:25 -0000 On Apr 8, 2014, at 1:03 AM, Jordan Hubbard wrote: >=20 > On Apr 7, 2014, at 8:03 PM, Warner Losh wrote: >=20 >> This is a desire, not a hard requirement. As such, there may be = missing bits if you chose to go this route. At least that=92s been the = notion in the past when using llvm features has come up. The notion for = this path has always been =91It is possible, but only a subset of the = functionality may be available.=94 >>=20 >> So if one were to add blocks, it would need a knob WITHOUT_BLOCKS = that would disable all functionality tied to them. For many = applications, this is a reasonable subset (based on my guesses at how = intrusive this would be to the system). And it might even be = automatically selected based on compiler support, but that=92s another = can of worms. >=20 > I=92m glad to hear that building with foreign compilers is more of a = desire than a hard constraint. I=92m still not sure what the scenario = would look like where libdispatch has been built WITHOUT_BLOCKS and is = now essentially API incompatible with the libdispatch compiled = WITH_BLOCKS (since you wouldn=92t get functions like dispatch_async() = but would get the dispatch_async_f() function which takes a function = pointer/context variable instead), but I guess that=92s a problem for = the hypothetical wacky-compiler user to figure out? Yes.=20 > You seemed to imply that when you said FreeBSD was already using a = number of custom compiler extensions, I just wasn=92t sure that this = extended as far as =93you can build FreeBSD, for the most part, with = your weird-ass compiler but you can also expect it to be a different = FreeBSD than what=92s in base for x86_64 (for example) because various = things will be switched off, API incompatible, or simply not compiled at = all.=94 The extensions right now are fairly modest. The biggest one being the = extended format checker for the kernel printf %b. You can expect it to = be a bit different than the amd64 FreeBSD right now if it is a different = architecture due to differing levels of pmap maturity and optimization, = for example. While most of the experience is the same, the further you = get away from the solidly Tier 1 platforms the more roll your own = FreeBSD becomes and the more handholding it requires. This doesn=92t = make that experience bad, mind you, just that more care is needed for = tier 2 and tier 3 platforms (whatever those terms really mean :). > If that=92s the case, then woohoo, when can I import libdispatch into = base? :) [I=92m being semi-facetious here, I realize there are a few = more steps to go]. If it is either completely disabled when the compiler doesn=92t support = blocks, or degrades gracefully automatically to avoiding blocks, I have = no problems with it. >> So to take this a step further=85 There=92s many levels of = integration here=85 First, there=92s the kernel, which is most often = the bit of code people want/need the special compiler for. Next, there=92s= having the feature available in user land. Finally, there=92d be a = wide-scale integration of this feature. I see very few programs in base = benefiting from libdispatch, honestly, but that doesn=92t mean the set = is empty. Do you have a longer write up on what you=92d like to do here? >=20 > Sure, I can write something up as part of my =93any objections to = doing this?=94 step, but just to summarize briefly for this = conversation: >=20 > 1. Libdispatch does not need to touch the kernel at all, nor does the = kernel require blocks support. The pthread_workqueue() stuff that = Stacey Son did all those years ago would be a nice optimization (this = lets libdispatch balance its thread pool loads across multiple = applications, not just within a single application) but it=92s not = strictly necessary, and it can also be added later without any consumer = of libdispatch (henceforth referred to by its other name of GCD, because = that=92s easier to type!) needing to know or care. >=20 > 2. Once we have GCD in base, I can then start looking at adding = libnotify and its associated notifyd daemon, which is a generic system = notification mechanism. >=20 > 3. Once libnotify is in base, I=92ll then add some basic notifications = to Libc, like timezone changes, hostname changes, and so on, = invalidating some of its internal caches in the face of said changes = (there are some fairly expensive stat() calls you can eliminate once you = have the far lighter-weight shared memory flag checking that notifyd = provides, which also helps save power by not walking around the = filesystem so much). The notifyd service can also translate filesystem = notifications into system notifications, which means services like samba = and netatalk can detect (cheaply) when something has changed out from = under them. I see no reason in principle that this couldn=92t be done. There might = be bumps along the way, but there always are=85 I=92d love to see this = stuff in FreeBSD, but I=92m also not in a position to devote time to it. > Beyond that, we probably need to talk about another daemon, which on = OS X is called configd, which deals with larger scope configuration = changes like network devices coming and going, default routes changing, = and so on. This does require some kernel up-call mechanisms, but it = will =93publish=94 its results in terms of notify(3) notifications so = applications can subscribe to those events the same way. If we can hack = usbd into doing the same for USB events, that=92s great, otherwise I = would imagine configd dealing with that sort of hardware notification = service as well. usbd? Where have you been. We=92ve not had a usbd in the base system = since before you started at Apple :). What you call configd is basically = handled by devd at the moment. It would be nice if devd grew = notification support, since right now it just has a tiny character = device interface with a workable, but imperfect, protocol. >> I see a continuum of answer here: If you want to modify the kernel = extensive to use blocks, then that=92s going to be a much bigger problem = than having a few daemons and a library in the tree that require them = which is a bigger problem than having a few daemons using it, but = optionally, and a library which is a bigger problem than a library in = the tree which is a bigger problem than the status quo. >=20 > I.. think=85 I follow that sentence enough to hope I=92ve answered it. = No blocks in the kernel. Nothing more than a few daemons and a library = and, at some point, a way of figuring out when the kernel has made = certain changes to the system configuration, if and where said changes = can=92t be intercepted and dealt with at the libc layer or otherwise = noticed by a user-land daemon which can post the notifications for any = subscribers of them. Yea, it was a tortured way of stating the continuum... > As to the subscribers, they won=92t have to use blocks unless they = choose to consume one or more of notify(3)=92s block APIs. The notify = mechanism was written with both old-style and new-style clients in mind. = If your daemon wants a signal when the notification it=92s subscribing = to changes, fine. If it wants some data on a file descriptor which it=92s= poll(3)/kqueue(3)ing, fine. See = https://developer.apple.com/library/mac/documentation/Darwin/Reference/Man= Pages/man3/notify.3.html and, of course, ignore the mach-specific = delivery methods as N/A. That=92s really cool stuff. I=92d love to see it. >> And 8.1 for libdispatch? Is it really so portable it would work with = our old, crappy gcc pre blocks update? >=20 > Sorry, misunderstanding. I was referring to the original slashdot / = phoronix article which stated that GCD would be coming to FreeBSD in = 8.1. Apparently, according to FreeBSD=92s own wiki = (https://wiki.freebsd.org/GCD) it works, too. Yea, I=92m not sure what happened with that. But the FreeBSD wiki is = editable by any developer at any time, or any friend of a developer that = creates an account, so with controls like that, there=92s bound to be = some inaccurate information there :) Warner