From owner-freebsd-hackers@FreeBSD.ORG Sun Dec 1 23:19:54 2013 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 63372E80; Sun, 1 Dec 2013 23:19:54 +0000 (UTC) Received: from mail-pb0-x234.google.com (mail-pb0-x234.google.com [IPv6:2607:f8b0:400e:c01::234]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 28B2B1F57; Sun, 1 Dec 2013 23:19:54 +0000 (UTC) Received: by mail-pb0-f52.google.com with SMTP id uo5so17653399pbc.11 for ; Sun, 01 Dec 2013 15:19:53 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=content-type:mime-version:subject:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to; bh=/lH2oZtp5dUpEp/C/QlTGIs4+z89aPXlT6NuyuULmNs=; b=oz0V/Nl9Tg2MTeMHqxKKRYHQjNSImPOT/gvUHYvNQw17CpycKRmG81nn5EVFCKwhEr VWtVImttRg0vEmRGMM8siHXp7aBPRgjY0uliP3TejnQ/hFPJdLftv9kHEOjB37qZyH0O Yrh1QoyLkX0COE6k4oVRXjqe5tU3USjeKGVVeLVX9P4OROK9reuQ8cdq3JZiGqYtzF3T 3CQ72ydR/GcwpE/YT5J556PJH4tgb2un4qRPma1+A2dgBfbvyPkme4DnB6i051vjfHlL gCcVOKw8I+qT/bQgwlcrewZc0OoTdtpJzATtq7hepaMUgqMJBsU6GtfHB9fmTwxdF031 gkPA== X-Received: by 10.66.197.135 with SMTP id iu7mr226096pac.149.1385939993701; Sun, 01 Dec 2013 15:19:53 -0800 (PST) Received: from [10.20.30.11] (75-101-82-48.static.sonic.net. [75.101.82.48]) by mx.google.com with ESMTPSA id sg1sm117872406pbb.16.2013.12.01.15.19.51 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Sun, 01 Dec 2013 15:19:52 -0800 (PST) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 7.0 \(1822\)) Subject: Extended Attributes and how to avoid them (was Re: O_XATTR support in FreeBSD?) From: Jordan Hubbard In-Reply-To: Date: Sun, 1 Dec 2013 15:19:50 -0800 Content-Transfer-Encoding: quoted-printable Message-Id: <92F46317-D62D-4E19-B687-2A392309A244@mail.turbofuzz.com> References: <718836647.19911209.1385302696963.JavaMail.root@uoguelph.ca> <706707CA-BD52-4814-BCCE-EB044B062BA6@kientzle.com> To: Lionel Cons X-Mailer: Apple Mail (2.1822) Cc: Rick Macklem , Cedric Blancher , Freebsd hackers list , Richard Yao , Pedro Giffuni X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.16 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 01 Dec 2013 23:19:54 -0000 On Dec 1, 2013, at 2:05 PM, Lionel Cons = wrote: > But this discussion is *not* about extended attributes, this > discussion is about Alternate Data Streams. Unfortunately the O_XATTR > discussion somehow started to cover the Linux "extended attribute > system", which is utterly useless in the intended use cases (as said, > no access through normal POSIX read(), write(), mmap(), no unlimited > size, no sparse data support (aka SEEK_HOLE, SEEK_DATA) etc etc). I think this discussion doesn't really *know* what it's about, frankly, = because there are so many possible avenues to choose from! :-) As we saw earlier, there is apparently some interest in supporting ADS = for Windows clients, though the question of how to actually add that = support seems primarily the job of Samba (or whatever BSD-licensed = equivalent someday emerges), so there's really not much to discuss there = from FreeBSD's perspective since FreeBSD itself has little to say on the = subject. If native CIFS support ever becomes a possibility, I'm sure it = will come up again! Then there's the whole topic of EAs (and I don't know who said Linux EAs = represented some sort of gold standard - I certainly didn't) and what = the intended use cases are. Let's stick with the intended (and = citable) use cases, if you please, because a lot of academic debate over = the years about "how EAs should work" has been, to be perfectly honest, = ultimately *pointless*. Academically speaking, there's nothing you can = do with an EA that you can't conceptually do just as well, if not = better, with a detached attribute database because academics don't have = to worry about their EAs working anywhere outside a laboratory setting! It's the *pragmatic* discussions and clearly defined use cases that = carry more weight (if not ALL the weight) - that's where you get into = real-world concerns about EAs and how to avoid them and their associated = files parting company, how to serialize and back them up, what clients = are *actually* going to use them and what APIs they need, etc. etc. Since you brought up POSIX APIs, let's talk about that for a second. = I've worked with EAs "in the field", as it were, a lot (a LOT) and no = one during my long history with them has ever demanded the ability to = call read() or write() on an EA, to mmap() one, or to store sparse data = in one. I would love to know which apps actually need to do that (and = why), because other than "unlimited size", none of those demands have = ever hit any bug database I've had access to. I'm also generally not = one to throw marketing numbers around in a technical conversation, but = with 72 million seats and over 1 million applications (and by all means = fact-check those numbers), if the ability to use EAs in that fashion = were truly necessary, I suspect I would have heard that early and often. = If anything, the trend has been in the other direction - people want a = simple file property getting/setting API that maybe uses EAs under the = covers or maybe it doesn't, all they know is that they can hand the API = a file handle (or path) and a dictionary and The Right Thing happens for = storing the EAs, the converse also being true for getting them. EAs = just are not first-class filesystem citizens and, frankly, they don't = really need to be in order to be "useful enough" for those situations = where an application or bit of OS middleware really needs a way of = storing some extended metadata for a file in a filesystem-neutral = fashion (and we've already covered the network filesystem and archiver = scenarios which make that important). I'll opine that If FreeBSD really wants to support EAs in a "useful = enough" way, then the best way of doing so is to stay focused on the = pragmatic "this our usage cases, and we are not afraid to describe them = in detail!" side of the street because, as I said, the academic = discussions generally don't lead anywhere but in circles. A pragmatic = approach will, conversely, lead to doing just the basic minimums and not = waste time implementing anything that won't actually be needed in = real-world scenarios. Heck, if we really want to get all academic about something here, let's = forget about EAs and ADS as comparatively uninteresting technologies = from the 90's and start talking instead about file object stores that = are far more flexible than what we have now! I don't want to have my filesystem view be necessarily hierarchical = (that should be a policy decision, not intrinsic to the filesystem = itself). I don't want any process to necessarily be able to see any = part of the file object space save that which I explicitly grant it or = its children. I don't want to have to think about where a file object = lives - I'd like it to be able to move around (memory, on-disk, "the = cloud", etc) purely in response to how "hot" it is without me having to = know or care about anything other than the object changing out from = under me (which should also be an intrinsic part of the filesystem = access APIs). I want file objects to be able to have arbitrary = properties of any type or size, and able to reference other file = objects, such that I don't have to keep side-stores around everywhere to = facilitate a lot of basic operations (like searching) that should be = intrinsic to the object store, or at least handled by a first class OS = service with the ability to be co-resident with it so things like = indexing are actually *efficient*. Can we have that discussion instead? It would be more fun. :-) - Jordan