Date: Fri, 15 Mar 2002 10:25:21 -0800 From: Terry Lambert <tlambert2@mindspring.com> To: Josh MacDonald <jmacd@CS.Berkeley.EDU> Cc: Parity Error <bootup@mail.ru>, freebsd-fs@FreeBSD.ORG, reiserfs-dev@namesys.com Subject: Re: metadata update durability ordering/soft updates Message-ID: <3C923C91.454D7710@mindspring.com> References: <E16lReK-000C3T-00@f10.mail.ru> <3C910C57.71C2D823@mindspring.com> <20020315065651.02637@helen.CS.Berkeley.EDU>
next in thread | previous in thread | raw e-mail | index | archive | help
Josh MacDonald wrote: > Terry, > > I'm not sure what you're talking about with regards to DOW and > ReiserFS. It doesn't sound right, and I'm pretty sure we're not using > anything like the patented DOW technique as you've described it. As usual, the patent claims are general enough to cover things; see: US: 5666532 US: 5642501 Here is the USPTO patent number search engine: http://164.195.100.11/netahtml/srchnum.htm > We are developing a transaction facility for many of the reasons > suggested at by the original post in this thread. Yes, I understand. > To summarize: > > - The file system has never made any guarantees. Yes it has. If you look at the atime/mtime/ctime update requirements for the OS, they are pretty blatant. THey just aren't enough to be able to blindly use them. > - You can use fsync() to stabilize a single file and its metadata > dependencies. Metadata stabilization should be automatic. What an fsync there does is really enforce ordering on metadata writes, by acting as a barrier. > - You can use two-phase commit above and beyond that. Yes, by implementing on top of fsync. > - If you're not doing the right thing, "then by definition, your > application can't have it's correctness effected... since it has no > correctness to lose." Yes. > - And, "the OS provides a set of services upon which reliable services > can be built, if they are correctly engineered." Yes. > All of these statements are true. Your attitude seems to be that this > is a fine state of affairs, that anyone who writes an application > should be fully informed of all these "transactional" issues, and that > anyone who is not fully informed of all these issues is a complete > moron if they expect to write reliable applications. No, I merely expect that a person who claims to be a craftsman should know his tools. > The problem is that you're asking way to much of the average > programmer, who doesn't understand transactions and isn't aware of how > little the operating system actually guarantees in this regard. It's not a problem with what I'm asking, or a problem with what the OS guarantees, it's a problem with "average programmers". BTW, I would disagree; I don't think that average programmers are that badly informed. If they are, then a CS degree is meaningless. > The other problem is that fsync() and two-phase-commit can seriously > limit application performance, unless you use highly sophisticated > techniques, which again rules out the average programmer. "Correct, fast, cheap. Pick two." > The fact is, it is very difficult to write "reliable services" on top > of the standard primitives, and it is not good enough to call people > morons if they don't understand this. 8-). My gut reaction was to write: You're right. We must also be compassionate, and train them how to properly ask ``Would you like fries with that?''. Frankly, I don't think it's possible to child-proof any career choice to the point that anyone can come in with zero assumptions or talent, and be productive. I personally have very little tolerance for people who get into any career field because of the money, rather than genuine interest in the field. It's my considered opinion that these people will not last out the next downturn, whenever that happens, and the world will not be a poorer place for it when they go off chasing the (then) more lucrative rewards in another field. Frankly, rewards are something that comes because of the work you do, not because of where you do it. It's like searching for the contact lens that you lost in the alley under the street-lamp "because the light is better". The whole dot-bomb thing happened because people wanted to be rewarded commensuarate with their job titles, rather than to their actual contributions to society (at large, or in the small of the company in which they were operating). If you think I regret the people with cardboard "will program for food" signs, think again. I might as well regret "Winter" for the effect it has on species survivability of tropical plants foolish people attempt to grow outdoors, in Ontario, Canada, or regret the effect that "Afternoon" has on Morning Glories. > There is a document describing our transactions design for ReiserFS > version 4, which is currently under development: > > http://namesys.com/txn-doc.html I've read it. I don't disagree, for that application domain, which is certainly a subset of all possible application domains (e.g. I'd never use transactions on a Usenet server). And just having it there doesn't mean that unclued people will automatically use it, if it require explicit invocation. > And somewhat off topic, I have demonstrated that using fsync() and > rename() as a means for reliable, atomic file updates can seriously > limit application performance and that having file system transactions > solves the problem. My point is that applications will perform > better, not worse, if the operating system helps construct reliable > services instead of this do-it-yourself approach. That's true as well, at least for applications that require that. I think your "average programmer too uninformed to know about building reliability from primitives" will be using those primitives, though, so long as they are optional. I also think making them non-optional is an error, in that it would perpetuate assumptions in that environment which were not valid to make in all environments. A scientist, even a computer scientist, needs to learn to think, and that involves coming at problems from first principles, among other things. FWIW: I pointed out that Soft Updates can be generalized to export a transaction interface, with a trivial amount of work, precisely because of the performance issues with barriers. That's the same reason I pointed out that DOW is inferior to Soft Updates: it introduces a draining barrier that interferes with concurrency. Given the patent status of DOW, it's really amazing to me that anyone would not opt for Soft Updates in any contest between the two for the ecological niche they fill, particularly in new work. > Master's thesis: > > http://prdownloads.sourceforge.net/xdelta/xdfs.pdf > > and the graph that shows it all: > > http://www.cs.berkeley.edu/~jmacd/xdfs-vs-rcs.eps Thanks for these references; I'll download them now, and read them later today. -- Terry To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-fs" in the body of the message
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?3C923C91.454D7710>