Date: Fri, 13 Apr 2012 19:34:30 -0700 From: Jason Evans <jasone@canonware.com> To: obrien@freebsd.org Cc: freebsd-current@freebsd.org Subject: Re: contrib/jemalloc Message-ID: <56C2E1FF-DEC4-47DA-A23B-BF81F5EE26FE@canonware.com> In-Reply-To: <20120413224303.GA89952@dragon.NUXI.org> References: <431CB493-836B-4DF4-AC42-A7C6ABF7DE3E@canonware.com> <20120412184114.GB71001@dragon.NUXI.org> <A29E26F5-2324-4DE2-BD61-7EBA3245AF43@canonware.com> <20120413224303.GA89952@dragon.NUXI.org>
next in thread | previous in thread | raw e-mail | index | archive | help
On Apr 13, 2012, at 3:43 PM, David O'Brien wrote: > On Thu, Apr 12, 2012 at 01:19:56PM -0700, Jason Evans wrote: >> On Apr 12, 2012, at 11:41 AM, David O'Brien wrote: > It looks like you could run './FREEBSD-upgrade extract' from a check = out > of ssh://svn.freebsd.org/base/vendor/jemalloc. >=20 > I'm unclear what the './FREEBSD-upgrade rediff' step is for. I'm = having > trouble following what "then regenerate diffs to update line offsets" = is > for. The 'rediff' step regenerates the diff so that it precisely matches the = differences as they apply to the code about to be imported. If this = step were omitted, patch fuzz would accumulate in FREEBSD-diffs. > How do you handle the difference between FreeBSD having > include/malloc_np.h and your GIT repo having > jemalloc/include/jemalloc/jemalloc.h[.in]? >=20 > Does './FREEBSD-upgrade extract' or './FREEBSD-upgrade rediff' patch > 'include/malloc_np.h' and 'include/stdlib.h'? > Especially with the '#define JEMALLOC_P(s) s' wrapping. > Is that part of the './FREEBSD-upgrade rediff' step? stdlib.h+malloc_np.h and jemalloc.h are different enough that they = require separate maintenance. Alas, not all programming can be = automated; if interfaces change, manual intervention is required. > contrib/jemalloc/FREEBSD-upgrade doesn't describe the "commit into > FreeBSD subversion repo" step, that I can see. So I am not sure what > gets committed vs. what's in git://canonware.com/jemalloc.git. Every difference that remains in the tree afterwards is to be committed. = That might require some 'svn add' and/or 'svn rm' commands prior to = 'svn commit'. >> The only upside to vendor branch merges I can think of is >> that if any jemalloc sources were manually modified between imports, >> merging would fail rather than silently overwriting the changes. >=20 > I feel that is an important check-and-balance. >=20 > Not following our established procedure also means that the average > developer(committer) and commercial consumer will have their = expectations > fails. One expects to be able to find the stock vendor sources in > ssh://svn.freebsd.org/base/vendor/ and to be able to find FreeBSD = local > changes to the sources thru 'svn diff' against > ssh://obrien@svn.freebsd.org/base. Regarding established procedure, I read through several of the = contrib/*/FREEBSD-upgrade files when trying to understand established = procedure, and the conclusion I came to is that many of the vendor = imports would be painful for a new person to take over. In contrast, = contrib/jemalloc/FREEBSD-upgrade provides nearly complete automation, = and documentation on how to deal with the manual parts of the process, = should the need arise. >> However, this presumes that changes aren't making it upstream. >=20 > That is not an unreasonable presumption. Unless contrib/jemalloc/ > is locked so that you are aware of every commit FreeBSD and > git://canonware.com/jemalloc.git can diverge at times. Today I updated the FREEBSD-upgrade script to take care of this = possibility. = http://people.freebsd.org/~jasone/patches/jemalloc_20120413a.patch One problem that doesn't fit neatly into the standard vendor import = procedure is that jemalloc.3 is a generated file. It isn't feasible to = keep FreeBSD-specific changes out of the vendor branch unless jemalloc's = build system is imported, or manual changes are made directly to the = generated file (yuck). In contrast, the FREEBSD-upgrade script leaves = all non-essential files out of the imports, resulting in a cleaner = import. Imagine we're looking at the svn history three years from now, and I've = croaked, leaving the FreeBSD copy of jemalloc to drift where it will. = svn might contain something like the following sequence of relevant = commits: - (jasone) Import jemalloc 7ca0fdfb85b2a9fc7a112e158892c098e004385b. - (jasone) Import jemalloc [some other git rev]. - (jasone) Import jemalloc [some other git rev]. - (jasone) Import jemalloc 3.0.0. - (jasone) Import jemalloc 3.0.1. - (obrien) Fix aligned_alloc() to stop snatching helpless children. - (jasone) Import jemalloc 3.2.0. - (obrien) Stop protecting children; they're not as helpless as they = appear. - (obrien) Import jemalloc 3.2.1. May jasone RIP. - (obrien) This allocator sucks. Gut it. What gets lost? As near as I can tell, the svn history tells the whole = story, and 'svn blame' even reflects FreeBSD-specific changes that = weren't made as part of an import commit (and FREEBSD-diffs provides a = full audit trail for those). It's worth mentioning that the less work imports are, the more often I = will be able to do them. I've put several days of effort into = streamlining this process so that FreeBSD's jemalloc won't languish. = I've been working more or less full time on jemalloc for the past six = weeks (thanks Facebook!), but I really have to wrap this up and get back = to other projects. Jason=
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?56C2E1FF-DEC4-47DA-A23B-BF81F5EE26FE>