Date: Mon, 26 May 2014 18:47:34 -0700 From: Nathan Whitehorn <nwhitehorn@freebsd.org> To: dteske@FreeBSD.org, 'Tijl Coosemans' <tijl@FreeBSD.org>, 'Warner Losh' <imp@bsdimp.com> Cc: 'Baptiste Daroussin' <bapt@FreeBSD.org>, src-committers@FreeBSD.org, 'Ian Lepore' <ian@FreeBSD.org>, svn-src-all@FreeBSD.org, 'Glen Barber' <gjb@FreeBSD.org>, svn-src-head@FreeBSD.org Subject: Re: svn commit: r266553 - head/release/scripts Message-ID: <5383EEB6.6010703@freebsd.org> In-Reply-To: <004b01cf7936$9fb11050$df1330f0$@FreeBSD.org> References: <201405221922.s4MJM4Y9025265@svn.freebsd.org> <537F6706.6070509@freebsd.org> <20140523153619.GF72340@ivaldir.etoilebsd.net> <537F6EBC.3080008@freebsd.org> <20140523162020.GG72340@ivaldir.etoilebsd.net> <C5A59513-AF58-4749-BCD7-F54BB6F56E90@gmail.com> <20140524165940.3c687553@kalimero.tijl.coosemans.org> <5380C311.60201@freebsd.org> <20140524185345.263f230d@kalimero.tijl.coosemans.org> <1400955835.1152.323.camel@revolution.hippie.lan> <5380EBA8.1030200@freebsd.org> <20140525011307.142b41ab@kalimero.tijl.coosemans.org> <3CCAFAD3-FABE-40EF-ABF9-815FE5826349@bsdimp.com> <9FE34CE4-C71F-4806-9EF6-30CB1051C62F@bsdimp.com> <20140526113502.239db74d@kalimero.tijl.coosemans.org> <5383522F.30108@freebsd.org> <004b01cf7936$9fb11050$df1330f0$@FreeBSD.org>
next in thread | previous in thread | raw e-mail | index | archive | help
On 05/26/14 16:02, dteske@FreeBSD.org wrote: > >> -----Original Message----- >> From: owner-src-committers@freebsd.org [mailto:owner-src- >> committers@freebsd.org] On Behalf Of Nathan Whitehorn >> Sent: Monday, May 26, 2014 7:40 AM >> To: Tijl Coosemans; Warner Losh >> Cc: Ian Lepore; Baptiste Daroussin; svn-src-head@freebsd.org; Glen Barber; >> svn-src-all@freebsd.org; src-committers@freebsd.org >> Subject: Re: svn commit: r266553 - head/release/scripts >> >> On 05/26/14 02:35, Tijl Coosemans wrote: >>> On Sat, 24 May 2014 19:00:18 -0600 Warner Losh wrote: >>>> On May 24, 2014, at 5:53 PM, Warner Losh <imp@bsdimp.com> wrote: >>>>> On May 24, 2014, at 5:13 PM, Tijl Coosemans <tijl@freebsd.org> wrote: >>>>>> There isn't necessarily any chroot environment. There's one >>>>>> kernel, two equally valid ABIs (ILP32 and LP64) and any binary like >>>>>> uname might use either of them. If uname -p returns a different >>>>>> result depending on which of these two ABIs it was compiled for >>>>>> that could be a problem for any script that uses it. >>>>> Well, it depends on what you want to do with the script, eh? If you >>>>> want to know the ABI of the native binary uname, that?s one thing. >>>>> But if you want to know the supported ABIs, you are doing it wrong by >> using uname. >>>>> You should be using sysctl kern.supported_abi. That will tell you >>>>> all the ABIs that you can install packages for on this machine, >>>>> which is what you really want to know. So I?m having trouble >>>>> connecting the dots between this and what you are saying here. >>>>> >>>>> I still am absolutely flabbergasted why the MACHINE_ARCH names >>>>> aren?t necessary and sufficient for packaging. I?ve yet to see any >>>>> coherent reason to not use them. >>>> Why do I care that they match? Good question. When I was doing >>>> FreeNAS, I looked at integrating pkgng into nanobsd. At the time this >>>> was quite difficult because every single architecture name was >>>> different between pkgng and MACHINE_ARCH. This would mean I?d have >>>> to drag around a huge table to know how to translate one to the other >>>> (there was no simple regex either, and things like mipsn32 wouldn?t >>>> have fit into the scheme at the time). I would very much like us to >>>> see us keep these names in sync and avoid large translation tables that >> are difficult to maintain. >>>> Now, do you need to get it from uname -p? No. If you want to parse >>>> elf files to get it, that?s fine, so long as the names map directly >>>> to the MACHINE_ARCH names that we?ve been using for years. They >>>> completely describe the universe of supported platforms. Are they >>>> perfect? No, around the edge there may be an odd-ball that?s possible >>>> to build, but is unsupported and likely doesn?t work at all. Have we >>>> learned from these mistakes? Yes. Anything that?s actively supported >>>> has a proper name. This name is needed, btw, so that any machine can >>>> self-host, a nice feature of the /usr/src system. >>> ABI consists of the following elements: >>> >>> - OS >>> - OS ABI version (major version number in FreeBSD) >>> - instruction set >>> - programming model (ILP32 or LP64) >>> - byte order (little/big endian) >>> >>> These are almost orthogonal dimensions in the sense that almost any >>> combination is possible. (A combination that isn't possible is a >>> 32-bit instruction set with LP64.) >>> >>> What you are asking for now is to combine two dimensions into one and >>> combination in this case means multiplication so if you have 3 >>> instruction sets and 2 programming models, the combined dimension >>> needs >>> 6 different values. You need to make the case for why you think this >>> is a good idea. For the past 20 years we got away with this because >>> on every installation of FreeBSD we only used one programming model at >>> a time. This is still the case for byte order of course. >>> >>> What I'm saying is to keep the option open for installations with >>> multiple programming models, where most binaries could use ILP32 and >>> only the ones that actually need a 64-bit address space use LP64. >>> You query the instruction set using uname and the programming models >>> using getconf. >>> >>> I suppose you could replace the "x86" in the pkg scheme with >>> i386/amd64, but then you'd still be talking about i386:32, amd64:32 >>> and amd64:64 instead of x86:32, x86:x32 and x86:64. >>> >> No. We support multiple "models" now and have for ten years. That's what >> MACHINE_ARCH is for: it defines the choice of the last three things you > list >> above. Specifically, a shared value of MACHINE_ARCH guarantees and OS >> version guarantees, in FreeBSD-land, complete binary compatibility of >> executables. Kernels support multiple ones, in general (e.g. i386 binaries > on >> amd64, powerpc binaries on powerpc64). They may support more in the >> future (x32 on amd64, potentially even cross-endian binaries). We have a >> nice flexible scheme in FreeBSD for supporting this. If you want to find > out >> the list of the things the installed kernel can run, check the >> kern.supported_archs sysctl. Simple. >> >> These strings are just as expressive as the ones in pkg. They are the >> standard. They're what external build systems test against, what the src, > doc, >> and ports trees use to define what to do universally. It's what users and > code >> expect. The wheel we've had for 20 years is perfectly good -- why invent a >> new, incompatible one? > +1 > > Trying to handle things like PR bin/187458 wherein I'm finding that if I > want to > have both a usable $PKG_ABI variable in my code _and_ not automatically > bootstrap pkg to get at said ABI, I'm going to have to do one of two things: > > a. pepper throughout my code -- everywhere I require $PKG_ABI to be set -- > some new code that will check if it is NULL (pkg hasn't been bootstrapped > yet > and thus cannot provide an ABI) and then prompt the user asking them if it > is > OK to bootstrap the thing. > > Alternatively... > > b. Build a table that maps from arch/march to proposed ABI > > Not ideal in my mindset, +1 to Nathan's cogent argument above and many > that came prior in the thread. I've written two patches today. The first (http://people.freebsd.org/~nwhitehorn/pkg_machinearch.diff) is to pkg itself and the second (http://people.freebsd.org/~nwhitehorn/pkg_bootstrap_machinearch.diff) is to the pkg bootstrapper in base. These switch pkg from using identifiers like "freebsd:11:arm:be:eabi:softfp" to identifiers like "FreeBSD:11:armeb", matching the canonical FreeBSD platform identifiers. The strings it uses can be predicted easily from scripts, as they are identical in all cases to the output of `uname -s`:`uname -r | cut -f 1 -d .`:`uname -p`. I tried to avoid changing much, so the patches are pretty short. Internally, the patch introduces a translation table to pkg that contains all extant FreeBSD and Dragonfly BSD architectures and moves between the ELF-based coding and MACHINE_ARCH values. This is kind of gross, but has the least possibility for regression, and can easily be changed behind the scenes later. Platform detection uses the same ELF-parsing code as before. The current/previous values are also kept so that the patched pkg can install a package marked either with an x86:64 or amd64-type architecture ID (symlinks will be needed for a little bit on the package server to allow both clients to work). Limited testing suggests it works well -- I can fetch and install packages fine. More testing would be great. One small issue is how to bootstrap the change for existing binary package users. The modified pkg can use packages with either architecture ID just, but the current one will barf on the FreeBSD:11:amd64 package containing its own update. There are a couple of options: manual instructions, marking that one package with the old-style architecture ID, etc. None should be more than slightly irritating, though. The least bumpy route, I think, is making directories with both the old and new names, but putting only one package in the old-named directory: a special intermediate version of pkg marked with the old architecture ID but able to install from the new one. Then you just have to deal with two rounds of updates without any other intervention, which is not so bad. -Nathan
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?5383EEB6.6010703>