Date: Mon, 11 May 2020 13:40:27 -0700 From: Mark Millard <marklmi@yahoo.com> To: "Patrick M. Hausen" <hausen@punkt.de> Cc: "info@netocean.de" <info@NetOcean.de>, FreeBSD ports <freebsd-ports@freebsd.org> Subject: Re: Poudriere - Compile ALL Ports Message-ID: <2611B3DD-43C3-420D-A909-A8C51EB7EC1E@yahoo.com> In-Reply-To: <FCCB041E-E67E-454B-B0BC-B8907E38D893@punkt.de> References: <625BBD86-9740-41D0-8199-CF1B3ACB5320.ref@yahoo.com> <625BBD86-9740-41D0-8199-CF1B3ACB5320@yahoo.com> <FCCB041E-E67E-454B-B0BC-B8907E38D893@punkt.de>
next in thread | previous in thread | raw e-mail | index | archive | help
On 2020-May-11, at 11:34, Patrick M. Hausen <hausen at punkt.de> wrote:
> I=E2=80=99m confused by this thread. isn=E2=80=99t the point of =
Poudriere
> bulk builds to
When I listed:
# poudriere jail -l
JAILNAME VERSION ARCH METHOD TIMESTAMP =
PATH
FBSDFSSDjail 13.0-CURRENT amd64 null 2018-11-09 =
08:58:53 /usr/obj/DESTDIRs/clang-amd64-installworld-poud
I probably should have explicitly pointed out the use of "METHOD"
"null" as the type of context I was covering.
Poudiere has many options. One of them for "poudriere jail" is:
-m method Specifies which method to use to create the jail. =
(default:http)
. . .
null
This option can be used to import an =
existing
directory that already contains an =
installed
system. The path must be specified =
with -M
path. It is expected that this =
directory be
installed to with the following:
/usr/src# make installworld =
DESTDIR=3DPATH DB_FROM_SRC=3D1
/usr/src# make distrib-dirs =
DESTDIR=3DPATH DB_FROM_SRC=3D1
/usr/src# make distribution =
DESTDIR=3DPATH DB_FROM_SRC=3D1
The path will be null-mounted during =
builds.
It will not be copied at the time of =
running
jail. Deleting the jail will attempt =
to re-
vert any files changed by poudriere.
. . .
This is one way of testing development contexts that
might not exist in svn yet (or ever). There is also
src=3DPATH and tar=3DPATH and the like.
Part of my guess was that such a "not from scratch"
context might be in use. The guess could be wrong
for all I know.
The rest of the below is tied the the "-m null" sort
of jail context: It is not covering all poudriere jail
methods.
> - start with a clean jail
If one manually makes the jail unclean by doing chroot and
things like installing ports, poudriere does not clean out
what one did: it takes the jail (mostly) as-is and uses it
to make the reference jail for the run. (That in turn is
used to make the jails for jobs.)
So it is the poudriere user's responsibility to have the
jail's content clean before starting poudriere. (Remember:
-m null as a poudriere jail context is the context being
covered.)
If started clean, poudriere will keep it clean. If started
dirty, poudriere will keep it dirty.
> - build the first port in the list and all dependencies
The order is dependencies have their packages built first
and dependencies are later used by being temporarily
installed in the job's jail before building what was
dependent. After the dependent package is packaged, the
temporary material is eliminated before the next port is
built in that jail.
> - store the packages for later incorporation into the pkg repo
Previously built packages are also used to install
what a dependent port's build depends on, even in the
same overall bulk run.
> - clean out the jail completely
There are multiple jails involved, especially for
parallel jobs.
poudriere makes a reference jail off the original user
jail (off of /usr/obj/DESTDIRs/clang-amd64-installworld-poud
in my example). It then makes jails for the parallel
jobs off the reference jail as I understand. These are what
get poudriere's equivalent of what a "clean out" would be
(between jobs that reuse the specific temporary jail).
For the most part poudriere leaves the likes of
/usr/obj/DESTDIRs/clang-amd64-installworld-poud alone and
works with its temporary reference and job jails.
> - start with the next port in the list
With parallel jobs and complicated dependencies,
the concept of "next" is itself somewhat complicated
but respects the dependency structure by not starting
anything too early for what all it depends on to be
available to be temporarily installed in job's
jail.
> - =E2=80=A6
>=20
> How can there ever be a build conflict unless there is a bug
> like port A depends on B and ImageMagick 7, while B is
> lagging somewhat behind and depends on ImageMagick 6?
If one has poudriere create the build's reference jail from
a user jail that already has ports installed, then the
reference and job jails will also already have those ports
installed. That in turn can lead to conflicts.
> But if ImageMagick 6 and 7 are not dependencies but intended
> target packages, doesn=E2=80=99t Poudriere build each of them in a =
shiny
> new empty environment?
The new environment is only empty if the original user jail
was already empty. (Again: the -m null poudriere jail context
is the intended coverage of the statements.)
> At least that=E2=80=99s what I thought our Poudriere was doing all day
> and night and we never had a build conflict like this. Bugs like
> the aforementioned aside.
I do not think that the original report listed what
options were in use for the poudriere jail involved,
such as -m method. One way to show some of that is
(using my example):
# poudriere jail -jFBSDFSSDjail -i
Jail name: FBSDFSSDjail
Jail version: 13.0-CURRENT
Jail arch: amd64
Jail method: null
Jail mount: /usr/obj/DESTDIRs/clang-amd64-installworld-poud
Jail fs: =20
Jail updated: 2018-11-09 08:58:53
I do a similar thing with the ports tree used:
# poudriere ports -l
PORTSTREE METHOD TIMESTAMP PATH
default null 2017-08-14 21:07:05 /usr/ports
Again a "null" METHOD where I'm supplying my own materials.
Again it allows for testing development materials that are
not checked into svn.
=3D=3D=3D
Mark Millard
marklmi at yahoo.com
( dsl-only.net went
away in early 2018-Mar)
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?2611B3DD-43C3-420D-A909-A8C51EB7EC1E>