From nobody Wed Jan 22 16:30:09 2025 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4YdV1B5RrKz5khsL for ; Wed, 22 Jan 2025 16:30:22 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: from mail-pl1-x62a.google.com (mail-pl1-x62a.google.com [IPv6:2607:f8b0:4864:20::62a]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "WR4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4YdV1B2WM5z3mkk for ; Wed, 22 Jan 2025 16:30:22 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-pl1-x62a.google.com with SMTP id d9443c01a7336-216401de828so124020685ad.3 for ; Wed, 22 Jan 2025 08:30:21 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bsdimp-com.20230601.gappssmtp.com; s=20230601; t=1737563420; x=1738168220; darn=freebsd.org; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=LcgPLuzb/oKMgx0S9DGMvvTGZUIUzcQdoqvZ0Sh7rFM=; b=Q/qerWgGAUmi+j2kmyCpslSMBe2UeVd3KOIsJa8mD7kr3OPNzDZvucMDxrpXqt5IxN pH8cSt9WYxEz2xn4L6irbSxZnk+jDS3MbsbH4vcpJf4uJLZ+BXDAcdGYNk8s5/lSPYJq LVkdKN8Ua2U6YZGWDgbzoA1/jGqBQg2WkMSlDDQpqS9+xf3M7+lj0gwPd2tzwIKkm8IG JkY27wm6fcZdz+iObefhXHiRlh+bqapWHCMKdAc+Y1L3FM5K061zyICK9idnQ+umHZvR cTfkEI+EF5nedV9A5DJL4HWvLnO7iRMmnVwAuzZTvY9+AmrgPTCSrCMmSCgF5bp7jqe2 Qryw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1737563420; x=1738168220; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=LcgPLuzb/oKMgx0S9DGMvvTGZUIUzcQdoqvZ0Sh7rFM=; b=GcdkDoH5UDYCqr+ddxN9mYW576YChDJV2lhCUDYhMjn3CTUK3HdMToKBG9Crw9ROHi 1RsE0Xt8jwI6+/63wMYoPiEf45NrG/0S2iCoR3x5nJGQC1tyYZ7IqI/I690A/sRwItWq Ao6WHqFS/ujK7OoqdCYOYjZoRTlv6Pf4SmBnx4NSvIJc+8G1rDddyE3NyjYf4U3AzwIY PUI32/LPPKIbKkESuMMJrpwuNrhmVoqiDx8utqJBvmspr8YXL3AZdKmH3YB9LXflpq/K XAq7vnD9xWcjHHF6J9W+ue212amEtn7M8i7K+PJU6O27pa1VVCoFUeo/QI7pO0KE+n// QBYw== X-Forwarded-Encrypted: i=1; AJvYcCW/xkV0yS6aOkUrt4/4b0/cAs7d6MjN4/RSC2qyIUtdHSYPtMD0v5AmZOYHmi/lP6yk1dwH9wEzFjtpTwVl/wM=@freebsd.org X-Gm-Message-State: AOJu0YwZSFKna8z4wRpbLtTv5Eoq29/b32inxlaklsFijtZGU/y8OpKm LHunQdtk+3lwE8pGmYj6z3Jhwl6r4p3z3aGyxRPchveDKPjXH1k2Scj+k57ec1insCqjPCsTIqD 1NGUw3aHIVU3GyQlRbsYfoxPu3MbGmQ+9lA6TjKRw6bbTraN5exEXcA== X-Gm-Gg: ASbGncsptYbW2BKZXmz+ikeaD9v0kHmbsfnh9bNkKDimWeubDlBcVmmj26P+haNgQoE 8tRnz0vND2bCBfPjY/oOCkJtwpKqQ1RxyopC0q75yAemyCByr8R1hI4qIvvuXgA== X-Google-Smtp-Source: AGHT+IEzmCoUGj/Rt+77inY76OYwtY9h4EJIr9Ah4H8u9Bi5rTmAss9SArUcAhChHGopsg23F3+4MdhIhhBDcfJFICs= X-Received: by 2002:a05:6a20:244c:b0:1e1:94a2:275c with SMTP id adf61e73a8af0-1eb21485c90mr38140969637.18.1737563419990; Wed, 22 Jan 2025 08:30:19 -0800 (PST) List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@FreeBSD.org MIME-Version: 1.0 References: <3782b90b-850f-4435-8b6d-bdb83753daea@app.fastmail.com> In-Reply-To: <3782b90b-850f-4435-8b6d-bdb83753daea@app.fastmail.com> From: Warner Losh Date: Wed, 22 Jan 2025 09:30:09 -0700 X-Gm-Features: AbW1kvYvheOyKucNndFqy8zNCWyCMKG0VlPEFia3SQYvkykWhm9nB4-n3bXtOIA Message-ID: Subject: Re: /usr/src and /usr/ports not git directories ? To: Dave Cottlehuber Cc: bob prohaska , Sulev-Madis Silber , freebsd-current Content-Type: multipart/alternative; boundary="0000000000006c7a30062c4e020d" X-Rspamd-Queue-Id: 4YdV1B2WM5z3mkk X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US] --0000000000006c7a30062c4e020d Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On Wed, Jan 22, 2025 at 1:23=E2=80=AFAM Dave Cottlehuber wrote: > On Tue, 21 Jan 2025, at 21:29, Warner Losh wrote: > >> Do you mean have the "install src" checkbox invoke git clone? > >> That seems like a better idea, at least to me. > > This wouldn't be ideal for small systems or any area with lousy > internet. > > > I think we should replace the populate /usr/src from a tarball with.... > > populate it > > with a tarball that represents a 1-deep checkout tree at the rev we > > built the release > > from. This lets users have the source, has minimal overhead and also > > lets users update > > or turn the shallow checkout into a deep one, etc. A shallow checkout > > is quite a bit > > less than a full tree, though still more than just the raw files. I've > > not done poking to > > see size comparisons. > > Can we ship a working gitup config? I've not used this yet, but this > would be a good option. > > Another option is to ship src.txz as we do today, but include a minimal > git config, such that `git fetch origin releng/14.2` would work. I like > this because we're not really reliant on a specific version of git > packfiles, and it's still just src as usual. > > Another option, IIRC Kyle mentioned this last year, is to use `git bundle= ` > https://git-scm.com/book/en/v2/Git-Tools-Bundling but I don't really know > enough about that. Perhaps that would be a reasonable option to fetch ove= r > network, in the installer? > > For the former, this seems to be sufficient: > > ``` > # /usr/src/ > .git/ > .git/branches/ > .git/objects/ > .git/refs/ > .git/HEAD ref: refs/heads/releng/14.2 > .git/description FreeBSD 14.2-RELEASE > .git/config see below > ``` > > ``` > # .git/config > [core] > repositoryformatversion =3D 0 > filemode =3D true > bare =3D false > logallrefupdates =3D true > > [remote "origin"] > url =3D https://git.freebsd.org/src.git > pushurl =3D git@gitrepo.freebsd.org:src.git > fetch =3D +refs/heads/releng/14.2:refs/remotes/origin/releng/14.2 > > [branch "releng/14.2"] > remote =3D origin > merge =3D refs/heads/releng/14.2 > ``` > > this yields ~ 350MiB raw, as txz its still 325MiB. > > ``` > $ git fetch --depth 1 --no-tags --jobs 32 --set-upstream origin > releng/14.2 > remote: Enumerating objects: 103451, done. > remote: Counting objects: 100% (103451/103451), done. > remote: Compressing objects: 100% (89088/89088), done. > remote: Total 103451 (delta 21659), reused 41475 (delta 10515), > pack-reused 0 (from 0) > Receiving objects: 100% (103451/103451), 344.67 MiB | 9.13 MiB/s, done. > Resolving deltas: 100% (21659/21659), done. > From https://git.freebsd.org/src > * branch releng/14.2 -> FETCH_HEAD > > $ du -chs .git > 348M .git > 348M total > ``` > > this isn't quite right, git switch / git reset --hard or similar might be > needed, > but its pretty close. > I think what you want is % git clone https://github.com/freebsd/freebsd-src --depth=3D1 -b releng/14= .2 --single-branch fred --bare % cd fred % mkdir .git % mv * .git % git config --local core.bare false This will clone 1 deep into the directory 'fred' and won't checkout a copy. If we make this a package, it's 325MB, as you say. That can then be extracted and then you can recover the tree with: % git checkout releng/14.2 which could be done with the pkg file when extracting to /usr/src. I've not measured ports. This would give the user the sources to the release in a package that's fairly minimal. And it would let them fetch other branches more cheaply than having no git repo at all if they are internet connected. It's a good balance between giving everything and taking up too much space. And it's only a little bit larger than the src pkg today. Warner --0000000000006c7a30062c4e020d Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable


On Wed, Jan 22,= 2025 at 1:23=E2=80=AFAM Dave Cottlehuber <dch@skunkwerks.at> wrote:
On Tue, 21 Jan 2025, at 21:29, Warner Losh wrote:<= br> >> Do you mean have the "install src" checkbox invoke git c= lone?
>> That seems like a better idea, at least to me.

This wouldn't be ideal for small systems or any area with lousy
internet.

> I think we should replace the populate /usr/src from a tarball with...= .
> populate it
> with a tarball that represents a 1-deep checkout tree at the rev we > built the release
> from. This lets users have the source, has minimal overhead and also <= br> > lets users update
> or turn the shallow checkout into a deep one, etc. A shallow checkout =
> is quite a bit
> less than a full tree, though still more than just the raw files. I= 9;ve
> not done poking to
> see size comparisons.

Can we ship a working gitup config? I've not used this yet, but this would be a good option.

Another option is to ship src.txz as we do today, but include a minimal
git config, such that `git fetch origin releng/14.2` would work. I like
this because we're not really reliant on a specific version of git
packfiles, and it's still just src as usual.

Another option, IIRC Kyle mentioned this last year, is to use `git bundle`<= br> https://git-scm.com/book/en/v2/Git-Tools-Bundling but I don't really know
enough about that. Perhaps that would be a reasonable option to fetch over<= br> network, in the installer?

For the former, this seems to be sufficient:

```
# /usr/src/
.git/
.git/branches/
.git/objects/
.git/refs/
.git/HEAD=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0ref: refs/h= eads/releng/14.2
.git/description=C2=A0 =C2=A0 =C2=A0 =C2=A0 FreeBSD 14.2-RELEASE
.git/config=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0see below
```

```
# .git/config
[core]
=C2=A0 =C2=A0 =C2=A0 =C2=A0 repositoryformatversion =3D 0
=C2=A0 =C2=A0 =C2=A0 =C2=A0 filemode =3D true
=C2=A0 =C2=A0 =C2=A0 =C2=A0 bare =3D false
=C2=A0 =C2=A0 =C2=A0 =C2=A0 logallrefupdates =3D true

[remote "origin"]
=C2=A0 =C2=A0 =C2=A0 =C2=A0 url =3D https://git.freebsd.org/src.git
=C2=A0 =C2=A0 =C2=A0 =C2=A0 pushurl =3D git@gitrepo.freebsd.org:src.git
=C2=A0 =C2=A0 =C2=A0 =C2=A0 fetch =3D +refs/heads/releng/14.2:refs/remotes/= origin/releng/14.2

[branch "releng/14.2"]
=C2=A0 =C2=A0 =C2=A0 =C2=A0 remote =3D origin
=C2=A0 =C2=A0 =C2=A0 =C2=A0 merge =3D refs/heads/releng/14.2
```

this yields ~ 350MiB raw, as txz its still 325MiB.

```
$ git fetch --depth 1=C2=A0 --no-tags --jobs 32 --set-upstream origin relen= g/14.2
remote: Enumerating objects: 103451, done.
remote: Counting objects: 100% (103451/103451), done.
remote: Compressing objects: 100% (89088/89088), done.
remote: Total 103451 (delta 21659), reused 41475 (delta 10515), pack-reused= 0 (from 0)
Receiving objects: 100% (103451/103451), 344.67 MiB | 9.13 MiB/s, done.
Resolving deltas: 100% (21659/21659), done.
>From https://git.freebsd.org/src
=C2=A0* branch=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 relen= g/14.2 -> FETCH_HEAD

$ du -chs .git
348M=C2=A0 =C2=A0 .git
348M=C2=A0 =C2=A0 total
```

this isn't quite right, git switch / git reset --hard or similar might = be needed,
but its pretty close.

I think what you = want is

%=C2=A0git clone https://github.com/freebsd/freebsd-src --dept= h=3D1 -b releng/14.2 --single-branch fred --bare
% cd fred
<= div>% mkdir .git
% mv * .git
% git config --local core.= bare false

This will clone 1 deep into the directo= ry 'fred' and won't checkout a copy. If
we make this = a package, it's 325MB, as you=C2=A0 say. That can then be extracted
and then you can recover the tree with:

% g= it checkout releng/14.2

which could be done with t= he pkg file when extracting to /usr/src. I've not measured
po= rts.

This would give the user the sources to the r= elease in a package that's fairly
minimal. And it would let t= hem fetch other branches more cheaply than
having no git repo at = all if they are internet connected. It's a good balance
betwe= en giving everything and taking up too much space. And it's only a litt= le bit
larger than the src pkg today.

Wa= rner

--0000000000006c7a30062c4e020d--