Date: Wed, 22 Jan 2025 09:30:09 -0700 From: Warner Losh <imp@bsdimp.com> To: Dave Cottlehuber <dch@skunkwerks.at> Cc: bob prohaska <fbsd@www.zefox.net>, Sulev-Madis Silber <freebsd-current-freebsd-org111@ketas.si.pri.ee>, freebsd-current <freebsd-current@freebsd.org> Subject: Re: /usr/src and /usr/ports not git directories ? Message-ID: <CANCZdfruU=GR5DFWL7RQA4_yjh2-NHOB669wsOSL1DzZ3Fs7%2BQ@mail.gmail.com> In-Reply-To: <3782b90b-850f-4435-8b6d-bdb83753daea@app.fastmail.com> References: <Z4vk3009iSwuzG4K@www.zefox.net> <Z4__B0EQM-ce0qPE@cell.glebi.us> <C509F94C-2AC2-414F-90C0-355C69869D72@ketas.si.pri.ee> <Z5AQ1GcwX_MZw69G@www.zefox.net> <CANCZdfoHUsZusqMg_gWN5mB9P3xByGv_GfELi9Dd63CHto1igw@mail.gmail.com> <3782b90b-850f-4435-8b6d-bdb83753daea@app.fastmail.com>
index | next in thread | previous in thread | raw e-mail
[-- Attachment #1 --] On Wed, Jan 22, 2025 at 1:23 AM Dave Cottlehuber <dch@skunkwerks.at> wrote: > On Tue, 21 Jan 2025, at 21:29, Warner Losh wrote: > >> Do you mean have the "install src" checkbox invoke git clone? > >> That seems like a better idea, at least to me. > > This wouldn't be ideal for small systems or any area with lousy > internet. > > > I think we should replace the populate /usr/src from a tarball with.... > > populate it > > with a tarball that represents a 1-deep checkout tree at the rev we > > built the release > > from. This lets users have the source, has minimal overhead and also > > lets users update > > or turn the shallow checkout into a deep one, etc. A shallow checkout > > is quite a bit > > less than a full tree, though still more than just the raw files. I've > > not done poking to > > see size comparisons. > > Can we ship a working gitup config? I've not used this yet, but this > would be a good option. > > Another option is to ship src.txz as we do today, but include a minimal > git config, such that `git fetch origin releng/14.2` would work. I like > this because we're not really reliant on a specific version of git > packfiles, and it's still just src as usual. > > Another option, IIRC Kyle mentioned this last year, is to use `git bundle` > https://git-scm.com/book/en/v2/Git-Tools-Bundling but I don't really know > enough about that. Perhaps that would be a reasonable option to fetch over > network, in the installer? > > For the former, this seems to be sufficient: > > ``` > # /usr/src/ > .git/ > .git/branches/ > .git/objects/ > .git/refs/ > .git/HEAD ref: refs/heads/releng/14.2 > .git/description FreeBSD 14.2-RELEASE > .git/config see below > ``` > > ``` > # .git/config > [core] > repositoryformatversion = 0 > filemode = true > bare = false > logallrefupdates = true > > [remote "origin"] > url = https://git.freebsd.org/src.git > pushurl = git@gitrepo.freebsd.org:src.git > fetch = +refs/heads/releng/14.2:refs/remotes/origin/releng/14.2 > > [branch "releng/14.2"] > remote = origin > merge = refs/heads/releng/14.2 > ``` > > this yields ~ 350MiB raw, as txz its still 325MiB. > > ``` > $ git fetch --depth 1 --no-tags --jobs 32 --set-upstream origin > releng/14.2 > remote: Enumerating objects: 103451, done. > remote: Counting objects: 100% (103451/103451), done. > remote: Compressing objects: 100% (89088/89088), done. > remote: Total 103451 (delta 21659), reused 41475 (delta 10515), > pack-reused 0 (from 0) > Receiving objects: 100% (103451/103451), 344.67 MiB | 9.13 MiB/s, done. > Resolving deltas: 100% (21659/21659), done. > From https://git.freebsd.org/src > * branch releng/14.2 -> FETCH_HEAD > > $ du -chs .git > 348M .git > 348M total > ``` > > this isn't quite right, git switch / git reset --hard or similar might be > needed, > but its pretty close. > I think what you want is % git clone https://github.com/freebsd/freebsd-src --depth=1 -b releng/14.2 --single-branch fred --bare % cd fred % mkdir .git % mv * .git % git config --local core.bare false This will clone 1 deep into the directory 'fred' and won't checkout a copy. If we make this a package, it's 325MB, as you say. That can then be extracted and then you can recover the tree with: % git checkout releng/14.2 which could be done with the pkg file when extracting to /usr/src. I've not measured ports. This would give the user the sources to the release in a package that's fairly minimal. And it would let them fetch other branches more cheaply than having no git repo at all if they are internet connected. It's a good balance between giving everything and taking up too much space. And it's only a little bit larger than the src pkg today. Warner [-- Attachment #2 --] <div dir="ltr"><div dir="ltr"><br></div><br><div class="gmail_quote gmail_quote_container"><div dir="ltr" class="gmail_attr">On Wed, Jan 22, 2025 at 1:23 AM Dave Cottlehuber <<a href="mailto:dch@skunkwerks.at">dch@skunkwerks.at</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">On Tue, 21 Jan 2025, at 21:29, Warner Losh wrote:<br> >> Do you mean have the "install src" checkbox invoke git clone?<br> >> That seems like a better idea, at least to me. <br> <br> This wouldn't be ideal for small systems or any area with lousy<br> internet.<br> <br> > I think we should replace the populate /usr/src from a tarball with.... <br> > populate it<br> > with a tarball that represents a 1-deep checkout tree at the rev we <br> > built the release<br> > from. This lets users have the source, has minimal overhead and also <br> > lets users update<br> > or turn the shallow checkout into a deep one, etc. A shallow checkout <br> > is quite a bit<br> > less than a full tree, though still more than just the raw files. I've <br> > not done poking to<br> > see size comparisons.<br> <br> Can we ship a working gitup config? I've not used this yet, but this<br> would be a good option.<br> <br> Another option is to ship src.txz as we do today, but include a minimal<br> git config, such that `git fetch origin releng/14.2` would work. I like<br> this because we're not really reliant on a specific version of git <br> packfiles, and it's still just src as usual.<br> <br> Another option, IIRC Kyle mentioned this last year, is to use `git bundle`<br> <a href="https://git-scm.com/book/en/v2/Git-Tools-Bundling" rel="noreferrer" target="_blank">https://git-scm.com/book/en/v2/Git-Tools-Bundling</a>; but I don't really know<br> enough about that. Perhaps that would be a reasonable option to fetch over<br> network, in the installer?<br> <br> For the former, this seems to be sufficient:<br> <br> ```<br> # /usr/src/<br> .git/<br> .git/branches/<br> .git/objects/<br> .git/refs/<br> .git/HEAD ref: refs/heads/releng/14.2<br> .git/description FreeBSD 14.2-RELEASE<br> .git/config see below<br> ```<br> <br> ```<br> # .git/config<br> [core]<br> repositoryformatversion = 0<br> filemode = true<br> bare = false<br> logallrefupdates = true<br> <br> [remote "origin"]<br> url = <a href="https://git.freebsd.org/src.git" rel="noreferrer" target="_blank">https://git.freebsd.org/src.git</a><br>; pushurl = git@gitrepo.freebsd.org:src.git<br> fetch = +refs/heads/releng/14.2:refs/remotes/origin/releng/14.2<br> <br> [branch "releng/14.2"]<br> remote = origin<br> merge = refs/heads/releng/14.2<br> ```<br> <br> this yields ~ 350MiB raw, as txz its still 325MiB.<br> <br> ```<br> $ git fetch --depth 1 --no-tags --jobs 32 --set-upstream origin releng/14.2<br> remote: Enumerating objects: 103451, done.<br> remote: Counting objects: 100% (103451/103451), done.<br> remote: Compressing objects: 100% (89088/89088), done.<br> remote: Total 103451 (delta 21659), reused 41475 (delta 10515), pack-reused 0 (from 0)<br> Receiving objects: 100% (103451/103451), 344.67 MiB | 9.13 MiB/s, done.<br> Resolving deltas: 100% (21659/21659), done.<br> >From <a href="https://git.freebsd.org/src" rel="noreferrer" target="_blank">https://git.freebsd.org/src</a><br>; * branch releng/14.2 -> FETCH_HEAD<br> <br> $ du -chs .git<br> 348M .git<br> 348M total<br> ```<br> <br> this isn't quite right, git switch / git reset --hard or similar might be needed,<br> but its pretty close.<br></blockquote><div><br></div><div>I think what you want is</div><div><br></div><div>% git clone <a href="https://github.com/freebsd/freebsd-src">https://github.com/freebsd/freebsd-src</a>; --depth=1 -b releng/14.2 --single-branch fred --bare</div><div>% cd fred</div><div>% mkdir .git</div><div>% mv * .git</div><div>% git config --local core.bare false</div><div><br></div><div>This will clone 1 deep into the directory 'fred' and won't checkout a copy. If</div><div>we make this a package, it's 325MB, as you say. That can then be extracted</div><div>and then you can recover the tree with:</div><div><br></div><div>% git checkout releng/14.2</div><div><br></div><div>which could be done with the pkg file when extracting to /usr/src. I've not measured</div><div>ports.</div><div><br></div><div>This would give the user the sources to the release in a package that's fairly</div><div>minimal. And it would let them fetch other branches more cheaply than</div><div>having no git repo at all if they are internet connected. It's a good balance</div><div>between giving everything and taking up too much space. And it's only a little bit</div><div>larger than the src pkg today.</div><div><br></div><div>Warner</div><div><br></div></div></div>help
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CANCZdfruU=GR5DFWL7RQA4_yjh2-NHOB669wsOSL1DzZ3Fs7%2BQ>