Date: Wed, 22 Jan 2025 09:30:09 -0700 From: Warner Losh <imp@bsdimp.com> To: Dave Cottlehuber <dch@skunkwerks.at> Cc: bob prohaska <fbsd@www.zefox.net>, Sulev-Madis Silber <freebsd-current-freebsd-org111@ketas.si.pri.ee>, freebsd-current <freebsd-current@freebsd.org> Subject: Re: /usr/src and /usr/ports not git directories ? Message-ID: <CANCZdfruU=GR5DFWL7RQA4_yjh2-NHOB669wsOSL1DzZ3Fs7%2BQ@mail.gmail.com> In-Reply-To: <3782b90b-850f-4435-8b6d-bdb83753daea@app.fastmail.com> References: <Z4vk3009iSwuzG4K@www.zefox.net> <Z4__B0EQM-ce0qPE@cell.glebi.us> <C509F94C-2AC2-414F-90C0-355C69869D72@ketas.si.pri.ee> <Z5AQ1GcwX_MZw69G@www.zefox.net> <CANCZdfoHUsZusqMg_gWN5mB9P3xByGv_GfELi9Dd63CHto1igw@mail.gmail.com> <3782b90b-850f-4435-8b6d-bdb83753daea@app.fastmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
--0000000000006c7a30062c4e020d Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On Wed, Jan 22, 2025 at 1:23=E2=80=AFAM Dave Cottlehuber <dch@skunkwerks.at= > wrote: > On Tue, 21 Jan 2025, at 21:29, Warner Losh wrote: > >> Do you mean have the "install src" checkbox invoke git clone? > >> That seems like a better idea, at least to me. > > This wouldn't be ideal for small systems or any area with lousy > internet. > > > I think we should replace the populate /usr/src from a tarball with.... > > populate it > > with a tarball that represents a 1-deep checkout tree at the rev we > > built the release > > from. This lets users have the source, has minimal overhead and also > > lets users update > > or turn the shallow checkout into a deep one, etc. A shallow checkout > > is quite a bit > > less than a full tree, though still more than just the raw files. I've > > not done poking to > > see size comparisons. > > Can we ship a working gitup config? I've not used this yet, but this > would be a good option. > > Another option is to ship src.txz as we do today, but include a minimal > git config, such that `git fetch origin releng/14.2` would work. I like > this because we're not really reliant on a specific version of git > packfiles, and it's still just src as usual. > > Another option, IIRC Kyle mentioned this last year, is to use `git bundle= ` > https://git-scm.com/book/en/v2/Git-Tools-Bundling but I don't really know > enough about that. Perhaps that would be a reasonable option to fetch ove= r > network, in the installer? > > For the former, this seems to be sufficient: > > ``` > # /usr/src/ > .git/ > .git/branches/ > .git/objects/ > .git/refs/ > .git/HEAD ref: refs/heads/releng/14.2 > .git/description FreeBSD 14.2-RELEASE > .git/config see below > ``` > > ``` > # .git/config > [core] > repositoryformatversion =3D 0 > filemode =3D true > bare =3D false > logallrefupdates =3D true > > [remote "origin"] > url =3D https://git.freebsd.org/src.git > pushurl =3D git@gitrepo.freebsd.org:src.git > fetch =3D +refs/heads/releng/14.2:refs/remotes/origin/releng/14.2 > > [branch "releng/14.2"] > remote =3D origin > merge =3D refs/heads/releng/14.2 > ``` > > this yields ~ 350MiB raw, as txz its still 325MiB. > > ``` > $ git fetch --depth 1 --no-tags --jobs 32 --set-upstream origin > releng/14.2 > remote: Enumerating objects: 103451, done. > remote: Counting objects: 100% (103451/103451), done. > remote: Compressing objects: 100% (89088/89088), done. > remote: Total 103451 (delta 21659), reused 41475 (delta 10515), > pack-reused 0 (from 0) > Receiving objects: 100% (103451/103451), 344.67 MiB | 9.13 MiB/s, done. > Resolving deltas: 100% (21659/21659), done. > From https://git.freebsd.org/src > * branch releng/14.2 -> FETCH_HEAD > > $ du -chs .git > 348M .git > 348M total > ``` > > this isn't quite right, git switch / git reset --hard or similar might be > needed, > but its pretty close. > I think what you want is % git clone https://github.com/freebsd/freebsd-src --depth=3D1 -b releng/14= .2 --single-branch fred --bare % cd fred % mkdir .git % mv * .git % git config --local core.bare false This will clone 1 deep into the directory 'fred' and won't checkout a copy. If we make this a package, it's 325MB, as you say. That can then be extracted and then you can recover the tree with: % git checkout releng/14.2 which could be done with the pkg file when extracting to /usr/src. I've not measured ports. This would give the user the sources to the release in a package that's fairly minimal. And it would let them fetch other branches more cheaply than having no git repo at all if they are internet connected. It's a good balance between giving everything and taking up too much space. And it's only a little bit larger than the src pkg today. Warner --0000000000006c7a30062c4e020d Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable <div dir=3D"ltr"><div dir=3D"ltr"><br></div><br><div class=3D"gmail_quote g= mail_quote_container"><div dir=3D"ltr" class=3D"gmail_attr">On Wed, Jan 22,= 2025 at 1:23=E2=80=AFAM Dave Cottlehuber <<a href=3D"mailto:dch@skunkwe= rks.at">dch@skunkwerks.at</a>> wrote:<br></div><blockquote class=3D"gmai= l_quote" style=3D"margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,20= 4,204);padding-left:1ex">On Tue, 21 Jan 2025, at 21:29, Warner Losh wrote:<= br> >> Do you mean have the "install src" checkbox invoke git c= lone?<br> >> That seems like a better idea, at least to me. <br> <br> This wouldn't be ideal for small systems or any area with lousy<br> internet.<br> <br> > I think we should replace the populate /usr/src from a tarball with...= . <br> > populate it<br> > with a tarball that represents a 1-deep checkout tree at the rev we <b= r> > built the release<br> > from. This lets users have the source, has minimal overhead and also <= br> > lets users update<br> > or turn the shallow checkout into a deep one, etc. A shallow checkout = <br> > is quite a bit<br> > less than a full tree, though still more than just the raw files. I= 9;ve <br> > not done poking to<br> > see size comparisons.<br> <br> Can we ship a working gitup config? I've not used this yet, but this<br= > would be a good option.<br> <br> Another option is to ship src.txz as we do today, but include a minimal<br> git config, such that `git fetch origin releng/14.2` would work. I like<br> this because we're not really reliant on a specific version of git <br> packfiles, and it's still just src as usual.<br> <br> Another option, IIRC Kyle mentioned this last year, is to use `git bundle`<= br> <a href=3D"https://git-scm.com/book/en/v2/Git-Tools-Bundling" rel=3D"norefe= rrer" target=3D"_blank">https://git-scm.com/book/en/v2/Git-Tools-Bundling</= a> but I don't really know<br> enough about that. Perhaps that would be a reasonable option to fetch over<= br> network, in the installer?<br> <br> For the former, this seems to be sufficient:<br> <br> ```<br> # /usr/src/<br> .git/<br> .git/branches/<br> .git/objects/<br> .git/refs/<br> .git/HEAD=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0ref: refs/h= eads/releng/14.2<br> .git/description=C2=A0 =C2=A0 =C2=A0 =C2=A0 FreeBSD 14.2-RELEASE<br> .git/config=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0see below<br> ```<br> <br> ```<br> # .git/config<br> [core]<br> =C2=A0 =C2=A0 =C2=A0 =C2=A0 repositoryformatversion =3D 0<br> =C2=A0 =C2=A0 =C2=A0 =C2=A0 filemode =3D true<br> =C2=A0 =C2=A0 =C2=A0 =C2=A0 bare =3D false<br> =C2=A0 =C2=A0 =C2=A0 =C2=A0 logallrefupdates =3D true<br> <br> [remote "origin"]<br> =C2=A0 =C2=A0 =C2=A0 =C2=A0 url =3D <a href=3D"https://git.freebsd.org/src.= git" rel=3D"noreferrer" target=3D"_blank">https://git.freebsd.org/src.git</= a><br> =C2=A0 =C2=A0 =C2=A0 =C2=A0 pushurl =3D git@gitrepo.freebsd.org:src.git<br> =C2=A0 =C2=A0 =C2=A0 =C2=A0 fetch =3D +refs/heads/releng/14.2:refs/remotes/= origin/releng/14.2<br> <br> [branch "releng/14.2"]<br> =C2=A0 =C2=A0 =C2=A0 =C2=A0 remote =3D origin<br> =C2=A0 =C2=A0 =C2=A0 =C2=A0 merge =3D refs/heads/releng/14.2<br> ```<br> <br> this yields ~ 350MiB raw, as txz its still 325MiB.<br> <br> ```<br> $ git fetch --depth 1=C2=A0 --no-tags --jobs 32 --set-upstream origin relen= g/14.2<br> remote: Enumerating objects: 103451, done.<br> remote: Counting objects: 100% (103451/103451), done.<br> remote: Compressing objects: 100% (89088/89088), done.<br> remote: Total 103451 (delta 21659), reused 41475 (delta 10515), pack-reused= 0 (from 0)<br> Receiving objects: 100% (103451/103451), 344.67 MiB | 9.13 MiB/s, done.<br> Resolving deltas: 100% (21659/21659), done.<br> >From <a href=3D"https://git.freebsd.org/src" rel=3D"noreferrer" target=3D"_= blank">https://git.freebsd.org/src</a><br>; =C2=A0* branch=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 relen= g/14.2 -> FETCH_HEAD<br> <br> $ du -chs .git<br> 348M=C2=A0 =C2=A0 .git<br> 348M=C2=A0 =C2=A0 total<br> ```<br> <br> this isn't quite right, git switch / git reset --hard or similar might = be needed,<br> but its pretty close.<br></blockquote><div><br></div><div>I think what you = want is</div><div><br></div><div>%=C2=A0git clone <a href=3D"https://github= .com/freebsd/freebsd-src">https://github.com/freebsd/freebsd-src</a>; --dept= h=3D1 -b releng/14.2 --single-branch fred --bare</div><div>% cd fred</div><= div>% mkdir .git</div><div>% mv * .git</div><div>% git config --local core.= bare false</div><div><br></div><div>This will clone 1 deep into the directo= ry 'fred' and won't checkout a copy. If</div><div>we make this = a package, it's 325MB, as you=C2=A0 say. That can then be extracted</di= v><div>and then you can recover the tree with:</div><div><br></div><div>% g= it checkout releng/14.2</div><div><br></div><div>which could be done with t= he pkg file when extracting to /usr/src. I've not measured</div><div>po= rts.</div><div><br></div><div>This would give the user the sources to the r= elease in a package that's fairly</div><div>minimal. And it would let t= hem fetch other branches more cheaply than</div><div>having no git repo at = all if they are internet connected. It's a good balance</div><div>betwe= en giving everything and taking up too much space. And it's only a litt= le bit</div><div>larger than the src pkg today.</div><div><br></div><div>Wa= rner</div><div><br></div></div></div> --0000000000006c7a30062c4e020d--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CANCZdfruU=GR5DFWL7RQA4_yjh2-NHOB669wsOSL1DzZ3Fs7%2BQ>