Date: Sun, 16 Jun 2024 19:26:36 -0700 From: Kevin Oberman <rkoberman@gmail.com> To: Thor Ablestar <thor@irk.ru> Cc: questions <questions@freebsd.org> Subject: Re: /usr/ports/distfiles/go insanely oversized Message-ID: <CAN6yY1ucqh9aszxHa2GgcqnFVxf22=Sdfh=P7soq0625CZj%2BYA@mail.gmail.com> In-Reply-To: <22e86a26-e2d5-47b3-8a28-26fde5c4d730@irk.ru> References: <22e86a26-e2d5-47b3-8a28-26fde5c4d730@irk.ru>
next in thread | previous in thread | raw e-mail | index | archive | help
--000000000000c300cf061b0cb2bf Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable I'm baffled. My distfiles with over 1200 ports installed total 9.1GB. This makes me wonder just why you are doing things like 'make checksum'? 'make install' will do the following: config, fetch, checksum, depends, extract, patch, configure, build, install= . That normally results in a single file in distfiles. Some ports require multiple dist files, usually for a port which maintains multiple versions or a metaport that contains distfiles for many ports included in the metaport. E.g. gnome. Still, this does not mean that there will be anything like what you are seeing. Can you explain exactly how you are doing this? And, why? Why not just use packages? Either pull down the packages you need or use poudriere or 'make package' to generate what you need? Either way will be far less resource intensive. Especially pulling down the pre-built packages. Clearly, something you're doing is not what you want. On Sun, Jun 16, 2024 at 2:16=E2=80=AFAM Thor Ablestar <thor@irk.ru> wrote: > Hello! > > I have a habit to keep /usr/ports/distfiles on separate SSD in order to > build packages on many different computers and OSes (12, 13 and now > 14.2). Usually i do the following: > > # cd /usr/ports/sometheme/somepackage > > # make checksum > > # make install > > so that I supervise the download of source files and then have some rest > while the port compiles. > > SSD became full, and I decided to copy data to the different HDD. Then I > have found that the /usr/ports/distfiles/go has an insane number of > subdirectories (some hundreds of thousands). I spent a full day in > attempts to copy them (successfully at last) and only some minutes to > copy the rest. > > Then I decided to find a total size of the distfiles with du. Now du > spent 5 minutes of machine time (top listing) and has shown 288 GB in go > subdir and 469 GB total. > > Then I have tried to compare some subdirectories in go directory, and it > seems that each subdirectory contains subdirectories of all their > dependencies (I am not sure). For instance, > > go/archivers_nfpm/pkg/mod/cache/download/ > github.com/!masterminds/goutils/@v > > contains the same files (name and size) as > > go/audio_gonic/pkg/mod/cache/download/github.com/!masterminds/goutils/@v > > What to do? How to manage it? If the problem cannot be solved I am going > just to ban the /usr/ports/distfiles/go dependent ports completely and > try to live without them. > > I had the same problem 2 years ago but it was not so severe as now. > > Sincerely yours, > > Anatoly > > > > --=20 Kevin Oberman, Part time kid herder and retired Network Engineer E-mail: rkoberman@gmail.com PGP Fingerprint: D03FB98AFA78E3B78C1694B318AB39EF1B055683 --000000000000c300cf061b0cb2bf Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable <div dir=3D"ltr"><div class=3D"gmail_default" style=3D"font-family:tahoma,s= ans-serif;font-size:small">I'm baffled. My distfiles with over 1200 por= ts installed total 9.1GB. This makes me wonder just why you are doing thing= s like 'make checksum'? 'make install' will do the followin= g:</div><div class=3D"gmail_default" style=3D"font-family:tahoma,sans-serif= ;font-size:small">config, fetch, checksum, depends, extract, patch, configu= re, build, install.<br></div><div class=3D"gmail_default" style=3D"font-fam= ily:tahoma,sans-serif;font-size:small"><br></div><div class=3D"gmail_defaul= t" style=3D"font-family:tahoma,sans-serif;font-size:small">That normally re= sults in a single file in distfiles. Some ports require multiple dist files= , usually for a port which maintains multiple versions or a metaport that c= ontains distfiles for many ports included in the metaport. E.g. gnome. Stil= l, this does not mean that there will be anything like what you are seeing.= </div><div class=3D"gmail_default" style=3D"font-family:tahoma,sans-serif;f= ont-size:small"><br></div><div class=3D"gmail_default" style=3D"font-family= :tahoma,sans-serif;font-size:small">Can you explain exactly how you are doi= ng this? And, why? Why not just use packages? Either pull down the packages= you need or use poudriere or 'make package' to generate what you n= eed? Either way will be far less resource intensive. Especially pulling dow= n the pre-built packages.</div><div class=3D"gmail_default" style=3D"font-f= amily:tahoma,sans-serif;font-size:small"><br></div><div class=3D"gmail_defa= ult" style=3D"font-family:tahoma,sans-serif;font-size:small">Clearly, somet= hing you're doing is not what you want.<br></div></div><br><div class= =3D"gmail_quote"><div dir=3D"ltr" class=3D"gmail_attr">On Sun, Jun 16, 2024= at 2:16=E2=80=AFAM Thor Ablestar <<a href=3D"mailto:thor@irk.ru">thor@i= rk.ru</a>> wrote:<br></div><blockquote class=3D"gmail_quote" style=3D"ma= rgin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:= 1ex">Hello!<br> <br> I have a habit to keep /usr/ports/distfiles on separate SSD in order to <br= > build packages on many different computers and OSes (12, 13 and now <br> 14.2). Usually i do the following:<br> <br> # cd /usr/ports/sometheme/somepackage<br> <br> # make checksum<br> <br> # make install<br> <br> so that I supervise the download of source files and then have some rest <b= r> while the port compiles.<br> <br> SSD became full, and I decided to copy data to the different HDD. Then I <b= r> have found that the /usr/ports/distfiles/go has an insane number of <br> subdirectories (some hundreds of thousands). I spent a full day in <br> attempts to copy them (successfully at last) and only some minutes to <br> copy the rest.<br> <br> Then I decided to find a total size of the distfiles with du. Now du <br> spent 5 minutes of machine time (top listing) and has shown 288 GB in go <b= r> subdir and 469 GB total.<br> <br> Then I have tried to compare some subdirectories in go directory, and it <b= r> seems that each subdirectory contains subdirectories of all their <br> dependencies (I am not sure). For instance,<br> <br> go/archivers_nfpm/pkg/mod/cache/download/<a href=3D"http://github.com/!mast= erminds/goutils/@v" rel=3D"noreferrer" target=3D"_blank">github.com/!master= minds/goutils/@v</a><br> <br> contains the same files (name and size) as<br> <br> go/audio_gonic/pkg/mod/cache/download/<a href=3D"http://github.com/!masterm= inds/goutils/@v" rel=3D"noreferrer" target=3D"_blank">github.com/!mastermin= ds/goutils/@v</a><br> <br> What to do? How to manage it? If the problem cannot be solved I am going <b= r> just to ban the /usr/ports/distfiles/go dependent ports completely and <br> try to live without them.<br> <br> I had the same problem 2 years ago but it was not so severe as now.<br> <br> Sincerely yours,<br> <br> Anatoly<br> <br> <br> <br> </blockquote></div><br clear=3D"all"><br><span class=3D"gmail_signature_pre= fix">-- </span><br><div dir=3D"ltr" class=3D"gmail_signature"><div dir=3D"l= tr"><div><div dir=3D"ltr"><div><div dir=3D"ltr"><div><div dir=3D"ltr">Kevin= Oberman, Part time kid herder and retired Network Engineer<br>E-mail: <a h= ref=3D"mailto:rkoberman@gmail.com" target=3D"_blank">rkoberman@gmail.com</a= ><br></div><div>PGP Fingerprint: D03FB98AFA78E3B78C1694B318AB39EF1B055683</= div></div></div></div></div></div></div></div> --000000000000c300cf061b0cb2bf--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAN6yY1ucqh9aszxHa2GgcqnFVxf22=Sdfh=P7soq0625CZj%2BYA>