Date: Mon, 17 Jun 2024 18:45:12 +0800 From: Thor Ablestar <thor@irk.ru> To: questions@freebsd.org Subject: Re: /usr/ports/distfiles/go insanely oversized Message-ID: <0ae81739-4d16-3c45-a7c9-71f849a369e6@irk.ru> In-Reply-To: <CAN6yY1ucqh9aszxHa2GgcqnFVxf22=Sdfh=P7soq0625CZj%2BYA@mail.gmail.com> References: <22e86a26-e2d5-47b3-8a28-26fde5c4d730@irk.ru> <CAN6yY1ucqh9aszxHa2GgcqnFVxf22=Sdfh=P7soq0625CZj%2BYA@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
[-- Attachment #1 --] Hello! On 6/17/24 10:26, Kevin Oberman wrote: > I'm baffled. My distfiles with over 1200 ports installed total 9.1GB. > This makes me wonder just why you are doing things like 'make > checksum'? 'make install' will do the following: > config, fetch, checksum, depends, extract, patch, configure, build, > install. "make fetch" and "make checksum" require supervision due to peculiarities of local Internet that I am afraid to discuss. After this, "make install" can be run unattended. > > That normally results in a single file in distfiles. Some ports > require multiple dist files, usually for a port which maintains > multiple versions or a metaport that contains distfiles for many ports > included in the metaport. E.g. gnome. Still, this does not mean that > there will be anything like what you are seeing. It is exactly so for everything but NOT for /usr/ports/distfiles/go I see that "make checksum" fetches and checksums not only the files listed in, say, /usr/ports/archivers/nfpm/distinfo but some (lots of) files placed in /usr/ports/distfiles/go/archivers_nfpm/pkg/mod/cache - and I cannot understand why. > > Can you explain exactly how you are doing this? For instance (I use archivers/nfpm as a testbed for this problem - it's the first one in alphabetical order): # cd /usr/ports/archivers/nfpm # make fetch # make checksum # make install > And, why? I am afraid of discussing the "why". Hint: HTTP error 451 > Why not just use packages? I do it often but not always. Sometimes I need ports. > Either pull down the packages you need or use poudriere or 'make > package' to generate what you need? Either way will be far less > resource intensive. Especially pulling down the pre-built packages. > > Clearly, something you're doing is not what you want. Yours, Anatoly > > On Sun, Jun 16, 2024 at 2:16 AM Thor Ablestar <thor@irk.ru> wrote: > > Hello! > > I have a habit to keep /usr/ports/distfiles on separate SSD in > order to > build packages on many different computers and OSes (12, 13 and now > 14.2). Usually i do the following: > > # cd /usr/ports/sometheme/somepackage > > # make checksum > > # make install > > so that I supervise the download of source files and then have > some rest > while the port compiles. > > SSD became full, and I decided to copy data to the different HDD. > Then I > have found that the /usr/ports/distfiles/go has an insane number of > subdirectories (some hundreds of thousands). I spent a full day in > attempts to copy them (successfully at last) and only some minutes to > copy the rest. > > Then I decided to find a total size of the distfiles with du. Now du > spent 5 minutes of machine time (top listing) and has shown 288 GB > in go > subdir and 469 GB total. > > Then I have tried to compare some subdirectories in go directory, > and it > seems that each subdirectory contains subdirectories of all their > dependencies (I am not sure). For instance, > > go/archivers_nfpm/pkg/mod/cache/download/github.com/!masterminds/goutils/@v > <http://github.com/!masterminds/goutils/@v>; > > contains the same files (name and size) as > > go/audio_gonic/pkg/mod/cache/download/github.com/!masterminds/goutils/@v > <http://github.com/!masterminds/goutils/@v>; > > What to do? How to manage it? If the problem cannot be solved I am > going > just to ban the /usr/ports/distfiles/go dependent ports completely > and > try to live without them. > > I had the same problem 2 years ago but it was not so severe as now. > > Sincerely yours, > > Anatoly > > > > > > -- > Kevin Oberman, Part time kid herder and retired Network Engineer > E-mail: rkoberman@gmail.com > PGP Fingerprint: D03FB98AFA78E3B78C1694B318AB39EF1B055683 [-- Attachment #2 --] <html> <head> <meta http-equiv="Content-Type" content="text/html; charset=UTF-8"> </head> <body> <p>Hello!<br> </p> <div class="moz-cite-prefix">On 6/17/24 10:26, Kevin Oberman wrote:<br> </div> <blockquote type="cite" cite="mid:CAN6yY1ucqh9aszxHa2GgcqnFVxf22=Sdfh=P7soq0625CZj+YA@mail.gmail.com"> <meta http-equiv="content-type" content="text/html; charset=UTF-8"> <div dir="ltr"> <div class="gmail_default" style="font-family:tahoma,sans-serif;font-size:small">I'm baffled. My distfiles with over 1200 ports installed total 9.1GB. This makes me wonder just why you are doing things like 'make checksum'? 'make install' will do the following:</div> <div class="gmail_default" style="font-family:tahoma,sans-serif;font-size:small">config, fetch, checksum, depends, extract, patch, configure, build, install.<br> </div> </div> </blockquote> "make fetch" and "make checksum" require supervision due to peculiarities of local Internet that I am afraid to discuss. After this, "make install" can be run unattended. <br> <blockquote type="cite" cite="mid:CAN6yY1ucqh9aszxHa2GgcqnFVxf22=Sdfh=P7soq0625CZj+YA@mail.gmail.com"> <div dir="ltr"> <div class="gmail_default" style="font-family:tahoma,sans-serif;font-size:small"><br> </div> <div class="gmail_default" style="font-family:tahoma,sans-serif;font-size:small">That normally results in a single file in distfiles. Some ports require multiple dist files, usually for a port which maintains multiple versions or a metaport that contains distfiles for many ports included in the metaport. E.g. gnome. Still, this does not mean that there will be anything like what you are seeing.</div> </div> </blockquote> <p>It is exactly so for everything but NOT for /usr/ports/distfiles/go <br> </p> <p>I see that "make checksum" fetches and checksums not only the files listed in, say, /usr/ports/archivers/nfpm/distinfo but some (lots of) files placed in /usr/ports/distfiles/go/archivers_nfpm/pkg/mod/cache - and I cannot understand why.<br> </p> <blockquote type="cite" cite="mid:CAN6yY1ucqh9aszxHa2GgcqnFVxf22=Sdfh=P7soq0625CZj+YA@mail.gmail.com"> <div dir="ltr"> <div class="gmail_default" style="font-family:tahoma,sans-serif;font-size:small"><br> </div> <div class="gmail_default" style="font-family:tahoma,sans-serif;font-size:small">Can you explain exactly how you are doing this?</div> </div> </blockquote> <p>For instance (I use archivers/nfpm as a testbed for this problem - it's the first one in alphabetical order):</p> <p># cd /usr/ports/archivers/nfpm</p> <p># make fetch</p> <p># make checksum</p> <p># make install<br> </p> <blockquote type="cite" cite="mid:CAN6yY1ucqh9aszxHa2GgcqnFVxf22=Sdfh=P7soq0625CZj+YA@mail.gmail.com"> <div dir="ltr"> <div class="gmail_default" style="font-family:tahoma,sans-serif;font-size:small"> And, why? </div> </div> </blockquote> I am afraid of discussing the "why". Hint: HTTP error 451<br> <blockquote type="cite" cite="mid:CAN6yY1ucqh9aszxHa2GgcqnFVxf22=Sdfh=P7soq0625CZj+YA@mail.gmail.com"> <div dir="ltr"> <div class="gmail_default" style="font-family:tahoma,sans-serif;font-size:small">Why not just use packages? </div> </div> </blockquote> I do it often but not always. Sometimes I need ports.<br> <blockquote type="cite" cite="mid:CAN6yY1ucqh9aszxHa2GgcqnFVxf22=Sdfh=P7soq0625CZj+YA@mail.gmail.com"> <div dir="ltr"> <div class="gmail_default" style="font-family:tahoma,sans-serif;font-size:small">Either pull down the packages you need or use poudriere or 'make package' to generate what you need? Either way will be far less resource intensive. Especially pulling down the pre-built packages.</div> <div class="gmail_default" style="font-family:tahoma,sans-serif;font-size:small"><br> </div> <div class="gmail_default" style="font-family:tahoma,sans-serif;font-size:small">Clearly, something you're doing is not what you want.<br> </div> </div> </blockquote> Yours, Anatoly<br> <blockquote type="cite" cite="mid:CAN6yY1ucqh9aszxHa2GgcqnFVxf22=Sdfh=P7soq0625CZj+YA@mail.gmail.com"><br> <div class="gmail_quote"> <div dir="ltr" class="gmail_attr">On Sun, Jun 16, 2024 at 2:16 AM Thor Ablestar <<a href="mailto:thor@irk.ru" moz-do-not-send="true" class="moz-txt-link-freetext">thor@irk.ru</a>> wrote:<br> </div> <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Hello!<br> <br> I have a habit to keep /usr/ports/distfiles on separate SSD in order to <br> build packages on many different computers and OSes (12, 13 and now <br> 14.2). Usually i do the following:<br> <br> # cd /usr/ports/sometheme/somepackage<br> <br> # make checksum<br> <br> # make install<br> <br> so that I supervise the download of source files and then have some rest <br> while the port compiles.<br> <br> SSD became full, and I decided to copy data to the different HDD. Then I <br> have found that the /usr/ports/distfiles/go has an insane number of <br> subdirectories (some hundreds of thousands). I spent a full day in <br> attempts to copy them (successfully at last) and only some minutes to <br> copy the rest.<br> <br> Then I decided to find a total size of the distfiles with du. Now du <br> spent 5 minutes of machine time (top listing) and has shown 288 GB in go <br> subdir and 469 GB total.<br> <br> Then I have tried to compare some subdirectories in go directory, and it <br> seems that each subdirectory contains subdirectories of all their <br> dependencies (I am not sure). For instance,<br> <br> go/archivers_nfpm/pkg/mod/cache/download/<a href="http://github.com/!masterminds/goutils/@v" rel="noreferrer" target="_blank" moz-do-not-send="true">github.com/!masterminds/goutils/@v</a><br> <br> contains the same files (name and size) as<br> <br> go/audio_gonic/pkg/mod/cache/download/<a href="http://github.com/!masterminds/goutils/@v" rel="noreferrer" target="_blank" moz-do-not-send="true">github.com/!masterminds/goutils/@v</a><br> <br> What to do? How to manage it? If the problem cannot be solved I am going <br> just to ban the /usr/ports/distfiles/go dependent ports completely and <br> try to live without them.<br> <br> I had the same problem 2 years ago but it was not so severe as now.<br> <br> Sincerely yours,<br> <br> Anatoly<br> <br> <br> <br> </blockquote> </div> <br clear="all"> <br> <span class="gmail_signature_prefix">-- </span><br> <div dir="ltr" class="gmail_signature"> <div dir="ltr"> <div> <div dir="ltr"> <div> <div dir="ltr"> <div> <div dir="ltr">Kevin Oberman, Part time kid herder and retired Network Engineer<br> E-mail: <a href="mailto:rkoberman@gmail.com" target="_blank" moz-do-not-send="true" class="moz-txt-link-freetext">rkoberman@gmail.com</a><br> </div> <div>PGP Fingerprint: D03FB98AFA78E3B78C1694B318AB39EF1B055683</div> </div> </div> </div> </div> </div> </div> </div> </blockquote> </body> </html>
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?0ae81739-4d16-3c45-a7c9-71f849a369e6>