Date: Wed, 14 Jan 2026 08:11:22 +0100 (CET) From: Ronald Klop <ronald-lists@klop.ws> To: ruby R53 <red_M95@proton.me> Cc: freebsd-current@freebsd.org Subject: Re: Change fetch addresses for Rust crates, Python packages and whatnot? Message-ID: <1503278086.9.1768374682910@localhost> In-Reply-To: <DRM-6bM0Q-ebeu_NeL6Ju4ZXLScQGlf_3hNxqhtw6e2kA8gwAV36SGgPD08G2H0H-xeChQPYAs6HRGeey6DaoD77gxqCCd_YFR81YgIfETk=@proton.me>
index | next in thread | previous in thread | raw e-mail
[-- Attachment #1 --] Van: ruby R53 <red_M95@proton.me> Datum: woensdag, 14 januari 2026 03:26 Aan: freebsd-current@freebsd.org Onderwerp: Change fetch addresses for Rust crates, Python packages and whatnot? > > I've been trying to compile www/chromium for literal days now, all because whenever `make` runs `fetch`, it goes to an address that's actually supposed to redirect to another one, but `fetch` instead keeps trying for a whole minute until it ultimately falls back to distcache.freebsd.org, which is another whole minute of waiting for the file to download. The cherry on top is that the file is, most of the time, just hundreds of kilobytes in size. > > Hell, I even had to manually download part of these individual files using that Links web browser··· it'd do it a lot faster than `fetch`, all because it actually followed the redirection and didn't just keep trying over and over. But of course I wouldn't manually download hundreds upon hundreds of archive files, this is the reason why such tool exists. > > So here's an example: currently one of the missing dependencies for www/chromium is GTK3, which depends on graphics/librsvg2-rust, which then depends on cargo-c. It's now trying to fetch dunce-1.0.5.crate, which is one of the crates required for it, from https://crates.io/api/v1/crates/dunce/1.0.5/download?dummy=/rust/crates/dunce-1.0.5.crate. Running `curl` on this link does literally nothing, no files are downloaded. But running it on any web browser instantly downloads the crate in question, all because that link actually redirects to https://static.crates.io/crates/dunce/dunce-1.0.5.crate. Even `curl`ing that new link does give the file I want. Same applies for Python packages. > > Now imagine this for all the remaining files··· if it takes about 2 minutes to download a single file, and there's 398 crate entries on cargo-c's `Makefile.crates`, that means it'd take a whole 796 minutes to download everything, which is literally more than half a day (~13.6 hours) :) > > > > Can you copy-paste the output of the following command in a reply? fetch -vv "https://crates.io/api/v1/crates/dunce/1.0.5/download?dummy=/rust/crates/dunce-1.0.5.crate" For me it immediately follows the redirect and downloads the file in half a second. See the output below. Regards, Ronald. -- time fetch -vv "https://crates.io/api/v1/crates/dunce/1.0.5/download?dummy=/rust/crates/dunce-1.0.5.crate" scheme: "https" user: "" password: "" host: "crates.io" port: "0" document: "/api/v1/crates/dunce/1.0.5/download?dummy=/rust/crates/dunce-1.0.5.crate" ---> crates.io:443 resolving server address: crates.io:443 SSL options: 82004850 Peer verification enabled Using OpenSSL default CA cert file and path Verify hostname TLSv1.3 connection established using TLS_AES_128_GCM_SHA256 Certificate subject: /CN=crates.io Certificate issuer: /C=US/O=Amazon/CN=Amazon RSA 2048 M01 requesting https://crates.io/api/v1/crates/dunce/1.0.5/download?dummy=/rust/crates/dunce-1.0.5.crate >>> GET /api/v1/crates/dunce/1.0.5/download?dummy=/rust/crates/dunce-1.0.5.crate HTTP/1.1 >>> Host: crates.io >>> Accept: */* >>> User-Agent: fetch libfetch/2.0 >>> Connection: close >>> <<< HTTP/1.1 302 Moved Temporarily <<< Content-Length: 0 <<< Connection: close content length: [0] <<< Access-Control-Allow-Origin: * <<< Date: Wed, 14 Jan 2026 07:10:06 GMT <<< Location: https://static.crates.io/crates/dunce/dunce-1.0.5.crate <<< Nel: {"report_to":"heroku-nel","response_headers":["Via"],"max_age":3600,"success_fraction":0.01,"failure_fraction":0.1} 302 redirect to https://static.crates.io/crates/dunce/dunce-1.0.5.crate scheme: "https" user: "" password: "" host: "static.crates.io" port: "0" document: "/crates/dunce/dunce-1.0.5.crate" <<< Report-To: {"group":"heroku-nel","endpoints":[{"url":"https://nel.heroku.com/reports?s=VOwdWzeBall4rSiByZHUwCe6zIbVFdxy%2FIgcGEUOIZo%3D\u0026sid=af571f24-03ee-46d1-9f90-ab9030c2c74c\u0026ts=1768374606"}],"max_age":3600} <<< Reporting-Endpoints: heroku-nel="https://nel.heroku.com/reports?s=VOwdWzeBall4rSiByZHUwCe6zIbVFdxy%2FIgcGEUOIZo%3D&sid=af571f24-03ee-46d1-9f90-ab9030c2c74c&ts=1768374606" <<< Server: Heroku <<< Strict-Transport-Security: max-age=31536000; includeSubDomains <<< Via: 1.1 heroku-router, 1.1 c2905f891f96a0ec9c7fab16916dbb46.cloudfront.net (CloudFront) <<< X-Cache: Miss from cloudfront <<< X-Amz-Cf-Pop: AMS58-P6 <<< X-Amz-Cf-Id: Xa10pwbYp4MCA5eToVa0shahNpzr2epWFUuQZS63bMpmbiDLKc2dmQ== <<< ---> static.crates.io:443 resolving server address: static.crates.io:443 SSL options: 82004850 Peer verification enabled Using OpenSSL default CA cert file and path Verify hostname TLSv1.3 connection established using TLS_AES_128_GCM_SHA256 Certificate subject: /CN=static.crates.io Certificate issuer: /C=BE/O=GlobalSign nv-sa/CN=GlobalSign Atlas R3 DV TLS CA 2025 Q3 requesting https://static.crates.io/crates/dunce/dunce-1.0.5.crate >>> GET /crates/dunce/dunce-1.0.5.crate HTTP/1.1 >>> Host: static.crates.io >>> Accept: */* >>> User-Agent: fetch libfetch/2.0 >>> Connection: close >>> <<< HTTP/1.1 200 <<< Connection: close <<< Content-Length: 8244 <<< etag: "198d7f7c37f632bc83502b28870b6d60" content length: [8244] <<< cache-control: public,max-age=31536000,immutable <<< x-amz-version-id: PO7xZdimM08F9npIqG8PYtpqpEuWdV9D <<< x-amz-id-2: Dj7mFoaqQJUc61mB1FcouUQcy0tajnacJvU+xOR3Be5//jo3GaSmdCwSSgtYyDMnYE4cPCiPv7PV9m7z1Jk0WazYHTFPbYhm <<< x-amz-request-id: KWXPQ4JP20HSYSY2 <<< date: Wed, 14 Jan 2026 00:09:56 GMT <<< x-amz-server-side-encryption: AES256 <<< x-amz-replication-status: COMPLETED <<< last-modified: Sun, 04 Aug 2024 01:36:47 GMT <<< accept-ranges: bytes last modified: [2024-08-04 01:36:47] <<< content-type: application/gzip <<< x-served-by: cache-ams21058-AMS <<< server: AmazonS3 <<< age: 25212 <<< x-cache: HIT <<< x-cache-hits: 148 <<< offset 0, length -1, size -1, clength 8244 local size / mtime: 8244 / 1722735407 remote size / mtime: 8244 / 1722735407 dunce-1.0.5.crate 8244 B 10 MBps 00s real 0m0.429s user 0m0.023s sys 0m0.027s [-- Attachment #2 --] <html><head></head><body><br> <p><strong>Van:</strong> ruby R53 <red_M95@proton.me><br> <strong>Datum:</strong> woensdag, 14 januari 2026 03:26<br> <strong>Aan:</strong> freebsd-current@freebsd.org<br> <strong>Onderwerp:</strong> Change fetch addresses for Rust crates, Python packages and whatnot?</p> <blockquote style="padding-right: 0px; padding-left: 5px; margin-left: 5px; border-left: #000000 2px solid; margin-right: 0px"> <div class="MessageRFC822Viewer" id="P"> <div class="TextPlainViewer" id="P.P">I've been trying to compile www/chromium for literal days now, all because whenever `make` runs `fetch`, it goes to an address that's actually supposed to redirect to another one, but `fetch` instead keeps trying for a whole minute until it ultimately falls back to distcache.freebsd.org, which is another whole minute of waiting for the file to download. The cherry on top is that the file is, most of the time, just hundreds of kilobytes in size.<br> <br> Hell, I even had to manually download part of these individual files using that Links web browser··· it'd do it a lot faster than `fetch`, all because it actually followed the redirection and didn't just keep trying over and over. But of course I wouldn't manually download hundreds upon hundreds of archive files, this is the reason why such tool exists.<br> <br> So here's an example: currently one of the missing dependencies for www/chromium is GTK3, which depends on graphics/librsvg2-rust, which then depends on cargo-c. It's now trying to fetch dunce-1.0.5.crate, which is one of the crates required for it, from <a href="https://crates.io/api/v1/crates/dunce/1.0.5/download?dummy=/rust/crates/dunce-1.0.5.crate">https://crates.io/api/v1/crates/dunce/1.0.5/download?dummy=/rust/crates/dunce-1.0.5.crate</a>. Running `curl` on this link does literally nothing, no files are downloaded. But running it on any web browser instantly downloads the crate in question, all because that link actually redirects to <a href="https://static.crates.io/crates/dunce/dunce-1.0.5.crate">https://static.crates.io/crates/dunce/dunce-1.0.5.crate</a>. Even `curl`ing that new link does give the file I want. Same applies for Python packages.<br> <br> Now imagine this for all the remaining files··· if it takes about 2 minutes to download a single file, and there's 398 crate entries on cargo-c's `Makefile.crates`, that means it'd take a whole 796 minutes to download everything, which is literally more than half a day (~13.6 hours) :)<br> </div> <hr></div> </blockquote> <br> <br> Can you copy-paste the output of the following command in a reply?<br> fetch -vv "https://crates.io/api/v1/crates/dunce/1.0.5/download?dummy=/rust/crates/dunce-1.0.5.crate"<br>; <br> For me it immediately follows the redirect and downloads the file in half a second. See the output below.<br> <br> Regards,<br> Ronald.<br> <br> <br> -- <pre>time fetch -vv "https://crates.io/api/v1/crates/dunce/1.0.5/download?dummy=/rust/crates/dunce-1.0.5.crate" scheme: "https" user: "" password: "" host: "crates.io" port: "0" document: "/api/v1/crates/dunce/1.0.5/download?dummy=/rust/crates/dunce-1.0.5.crate" ---> crates.io:443 resolving server address: crates.io:443 SSL options: 82004850 Peer verification enabled Using OpenSSL default CA cert file and path Verify hostname TLSv1.3 connection established using TLS_AES_128_GCM_SHA256 Certificate subject: /CN=crates.io Certificate issuer: /C=US/O=Amazon/CN=Amazon RSA 2048 M01 requesting https://crates.io/api/v1/crates/dunce/1.0.5/download?dummy=/rust/crates/dunce-1.0.5.crate >>> GET /api/v1/crates/dunce/1.0.5/download?dummy=/rust/crates/dunce-1.0.5.crate HTTP/1.1 >>> Host: crates.io >>> Accept: */* >>> User-Agent: fetch libfetch/2.0 >>> Connection: close >>> <<< HTTP/1.1 302 Moved Temporarily <<< Content-Length: 0 <<< Connection: close content length: [0] <<< Access-Control-Allow-Origin: * <<< Date: Wed, 14 Jan 2026 07:10:06 GMT <<< Location: https://static.crates.io/crates/dunce/dunce-1.0.5.crate <<< Nel: {"report_to":"heroku-nel","response_headers":["Via"],"max_age":3600,"success_fraction":0.01,"failure_fraction":0.1} 302 redirect to https://static.crates.io/crates/dunce/dunce-1.0.5.crate scheme: "https" user: "" password: "" host: "static.crates.io" port: "0" document: "/crates/dunce/dunce-1.0.5.crate" <<< Report-To: {"group":"heroku-nel","endpoints":[{"url":"https://nel.heroku.com/reports?s=VOwdWzeBall4rSiByZHUwCe6zIbVFdxy%2FIgcGEUOIZo%3D\u0026sid=af571f24-03ee-46d1-9f90-ab9030c2c74c\u0026ts=1768374606"}],"max_age":3600} <<< Reporting-Endpoints: heroku-nel="https://nel.heroku.com/reports?s=VOwdWzeBall4rSiByZHUwCe6zIbVFdxy%2FIgcGEUOIZo%3D&sid=af571f24-03ee-46d1-9f90-ab9030c2c74c&ts=1768374606" <<< Server: Heroku <<< Strict-Transport-Security: max-age=31536000; includeSubDomains <<< Via: 1.1 heroku-router, 1.1 c2905f891f96a0ec9c7fab16916dbb46.cloudfront.net (CloudFront) <<< X-Cache: Miss from cloudfront <<< X-Amz-Cf-Pop: AMS58-P6 <<< X-Amz-Cf-Id: Xa10pwbYp4MCA5eToVa0shahNpzr2epWFUuQZS63bMpmbiDLKc2dmQ== <<< ---> static.crates.io:443 resolving server address: static.crates.io:443 SSL options: 82004850 Peer verification enabled Using OpenSSL default CA cert file and path Verify hostname TLSv1.3 connection established using TLS_AES_128_GCM_SHA256 Certificate subject: /CN=static.crates.io Certificate issuer: /C=BE/O=GlobalSign nv-sa/CN=GlobalSign Atlas R3 DV TLS CA 2025 Q3 requesting https://static.crates.io/crates/dunce/dunce-1.0.5.crate >>> GET /crates/dunce/dunce-1.0.5.crate HTTP/1.1 >>> Host: static.crates.io >>> Accept: */* >>> User-Agent: fetch libfetch/2.0 >>> Connection: close >>> <<< HTTP/1.1 200 <<< Connection: close <<< Content-Length: 8244 <<< etag: "198d7f7c37f632bc83502b28870b6d60" content length: [8244] <<< cache-control: public,max-age=31536000,immutable <<< x-amz-version-id: PO7xZdimM08F9npIqG8PYtpqpEuWdV9D <<< x-amz-id-2: Dj7mFoaqQJUc61mB1FcouUQcy0tajnacJvU+xOR3Be5//jo3GaSmdCwSSgtYyDMnYE4cPCiPv7PV9m7z1Jk0WazYHTFPbYhm <<< x-amz-request-id: KWXPQ4JP20HSYSY2 <<< date: Wed, 14 Jan 2026 00:09:56 GMT <<< x-amz-server-side-encryption: AES256 <<< x-amz-replication-status: COMPLETED <<< last-modified: Sun, 04 Aug 2024 01:36:47 GMT <<< accept-ranges: bytes last modified: [2024-08-04 01:36:47] <<< content-type: application/gzip <<< x-served-by: cache-ams21058-AMS <<< server: AmazonS3 <<< age: 25212 <<< x-cache: HIT <<< x-cache-hits: 148 <<< offset 0, length -1, size -1, clength 8244 local size / mtime: 8244 / 1722735407 remote size / mtime: 8244 / 1722735407 dunce-1.0.5.crate 8244 B 10 MBps 00s real 0m0.429s user 0m0.023s sys 0m0.027s</pre></body></html>home | help
Want to link to this message? Use this
URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?1503278086.9.1768374682910>