From nobody Wed Jan 14 07:11:22 2026 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4drcjW0w6Dz6PHjB for ; Wed, 14 Jan 2026 07:11:27 +0000 (UTC) (envelope-from ronald-lists@klop.ws) Received: from smtp-relay-int.realworks.nl (smtp-relay-int.realworks.nl [31.134.205.98]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4drcjV5K8Gz3D5B for ; Wed, 14 Jan 2026 07:11:26 +0000 (UTC) (envelope-from ronald-lists@klop.ws) Authentication-Results: mx1.freebsd.org; none Received: from crmpreview4.colo2.realworks.nl (localhost [127.0.0.1]) by crmpreview4.colo2.realworks.nl (Postfix) with ESMTP id E74961C0378; Wed, 14 Jan 2026 08:11:22 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=klop.ws; s=rw2; t=1768374683; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=sJwPNjo7+YsBVoB37toejX+2WcBzM3GDHVMx4l5eZ6M=; b=cEfiu8hTeVlbB6+8ceuSd3eZk4tk0iu+QlAIglp4j3eGoKBL12adjxcm82RXPowgxXRBtq 8pjH0wdJCZRWe4W1grzgmp/HNQZQY9ocBsAhej6cRwukAucJlGnx8zpEEVI6F04FDYiiet xdmOdyR2dzkrcO1zp+C0jyRRZqnURVity7kH0ketfzThHXuqmJREu0E7fCrYvxueq5BUjn X9dhEczLzx8W4yIbKbBRMePHAgdE3Iu7G+VucxCVdi3qNleOWIvairyWngGxbZl1++K+qS 3wYwCzf5/AySt0nYqQTtRwA1DXlQsEyGZj3Qs3N98BNCbVeKK2pR3nTp08H4Kw== Date: Wed, 14 Jan 2026 08:11:22 +0100 (CET) From: Ronald Klop To: ruby R53 Cc: freebsd-current@freebsd.org Message-ID: <1503278086.9.1768374682910@localhost> In-Reply-To: References: Subject: Re: Change fetch addresses for Rust crates, Python packages and whatnot? List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@FreeBSD.org MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="----=_Part_8_92420581.1768374682870" X-Mailer: Realworks (778.35) Importance: Normal X-Priority: 3 (Normal) X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:51088, ipnet:31.134.200.0/21, country:NL] X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Rspamd-Queue-Id: 4drcjV5K8Gz3D5B ------=_Part_8_92420581.1768374682870 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: quoted-printable Van: ruby R53 Datum: woensdag, 14 januari 2026 03:26 Aan: freebsd-current@freebsd.org Onderwerp: Change fetch addresses for Rust crates, Python packages and what= not? >=20 > I've been trying to compile www/chromium for literal days now, all becaus= e whenever `make` runs `fetch`, it goes to an address that's actually suppo= sed to redirect to another one, but `fetch` instead keeps trying for a whol= e minute until it ultimately falls back to distcache.freebsd.org, which is = another whole minute of waiting for the file to download. The cherry on top= is that the file is, most of the time, just hundreds of kilobytes in size. >=20 > Hell, I even had to manually download part of these individual files usin= g that Links web browser=C2=B7=C2=B7=C2=B7 it'd do it a lot faster than `fe= tch`, all because it actually followed the redirection and didn't just keep= trying over and over. But of course I wouldn't manually download hundreds = upon hundreds of archive files, this is the reason why such tool exists. >=20 > So here's an example: currently one of the missing dependencies for www/c= hromium is GTK3, which depends on graphics/librsvg2-rust, which then depend= s on cargo-c. It's now trying to fetch dunce-1.0.5.crate, which is one of t= he crates required for it, from https://crates.io/api/v1/crates/dunce/1.0.5= /download?dummy=3D/rust/crates/dunce-1.0.5.crate. Running `curl` on this li= nk does literally nothing, no files are downloaded. But running it on any w= eb browser instantly downloads the crate in question, all because that link= actually redirects to https://static.crates.io/crates/dunce/dunce-1.0.5.cr= ate. Even `curl`ing that new link does give the file I want. Same applies f= or Python packages. >=20 > Now imagine this for all the remaining files=C2=B7=C2=B7=C2=B7 if it take= s about 2 minutes to download a single file, and there's 398 crate entries = on cargo-c's `Makefile.crates`, that means it'd take a whole 796 minutes to= download everything, which is literally more than half a day (~13.6 hours)= :) > =20 >=20 >=20 >=20 Can you copy-paste the output of the following command in a reply? fetch -vv "https://crates.io/api/v1/crates/dunce/1.0.5/download?dummy=3D/ru= st/crates/dunce-1.0.5.crate" For me it immediately follows the redirect and downloads the file in half a= second. See the output below. Regards, Ronald. --=20 time fetch -vv "https://crates.io/api/v1/crates/dunce/1.0.5/download?dummy= =3D/rust/crates/dunce-1.0.5.crate" scheme: "https" user: "" password: "" host: "crates.io" port: "0" document: "/api/v1/crates/dunce/1.0.5/download?dummy=3D/rust/crates/dunce-1= .0.5.crate" ---> crates.io:443 resolving server address: crates.io:443 SSL options: 82004850 Peer verification enabled Using OpenSSL default CA cert file and path Verify hostname TLSv1.3 connection established using TLS_AES_128_GCM_SHA256 Certificate subject: /CN=3Dcrates.io Certificate issuer: /C=3DUS/O=3DAmazon/CN=3DAmazon RSA 2048 M01 requesting https://crates.io/api/v1/crates/dunce/1.0.5/download?dummy=3D/ru= st/crates/dunce-1.0.5.crate >>> GET /api/v1/crates/dunce/1.0.5/download?dummy=3D/rust/crates/dunce-1.0.= 5.crate HTTP/1.1 >>> Host: crates.io >>> Accept: */* >>> User-Agent: fetch libfetch/2.0 >>> Connection: close >>>=20 <<< HTTP/1.1 302 Moved Temporarily <<< Content-Length: 0 <<< Connection: close content length: [0] <<< Access-Control-Allow-Origin: * <<< Date: Wed, 14 Jan 2026 07:10:06 GMT <<< Location: https://static.crates.io/crates/dunce/dunce-1.0.5.crate <<< Nel: {"report_to":"heroku-nel","response_headers":["Via"],"max_age":360= 0,"success_fraction":0.01,"failure_fraction":0.1} 302 redirect to https://static.crates.io/crates/dunce/dunce-1.0.5.crate scheme: "https" user: "" password: "" host: "static.crates.io" port: "0" document: "/crates/dunce/dunce-1.0.5.crate" <<< Report-To: {"group":"heroku-nel","endpoints":[{"url":"https://nel.herok= u.com/reports?s=3DVOwdWzeBall4rSiByZHUwCe6zIbVFdxy%2FIgcGEUOIZo%3D\u0026sid= =3Daf571f24-03ee-46d1-9f90-ab9030c2c74c\u0026ts=3D1768374606"}],"max_age":3= 600} <<< Reporting-Endpoints: heroku-nel=3D"https://nel.heroku.com/reports?s=3DV= OwdWzeBall4rSiByZHUwCe6zIbVFdxy%2FIgcGEUOIZo%3D&sid=3Daf571f24-03ee-46d1-9f= 90-ab9030c2c74c&ts=3D1768374606" <<< Server: Heroku <<< Strict-Transport-Security: max-age=3D31536000; includeSubDomains <<< Via: 1.1 heroku-router, 1.1 c2905f891f96a0ec9c7fab16916dbb46.cloudfront= .net (CloudFront) <<< X-Cache: Miss from cloudfront <<< X-Amz-Cf-Pop: AMS58-P6 <<< X-Amz-Cf-Id: Xa10pwbYp4MCA5eToVa0shahNpzr2epWFUuQZS63bMpmbiDLKc2dmQ=3D= =3D <<<=20 ---> static.crates.io:443 resolving server address: static.crates.io:443 SSL options: 82004850 Peer verification enabled Using OpenSSL default CA cert file and path Verify hostname TLSv1.3 connection established using TLS_AES_128_GCM_SHA256 Certificate subject: /CN=3Dstatic.crates.io Certificate issuer: /C=3DBE/O=3DGlobalSign nv-sa/CN=3DGlobalSign Atlas R3 D= V TLS CA 2025 Q3 requesting https://static.crates.io/crates/dunce/dunce-1.0.5.crate >>> GET /crates/dunce/dunce-1.0.5.crate HTTP/1.1 >>> Host: static.crates.io >>> Accept: */* >>> User-Agent: fetch libfetch/2.0 >>> Connection: close >>>=20 <<< HTTP/1.1 200=20 <<< Connection: close <<< Content-Length: 8244 <<< etag: "198d7f7c37f632bc83502b28870b6d60" content length: [8244] <<< cache-control: public,max-age=3D31536000,immutable <<< x-amz-version-id: PO7xZdimM08F9npIqG8PYtpqpEuWdV9D <<< x-amz-id-2: Dj7mFoaqQJUc61mB1FcouUQcy0tajnacJvU+xOR3Be5//jo3GaSmdCwSSgt= YyDMnYE4cPCiPv7PV9m7z1Jk0WazYHTFPbYhm <<< x-amz-request-id: KWXPQ4JP20HSYSY2 <<< date: Wed, 14 Jan 2026 00:09:56 GMT <<< x-amz-server-side-encryption: AES256 <<< x-amz-replication-status: COMPLETED <<< last-modified: Sun, 04 Aug 2024 01:36:47 GMT <<< accept-ranges: bytes last modified: [2024-08-04 01:36:47] <<< content-type: application/gzip <<< x-served-by: cache-ams21058-AMS <<< server: AmazonS3 <<< age: 25212 <<< x-cache: HIT <<< x-cache-hits: 148 <<<=20 offset 0, length -1, size -1, clength 8244 local size / mtime: 8244 / 1722735407 remote size / mtime: 8244 / 1722735407 dunce-1.0.5.crate 8244 B 10 MBps = 00s real 0m0.429s user 0m0.023s sys 0m0.027s ------=_Part_8_92420581.1768374682870 Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: quoted-printable

Van: ruby R53 <red_M95@proton.me>
Datum: woensdag, 14 januari 2026 03:26
Aan: freebsd-current@freebsd.org
Onderwerp: Change fetch addresses for Rust crates, Python = packages and whatnot?

I've been trying to compile www/c= hromium for literal days now, all because whenever `make` runs `fetch`, it = goes to an address that's actually supposed to redirect to another one, but= `fetch` instead keeps trying for a whole minute until it ultimately falls = back to distcache.freebsd.org, which is another whole minute of waiting for= the file to download. The cherry on top is that the file is, most of the t= ime, just hundreds of kilobytes in size.

Hell, I even had to manually download part of these individual files using = that Links web browser=C2=B7=C2=B7=C2=B7 it'd do it a lot faster than `fetc= h`, all because it actually followed the redirection and didn't just keep t= rying over and over. But of course I wouldn't manually download hundreds up= on hundreds of archive files, this is the reason why such tool exists.

So here's an example: currently one of the missing dependencies for www/chr= omium is GTK3, which depends on graphics/librsvg2-rust, which then depends = on cargo-c. It's now trying to fetch dunce-1.0.5.crate, which is one of the= crates required for it, from https://crates.i= o/api/v1/crates/dunce/1.0.5/download?dummy=3D/rust/crates/dunce-1.0.5.crate= . Running `curl` on this link does literally nothing, no files are down= loaded. But running it on any web browser instantly downloads the crate in = question, all because that link actually redirects to https://static.crates.io/cra= tes/dunce/dunce-1.0.5.crate. Even `curl`ing that new link does give the= file I want. Same applies for Python packages.

Now imagine this for all the remaining files=C2=B7=C2=B7=C2=B7 if it takes = about 2 minutes to download a single file, and there's 398 crate entries on= cargo-c's `Makefile.crates`, that means it'd take a whole 796 minutes to d= ownload everything, which is literally more than half a day (~13.6 hours) := )
 


Can you copy-paste the output of the following command in a reply?
fetch -vv "https://crates.io/api/v1/crates/dunce/1.0.5/download?dummy=3D/ru= st/crates/dunce-1.0.5.crate"

For me it immediately follows the redirect and downloads the file in half a= second. See the output below.

Regards,
Ronald.


--  
time fetch -vv "https://crates.io/api/v1/crates/dunce/1.0.5/download?d=
ummy=3D/rust/crates/dunce-1.0.5.crate"
scheme:   "https"
user:     ""
password: ""
host:     "crates.io"
port:     "0"
document: "/api/v1/crates/dunce/1.0.5/download?dummy=3D/rust/crates/dunce-1=
.0.5.crate"
---> crates.io:443
resolving server address: crates.io:443
SSL options: 82004850
Peer verification enabled
Using OpenSSL default CA cert file and path
Verify hostname
TLSv1.3 connection established using TLS_AES_128_GCM_SHA256
Certificate subject: /CN=3Dcrates.io
Certificate issuer: /C=3DUS/O=3DAmazon/CN=3DAmazon RSA 2048 M01
requesting https://crates.io/api/v1/crates/dunce/1.0.5/download?dummy=3D/ru=
st/crates/dunce-1.0.5.crate
>>> GET /api/v1/crates/dunce/1.0.5/download?dummy=3D/rust/crates/d=
unce-1.0.5.crate HTTP/1.1
>>> Host: crates.io
>>> Accept: */*
>>> User-Agent: fetch libfetch/2.0
>>> Connection: close
>>>=20
<<< HTTP/1.1 302 Moved Temporarily
<<< Content-Length: 0
<<< Connection: close
content length: [0]
<<< Access-Control-Allow-Origin: *
<<< Date: Wed, 14 Jan 2026 07:10:06 GMT
<<< Location: https://static.crates.io/crates/dunce/dunce-1.0.5.cr=
ate
<<< Nel: {"report_to":"heroku-nel","response_headers":["Via"],"max=
_age":3600,"success_fraction":0.01,"failure_fraction":0.1}
302 redirect to https://static.crates.io/crates/dunce/dunce-1.0.5.crate
scheme:   "https"
user:     ""
password: ""
host:     "static.crates.io"
port:     "0"
document: "/crates/dunce/dunce-1.0.5.crate"
<<< Report-To: {"group":"heroku-nel","endpoints":[{"url":"https://=
nel.heroku.com/reports?s=3DVOwdWzeBall4rSiByZHUwCe6zIbVFdxy%2FIgcGEUOIZo%3D=
\u0026sid=3Daf571f24-03ee-46d1-9f90-ab9030c2c74c\u0026ts=3D1768374606"}],"m=
ax_age":3600}
<<< Reporting-Endpoints: heroku-nel=3D"https://nel.heroku.com/repo=
rts?s=3DVOwdWzeBall4rSiByZHUwCe6zIbVFdxy%2FIgcGEUOIZo%3D&sid=3Daf571f24=
-03ee-46d1-9f90-ab9030c2c74c&ts=3D1768374606"
<<< Server: Heroku
<<< Strict-Transport-Security: max-age=3D31536000; includeSubDomai=
ns
<<< Via: 1.1 heroku-router, 1.1 c2905f891f96a0ec9c7fab16916dbb46.c=
loudfront.net (CloudFront)
<<< X-Cache: Miss from cloudfront
<<< X-Amz-Cf-Pop: AMS58-P6
<<< X-Amz-Cf-Id: Xa10pwbYp4MCA5eToVa0shahNpzr2epWFUuQZS63bMpmbiDLK=
c2dmQ=3D=3D
<<<=20
---> static.crates.io:443
resolving server address: static.crates.io:443
SSL options: 82004850
Peer verification enabled
Using OpenSSL default CA cert file and path
Verify hostname
TLSv1.3 connection established using TLS_AES_128_GCM_SHA256
Certificate subject: /CN=3Dstatic.crates.io
Certificate issuer: /C=3DBE/O=3DGlobalSign nv-sa/CN=3DGlobalSign Atlas R3 D=
V TLS CA 2025 Q3
requesting https://static.crates.io/crates/dunce/dunce-1.0.5.crate
>>> GET /crates/dunce/dunce-1.0.5.crate HTTP/1.1
>>> Host: static.crates.io
>>> Accept: */*
>>> User-Agent: fetch libfetch/2.0
>>> Connection: close
>>>=20
<<< HTTP/1.1 200=20
<<< Connection: close
<<< Content-Length: 8244
<<< etag: "198d7f7c37f632bc83502b28870b6d60"
content length: [8244]
<<< cache-control: public,max-age=3D31536000,immutable
<<< x-amz-version-id: PO7xZdimM08F9npIqG8PYtpqpEuWdV9D
<<< x-amz-id-2: Dj7mFoaqQJUc61mB1FcouUQcy0tajnacJvU+xOR3Be5//jo3Ga=
SmdCwSSgtYyDMnYE4cPCiPv7PV9m7z1Jk0WazYHTFPbYhm
<<< x-amz-request-id: KWXPQ4JP20HSYSY2
<<< date: Wed, 14 Jan 2026 00:09:56 GMT
<<< x-amz-server-side-encryption: AES256
<<< x-amz-replication-status: COMPLETED
<<< last-modified: Sun, 04 Aug 2024 01:36:47 GMT
<<< accept-ranges: bytes
last modified: [2024-08-04 01:36:47]
<<< content-type: application/gzip
<<< x-served-by: cache-ams21058-AMS
<<< server: AmazonS3
<<< age: 25212
<<< x-cache: HIT
<<< x-cache-hits: 148
<<<=20
offset 0, length -1, size -1, clength 8244
local size / mtime: 8244 / 1722735407
remote size / mtime: 8244 / 1722735407
dunce-1.0.5.crate                                     8244  B   10 MBps    =
00s

real    0m0.429s
user    0m0.023s
sys    0m0.027s
------=_Part_8_92420581.1768374682870--