Date: Fri, 3 Apr 2020 19:36:19 +0200 From: Dimitry Andric <dim@FreeBSD.org> To: Kurt Jaeger <pi@freebsd.org> Cc: Lev Serebryakov <lev@freebsd.org>, ports@freebsd.org Subject: Re: fetch is tarpitted by Texas Instruments and/or Akamai and can not download distfiles for TI-related ports. Message-ID: <88F7E459-73D5-444C-A473-85C78A4484C5@FreeBSD.org> In-Reply-To: <20200403130821.GB37073@home.opsec.eu> References: <04228bf1-5d14-8458-37d3-66ec28645763@FreeBSD.org> <20200403130821.GB37073@home.opsec.eu>
next in thread | previous in thread | raw e-mail | index | archive | help
--Apple-Mail=_C0974ACC-C957-4724-9C50-0F69FBFABA09 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=us-ascii On 3 Apr 2020, at 15:08, Kurt Jaeger <pi@freebsd.org> wrote: >=20 > Hi! >=20 >> It is true for both IPv4 and IPv6, and for Russia and Germany (I can >> not test from USA, though). >=20 > I tried it from freefall, same problem. Typically one of those "Endpoint Protection" products, they have some = sort of whitelist or blacklist of bad user agents. Using "curl/7.68.0" = as user agent works: $ fetch -vvv --user-agent=3D"curl/7.68.0" = http://www.ti.com/lit/ug/slau646e/slau646e.pdf scheme: "http" user: "" password: "" host: "www.ti.com" port: "0" document: "/lit/ug/slau646e/slau646e.pdf" ---> www.ti.com:80 resolving server address: www.ti.com:80 requesting http://www.ti.com/lit/ug/slau646e/slau646e.pdf >>> GET /lit/ug/slau646e/slau646e.pdf HTTP/1.1 >>> Host: www.ti.com >>> Accept: */* >>> User-Agent: curl/7.68.0 >>> Connection: close >>> <<< HTTP/1.1 200 OK <<< Server: Apache <<< Last-Modified: Wed, 26 Jun 2019 21:34:37 GMT <<< ETag: "1551ab-58c40cfeb6c6f" last modified: [2019-06-26 21:34:37] <<< Accept-Ranges: bytes <<< Content-Length: 1397163 <<< Cache-Control: max-age=3D60 content length: [1397163] <<< Expires: Thu, 30 Jan 2020 09:27:51 GMT <<< Content-Type: application/pdf <<< Date: Fri, 03 Apr 2020 17:30:20 GMT <<< Connection: close <<< Set-Cookie: = ti_geo=3Dcountry=3DXX|city=3DYYYYYYY|continent=3DEU|tc_ip=3Dn.m.o.p; = path=3D/; domain=3D.ti.com <<< Set-Cookie: ti_rid=3D3110b500; path=3D/; domain=3D.ti.com <<< Set-Cookie: ti_ua=3Dcurl%2f7.68.0; path=3D/; domain=3D.ti.com <<< Set-Cookie: ti_ak_id=3D67fba42067fba42067fba4205e8772ac3110b500; = expires=3DSat, 03-Apr-2021 17:30:20 GMT; path=3D/; domain=3D.ti.com <<< Set-Cookie: ti_ridh=3Dexpired; path=3D/; domain=3Dti.com; = expires=3DThu, 1-Aug-2019 06:14:04 GMT <<< Set-Cookie: ti_bm=3D; path=3D/; domain=3D.ti.com <<< offset 0, length -1, size -1, clength 1397163 local size / mtime: 1397163 / 1585934997 remote size / mtime: 1397163 / 1561584877 slau646e.pdf 1364 kB 15 MBps = 00s You can immediately see they're trying to violate cookie laws. ;-) The = "ti_ua" cookie stores the user agent, and I guess "tk_ak_id" is the = Akamai ID (a.k.a. personally identifiable information). -Dimitry --Apple-Mail=_C0974ACC-C957-4724-9C50-0F69FBFABA09 Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=signature.asc Content-Type: application/pgp-signature; name=signature.asc Content-Description: Message signed with OpenPGP -----BEGIN PGP SIGNATURE----- Version: GnuPG/MacGPG2 v2.2 iF0EARECAB0WIQR6tGLSzjX8bUI5T82wXqMKLiCWowUCXod0EwAKCRCwXqMKLiCW o3pKAKDYUj8CokLhlH9Cn2oEpgHi+94KdwCfUaoV642XozaxpsDkCUQB3WIs2FA= =TS4h -----END PGP SIGNATURE----- --Apple-Mail=_C0974ACC-C957-4724-9C50-0F69FBFABA09--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?88F7E459-73D5-444C-A473-85C78A4484C5>