From owner-freebsd-ports@freebsd.org Fri Apr 3 17:36:49 2020 Return-Path: Delivered-To: freebsd-ports@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id F0A22262470 for ; Fri, 3 Apr 2020 17:36:49 +0000 (UTC) (envelope-from dim@FreeBSD.org) Received: from mailman.nyi.freebsd.org (unknown [127.0.1.3]) by mx1.freebsd.org (Postfix) with ESMTP id 48v6Yr1MQWz4d2d for ; Fri, 3 Apr 2020 17:36:48 +0000 (UTC) (envelope-from dim@FreeBSD.org) Received: by mailman.nyi.freebsd.org (Postfix) id C4C9A26246F; Fri, 3 Apr 2020 17:36:42 +0000 (UTC) Delivered-To: ports@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id C2AB426246E for ; Fri, 3 Apr 2020 17:36:42 +0000 (UTC) (envelope-from dim@FreeBSD.org) Received: from smtp.freebsd.org (smtp.freebsd.org [IPv6:2610:1c1:1:606c::24b:4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 48v6Yh2qWXz4d0Z; Fri, 3 Apr 2020 17:36:40 +0000 (UTC) (envelope-from dim@FreeBSD.org) Received: from tensor.andric.com (tensor.andric.com [87.251.56.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "tensor.andric.com", Issuer "Let's Encrypt Authority X3" (verified OK)) (Authenticated sender: dim) by smtp.freebsd.org (Postfix) with ESMTPSA id 18D4819269; Fri, 3 Apr 2020 17:36:32 +0000 (UTC) (envelope-from dim@FreeBSD.org) Received: from [IPv6:2001:470:7a58::8873:afed:f4c0:942f] (unknown [IPv6:2001:470:7a58:0:8873:afed:f4c0:942f]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by tensor.andric.com (Postfix) with ESMTPSA id E19435ACE6; Fri, 3 Apr 2020 19:36:29 +0200 (CEST) From: Dimitry Andric Message-Id: <88F7E459-73D5-444C-A473-85C78A4484C5@FreeBSD.org> Content-Type: multipart/signed; boundary="Apple-Mail=_C0974ACC-C957-4724-9C50-0F69FBFABA09"; protocol="application/pgp-signature"; micalg=pgp-sha1 Mime-Version: 1.0 (Mac OS X Mail 12.4 \(3445.104.14\)) Subject: Re: fetch is tarpitted by Texas Instruments and/or Akamai and can not download distfiles for TI-related ports. Date: Fri, 3 Apr 2020 19:36:19 +0200 In-Reply-To: <20200403130821.GB37073@home.opsec.eu> Cc: Lev Serebryakov , ports@freebsd.org To: Kurt Jaeger References: <04228bf1-5d14-8458-37d3-66ec28645763@FreeBSD.org> <20200403130821.GB37073@home.opsec.eu> X-Mailer: Apple Mail (2.3445.104.14) X-BeenThere: freebsd-ports@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Porting software to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 03 Apr 2020 17:36:50 -0000 --Apple-Mail=_C0974ACC-C957-4724-9C50-0F69FBFABA09 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=us-ascii On 3 Apr 2020, at 15:08, Kurt Jaeger wrote: >=20 > Hi! >=20 >> It is true for both IPv4 and IPv6, and for Russia and Germany (I can >> not test from USA, though). >=20 > I tried it from freefall, same problem. Typically one of those "Endpoint Protection" products, they have some = sort of whitelist or blacklist of bad user agents. Using "curl/7.68.0" = as user agent works: $ fetch -vvv --user-agent=3D"curl/7.68.0" = http://www.ti.com/lit/ug/slau646e/slau646e.pdf scheme: "http" user: "" password: "" host: "www.ti.com" port: "0" document: "/lit/ug/slau646e/slau646e.pdf" ---> www.ti.com:80 resolving server address: www.ti.com:80 requesting http://www.ti.com/lit/ug/slau646e/slau646e.pdf >>> GET /lit/ug/slau646e/slau646e.pdf HTTP/1.1 >>> Host: www.ti.com >>> Accept: */* >>> User-Agent: curl/7.68.0 >>> Connection: close >>> <<< HTTP/1.1 200 OK <<< Server: Apache <<< Last-Modified: Wed, 26 Jun 2019 21:34:37 GMT <<< ETag: "1551ab-58c40cfeb6c6f" last modified: [2019-06-26 21:34:37] <<< Accept-Ranges: bytes <<< Content-Length: 1397163 <<< Cache-Control: max-age=3D60 content length: [1397163] <<< Expires: Thu, 30 Jan 2020 09:27:51 GMT <<< Content-Type: application/pdf <<< Date: Fri, 03 Apr 2020 17:30:20 GMT <<< Connection: close <<< Set-Cookie: = ti_geo=3Dcountry=3DXX|city=3DYYYYYYY|continent=3DEU|tc_ip=3Dn.m.o.p; = path=3D/; domain=3D.ti.com <<< Set-Cookie: ti_rid=3D3110b500; path=3D/; domain=3D.ti.com <<< Set-Cookie: ti_ua=3Dcurl%2f7.68.0; path=3D/; domain=3D.ti.com <<< Set-Cookie: ti_ak_id=3D67fba42067fba42067fba4205e8772ac3110b500; = expires=3DSat, 03-Apr-2021 17:30:20 GMT; path=3D/; domain=3D.ti.com <<< Set-Cookie: ti_ridh=3Dexpired; path=3D/; domain=3Dti.com; = expires=3DThu, 1-Aug-2019 06:14:04 GMT <<< Set-Cookie: ti_bm=3D; path=3D/; domain=3D.ti.com <<< offset 0, length -1, size -1, clength 1397163 local size / mtime: 1397163 / 1585934997 remote size / mtime: 1397163 / 1561584877 slau646e.pdf 1364 kB 15 MBps = 00s You can immediately see they're trying to violate cookie laws. ;-) The = "ti_ua" cookie stores the user agent, and I guess "tk_ak_id" is the = Akamai ID (a.k.a. personally identifiable information). -Dimitry --Apple-Mail=_C0974ACC-C957-4724-9C50-0F69FBFABA09 Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=signature.asc Content-Type: application/pgp-signature; name=signature.asc Content-Description: Message signed with OpenPGP -----BEGIN PGP SIGNATURE----- Version: GnuPG/MacGPG2 v2.2 iF0EARECAB0WIQR6tGLSzjX8bUI5T82wXqMKLiCWowUCXod0EwAKCRCwXqMKLiCW o3pKAKDYUj8CokLhlH9Cn2oEpgHi+94KdwCfUaoV642XozaxpsDkCUQB3WIs2FA= =TS4h -----END PGP SIGNATURE----- --Apple-Mail=_C0974ACC-C957-4724-9C50-0F69FBFABA09--