From owner-freebsd-current@FreeBSD.ORG Mon Jun 21 11:38:48 2010 Return-Path: Delivered-To: freebsd-current@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id A42941065674 for ; Mon, 21 Jun 2010 11:38:48 +0000 (UTC) (envelope-from tevans.uk@googlemail.com) Received: from mail-fx0-f54.google.com (mail-fx0-f54.google.com [209.85.161.54]) by mx1.freebsd.org (Postfix) with ESMTP id 2FD498FC25 for ; Mon, 21 Jun 2010 11:38:47 +0000 (UTC) Received: by fxm7 with SMTP id 7so2170876fxm.13 for ; Mon, 21 Jun 2010 04:38:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlemail.com; s=gamma; h=domainkey-signature:mime-version:received:received:in-reply-to :references:date:message-id:subject:from:to:cc:content-type; bh=i5yJf+gMDmMjZlWjyeVa04peEMrRtem+NJZ1jBcrYN4=; b=rEpcOv9AZ7bG39mKaklfwqkhKWefkq+WRUsCAdp9JzGs1YlxuFT6tzh2ZGyX1USJDH iP3zDfaHhovC9bNGQp8l0ZWBnCN4/qwgRhOhCweqHuts+30P+LEiQ1Gn/K6YlHyc3ror fxXKVJV6XSHWqqFE3/uL+1VuXfTBQPep7J8kI= DomainKey-Signature: a=rsa-sha1; c=nofws; d=googlemail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; b=HlE4v93qpnbL/hopbvHl/jqARWUbGaxxVh2vSSTcPh8LK5Ji4dNZSz0f1+/fHtsO8m 0As6fOEe+J4UoHEBdjHAnmonhySS7+O7zjj5WOrQCOfHCyZwZDcWLz0B01dvAe4YbR3u Ya+jlxQ/s6giVLEpQEDkpT5PxZP6rCmQa8CuY= MIME-Version: 1.0 Received: by 10.239.183.146 with SMTP id u18mr342910hbg.163.1277120326921; Mon, 21 Jun 2010 04:38:46 -0700 (PDT) Received: by 10.239.185.1 with HTTP; Mon, 21 Jun 2010 04:38:46 -0700 (PDT) In-Reply-To: <20100621130743.4df77343@ernst.jennejohn.org> References: <20100621101046.GA76036@droso.net> <20100621130743.4df77343@ernst.jennejohn.org> Date: Mon, 21 Jun 2010 12:38:46 +0100 Message-ID: From: Tom Evans To: gljennjohn@googlemail.com, des@des.no Content-Type: multipart/mixed; boundary=001485f428486bede1048988bf37 Cc: freebsd-current Subject: Re: Ports doesnt respect fetch environment settings X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 21 Jun 2010 11:38:48 -0000 --001485f428486bede1048988bf37 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable On Mon, Jun 21, 2010 at 12:07 PM, Gary Jennejohn wrote: > Yes. =C2=A0When you ran fetch by hand you didn't have the -ApRr on the CL= . > Could it be that the 'p' flag is causing problems? > > Try running fetch by hand again with those flags and see what happens. > If it fails, try removing the 'p' flag. > > -- > Gary Jennejohn > Yes, I went through the same logic - its not the 'p' flag, its the 'A' flag, which is supposed to prevent it following 302 redirects. In this case, it refuses to retry the request when it receives a 407. > # /usr/bin/fetch -ApRr -v -S 37867 http://googlecl.googlecode.com/files/g= ooglecl-0.9.5.tar.gz looking up proxy connecting to proxy:3128 requesting http://googlecl.googlecode.com/files/googlecl-0.9.5.tar.gz proxy requires authorization fetch: http://googlecl.googlecode.com/files/googlecl-0.9.5.tar.gz: Proxy Authentication Required root@strangepork '12:13:28' '/usr/ports/net/googlecl' > # /usr/bin/fetch -pRr -v -S 37867 http://googlecl.googlecode.com/files/go= oglecl-0.9.5.tar.gz looking up proxy connecting to proxy:3128 requesting http://googlecl.googlecode.com/files/googlecl-0.9.5.tar.gz proxy requires authorization looking up proxy connecting to proxy:3128 requesting http://googlecl.googlecode.com/files/googlecl-0.9.5.tar.gz local size / mtime: 37867 / 1276839258 remote size / mtime: 37867 / 1276839258 That doesn't seem right! Looking in lib/libfetch/http.c it tries to fetch the file in a loop. libfetch first tries without proxy authentication, even if you specify it in your environment. If the request fails due to proxy authentication being required, it sets a flag to add proxy auth details next time through the loop, and continues. If the '-A' flag is set however, it will only go through this loop one time, and so does not attempt to use the supplied proxy authentication. Comments in the source code imply that this is a change in behaviour introduced at the start of the year to support digest authentication; prior to that it would have attempted proxy auth on the first request. The flag for '-A' is documented as: -A Do not automatically follow =E2=80=98=E2=80=98temporary=E2=80= =99=E2=80=99 (302) redirects. Some broken Web sites will return a redirect instead of a not=E2=80=90found error when the requested object does not exist. where as the behaviour is: Do not attempt to download this file more than once, for any reason. Having seen this, the bug is that we wish to go thru the loop one more time to retry with proxy authentication added, but the loop may exit on the next iteration. This diff allows it to go through the loop one more time, and now fetches the file correctly. Incidentally, having fixed fetch to work with '-A' thru a proxy that requires proxy auth, I now dont require anything in FETCH_ENV or FETCH_*_ARGS, it works correctly with the PROXY_* environment variables. Patch attached. Cheers Tom --001485f428486bede1048988bf37 Content-Type: text/plain; charset=US-ASCII; name="lib::libfetch::http.c.diff.txt" Content-Disposition: attachment; filename="lib::libfetch::http.c.diff.txt" Content-Transfer-Encoding: base64 X-Attachment-Id: f_gap8fppp0 SW5kZXg6IC91c3Ivc3JjL2xpYi9saWJmZXRjaC9odHRwLmMKPT09PT09PT09PT09PT09PT09PT09 PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PQpSQ1MgZmlsZTog L2hvbWUvbmN2cy9zcmMvbGliL2xpYmZldGNoL2h0dHAuYyx2CnJldHJpZXZpbmcgcmV2aXNpb24g MS43OC4yLjUKZGlmZiAtdSAtcjEuNzguMi41IGh0dHAuYwotLS0gL3Vzci9zcmMvbGliL2xpYmZl dGNoL2h0dHAuYwkyNyBKYW4gMjAxMCAxNDo1NDo0OCAtMDAwMAkxLjc4LjIuNQorKysgL3Vzci9z cmMvbGliL2xpYmZldGNoL2h0dHAuYwkyMSBKdW4gMjAxMCAxMTozMDozMiAtMDAwMApAQCAtMTcx MCw2ICsxNzEwLDcgQEAKIAkJCQlnb3RvIG91Y2g7CiAJCQl9CiAJCQkvKiB0cnkgYWdhaW4sIGJ1 dCBzZW5kIHRoZSBwYXNzd29yZCB0aGlzIHRpbWUgKi8KKwkJCSsrbjsKIAkJCWlmICh2ZXJib3Nl KQogCQkJCWZldGNoX2luZm8oInByb3h5IHJlcXVpcmVzIGF1dGhvcml6YXRpb24iKTsKIAkJCWJy ZWFrOwo= --001485f428486bede1048988bf37--