From owner-freebsd-bugs@FreeBSD.ORG Sun Oct 7 13:20:12 2012 Return-Path: Delivered-To: freebsd-bugs@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 7F4201065670 for ; Sun, 7 Oct 2012 13:20:12 +0000 (UTC) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 54B988FC0A for ; Sun, 7 Oct 2012 13:20:12 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.5/8.14.5) with ESMTP id q97DKCYm048615 for ; Sun, 7 Oct 2012 13:20:12 GMT (envelope-from gnats@freefall.freebsd.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.5/8.14.5/Submit) id q97DKCns048614; Sun, 7 Oct 2012 13:20:12 GMT (envelope-from gnats) Resent-Date: Sun, 7 Oct 2012 13:20:12 GMT Resent-Message-Id: <201210071320.q97DKCns048614@freefall.freebsd.org> Resent-From: FreeBSD-gnats-submit@FreeBSD.org (GNATS Filer) Resent-To: freebsd-bugs@FreeBSD.org Resent-Reply-To: FreeBSD-gnats-submit@FreeBSD.org, Garrett Cooper Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 3D26C106566B for ; Sun, 7 Oct 2012 13:14:43 +0000 (UTC) (envelope-from nobody@FreeBSD.org) Received: from red.freebsd.org (red.freebsd.org [IPv6:2001:4f8:fff6::22]) by mx1.freebsd.org (Postfix) with ESMTP id 24CE48FC14 for ; Sun, 7 Oct 2012 13:14:43 +0000 (UTC) Received: from red.freebsd.org (localhost [127.0.0.1]) by red.freebsd.org (8.14.5/8.14.5) with ESMTP id q97DEg8O063427 for ; Sun, 7 Oct 2012 13:14:42 GMT (envelope-from nobody@red.freebsd.org) Received: (from nobody@localhost) by red.freebsd.org (8.14.5/8.14.5/Submit) id q97DEgaO063426; Sun, 7 Oct 2012 13:14:42 GMT (envelope-from nobody) Message-Id: <201210071314.q97DEgaO063426@red.freebsd.org> Date: Sun, 7 Oct 2012 13:14:42 GMT From: Garrett Cooper To: freebsd-gnats-submit@FreeBSD.org X-Send-Pr-Version: www-3.1 Cc: Subject: bin/172451: [fetch] Not properly following redirects; causes unnecessary manual intervention with ports installs, etc X-BeenThere: freebsd-bugs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Bug reports List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 07 Oct 2012 13:20:12 -0000 >Number: 172451 >Category: bin >Synopsis: [fetch] Not properly following redirects; causes unnecessary manual intervention with ports installs, etc >Confidential: no >Severity: non-critical >Priority: low >Responsible: freebsd-bugs >State: open >Quarter: >Keywords: >Date-Required: >Class: sw-bug >Submitter-Id: current-users >Arrival-Date: Sun Oct 07 13:20:11 UTC 2012 >Closed-Date: >Last-Modified: >Originator: Garrett Cooper >Release: 9.1-STABLE >Organization: EMC Isilon >Environment: >Description: I have a fair amount of issues with fetch/libfetch and HTTP redirects. Some servers do it properly, others don't. A busted example (one of samba.org's HTTP servers) is as follows: $ fetch -vv -AFpr http://us1.samba.org/samba/ftp/stable/samba-3.6.8.tar.gz scheme: [http] user: [] password: [] host: [us1.samba.org] port: [0] document: [/samba/ftp/stable/samba-3.6.8.tar.gz] ---> us1.samba.org:80 looking up us1.samba.org connecting to us1.samba.org:80 requesting http://us1.samba.org/samba/ftp/stable/samba-3.6.8.tar.gz >How-To-Repeat: fetch -AFpr http://us1.samba.org/samba/ftp/stable/samba-3.6.8.tar.gz >Fix: Patch attached with submission follows: Index: lib/libfetch/http.c =================================================================== --- lib/libfetch/http.c (revision 241309) +++ lib/libfetch/http.c (working copy) @@ -1524,8 +1526,7 @@ /* try the provided URL first */ url = URL; - /* if the A flag is set, we only get one try */ - n = noredirect ? 1 : MAX_REDIRECT; + n = MAX_REDIRECT; i = 0; e = HTTP_PROTOCOL_ERROR; @@ -1772,6 +1774,15 @@ case hdr_location: if (!HTTP_REDIRECT(conn->err)) break; + /* + * Don't redirect due to bad server + * paranoia. + */ + if (noredirect && + conn->err == HTTP_MOVED_TEMP) { + n = 1; + break; + } if (new) free(new); if (verbose) >Release-Note: >Audit-Trail: >Unformatted: >>> GET /samba/ftp/stable/samba-3.6.8.tar.gz HTTP/1.1 >>> Host: us1.samba.org >>> User-Agent: fetch libfetch/2.0 >>> Connection: close >>> <<< HTTP/1.1 301 Moved Permanently <<< Date: Sun, 07 Oct 2012 11:14:28 GMT <<< Server: Apache <<< Location: http://www.samba.org/samba/ftp/stable/samba-3.6.8.tar.gz <<< Vary: Accept-Encoding 301 redirect to http://www.samba.org/samba/ftp/stable/samba-3.6.8.tar.gz scheme: [http] user: [] password: [] host: [www.samba.org] port: [0] document: [/samba/ftp/stable/samba-3.6.8.tar.gz] <<< Content-Length: 331 <<< Connection: close content length: [331] <<< Content-Type: text/html; charset=iso-8859-1 <<< fetch: http://us1.samba.org/samba/ftp/stable/samba-3.6.8.tar.gz: Moved Permanently A working example with samba.org (without the -A flag): $ fetch -vv -Fpr http://us1.samba.org/samba/ftp/stable/samba-3.6.8.tar.gz scheme: [http] user: [] password: [] host: [us1.samba.org] port: [0] document: [/samba/ftp/stable/samba-3.6.8.tar.gz] ---> us1.samba.org:80 looking up us1.samba.org connecting to us1.samba.org:80 requesting http://us1.samba.org/samba/ftp/stable/samba-3.6.8.tar.gz >>> GET /samba/ftp/stable/samba-3.6.8.tar.gz HTTP/1.1 >>> Host: us1.samba.org >>> User-Agent: fetch libfetch/2.0 >>> Connection: close >>> <<< HTTP/1.1 301 Moved Permanently <<< Date: Sun, 07 Oct 2012 11:25:32 GMT <<< Server: Apache <<< Location: http://www.samba.org/samba/ftp/stable/samba-3.6.8.tar.gz <<< Vary: Accept-Encoding 301 redirect to http://www.samba.org/samba/ftp/stable/samba-3.6.8.tar.gz scheme: [http] user: [] password: [] host: [www.samba.org] port: [0] document: [/samba/ftp/stable/samba-3.6.8.tar.gz] <<< Content-Length: 331 <<< Connection: close content length: [331] <<< Content-Type: text/html; charset=iso-8859-1 <<< ---> www.samba.org:80 looking up www.samba.org connecting to www.samba.org:80 requesting http://www.samba.org/samba/ftp/stable/samba-3.6.8.tar.gz >>> GET /samba/ftp/stable/samba-3.6.8.tar.gz HTTP/1.1 >>> Host: www.samba.org >>> User-Agent: fetch libfetch/2.0 >>> Connection: close >>> <<< HTTP/1.1 301 Moved Permanently <<< Date: Sun, 07 Oct 2012 11:25:32 GMT <<< Server: Apache <<< Location: http://ftp.samba.org/pub/samba/stable/samba-3.6.8.tar.gz <<< Vary: Accept-Encoding 301 redirect to http://ftp.samba.org/pub/samba/stable/samba-3.6.8.tar.gz scheme: [http] user: [] password: [] host: [ftp.samba.org] port: [0] document: [/pub/samba/stable/samba-3.6.8.tar.gz] <<< Content-Length: 327 <<< Connection: close content length: [327] <<< Content-Type: text/html; charset=iso-8859-1 <<< ---> ftp.samba.org:80 looking up ftp.samba.org connecting to ftp.samba.org:80 requesting http://ftp.samba.org/pub/samba/stable/samba-3.6.8.tar.gz >>> GET /pub/samba/stable/samba-3.6.8.tar.gz HTTP/1.1 >>> Host: ftp.samba.org >>> User-Agent: fetch libfetch/2.0 >>> Connection: close >>> <<< HTTP/1.1 200 OK <<< Date: Sun, 07 Oct 2012 11:25:32 GMT <<< Server: Apache <<< Last-Modified: Mon, 17 Sep 2012 07:13:13 GMT <<< ETag: "125001d-207e10e-4c9e081b7b440" last modified: [2012-09-17 07:13:13] <<< Accept-Ranges: bytes <<< Content-Length: 34070798 <<< Connection: close content length: [34070798] <<< Content-Type: application/x-gzip <<< Content-Encoding: x-gzip <<< offset 0, length -1, size -1, clength 34070798 remote size / mtime: 34070798 / 1347865993 samba-3.6.8.tar.gz 2% of 32 MB 659 kBps^C fetch: transfer interrupted I think that the issue is that HTTP 3XX codes are blacklisted with the -A flag too liberally -- in particular 301, 302, and 307 should be followed per RFC-2616 [1]. The attached patch does that exactly (and also adds support for HTTP error code 305 -- us e proxy -- although scrubbing that before commit is fine if it should be in another bug). FWIW that's what the documentation for fetch(1) suggests as well... -A Do not automatically follow ``temporary'' (302) redirects. Some broken Web sites will return a redirect instead of a not-found error when the requested object does not exist. fetch(3) doesn't have anything to say on the matter (and in fact doesn't document all of the supported flags)... HTTP SCHEME The fetchXGetHTTP(), fetchGetHTTP() and fetchPutHTTP() functions imple- ment the HTTP/1.1 protocol. With a little luck, there is even a chance that they comply with RFC2616 and RFC2617. If the `d' (direct) flag is specified, fetchXGetHTTP(), fetchGetHTTP() and fetchPutHTTP() will use a direct connection even if a proxy server is defined. If the `i' (if-modified-since) flag is specified, and the ims_time field is set in struct url, then fetchXGetHTTP() and fetchGetHTTP() will send a conditional If-Modified-Since HTTP header to only fetch the content if it is newer than ims_time. Since there seems to be no good way of implementing the HTTP PUT method in a manner consistent with the rest of the fetch library, fetchPutHTTP() is currently unimplemented. After the patch is applied things function as I would expect: $ fetch -vv -AFpr http://us1.samba.org/samba/ftp/stable/samba-3.6.8.tar.gz scheme: [http] user: [] password: [] host: [us1.samba.org] port: [0] document: [/samba/ftp/stable/samba-3.6.8.tar.gz] ---> us1.samba.org:80 looking up us1.samba.org connecting to us1.samba.org:80 requesting http://us1.samba.org/samba/ftp/stable/samba-3.6.8.tar.gz >>> GET /samba/ftp/stable/samba-3.6.8.tar.gz HTTP/1.1 >>> Host: us1.samba.org >>> User-Agent: fetch libfetch/2.0 >>> Range: bytes=827392- >>> Connection: close >>> <<< HTTP/1.1 301 Moved Permanently <<< Date: Sun, 07 Oct 2012 12:36:27 GMT <<< Server: Apache <<< Location: http://www.samba.org/samba/ftp/stable/samba-3.6.8.tar.gz <<< Vary: Accept-Encoding 301 redirect to http://www.samba.org/samba/ftp/stable/samba-3.6.8.tar.gz scheme: [http] user: [] password: [] host: [www.samba.org] port: [0] document: [/samba/ftp/stable/samba-3.6.8.tar.gz] <<< Content-Length: 331 <<< Connection: close content length: [331] <<< Content-Type: text/html; charset=iso-8859-1 <<< ---> www.samba.org:80 looking up www.samba.org connecting to www.samba.org:80 requesting http://www.samba.org/samba/ftp/stable/samba-3.6.8.tar.gz >>> GET /samba/ftp/stable/samba-3.6.8.tar.gz HTTP/1.1 >>> Host: www.samba.org >>> User-Agent: fetch libfetch/2.0 >>> Range: bytes=827392- >>> Connection: close >>> <<< HTTP/1.1 301 Moved Permanently <<< Date: Sun, 07 Oct 2012 12:36:28 GMT <<< Server: Apache <<< Location: http://ftp.samba.org/pub/samba/stable/samba-3.6.8.tar.gz <<< Vary: Accept-Encoding 301 redirect to http://ftp.samba.org/pub/samba/stable/samba-3.6.8.tar.gz scheme: [http] user: [] password: [] host: [ftp.samba.org] port: [0] document: [/pub/samba/stable/samba-3.6.8.tar.gz] <<< Content-Length: 327 <<< Connection: close content length: [327] <<< Content-Type: text/html; charset=iso-8859-1 <<< ---> ftp.samba.org:80 looking up ftp.samba.org connecting to ftp.samba.org:80 requesting http://ftp.samba.org/pub/samba/stable/samba-3.6.8.tar.gz >>> GET /pub/samba/stable/samba-3.6.8.tar.gz HTTP/1.1 >>> Host: ftp.samba.org >>> User-Agent: fetch libfetch/2.0 >>> Range: bytes=827392- >>> Connection: close >>> <<< HTTP/1.1 206 Partial Content <<< Date: Sun, 07 Oct 2012 12:36:28 GMT <<< Server: Apache <<< Last-Modified: Mon, 17 Sep 2012 07:13:13 GMT <<< ETag: "125001d-207e10e-4c9e081b7b440" last modified: [2012-09-17 07:13:13] <<< Accept-Ranges: bytes <<< Content-Length: 33243406 <<< Content-Range: bytes 827392-34070797/34070798 content length: [33243406] <<< Connection: close content range: [827392-34070797/34070798] <<< Content-Type: application/x-gzip <<< Content-Encoding: x-gzip <<< offset 827392, length 33243406, size 34070798, clength 33243406 local size / mtime: 827392 / 1347865993 remote size / mtime: 34070798 / 1347865993 samba-3.6.8.tar.gz 100% of 32 MB 2301 kBps 00m00s And removing the HTTP Error 302 clause I get behavior I would expect from fetch(1): $ fetch -AFpr http://us1.samba.org/samba/ftp/stable/samba-3.6.8.tar.gz fetch: http://us1.samba.org/samba/ftp/stable/samba-3.6.8.tar.gz: Moved Permanently References: 1. http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html