From owner-freebsd-bugs@FreeBSD.ORG Wed Oct 17 17:50:02 2007 Return-Path: Delivered-To: freebsd-bugs@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 335E716A417 for ; Wed, 17 Oct 2007 17:50:02 +0000 (UTC) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 100F713C481 for ; Wed, 17 Oct 2007 17:50:02 +0000 (UTC) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (gnats@localhost [127.0.0.1]) by freefall.freebsd.org (8.14.1/8.14.1) with ESMTP id l9HHo1DE092763 for ; Wed, 17 Oct 2007 17:50:01 GMT (envelope-from gnats@freefall.freebsd.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.1/8.14.1/Submit) id l9HHo1UW092762; Wed, 17 Oct 2007 17:50:01 GMT (envelope-from gnats) Resent-Date: Wed, 17 Oct 2007 17:50:01 GMT Resent-Message-Id: <200710171750.l9HHo1UW092762@freefall.freebsd.org> Resent-From: FreeBSD-gnats-submit@FreeBSD.org (GNATS Filer) Resent-To: freebsd-bugs@FreeBSD.org Resent-Reply-To: FreeBSD-gnats-submit@FreeBSD.org, Fabian Keil Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id B7AFF16A4E1 for ; Wed, 17 Oct 2007 17:41:55 +0000 (UTC) (envelope-from nobody@FreeBSD.org) Received: from www.freebsd.org (www.freebsd.org [IPv6:2001:4f8:fff6::21]) by mx1.freebsd.org (Postfix) with ESMTP id 8664113C4AC for ; Wed, 17 Oct 2007 17:41:54 +0000 (UTC) (envelope-from nobody@FreeBSD.org) Received: from www.freebsd.org (localhost [127.0.0.1]) by www.freebsd.org (8.14.1/8.14.1) with ESMTP id l9HHfgJX070162 for ; Wed, 17 Oct 2007 17:41:42 GMT (envelope-from nobody@www.freebsd.org) Received: (from nobody@localhost) by www.freebsd.org (8.14.1/8.14.1/Submit) id l9HHfgdE070161; Wed, 17 Oct 2007 17:41:42 GMT (envelope-from nobody) Message-Id: <200710171741.l9HHfgdE070161@www.freebsd.org> Date: Wed, 17 Oct 2007 17:41:42 GMT From: Fabian Keil To: freebsd-gnats-submit@FreeBSD.org X-Send-Pr-Version: www-3.1 Cc: Subject: bin/117277: [PATCH] fetch's resume mode doesn't verify that it actually got partial content X-BeenThere: freebsd-bugs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Bug reports List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 17 Oct 2007 17:50:02 -0000 >Number: 117277 >Category: bin >Synopsis: [PATCH] fetch's resume mode doesn't verify that it actually got partial content >Confidential: no >Severity: non-critical >Priority: low >Responsible: freebsd-bugs >State: open >Quarter: >Keywords: >Date-Required: >Class: sw-bug >Submitter-Id: current-users >Arrival-Date: Wed Oct 17 17:50:01 UTC 2007 >Closed-Date: >Last-Modified: >Originator: Fabian Keil >Release: RELENG_7 >Organization: >Environment: FreeBSD TP51.local 7.0-PRERELEASE FreeBSD 7.0-PRERELEASE #1: Mon Oct 15 18:31:16 CEST 2007 fk@TP51.local:/usr/obj/usr/src/sys/THINKPAD i386 >Description: When resuming a file over HTTP, fetch(1) treats complete retransmits like partial content and appends the whole response to the already existing part that was fetched earlier. The problem shows if the server doesn't support Range requests or if the proxy strips out the Range header to prevent range mismatches (which could happen if the proxy modified the first response): fk@TP51 /tank/fetch-tests $fetch -r http://10.0.0.1/BrooksDavis.EuroBSDCon.2007.avi BrooksDavis.EuroBSDCon.2007.avi 30% of 92 MB 8440 kBps^C fetch: transfer interrupted fk@TP51 /tank/fetch-tests $fetch -r http://10.0.0.1/BrooksDavis.EuroBSDCon.2007.avi BrooksDavis.EuroBSDCon.2007.avi 100% of 92 MB 17 MBps fk@TP51 /tank/fetch-tests $ls -lh total 123649 -rw-r--r-- 1 fk wheel 121M Sep 25 21:01 BrooksDavis.EuroBSDCon.2007.avi >How-To-Repeat: Resume a file from a HTTP server that doesn't support Range requests or with a proxy that removes the Range header. >Fix: While fetch(3) doesn't pass the HTTP status code to fetch(1), the problem can be solved by checking the content offset. If the offset is zero, the response contains the whole file and the already existing part of it has to be overwritten. With the attached patch it works as expected: fk@TP51 /tank/fetch-tests $rm BrooksDavis.EuroBSDCon.2007.avi fk@TP51 /tank/fetch-tests $fetch -r http://10.0.0.1/BrooksDavis.EuroBSDCon.2007.avi BrooksDavis.EuroBSDCon.2007.avi 58% of 92 MB 15 MBps^C fetch: transfer interrupted fk@TP51 /tank/fetch-tests $fetch -r http://10.0.0.1/BrooksDavis.EuroBSDCon.2007.avi BrooksDavis.EuroBSDCon.2007.avi 100% of 92 MB 13 MBps fk@TP51 /tank/fetch-tests $ls -lh total 95081 -rw-r--r-- 1 fk wheel 93M Sep 25 21:01 BrooksDavis.EuroBSDCon.2007.avi Patch attached with submission follows: --- .zfs/snapshot/2007-10-15/usr.bin/fetch/fetch.c 2006-11-10 23:05:41.000000000 +0100 +++ usr.bin/fetch/fetch.c 2007-10-16 14:21:20.221581714 +0200 @@ -488,8 +488,11 @@ if (o_stdout) { /* output to stdout */ of = stdout; - } else if (r_flag && sb.st_size != -1) { - /* resume mode, local file exists */ + } else if (r_flag && sb.st_size != -1 && url->offset) { + /* + * resume mode, local file exists and we + * actually received partial content as requested + */ if (!F_flag && us.mtime && sb.st_mtime != us.mtime) { /* no match! have to refetch */ fclose(f); >Release-Note: >Audit-Trail: >Unformatted: