From owner-freebsd-questions@FreeBSD.ORG Sat Jul 18 17:09:55 2009 Return-Path: Delivered-To: freebsd-questions@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 9ADDE1065670 for ; Sat, 18 Jul 2009 17:09:55 +0000 (UTC) (envelope-from brampton@gmail.com) Received: from mail-ew0-f220.google.com (mail-ew0-f220.google.com [209.85.219.220]) by mx1.freebsd.org (Postfix) with ESMTP id 1C1838FC28 for ; Sat, 18 Jul 2009 17:09:54 +0000 (UTC) (envelope-from brampton@gmail.com) Received: by ewy20 with SMTP id 20so1252359ewy.43 for ; Sat, 18 Jul 2009 10:09:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:sender:received:in-reply-to :references:date:x-google-sender-auth:message-id:subject:from:to:cc :content-type:content-transfer-encoding; bh=m/sEar1du8TA4EKywMV86FzlK0f460hzQ6tbuc02Zmo=; b=SePZSs/Pjn8aVTiH1Wx2bd22ABsLG1bLlfDoyLFmySfy54cHcTJIQZ8tcfka4bDNeg 7p6s47TQgVJGkaywLrjbXhI2PezF/SA8QEIdhpnX2EoUGyrP4tou5WOP/NpieuWxFj+S XUz5rh5r6tHEWYhA+xa3tC88Nm2D2HpbQmwE0= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type :content-transfer-encoding; b=QTrulIzWJIfbiPpA0HPbgax6xeauLXxOU96L9qZ7qUWNm9R3LLA+zAPabvnsmNG5iV NNtfRtkn5pIa8g0Jv3HbP9U3YgAzbV+INAMt6COtclKU9utdZihm0Q8DV915EsQ0Q8Bm B5dNd9gzGT8tg6qhF6aoArfs5lgurAcRxBQOw= MIME-Version: 1.0 Sender: brampton@gmail.com Received: by 10.216.25.209 with SMTP id z59mr637976wez.204.1247936994183; Sat, 18 Jul 2009 10:09:54 -0700 (PDT) In-Reply-To: <20090718093237.Y19472@cloud.ccsf.cc.ca.us> References: <20090717144049.C35992@cloud.ccsf.cc.ca.us> <20090718093237.Y19472@cloud.ccsf.cc.ca.us> Date: Sat, 18 Jul 2009 18:09:54 +0100 X-Google-Sender-Auth: 779c23ffa7537040 Message-ID: From: Andrew Brampton To: "Joe R. Jah" Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Cc: freebsd-questions@freebsd.org Subject: Re: OT: wget bug X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 18 Jul 2009 17:09:55 -0000 2009/7/18 Joe R. Jah : > Thank you Andrew. =C2=A0Yes the server is truly returning 401. =C2=A0I ha= ve already > reconfigured wget to download everything regardless of their timestamp, > but it's a waste of bandwidth, because most of the site is unchanged. > > Do you know of any workaround in wget, or an alternative tool to ONLY > download newer files by http? > Joe, There are two ways to check if the file has been changed. One, read the time the file was last changed, or two, read the file and compare it to a old copy. Wget was obviously trying to do option 1 but this is denied by the remote server. You most likely could get it to do option 2, however by doing so you are wasting bandwidth downloading unchanged files just to check if they had been changed. If you have control over the remote webserver, then the simplest way to solve this problem is to configure the webserver not to return 401 when wget sends the If-Modified-Since HTTP header. A better solution, again assuming you have control of the remote server, is to use "rsync" as it is designed for this kind of task. If you don't have control over the remote server, then you are stuck with your current solution. Andrew