From owner-freebsd-questions@FreeBSD.ORG Sat Jul 18 12:19:25 2009 Return-Path: Delivered-To: freebsd-questions@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id A13FA106564A for ; Sat, 18 Jul 2009 12:19:25 +0000 (UTC) (envelope-from brampton@gmail.com) Received: from mail-ew0-f220.google.com (mail-ew0-f220.google.com [209.85.219.220]) by mx1.freebsd.org (Postfix) with ESMTP id 3234C8FC0C for ; Sat, 18 Jul 2009 12:19:25 +0000 (UTC) (envelope-from brampton@gmail.com) Received: by ewy20 with SMTP id 20so1168741ewy.43 for ; Sat, 18 Jul 2009 05:19:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:sender:received:in-reply-to :references:date:x-google-sender-auth:message-id:subject:from:to:cc :content-type:content-transfer-encoding; bh=PmylwwymYSplsXNPXRxeCNxJMV7yJid3Z/4K3EtxIrQ=; b=pY5f+uFrRTkcqiIatI9D21wFFXjn1Y+OdkTyFRQWd3XwTl4NFp2W967Oigc8JChKEs kChZvhaZNMLVG/fMJu4xRXiEF7PgLr5p8PTeeQIH32B0F/lj1E2K6gscmc0OuR6ZGk/V jrVWrH84OL3tsYo5SJNBCQRTFrj/2soMlcRmQ= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type :content-transfer-encoding; b=BCQcVJ+6HzEh6Em2LjYUC/XmHg6agRcc1btIDr1hw63Am4Hg4JcCSdehrZq0mihjns BstwFd+7Bjh4PLbxHfgaXZAdhzgmDXiWbBi9doE5uvRF4lasaM/qykXAq37U1CkNqnFe nTYlrs6x9Yx3SsIExuqYcLD3awIqnNrrnNDrs= MIME-Version: 1.0 Sender: brampton@gmail.com Received: by 10.216.11.138 with SMTP id 10mr591843wex.51.1247917927493; Sat, 18 Jul 2009 04:52:07 -0700 (PDT) In-Reply-To: <20090717144049.C35992@cloud.ccsf.cc.ca.us> References: <20090717144049.C35992@cloud.ccsf.cc.ca.us> Date: Sat, 18 Jul 2009 12:52:07 +0100 X-Google-Sender-Auth: c8ce5a5acce3042f Message-ID: From: Andrew Brampton To: "Joe R. Jah" Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit Cc: freebsd-questions@freebsd.org Subject: Re: OT: wget bug X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 18 Jul 2009 12:19:25 -0000 2009/7/17 Joe R. Jah : > > Hello all, > > I want to wget a site at regular intervals and only get the updated pages, > so I use the this wget command line: > > wget -b -m -nH http://host.domain/Directory/file.html > > It works fine on the first try, but it fails on subsequent tries with the > following error message: > > --8<-- > Connecting to host.domain ... connected. > HTTP request sent, awaiting response... 401 Unauthorized > Authorization failed. > --8<-- This to me seems like the remote server is replying with 401. Perhaps wget is sending the If-Modified-Since HTTP header, and the remote server does not support this. I would confirm this by running tcpdump (or wireshark) to sniff the traffic and see what the remote server is replying with. If the remote server is truly returning 401, then you might either need to use an alternative tool, or configure wget differently. Hope this helps Andrew