Date: Sat, 18 Jul 2009 23:11:58 -0700 (PDT) From: "Joe R. Jah" <jjah@cloud.ccsf.cc.ca.us> To: Andrew Brampton <brampton+freebsd@gmail.com> Cc: freebsd-questions@freebsd.org Subject: Re: OT: wget bug Message-ID: <20090718230601.F10250@cloud.ccsf.cc.ca.us> In-Reply-To: <d41814900907181009q1d4f4b62k9818607a65421419@mail.gmail.com> References: <20090717144049.C35992@cloud.ccsf.cc.ca.us> <d41814900907180452p29244911nd2570909e7274791@mail.gmail.com> <20090718093237.Y19472@cloud.ccsf.cc.ca.us> <d41814900907181009q1d4f4b62k9818607a65421419@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
This message is in MIME format. The first part should be readable text,
while the remaining parts are likely unreadable without MIME-aware tools.
--0-306400255-1247983918=:10250
Content-Type: TEXT/PLAIN; charset=UTF-8
Content-Transfer-Encoding: QUOTED-PRINTABLE
On Sat, 18 Jul 2009, Andrew Brampton wrote:
> Date: Sat, 18 Jul 2009 18:09:54 +0100
> From: Andrew Brampton <brampton+freebsd@gmail.com>
> To: Joe R. Jah <jjah@cloud.ccsf.cc.ca.us>
> Cc: freebsd-questions@freebsd.org
> Subject: Re: OT: wget bug
>
> 2009/7/18 Joe R. Jah <jjah@cloud.ccsf.cc.ca.us>:
> > Thank you Andrew. =C2=A0Yes the server is truly returning 401. =C2=A0I =
have already
> > reconfigured wget to download everything regardless of their timestamp,
> > but it's a waste of bandwidth, because most of the site is unchanged.
> >
> > Do you know of any workaround in wget, or an alternative tool to ONLY
> > download newer files by http?
> >
>
> Joe,
> There are two ways to check if the file has been changed. One, read
> the time the file was last changed, or two, read the file and compare
> it to a old copy. Wget was obviously trying to do option 1 but this is
> denied by the remote server. You most likely could get it to do option
> 2, however by doing so you are wasting bandwidth downloading unchanged
> files just to check if they had been changed.
>
> If you have control over the remote webserver, then the simplest way
> to solve this problem is to configure the webserver not to return 401
> when wget sends the If-Modified-Since HTTP header. A better solution,
> again assuming you have control of the remote server, is to use
> "rsync" as it is designed for this kind of task.
>
> If you don't have control over the remote server, then you are stuck
> with your current solution.
>
> Andrew
Thank you Andrew.
Regards,
Joe
--=20
_/ _/_/_/ _/ ____________ __o
_/ _/ _/ _/ ______________ _-\<,_
_/ _/ _/_/_/ _/ _/ ......(_)/ (_)
_/_/ oe _/ _/. _/_/ ah jjah@cloud.ccsf.cc.ca.us
--0-306400255-1247983918=:10250--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20090718230601.F10250>