Date: Thu, 7 Jun 2007 10:06:01 +0200 From: David Naylor <blackdragon@highveldmail.co.za> To: freebsd-hackers@freebsd.org Subject: fetch: Resume errors, help! Message-ID: <200706071006.12201.blackdragon@highveldmail.co.za>
next in thread | raw e-mail | index | archive | help
--nextPart3011027.MtEEHuugcO
Content-Type: text/plain;
charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
Content-Disposition: inline
Hi,
I have been having a very annoying problem with fetch and the ports distfil=
es=20
fetching. Living in South Africa internet access is not what it could be, =
as=20
such I have to use a proxy to access the internet. On to the problem:
When fetch tries to resume from an ftp site that does not support resuming =
(or=20
fetch lacks the ability in these specific cases, or the proxy + fetch=20
faulted) it appends to the file but restarts the transfer from 0. This=20
results in a file that is too big but does contain a complete copy of the=20
file. A solution is simple:
es =3D Expected size of the file
as =3D Actual size of the file
f =3D File name
# dd if=3Df of=3Df~ bs=3D$((as - es)) skip=3D1
# mv f~ f
This is very inconvenient and a problem when trying to download files in=20
batch. One gets interrupted and stops everything else. As such I looked=20
into fetch.c and tried to fix the problem. See below for my patch. A=20
problem is that how to handle the -R flag (no overwrite file flag). I have=
=20
three solutions:
1) Ignore flag and overwrite the file anyway
2) Do not overwrite and abort
3) Specify above behaviour with an additional flag
Given that the ports and having many mirrors it is possible to find a site=
=20
that does support resuming, and thus will save in bandwidth (important for=
=20
me). However it is possible that there are no alternatives or all sites do=
=20
not support resume. =20
In the patch the commented out section is solution 2, the implemented versi=
on=20
is solution 1. =20
Any advice will be welcome, once a satisfactory solution has been achieved =
I=20
will submit a PR with the patch? =20
Thank you
David
P.S. I've been using FreeBSD for a year or two however I am very new at=20
writing patches for it, this will be my first so all help and mentoring wil=
l=20
be very welcome. =20
Patch file:
=2D-- usr.bin/fetch/fetch.c.orig 2007-06-07 09:17:18.000000000 +0200
+++ usr.bin/fetch/fetch.c 2007-06-07 09:18:39.000000000 +0200
@@ -510,8 +510,36 @@
(intmax_t)sb.st_size, (intmax_t)us.size);
goto failure;
}
+// /* check fetched offset is as expected */
+// if (url->offset !=3D sb.st_size) {
+// if (R_flag) {
+// /* unable to resume */
+// warnx("%s: unexpected offset, "
+// "expected %jd bytes, got %jd "
+// "bytes", (intmax_t)sb.st_size,=20
+// (intmax_t)url->offset);
+// goto failure_keep;
+// } else if (url->offset !=3D 0)
+// /* if not we need to restart from scratch */
+// goto restart;
+// }
+// if ((of =3D fopen(path, "a")) =3D=3D NULL) {
+// warn("%s: fopen()", path);
+// goto failure;
+// }
+// /* check feteched offset is either as expected or 0 */
+ if (url->offset !=3D sb.st_size || url->offset !=3D 0)
+ /* if not we need to restart from scratch */
+ goto restart;
+ /*
+ * NOTE: R_flag is ignored and local file is over=20
+ * written *not* deleted as it is the only way
+ * to resume transfer of this file from this=20
+ * URL.
+ */
/* we got it, open local file */
=2D if ((of =3D fopen(path, "a")) =3D=3D NULL) {
+ if ((of =3D url->offset =3D=3D 0 ? fopen(path, "w") :=20
+ fopen(path, "a")) =3D=3D NULL) {
warn("%s: fopen()", path);
goto failure;
}
@@ -536,6 +564,7 @@
goto success;
}
=20
+restart:
if (of =3D=3D NULL) {
/*
* We don't yet have an output file; either this is a
--nextPart3011027.MtEEHuugcO
Content-Type: application/pgp-signature
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.5 (GNU/Linux)
iD8DBQBGZ7x0yqzxLKpyZI8RAl/EAJ0Rqk0h1a4yBPnbG9mOwLx6nA+qwgCgs2DH
GSWFmYQd8+84NdIdlYJMvJA=
=lCsd
-----END PGP SIGNATURE-----
--nextPart3011027.MtEEHuugcO--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?200706071006.12201.blackdragon>