Date: Wed, 1 Apr 2015 00:33:20 +0200 From: Baptiste Daroussin <bapt@FreeBSD.org> To: sg-ball@laposte.net Cc: freebsd-pkg@freebsd.org Subject: Re: Problem with pkg behind a chunking proxy Message-ID: <20150331223320.GJ30115@ivaldir.etoilebsd.net> In-Reply-To: <1476070491.31317997.1427838521190.JavaMail.zimbra@laposte.net> References: <902242756.31277524.1427836544530.JavaMail.zimbra@laposte.net> <1476070491.31317997.1427838521190.JavaMail.zimbra@laposte.net>
next in thread | previous in thread | raw e-mail | index | archive | help
--OX7PPUk8qMPP4++R Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Tue, Mar 31, 2015 at 11:48:41PM +0200, sg-ball@laposte.net wrote: > Some times ago I tried to install a FreeBSD 10.1 Release at work, where i= nternal network is isolated from internet through a corporate proxy. And I = could not make pkg work correctly. >=20 > I finally confirmed the problem was caused by the proxy using Transfer-En= coding: chunked instead of giving in first place a ContentLength header by = using a python script acting as a proxy and forcing either chunked or not c= hunked response, and filed a bug report : https://bugs.freebsd.org/bugzilla= /show_bug.cgi?id=3D198772 (got no followup till now ...) >=20 > I could dive in pkg 1.4.12 source code (thanks to the port) and found tha= t the real cause was that in libpkg/fetch.c function pkg_fetch_file_to_fd u= sed > remote =3D fetchXGet(u, &st, sbuf_data(fetchOpts)); > to get remote file meta data in st (notably the file size) but did not te= st that st.size was not -1 (what it is when using Transfer-Encoding: chunke= d), so the following loop getting data up to st.size simply did not run. >=20 > So I could make some minimal changes to this file (fetch.c) to accept st.= size =3D=3D -1. Currently it simply writes "Fetching file: 0%" until the fi= le is fully downloaded and then write full line with "100% size speed time"= =2E Of course it runs unchanged if it gets the file size through a ContentL= ength header. >=20 > It would certainly be better to trace the size all along the download, bu= t it would require changes to libpkg/event.c as well, and I prefered to lim= it the changes for now. For anyone interested, I join the patch to this mai= l. >=20 > My questions now are : > - is this the correct way to propose a patch ? > - would it be better to propose the patch for pkg-devel ? > - or should I use directly github a submit a pull request ? > - or the feature must first be discussed here ? > - or ... >=20 > I think this feature could be interesting since Transfer-Encoding: chunke= d is valid as HTTP 1.1 but I'm not used to submitting patches even if I've = been using FreeBSD since release 3.x First thank you for the patch! fixing this is on my TODO like forever. Second sorry to not have followed up on the bug tracker, I am very busy in = lot of areas. So usually the feature should be discussed in pkg@FreeBSD.org or in #pkgng (freenode) or on github issue, I have to confess that I do track more often github issue tracker than freebsd's bugzilla for pkg related issues. (which= is bad I should more track both). Concerning your feature it is very welcome :) usually we do like pull reque= st on github. Note that you are just in time for pkg 1.5 which will be released s= oon :) The preferred way is usually pull request, but I'm flexible if the submitte= r is anti github then I can just grab patches. That part of the code hasn't been modified since pkg 1.4 so the patch should apply just fine. Note that the bootstrap /usr/sbin/pkg in base suffer the same problems so t= hat patch will also apply there (maybe needed to be tuned a bit) So as a conclusion first make a pull request on github, then once it is well tested enough you can provide a patch for base (or I'll do) if so open a ti= cket in the bug tracker for base and do not hesitate to harass me :) Thanks again, Bapt > --- old/fetch.c 2015-02-13 20:34:33.000000000 +0100 > +++ new/fetch.c 2015-03-31 19:41:25.000000000 +0200 > @@ -618,30 +618,50 @@ > =20 > pkg_emit_fetch_begin(url); > pkg_emit_progress_start(NULL); > - while (done < sz) { > - int to_read =3D MIN(sizeof(buf), sz - done); > + if (sz > 0) { > + while (done < sz) { > + int to_read =3D MIN(sizeof(buf), sz - done); > + > + pkg_debug(1, "Reading status: want read %d over %d, %d already done", > + to_read, sz, done); > + if ((r =3D fread(buf, 1, to_read, remote)) < 1) > + break; > + > + if (write(dest, buf, r) !=3D r) { > + pkg_emit_errno("write", ""); > + retcode =3D EPKG_FATAL; > + goto cleanup; > + } > =20 > - pkg_debug(1, "Reading status: want read %d over %d, %d already done", > - to_read, sz, done); > - if ((r =3D fread(buf, 1, to_read, remote)) < 1) > - break; > + done +=3D r; > + pkg_debug(1, "Read status: %d over %d", done, sz); > =20 > - if (write(dest, buf, r) !=3D r) { > - pkg_emit_errno("write", ""); > + pkg_emit_progress_tick(done, sz); > + } > + > + if (done < sz) { > + pkg_emit_error("An error occurred while fetching package"); > retcode =3D EPKG_FATAL; > goto cleanup; > } > - > - done +=3D r; > - pkg_debug(1, "Read status: %d over %d", done, sz); > - > - pkg_emit_progress_tick(done, sz); > - } > - > - if (done < sz) { > - pkg_emit_error("An error occurred while fetching package"); > - retcode =3D EPKG_FATAL; > - goto cleanup; > + } > + else { > + while ((r =3D fread(buf, 1, sizeof(buf), remote)) > 0) { > + if (write(dest, buf, r) !=3D r) { > + pkg_emit_errno("write", ""); > + retcode =3D EPKG_FATAL; > + goto cleanup; > + } > + done +=3D r; > + } > + if (r !=3D 0) { > + pkg_emit_error("An error occurred while fetching package"); > + retcode =3D EPKG_FATAL; > + goto cleanup; > + } > + else { > + pkg_emit_progress_tick(done, done); > + } > } > pkg_emit_fetch_finished(url); > =20 > _______________________________________________ > freebsd-pkg@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-pkg > To unsubscribe, send any mail to "freebsd-pkg-unsubscribe@freebsd.org" --OX7PPUk8qMPP4++R Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v1 iEYEARECAAYFAlUbILAACgkQ8kTtMUmk6ExHFgCgnml9fSE/3GmZZoYDBhnj3wQL fYwAoI7XCC6vfE/nownZ9mnxc6MSmVYw =Grdl -----END PGP SIGNATURE----- --OX7PPUk8qMPP4++R--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20150331223320.GJ30115>