From owner-freebsd-pkg@FreeBSD.ORG Tue Mar 31 22:33:26 2015 Return-Path: Delivered-To: freebsd-pkg@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id C64C0F81 for ; Tue, 31 Mar 2015 22:33:26 +0000 (UTC) Received: from mail-yk0-x22f.google.com (mail-yk0-x22f.google.com [IPv6:2607:f8b0:4002:c07::22f]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 78E64279 for ; Tue, 31 Mar 2015 22:33:26 +0000 (UTC) Received: by ykfa3 with SMTP id a3so5613938ykf.0 for ; Tue, 31 Mar 2015 15:33:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=sender:date:from:to:cc:subject:message-id:references:mime-version :content-type:content-disposition:in-reply-to:user-agent; bh=X7PvpwKzmT2q0fIdOAi+Ksx2LxlmriM6C8LxGaDnY5g=; b=ZMPp/wG0ImtE/tYSNJj6LNLtXHLBOOFtgU8Cmiam+iHi8pSRceS/rry2ZA1kSPaq1l 8lyZOzDiMK0RPqkCdAjrUJJFyqGlrvzmNH8ubMDs/yLd+1XEOx1Ig9fHJC9uv8PeuF1S F8J9wW9SBiydB3LqO9pg2Zzp6pDhBuMP5zJAtIyCyMRIqI9F2Jkt0WfdqGsELM88Quw5 CGUBZE0P0qsR9bRdYvmEVuNQQg0Oe7khPuSOUNXM0fXAU7wvt/RVnLg33hNbjJEXoF/7 DlgzEP4CHha0pBI4ubdikFpyYh3qiE1j4v3Yfs13ByjW3R3mho04QdvjG2MVhgSmcmpv /PqA== X-Received: by 10.52.138.17 with SMTP id qm17mr16016573vdb.59.1427841205599; Tue, 31 Mar 2015 15:33:25 -0700 (PDT) Received: from ivaldir.etoilebsd.net ([2001:41d0:8:db4c::1]) by mx.google.com with ESMTPSA id cf6sm2601480vdc.15.2015.03.31.15.33.23 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 31 Mar 2015 15:33:24 -0700 (PDT) Sender: Baptiste Daroussin Date: Wed, 1 Apr 2015 00:33:20 +0200 From: Baptiste Daroussin To: sg-ball@laposte.net Subject: Re: Problem with pkg behind a chunking proxy Message-ID: <20150331223320.GJ30115@ivaldir.etoilebsd.net> References: <902242756.31277524.1427836544530.JavaMail.zimbra@laposte.net> <1476070491.31317997.1427838521190.JavaMail.zimbra@laposte.net> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="OX7PPUk8qMPP4++R" Content-Disposition: inline In-Reply-To: <1476070491.31317997.1427838521190.JavaMail.zimbra@laposte.net> User-Agent: Mutt/1.5.23 (2014-03-12) Cc: freebsd-pkg@freebsd.org X-BeenThere: freebsd-pkg@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Binary package management and package tools discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 31 Mar 2015 22:33:26 -0000 --OX7PPUk8qMPP4++R Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Tue, Mar 31, 2015 at 11:48:41PM +0200, sg-ball@laposte.net wrote: > Some times ago I tried to install a FreeBSD 10.1 Release at work, where i= nternal network is isolated from internet through a corporate proxy. And I = could not make pkg work correctly. >=20 > I finally confirmed the problem was caused by the proxy using Transfer-En= coding: chunked instead of giving in first place a ContentLength header by = using a python script acting as a proxy and forcing either chunked or not c= hunked response, and filed a bug report : https://bugs.freebsd.org/bugzilla= /show_bug.cgi?id=3D198772 (got no followup till now ...) >=20 > I could dive in pkg 1.4.12 source code (thanks to the port) and found tha= t the real cause was that in libpkg/fetch.c function pkg_fetch_file_to_fd u= sed > remote =3D fetchXGet(u, &st, sbuf_data(fetchOpts)); > to get remote file meta data in st (notably the file size) but did not te= st that st.size was not -1 (what it is when using Transfer-Encoding: chunke= d), so the following loop getting data up to st.size simply did not run. >=20 > So I could make some minimal changes to this file (fetch.c) to accept st.= size =3D=3D -1. Currently it simply writes "Fetching file: 0%" until the fi= le is fully downloaded and then write full line with "100% size speed time"= =2E Of course it runs unchanged if it gets the file size through a ContentL= ength header. >=20 > It would certainly be better to trace the size all along the download, bu= t it would require changes to libpkg/event.c as well, and I prefered to lim= it the changes for now. For anyone interested, I join the patch to this mai= l. >=20 > My questions now are : > - is this the correct way to propose a patch ? > - would it be better to propose the patch for pkg-devel ? > - or should I use directly github a submit a pull request ? > - or the feature must first be discussed here ? > - or ... >=20 > I think this feature could be interesting since Transfer-Encoding: chunke= d is valid as HTTP 1.1 but I'm not used to submitting patches even if I've = been using FreeBSD since release 3.x First thank you for the patch! fixing this is on my TODO like forever. Second sorry to not have followed up on the bug tracker, I am very busy in = lot of areas. So usually the feature should be discussed in pkg@FreeBSD.org or in #pkgng (freenode) or on github issue, I have to confess that I do track more often github issue tracker than freebsd's bugzilla for pkg related issues. (which= is bad I should more track both). Concerning your feature it is very welcome :) usually we do like pull reque= st on github. Note that you are just in time for pkg 1.5 which will be released s= oon :) The preferred way is usually pull request, but I'm flexible if the submitte= r is anti github then I can just grab patches. That part of the code hasn't been modified since pkg 1.4 so the patch should apply just fine. Note that the bootstrap /usr/sbin/pkg in base suffer the same problems so t= hat patch will also apply there (maybe needed to be tuned a bit) So as a conclusion first make a pull request on github, then once it is well tested enough you can provide a patch for base (or I'll do) if so open a ti= cket in the bug tracker for base and do not hesitate to harass me :) Thanks again, Bapt > --- old/fetch.c 2015-02-13 20:34:33.000000000 +0100 > +++ new/fetch.c 2015-03-31 19:41:25.000000000 +0200 > @@ -618,30 +618,50 @@ > =20 > pkg_emit_fetch_begin(url); > pkg_emit_progress_start(NULL); > - while (done < sz) { > - int to_read =3D MIN(sizeof(buf), sz - done); > + if (sz > 0) { > + while (done < sz) { > + int to_read =3D MIN(sizeof(buf), sz - done); > + > + pkg_debug(1, "Reading status: want read %d over %d, %d already done", > + to_read, sz, done); > + if ((r =3D fread(buf, 1, to_read, remote)) < 1) > + break; > + > + if (write(dest, buf, r) !=3D r) { > + pkg_emit_errno("write", ""); > + retcode =3D EPKG_FATAL; > + goto cleanup; > + } > =20 > - pkg_debug(1, "Reading status: want read %d over %d, %d already done", > - to_read, sz, done); > - if ((r =3D fread(buf, 1, to_read, remote)) < 1) > - break; > + done +=3D r; > + pkg_debug(1, "Read status: %d over %d", done, sz); > =20 > - if (write(dest, buf, r) !=3D r) { > - pkg_emit_errno("write", ""); > + pkg_emit_progress_tick(done, sz); > + } > + > + if (done < sz) { > + pkg_emit_error("An error occurred while fetching package"); > retcode =3D EPKG_FATAL; > goto cleanup; > } > - > - done +=3D r; > - pkg_debug(1, "Read status: %d over %d", done, sz); > - > - pkg_emit_progress_tick(done, sz); > - } > - > - if (done < sz) { > - pkg_emit_error("An error occurred while fetching package"); > - retcode =3D EPKG_FATAL; > - goto cleanup; > + } > + else { > + while ((r =3D fread(buf, 1, sizeof(buf), remote)) > 0) { > + if (write(dest, buf, r) !=3D r) { > + pkg_emit_errno("write", ""); > + retcode =3D EPKG_FATAL; > + goto cleanup; > + } > + done +=3D r; > + } > + if (r !=3D 0) { > + pkg_emit_error("An error occurred while fetching package"); > + retcode =3D EPKG_FATAL; > + goto cleanup; > + } > + else { > + pkg_emit_progress_tick(done, done); > + } > } > pkg_emit_fetch_finished(url); > =20 > _______________________________________________ > freebsd-pkg@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-pkg > To unsubscribe, send any mail to "freebsd-pkg-unsubscribe@freebsd.org" --OX7PPUk8qMPP4++R Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v1 iEYEARECAAYFAlUbILAACgkQ8kTtMUmk6ExHFgCgnml9fSE/3GmZZoYDBhnj3wQL fYwAoI7XCC6vfE/nownZ9mnxc6MSmVYw =Grdl -----END PGP SIGNATURE----- --OX7PPUk8qMPP4++R--