From owner-freebsd-pkg@FreeBSD.ORG Tue Mar 31 22:12:03 2015 Return-Path: Delivered-To: freebsd-pkg@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 984BE70F for ; Tue, 31 Mar 2015 22:12:03 +0000 (UTC) Received: from smtp.laposte.net (smtpoutz13.laposte.net [194.117.213.172]) (using TLSv1.1 with cipher ECDHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 455F6F8D for ; Tue, 31 Mar 2015 22:12:02 +0000 (UTC) Received: from smtp.laposte.net (localhost [127.0.0.1]) by lpn-prd-vrout001 (Postfix) with ESMTP id 6E3B74E4103C for ; Tue, 31 Mar 2015 23:48:41 +0200 (CEST) Received: from smtp.laposte.net (localhost [127.0.0.1]) by lpn-prd-vrout001 (Postfix) with ESMTP id 5192D4E4104E for ; Tue, 31 Mar 2015 23:48:41 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=laposte.net; s=mail1; t=1427838521; bh=8KL2Y1uumIK2CnDqyHYuCYp09bDofH2ygBRnWPrS9H4=; h=Date:From:To:In-Reply-To:Subject; b=XEV+YZdLWA2sHatNH7kEP5hJOOiSrG7pE352tYeALjxjZ6v4FsNeUPfDDwgMY27Pc +Zx7MfPwVL11DvzGIL4Qs86ywc+ez3K5Q6KigGBelrfUA6SOKbYKlElBghpWtRCMiT oh/tLyW7xrKMcty/JLGUdfmduON25XgTg3SizeVEbK0OmytTt+CcYiVbIOyo/tqzVk bAndleJOrONcWTY6LSLWOX7RClY2BGf16d3xRPJRyFKpgk8Gg3yPd3WDT18C6vW8cs +VqWcH+AIRfD1QZGxp4hxHhHkmeHFFWgH9efVq5SB2z0V0hz19yY4vt67xgvMFNIcD jF/dmaqLRv+Sg== Received: from lpn-prd-mstr033.laposte (lpn-prd-mstr033.laposte [10.128.59.34]) by lpn-prd-vrout001 (Postfix) with ESMTP id 3F1154E4103C; Tue, 31 Mar 2015 23:48:41 +0200 (CEST) Date: Tue, 31 Mar 2015 23:48:41 +0200 (CEST) From: sg-ball@laposte.net To: freebsd-pkg@freebsd.org Message-ID: <1476070491.31317997.1427838521190.JavaMail.zimbra@laposte.net> In-Reply-To: <902242756.31277524.1427836544530.JavaMail.zimbra@laposte.net> Subject: Problem with pkg behind a chunking proxy MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="----=_Part_31317995_1432316153.1427838521189" X-Originating-IP: [78.115.104.177] X-Mailer: Zimbra 8.0.6_GA_5922 (ZimbraWebClient - FF36 ([unknown])/La Poste) Thread-Topic: Problem with pkg behind a chunking proxy Thread-Index: wNkeSzg4yYWCgb4Wgj0UFXRVBYrR7Q== X-VR-SrcIP: [78.115.104.177] X-VR-FullState: 0 X-VR-Score: 0 X-VR-Cause-1: gggruggvucftvghtrhhoucdtuddrfeektddrudekgddtkecutefuodetggdotefrucfrrhhofhhilhgv X-VR-Cause-2: mecunfetrffquffvgfenuceurghilhhouhhtmecuhedttdenucenucfjughrpeffhffvkfgjufggtghi X-VR-Cause-3: ofhtsehmtdgtsgertdejnecuhfhrohhmpehsghdqsggrlhhlsehlrghpohhsthgvrdhnvghtnecuffho X-VR-Cause-4: mhgrihhnpehfrhgvvggsshgurdhorhhgnecurfgrrhgrmhepmhhouggvpehsmhhtphhouhhtpdhhvghl X-VR-Cause-5: oheplhhpnhdqphhrugdqmhhsthhrtdeffedrphhrohhsohguihgvpdhinhgvthepuddtrdduvdekrdeh X-VR-Cause-6: ledrfeegpdhmrghilhhfrhhomhepshhgqdgsrghllheslhgrphhoshhtvgdrnhgvthdprhgtphhtthho X-VR-Cause-7: pehsghdqsggrlhhlsehlrghpohhsthgvrdhnvght X-VR-AvState: No X-VR-State: 0 X-BeenThere: freebsd-pkg@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Binary package management and package tools discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 31 Mar 2015 22:12:03 -0000 ------=_Part_31317995_1432316153.1427838521189 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit Some times ago I tried to install a FreeBSD 10.1 Release at work, where internal network is isolated from internet through a corporate proxy. And I could not make pkg work correctly. I finally confirmed the problem was caused by the proxy using Transfer-Encoding: chunked instead of giving in first place a ContentLength header by using a python script acting as a proxy and forcing either chunked or not chunked response, and filed a bug report : https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=198772 (got no followup till now ...) I could dive in pkg 1.4.12 source code (thanks to the port) and found that the real cause was that in libpkg/fetch.c function pkg_fetch_file_to_fd used remote = fetchXGet(u, &st, sbuf_data(fetchOpts)); to get remote file meta data in st (notably the file size) but did not test that st.size was not -1 (what it is when using Transfer-Encoding: chunked), so the following loop getting data up to st.size simply did not run. So I could make some minimal changes to this file (fetch.c) to accept st.size == -1. Currently it simply writes "Fetching file: 0%" until the file is fully downloaded and then write full line with "100% size speed time". Of course it runs unchanged if it gets the file size through a ContentLength header. It would certainly be better to trace the size all along the download, but it would require changes to libpkg/event.c as well, and I prefered to limit the changes for now. For anyone interested, I join the patch to this mail. My questions now are : - is this the correct way to propose a patch ? - would it be better to propose the patch for pkg-devel ? - or should I use directly github a submit a pull request ? - or the feature must first be discussed here ? - or ... I think this feature could be interesting since Transfer-Encoding: chunked is valid as HTTP 1.1 but I'm not used to submitting patches even if I've been using FreeBSD since release 3.x ------=_Part_31317995_1432316153.1427838521189 Content-Type: text/x-patch; name=pkg_1_4_14_libpkg_fetch.patch Content-Disposition: attachment; filename=pkg_1_4_14_libpkg_fetch.patch Content-Transfer-Encoding: base64 LS0tIG9sZC9mZXRjaC5jCTIwMTUtMDItMTMgMjA6MzQ6MzMuMDAwMDAwMDAwICswMTAwCisrKyBu ZXcvZmV0Y2guYwkyMDE1LTAzLTMxIDE5OjQxOjI1LjAwMDAwMDAwMCArMDIwMApAQCAtNjE4LDMw ICs2MTgsNTAgQEAKIAogCXBrZ19lbWl0X2ZldGNoX2JlZ2luKHVybCk7CiAJcGtnX2VtaXRfcHJv Z3Jlc3Nfc3RhcnQoTlVMTCk7Ci0Jd2hpbGUgKGRvbmUgPCBzeikgewotCQlpbnQgdG9fcmVhZCA9 IE1JTihzaXplb2YoYnVmKSwgc3ogLSBkb25lKTsKKyAgICAgICAgaWYgKHN6ID4gMCkgeworCQl3 aGlsZSAoZG9uZSA8IHN6KSB7CisJCQlpbnQgdG9fcmVhZCA9IE1JTihzaXplb2YoYnVmKSwgc3og LSBkb25lKTsKKworCQkJcGtnX2RlYnVnKDEsICJSZWFkaW5nIHN0YXR1czogd2FudCByZWFkICVk IG92ZXIgJWQsICVkIGFscmVhZHkgZG9uZSIsCisJCQkJdG9fcmVhZCwgc3osIGRvbmUpOworCQkJ aWYgKChyID0gZnJlYWQoYnVmLCAxLCB0b19yZWFkLCByZW1vdGUpKSA8IDEpCisJCQkJYnJlYWs7 CisKKwkJCWlmICh3cml0ZShkZXN0LCBidWYsIHIpICE9IHIpIHsKKwkJCQlwa2dfZW1pdF9lcnJu bygid3JpdGUiLCAiIik7CisJCQkJcmV0Y29kZSA9IEVQS0dfRkFUQUw7CisJCQkJZ290byBjbGVh bnVwOworCQkJfQogCi0JCXBrZ19kZWJ1ZygxLCAiUmVhZGluZyBzdGF0dXM6IHdhbnQgcmVhZCAl ZCBvdmVyICVkLCAlZCBhbHJlYWR5IGRvbmUiLAotCQkJdG9fcmVhZCwgc3osIGRvbmUpOwotCQlp ZiAoKHIgPSBmcmVhZChidWYsIDEsIHRvX3JlYWQsIHJlbW90ZSkpIDwgMSkKLQkJCWJyZWFrOwor CQkJZG9uZSArPSByOworCQkJcGtnX2RlYnVnKDEsICJSZWFkIHN0YXR1czogJWQgb3ZlciAlZCIs IGRvbmUsIHN6KTsKIAotCQlpZiAod3JpdGUoZGVzdCwgYnVmLCByKSAhPSByKSB7Ci0JCQlwa2df ZW1pdF9lcnJubygid3JpdGUiLCAiIik7CisJCQlwa2dfZW1pdF9wcm9ncmVzc190aWNrKGRvbmUs IHN6KTsKKwkJfQorCisJCWlmIChkb25lIDwgc3opIHsKKwkJCXBrZ19lbWl0X2Vycm9yKCJBbiBl cnJvciBvY2N1cnJlZCB3aGlsZSBmZXRjaGluZyBwYWNrYWdlIik7CiAJCQlyZXRjb2RlID0gRVBL R19GQVRBTDsKIAkJCWdvdG8gY2xlYW51cDsKIAkJfQotCi0JCWRvbmUgKz0gcjsKLQkJcGtnX2Rl YnVnKDEsICJSZWFkIHN0YXR1czogJWQgb3ZlciAlZCIsIGRvbmUsIHN6KTsKLQotCQlwa2dfZW1p dF9wcm9ncmVzc190aWNrKGRvbmUsIHN6KTsKLQl9Ci0KLQlpZiAoZG9uZSA8IHN6KSB7Ci0JCXBr Z19lbWl0X2Vycm9yKCJBbiBlcnJvciBvY2N1cnJlZCB3aGlsZSBmZXRjaGluZyBwYWNrYWdlIik7 Ci0JCXJldGNvZGUgPSBFUEtHX0ZBVEFMOwotCQlnb3RvIGNsZWFudXA7CisgICAgICAgIH0KKwll bHNlIHsKKwkJd2hpbGUgKChyID0gZnJlYWQoYnVmLCAxLCBzaXplb2YoYnVmKSwgcmVtb3RlKSkg PiAwKSB7CisJCQlpZiAod3JpdGUoZGVzdCwgYnVmLCByKSAhPSByKSB7CisJCQkJcGtnX2VtaXRf ZXJybm8oIndyaXRlIiwgIiIpOworCQkJCXJldGNvZGUgPSBFUEtHX0ZBVEFMOworCQkJCWdvdG8g Y2xlYW51cDsKKwkJCX0KKwkJCWRvbmUgKz0gcjsKKwkJfQorCQlpZiAociAhPSAwKSB7CisJCQlw a2dfZW1pdF9lcnJvcigiQW4gZXJyb3Igb2NjdXJyZWQgd2hpbGUgZmV0Y2hpbmcgcGFja2FnZSIp OworCQkJcmV0Y29kZSA9IEVQS0dfRkFUQUw7CisJCQlnb3RvIGNsZWFudXA7CisJCX0KKwkJZWxz ZSB7CisJCQlwa2dfZW1pdF9wcm9ncmVzc190aWNrKGRvbmUsIGRvbmUpOworCQl9CiAJfQogCXBr Z19lbWl0X2ZldGNoX2ZpbmlzaGVkKHVybCk7CiAK ------=_Part_31317995_1432316153.1427838521189--