Date: Tue, 31 Mar 2015 23:48:41 +0200 (CEST) From: sg-ball@laposte.net To: freebsd-pkg@freebsd.org Subject: Problem with pkg behind a chunking proxy Message-ID: <1476070491.31317997.1427838521190.JavaMail.zimbra@laposte.net> In-Reply-To: <902242756.31277524.1427836544530.JavaMail.zimbra@laposte.net>
next in thread | previous in thread | raw e-mail | index | archive | help
------=_Part_31317995_1432316153.1427838521189 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit Some times ago I tried to install a FreeBSD 10.1 Release at work, where internal network is isolated from internet through a corporate proxy. And I could not make pkg work correctly. I finally confirmed the problem was caused by the proxy using Transfer-Encoding: chunked instead of giving in first place a ContentLength header by using a python script acting as a proxy and forcing either chunked or not chunked response, and filed a bug report : https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=198772 (got no followup till now ...) I could dive in pkg 1.4.12 source code (thanks to the port) and found that the real cause was that in libpkg/fetch.c function pkg_fetch_file_to_fd used remote = fetchXGet(u, &st, sbuf_data(fetchOpts)); to get remote file meta data in st (notably the file size) but did not test that st.size was not -1 (what it is when using Transfer-Encoding: chunked), so the following loop getting data up to st.size simply did not run. So I could make some minimal changes to this file (fetch.c) to accept st.size == -1. Currently it simply writes "Fetching file: 0%" until the file is fully downloaded and then write full line with "100% size speed time". Of course it runs unchanged if it gets the file size through a ContentLength header. It would certainly be better to trace the size all along the download, but it would require changes to libpkg/event.c as well, and I prefered to limit the changes for now. For anyone interested, I join the patch to this mail. My questions now are : - is this the correct way to propose a patch ? - would it be better to propose the patch for pkg-devel ? - or should I use directly github a submit a pull request ? - or the feature must first be discussed here ? - or ... I think this feature could be interesting since Transfer-Encoding: chunked is valid as HTTP 1.1 but I'm not used to submitting patches even if I've been using FreeBSD since release 3.x ------=_Part_31317995_1432316153.1427838521189 Content-Type: text/x-patch; name=pkg_1_4_14_libpkg_fetch.patch Content-Disposition: attachment; filename=pkg_1_4_14_libpkg_fetch.patch Content-Transfer-Encoding: base64 LS0tIG9sZC9mZXRjaC5jCTIwMTUtMDItMTMgMjA6MzQ6MzMuMDAwMDAwMDAwICswMTAwCisrKyBu ZXcvZmV0Y2guYwkyMDE1LTAzLTMxIDE5OjQxOjI1LjAwMDAwMDAwMCArMDIwMApAQCAtNjE4LDMw ICs2MTgsNTAgQEAKIAogCXBrZ19lbWl0X2ZldGNoX2JlZ2luKHVybCk7CiAJcGtnX2VtaXRfcHJv Z3Jlc3Nfc3RhcnQoTlVMTCk7Ci0Jd2hpbGUgKGRvbmUgPCBzeikgewotCQlpbnQgdG9fcmVhZCA9 IE1JTihzaXplb2YoYnVmKSwgc3ogLSBkb25lKTsKKyAgICAgICAgaWYgKHN6ID4gMCkgeworCQl3 aGlsZSAoZG9uZSA8IHN6KSB7CisJCQlpbnQgdG9fcmVhZCA9IE1JTihzaXplb2YoYnVmKSwgc3og LSBkb25lKTsKKworCQkJcGtnX2RlYnVnKDEsICJSZWFkaW5nIHN0YXR1czogd2FudCByZWFkICVk IG92ZXIgJWQsICVkIGFscmVhZHkgZG9uZSIsCisJCQkJdG9fcmVhZCwgc3osIGRvbmUpOworCQkJ aWYgKChyID0gZnJlYWQoYnVmLCAxLCB0b19yZWFkLCByZW1vdGUpKSA8IDEpCisJCQkJYnJlYWs7 CisKKwkJCWlmICh3cml0ZShkZXN0LCBidWYsIHIpICE9IHIpIHsKKwkJCQlwa2dfZW1pdF9lcnJu bygid3JpdGUiLCAiIik7CisJCQkJcmV0Y29kZSA9IEVQS0dfRkFUQUw7CisJCQkJZ290byBjbGVh bnVwOworCQkJfQogCi0JCXBrZ19kZWJ1ZygxLCAiUmVhZGluZyBzdGF0dXM6IHdhbnQgcmVhZCAl ZCBvdmVyICVkLCAlZCBhbHJlYWR5IGRvbmUiLAotCQkJdG9fcmVhZCwgc3osIGRvbmUpOwotCQlp ZiAoKHIgPSBmcmVhZChidWYsIDEsIHRvX3JlYWQsIHJlbW90ZSkpIDwgMSkKLQkJCWJyZWFrOwor CQkJZG9uZSArPSByOworCQkJcGtnX2RlYnVnKDEsICJSZWFkIHN0YXR1czogJWQgb3ZlciAlZCIs IGRvbmUsIHN6KTsKIAotCQlpZiAod3JpdGUoZGVzdCwgYnVmLCByKSAhPSByKSB7Ci0JCQlwa2df ZW1pdF9lcnJubygid3JpdGUiLCAiIik7CisJCQlwa2dfZW1pdF9wcm9ncmVzc190aWNrKGRvbmUs IHN6KTsKKwkJfQorCisJCWlmIChkb25lIDwgc3opIHsKKwkJCXBrZ19lbWl0X2Vycm9yKCJBbiBl cnJvciBvY2N1cnJlZCB3aGlsZSBmZXRjaGluZyBwYWNrYWdlIik7CiAJCQlyZXRjb2RlID0gRVBL R19GQVRBTDsKIAkJCWdvdG8gY2xlYW51cDsKIAkJfQotCi0JCWRvbmUgKz0gcjsKLQkJcGtnX2Rl YnVnKDEsICJSZWFkIHN0YXR1czogJWQgb3ZlciAlZCIsIGRvbmUsIHN6KTsKLQotCQlwa2dfZW1p dF9wcm9ncmVzc190aWNrKGRvbmUsIHN6KTsKLQl9Ci0KLQlpZiAoZG9uZSA8IHN6KSB7Ci0JCXBr Z19lbWl0X2Vycm9yKCJBbiBlcnJvciBvY2N1cnJlZCB3aGlsZSBmZXRjaGluZyBwYWNrYWdlIik7 Ci0JCXJldGNvZGUgPSBFUEtHX0ZBVEFMOwotCQlnb3RvIGNsZWFudXA7CisgICAgICAgIH0KKwll bHNlIHsKKwkJd2hpbGUgKChyID0gZnJlYWQoYnVmLCAxLCBzaXplb2YoYnVmKSwgcmVtb3RlKSkg PiAwKSB7CisJCQlpZiAod3JpdGUoZGVzdCwgYnVmLCByKSAhPSByKSB7CisJCQkJcGtnX2VtaXRf ZXJybm8oIndyaXRlIiwgIiIpOworCQkJCXJldGNvZGUgPSBFUEtHX0ZBVEFMOworCQkJCWdvdG8g Y2xlYW51cDsKKwkJCX0KKwkJCWRvbmUgKz0gcjsKKwkJfQorCQlpZiAociAhPSAwKSB7CisJCQlw a2dfZW1pdF9lcnJvcigiQW4gZXJyb3Igb2NjdXJyZWQgd2hpbGUgZmV0Y2hpbmcgcGFja2FnZSIp OworCQkJcmV0Y29kZSA9IEVQS0dfRkFUQUw7CisJCQlnb3RvIGNsZWFudXA7CisJCX0KKwkJZWxz ZSB7CisJCQlwa2dfZW1pdF9wcm9ncmVzc190aWNrKGRvbmUsIGRvbmUpOworCQl9CiAJfQogCXBr Z19lbWl0X2ZldGNoX2ZpbmlzaGVkKHVybCk7CiAK ------=_Part_31317995_1432316153.1427838521189--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?1476070491.31317997.1427838521190.JavaMail.zimbra>