Date: Tue, 31 Mar 2015 23:48:41 +0200 (CEST) From: sg-ball@laposte.net To: freebsd-pkg@freebsd.org Subject: Problem with pkg behind a chunking proxy Message-ID: <1476070491.31317997.1427838521190.JavaMail.zimbra@laposte.net> In-Reply-To: <902242756.31277524.1427836544530.JavaMail.zimbra@laposte.net>
next in thread | previous in thread | raw e-mail | index | archive | help
[-- Attachment #1 --] Some times ago I tried to install a FreeBSD 10.1 Release at work, where internal network is isolated from internet through a corporate proxy. And I could not make pkg work correctly. I finally confirmed the problem was caused by the proxy using Transfer-Encoding: chunked instead of giving in first place a ContentLength header by using a python script acting as a proxy and forcing either chunked or not chunked response, and filed a bug report : https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=198772 (got no followup till now ...) I could dive in pkg 1.4.12 source code (thanks to the port) and found that the real cause was that in libpkg/fetch.c function pkg_fetch_file_to_fd used remote = fetchXGet(u, &st, sbuf_data(fetchOpts)); to get remote file meta data in st (notably the file size) but did not test that st.size was not -1 (what it is when using Transfer-Encoding: chunked), so the following loop getting data up to st.size simply did not run. So I could make some minimal changes to this file (fetch.c) to accept st.size == -1. Currently it simply writes "Fetching file: 0%" until the file is fully downloaded and then write full line with "100% size speed time". Of course it runs unchanged if it gets the file size through a ContentLength header. It would certainly be better to trace the size all along the download, but it would require changes to libpkg/event.c as well, and I prefered to limit the changes for now. For anyone interested, I join the patch to this mail. My questions now are : - is this the correct way to propose a patch ? - would it be better to propose the patch for pkg-devel ? - or should I use directly github a submit a pull request ? - or the feature must first be discussed here ? - or ... I think this feature could be interesting since Transfer-Encoding: chunked is valid as HTTP 1.1 but I'm not used to submitting patches even if I've been using FreeBSD since release 3.x [-- Attachment #2 --] --- old/fetch.c 2015-02-13 20:34:33.000000000 +0100 +++ new/fetch.c 2015-03-31 19:41:25.000000000 +0200 @@ -618,30 +618,50 @@ pkg_emit_fetch_begin(url); pkg_emit_progress_start(NULL); - while (done < sz) { - int to_read = MIN(sizeof(buf), sz - done); + if (sz > 0) { + while (done < sz) { + int to_read = MIN(sizeof(buf), sz - done); + + pkg_debug(1, "Reading status: want read %d over %d, %d already done", + to_read, sz, done); + if ((r = fread(buf, 1, to_read, remote)) < 1) + break; + + if (write(dest, buf, r) != r) { + pkg_emit_errno("write", ""); + retcode = EPKG_FATAL; + goto cleanup; + } - pkg_debug(1, "Reading status: want read %d over %d, %d already done", - to_read, sz, done); - if ((r = fread(buf, 1, to_read, remote)) < 1) - break; + done += r; + pkg_debug(1, "Read status: %d over %d", done, sz); - if (write(dest, buf, r) != r) { - pkg_emit_errno("write", ""); + pkg_emit_progress_tick(done, sz); + } + + if (done < sz) { + pkg_emit_error("An error occurred while fetching package"); retcode = EPKG_FATAL; goto cleanup; } - - done += r; - pkg_debug(1, "Read status: %d over %d", done, sz); - - pkg_emit_progress_tick(done, sz); - } - - if (done < sz) { - pkg_emit_error("An error occurred while fetching package"); - retcode = EPKG_FATAL; - goto cleanup; + } + else { + while ((r = fread(buf, 1, sizeof(buf), remote)) > 0) { + if (write(dest, buf, r) != r) { + pkg_emit_errno("write", ""); + retcode = EPKG_FATAL; + goto cleanup; + } + done += r; + } + if (r != 0) { + pkg_emit_error("An error occurred while fetching package"); + retcode = EPKG_FATAL; + goto cleanup; + } + else { + pkg_emit_progress_tick(done, done); + } } pkg_emit_fetch_finished(url);
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?1476070491.31317997.1427838521190.JavaMail.zimbra>