From owner-freebsd-pkg@freebsd.org  Wed Nov  2 02:54:17 2016
Return-Path: <owner-freebsd-pkg@freebsd.org>
Delivered-To: freebsd-pkg@mailman.ysv.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org
 [IPv6:2001:1900:2254:206a::19:1])
 by mailman.ysv.freebsd.org (Postfix) with ESMTP id 88162C2974C
 for <freebsd-pkg@mailman.ysv.freebsd.org>;
 Wed,  2 Nov 2016 02:54:17 +0000 (UTC)
 (envelope-from freeglider@gmail.com)
Received: from mail-vk0-x22e.google.com (mail-vk0-x22e.google.com
 [IPv6:2607:f8b0:400c:c05::22e])
 (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits))
 (Client CN "smtp.gmail.com",
 Issuer "Google Internet Authority G2" (verified OK))
 by mx1.freebsd.org (Postfix) with ESMTPS id 1B28A1BB5
 for <freebsd-pkg@freebsd.org>; Wed,  2 Nov 2016 02:54:17 +0000 (UTC)
 (envelope-from freeglider@gmail.com)
Received: by mail-vk0-x22e.google.com with SMTP id w194so2229668vkw.2
 for <freebsd-pkg@freebsd.org>; Tue, 01 Nov 2016 19:54:16 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113;
 h=mime-version:from:date:message-id:subject:to;
 bh=d7+xUi1xCBqouhuUUYvN8KRp2zMEpuJ1s74VTF0VFyU=;
 b=e6HPzHtOlRtmt+gjgiZu/5leMzLO/c8jsxqIyeEVdXSUEpD3bHxB6C7UEHKrwuGQ0L
 gLIJ3ZDwDBKK07QaGwLnEB+fFdL/ONcgzAM4mB03IynTvP7/U/XtuwyrEB2yCQiV0+lU
 aQga0KgEmAMjJKWrHTYOt+PbC8jS573wN/zIc9/BLO7vsG4iowUTpX53Km7Cva+RrD/T
 eoc7go7qypStBVFV2LCpWKjL+jXTf1xANqnpKUM5nPhTlgLZvDn83c8EG/OWqKec8Pou
 m0gceg3FRM5a9E8q56OQ7PN87KFgN7gBKmzFu9kgPbJ7r6cMP9nB+cVkA/3F4rK2YI7F
 dM0Q==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=1e100.net; s=20130820;
 h=x-gm-message-state:mime-version:from:date:message-id:subject:to;
 bh=d7+xUi1xCBqouhuUUYvN8KRp2zMEpuJ1s74VTF0VFyU=;
 b=Ooad64j10J84m3pdBkDTioQM+BeTldfnR/xPnB50mHLVw4Zi8I6QBw14NSOGWDhyhC
 mN6dB7fQwstC8blpdhzMLY7StybnJmnoU3cKbxziVnzmrAE+wRuABHOe1J+3AT9nuxJC
 WuRBoI5raIyLIW1z/6XJFZeS7GdztAQSXhqs6qy5QP76AMgOr1z8ymcDbFZptTNc2lDB
 c8s9qk3efKCZCgsfIht9d/6+cLgCINCfgD6VuVIzCEBbzGfkxmjxXqYkkNoWaqq31mOm
 6mdpY62c3Qmoq0djd+vc9i07jv/gxQ4Kp+tFtK0hCD7eztRqhAY5DuGTUvDiTrcAG5BE
 iHJw==
X-Gm-Message-State: ABUngvfA2rYZhJV+W08CwAauV10l246NBcJAYdgaYkffAhdLwBJKwPDjlZvPJcuExGXpHC63X1zfnfasv2ifbQ==
X-Received: by 10.31.65.3 with SMTP id o3mr923722vka.3.1478055255834; Tue, 01
 Nov 2016 19:54:15 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.176.5.234 with HTTP; Tue, 1 Nov 2016 19:53:55 -0700 (PDT)
From: Evgeny Petrov <freeglider@gmail.com>
Date: Wed, 2 Nov 2016 05:53:55 +0300
Message-ID: <CAGsiuDPffyr_Jfg-qy=nP7FmrwPpePAQoV6K5CbQ1jyvU64ubA@mail.gmail.com>
Subject: pkg(7) is not bootstrapping, possible error in working with libfetch
To: freebsd-pkg@freebsd.org
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable
X-Content-Filtered-By: Mailman/MimeDel 2.1.23
X-BeenThere: freebsd-pkg@freebsd.org
X-Mailman-Version: 2.1.23
Precedence: list
List-Id: Binary package management and package tools discussion
 <freebsd-pkg.freebsd.org>
List-Unsubscribe: <https://lists.freebsd.org/mailman/options/freebsd-pkg>,
 <mailto:freebsd-pkg-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-pkg/>
List-Post: <mailto:freebsd-pkg@freebsd.org>
List-Help: <mailto:freebsd-pkg-request@freebsd.org?subject=help>
List-Subscribe: <https://lists.freebsd.org/mailman/listinfo/freebsd-pkg>,
 <mailto:freebsd-pkg-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Wed, 02 Nov 2016 02:54:17 -0000

Installing using packer and virtualbox 5.028 r111378 of
FreeBSD-10.3-RELEASE-amd64-disc1.iso using bsdinstall with
=E2=80=8B
the following script:
---

PARTITIONS=3D"ada0 { 29G freebsd-ufs /, 5G freebsd-swap, 10G freebsd-ufs /v=
ar
}"
DISTRIBUTIONS=3D"base.txz kernel.txz"
#!/bin/sh
echo 'WITHOUT_X11=3D"YES"' >> /etc/make.conf
echo 'OPTIONS_UNSET=3DX11' >> /etc/make.conf
echo 'nameserver 8.8.8.8' >> /etc/resolv.conf
cat >> /etc/rc.conf <<EOF
ifconfig_em0=3D"DHCP"
sshd_enable=3D"YES"

dumpdev=3D"NO"
EOF

env ASSUME_ALWAYS_YES=3D1 pkg bootstrap #       <<stops here
pkg update
pkg install -y sudo

[.....snip.....]

reboot
---

This stops at bootstrapping pkg with the message:

---
Signature for pkg not available.
pkg: Error fetching
http://pkg.FreeBSD.org/FreeBSD:10:amd64/quarterly/Latest/pkg.txz.sig:
Connection reset by peer
---

If I break the script and chroot /mnt - I can get the file using fetch(1)
without problem.
Adding two fetches(for txz and sig files) and "pkg add pkg.txz" to my
script solved the problem.

As I've found out using gdb - the problem is that pkg passes size=3D=3D1024=
0 to
fread() function (pkg.c:242), which leads
=E2=80=8B =E2=80=8B
to two iterations of (fread.c:100) while{} loop.

First iteration reads whole file (pkg.txz.sig 727bytes).

Second iteration sets error flag due to connection reset, and returns -1.
This triggers the message(ferror() inpkg.c:254) while the temporary file
/tmp/pkg.txz.sig.[random symbols] is already there and filled with the
remotefile content.

The reason fetch(1) is working - they use their buffer size only if it's
less than remote file size.
Otherwise they set it to the size of remote file (=3D=3D 727bytes in my cas=
e)
=E2=80=8B =E2=80=8B
(fetch/fetch.c:720-724).
So the __srefill (fread.c:106) is called only once not trying to read from
closed connection.

If I lower the bandwidth to 10kbit/s using "ipfw pipe" pkg(7) fails at the
pkg.txz stage, "Connection reset by peer".
If I run multiple times without bw limiting - approximately 1/20 of tries
succedes.

It seems to be race condition - if connection is closed after the attempt
of second buffer refill - it sets EOF and continiues to run, if connection
is closed eariler - it returns error and unlink temporary file and exits.

Tcpdump capture can be found here:
https://gist.github.com/petruchito/82877b1954e7dc8f4502a696683c9385

There's more detailed description in the following StackOverflow question:
http://stackoverflow.com/questions/40209220/freebsd-pkg-suddenly-stopped-bo=
otstrapping



--
Sincerely yours,
Evgeny Petrov