From owner-freebsd-pkg@freebsd.org Sun Jun 12 12:58:59 2016 Return-Path: Delivered-To: freebsd-pkg@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id E28B9AF0941 for ; Sun, 12 Jun 2016 12:58:59 +0000 (UTC) (envelope-from trashcan@ellael.org) Received: from mx2.enfer-du-nord.net (mx2.enfer-du-nord.net [IPv6:2001:41d0:d:3049:1:1:0:1]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 549AC2BDE; Sun, 12 Jun 2016 12:58:59 +0000 (UTC) (envelope-from trashcan@ellael.org) Received: from [IPv6:2003:8c:2e30:4901:2880:ee64:d058:1b4f] (p2003008C2E3049012880EE64D0581B4F.dip0.t-ipconnect.de [IPv6:2003:8c:2e30:4901:2880:ee64:d058:1b4f]) by mx2.enfer-du-nord.net (Postfix) with ESMTPSA id 3rSGFD54X1z3Sy; Sun, 12 Jun 2016 14:58:56 +0200 (CEST) Content-Type: text/plain; charset=utf-8 Mime-Version: 1.0 (Mac OS X Mail 9.3 \(3124\)) Subject: Re: pkg 1.8.4 seems to want to reintall lots of packages From: Michael Grimm In-Reply-To: <20160610191504.mkkcufwes426pt7f@ivaldir.etoilebsd.net> Date: Sun, 12 Jun 2016 14:58:55 +0200 Cc: freebsd-pkg@FreeBSD.org Content-Transfer-Encoding: quoted-printable Message-Id: <07EA9C70-C02D-4133-9BA5-F47CDAF61739@ellael.org> References: <201606082233.u58MXHWk010878@gw.catspoiler.org> <20160610191504.mkkcufwes426pt7f@ivaldir.etoilebsd.net> To: Baptiste Daroussin X-Virus-Scanned: clamav-milter 0.99.2 at mail X-Virus-Status: Clean X-Mailer: Apple Mail (2.3124) X-BeenThere: freebsd-pkg@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Binary package management and package tools discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 12 Jun 2016 12:59:00 -0000 Baptiste Daroussin wrote: >=20 > On Wed, Jun 08, 2016 at 03:33:17PM -0700, Don Lewis wrote: >> All of the sudden, pkg seems to want to reinstall lots of packages, = but >> it doesn't say why. > Fixed in 1.8.5, it was reinstalling some reverse dependencies sorry = about that Now, forced reinstallation of ports stopped working for me: test> pkg -v 1.8.5 Regular upgrade, fine because there is nothing to upgrade: test> sudo ezjail-admin console -e 'pkg upgrade -y' test Updating poudriere repository catalogue... poudriere repository is up-to-date. All repositories are up-to-date. Checking for upgrades (0 candidates): 100% Processing candidates (0 candidates): 100% Checking integrity... done (0 conflicting) Your packages are up to date. Forced upgrade =E2=80=A6 test> sudo ezjail-admin console -e 'pkg upgrade -y -f' test Updating poudriere repository catalogue... poudriere repository is up-to-date. All repositories are up-to-date. !--> Checking for upgrades (16 candidates): 100% !--> Processing candidates (16 candidates): 100% Checking integrity... done (0 conflicting) Your packages are up to date. =E2=80=A6 without a reinstallation of a single port, though. Regards, Michael= From owner-freebsd-pkg@freebsd.org Sun Jun 12 22:45:28 2016 Return-Path: Delivered-To: freebsd-pkg@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 527F2AF1ECA for ; Sun, 12 Jun 2016 22:45:28 +0000 (UTC) (envelope-from baptiste.daroussin@gmail.com) Received: from mail-wm0-x230.google.com (mail-wm0-x230.google.com [IPv6:2a00:1450:400c:c09::230]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id DC0E22E03 for ; Sun, 12 Jun 2016 22:45:27 +0000 (UTC) (envelope-from baptiste.daroussin@gmail.com) Received: by mail-wm0-x230.google.com with SMTP id n184so55804764wmn.1 for ; Sun, 12 Jun 2016 15:45:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=sender:date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to:user-agent; bh=31gCR42xL9HdOtsVC/rgz2De3yCCEl6RuiDGNUMG9hw=; b=e27+JRnCmv17xn3pMVxUCKTDoDma99UPobQBV2WHCvizkv7iZZ4EVKYP5Y9L/qn98f ym8rCa4az27hK+9r/yeEvZDRUGmQj3qS5jeUgtIXZ9vq4CQcczvmwT5DlTtdaLK6deeB h1avmq1iGuURlZhXU45gJifBEuTaGBwaU0H3dDbFeqmwhEzhcvrIsCKtVRRZ4Tcrbiyr cnzcJZpbATszOp6LxwuS1uRBdikpsA8TJMizx4dxI5zfKiuhG8+3gXs9gJNLgdTw/AA4 hdCd0uKV92iBBTs7QTufiiqqv29DPZOL2hD3+HURes6yQ6ISkBDm9pilsgCAQNMffGFE Muwg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:sender:date:from:to:cc:subject:message-id :references:mime-version:content-disposition:in-reply-to:user-agent; bh=31gCR42xL9HdOtsVC/rgz2De3yCCEl6RuiDGNUMG9hw=; b=cMngzG+yZjqBNXc/Zbso0MGOGd7jgtL3HAyT+cnD7ouBoopk+guWRYFUY5YhbqEblV y1pyn445CO070jSj7cVxBwp03x4bf3ryrNGhB3756SoxXcSr9iiVgb2qLN3GtpPMhRqE K3KqCIM5ypkWJwWDp0zdsfyKnwXOmjRjd0JNCHuJuAw66zIEyQKA5W9/4SZTm0YlC1jc kPY1W3+aKsqat6H03auTAxG35xjYcorZFWpDAxPYP3wj5X1Da1/S9qBsvDBKaxdZXpRN V1hEH/gupprD1CXmo16WElJSjyd16zGWEqu6F+sl/CePpI4I8Sm2vZmj4OTrd9mStfE7 RYWw== X-Gm-Message-State: ALyK8tImbvWcv+ApjEi4bx4htVqPFzPErVgIX6ptDAs951t/hxEaFwIMKynfxYR8ffp6IQ== X-Received: by 10.28.89.6 with SMTP id n6mr8296782wmb.4.1465771526305; Sun, 12 Jun 2016 15:45:26 -0700 (PDT) Received: from ivaldir.etoilebsd.net ([2001:41d0:8:db4c::1]) by smtp.gmail.com with ESMTPSA id k127sm11028099wmf.21.2016.06.12.15.45.24 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sun, 12 Jun 2016 15:45:25 -0700 (PDT) Sender: Baptiste Daroussin Date: Mon, 13 Jun 2016 00:45:24 +0200 From: Baptiste Daroussin To: Michael Grimm Cc: freebsd-pkg@FreeBSD.org Subject: Re: pkg 1.8.4 seems to want to reintall lots of packages Message-ID: <20160612224524.6ns3lgwt4ig47dym@ivaldir.etoilebsd.net> References: <201606082233.u58MXHWk010878@gw.catspoiler.org> <20160610191504.mkkcufwes426pt7f@ivaldir.etoilebsd.net> <07EA9C70-C02D-4133-9BA5-F47CDAF61739@ellael.org> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="kuholbfgctyl6an4" Content-Disposition: inline In-Reply-To: <07EA9C70-C02D-4133-9BA5-F47CDAF61739@ellael.org> User-Agent: Mutt/1.6.1-neo (2016-05-23) X-BeenThere: freebsd-pkg@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Binary package management and package tools discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 12 Jun 2016 22:45:28 -0000 --kuholbfgctyl6an4 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Sun, Jun 12, 2016 at 02:58:55PM +0200, Michael Grimm wrote: > Baptiste Daroussin wrote: > >=20 > > On Wed, Jun 08, 2016 at 03:33:17PM -0700, Don Lewis wrote: >=20 > >> All of the sudden, pkg seems to want to reinstall lots of packages, but > >> it doesn't say why. >=20 > > Fixed in 1.8.5, it was reinstalling some reverse dependencies sorry abo= ut that >=20 > Now, forced reinstallation of ports stopped working for me: >=20 > test> pkg -v > 1.8.5 >=20 > Regular upgrade, fine because there is nothing to upgrade: >=20 > test> sudo ezjail-admin console -e 'pkg upgrade -y' test > Updating poudriere repository catalogue... > poudriere repository is up-to-date. > All repositories are up-to-date. > Checking for upgrades (0 candidates): 100% > Processing candidates (0 candidates): 100% > Checking integrity... done (0 conflicting) > Your packages are up to date. >=20 > Forced upgrade =E2=80=A6 >=20 > test> sudo ezjail-admin console -e 'pkg upgrade -y -f' test > Updating poudriere repository catalogue... > poudriere repository is up-to-date. > All repositories are up-to-date. > !--> Checking for upgrades (16 candidates): 100% > !--> Processing candidates (16 candidates): 100% > Checking integrity... done (0 conflicting) > Your packages are up to date. >=20 > =E2=80=A6 without a reinstallation of a single port, though. >=20 > Regards, > Michael Fixed, Bapt --kuholbfgctyl6an4 Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQIcBAEBCAAGBQJXXeX3AAoJEGOJi9zxtz5aP8sQAInI5V74hZ2TpK1Sr4QeWoa8 V+O0h787/N2PQyQ31zbIdXBmWcyqIxabGoZq+hwrC5ZK7S71zGcC1Ade5gfCVWoE ZMZBVv2pCStg6faLEGb4vFNwHhc6Eqg4uHSuKsy60zX7oisWvF+4hPkyKAjXv7MT M5n/aSREN+XYZoAudtYPMOnPLiPObtE14Q/TjeicdB8DMxwKERZiaR8uwB6V+UYz EDDTm7Qox/stU0G/a5hhZMcTAXtmObLrABnCcV8AcrQZkWmyGm+xsyvjEoAYi/h9 NO6RRIuDVEJGnKjp45roeBF5JUpsJtps5YBQrAL/lY3/pi7xmD/o0o38RKRl5Oh/ VcRxNO+/dS9ftOzBMTdMKgdd6vNLw+V9Iz8aGy/UN6LKxKfUX9N7EBepxH/opTXl 6xX7S8O4Op0Tq4pjOEJcMRZmLdyT5Gh1AxhZNXNcT71guAhBYfpm9j8mKAa92o/R ZdMxwQ8FcBh2GqL4zwbuMuS7cwnpAnPvTr3Vne9my9kXMy64gllQp+2z5JGQzbpd HL/+FLSzDXlKGbYd8yi3u+rgB4LXGlzRisPaOtImT7BjeAOQuhkOGfcg8gcIgMD6 jUJqF1uo6EzyvBw1L+3HNWbCNAuv73urr5ViIdcLM005pWzscLSsuaX7HiwqKlVv YqX/MaNsUUJ7LrcYpRuD =PLsQ -----END PGP SIGNATURE----- --kuholbfgctyl6an4-- From owner-freebsd-pkg@freebsd.org Mon Jun 13 05:23:24 2016 Return-Path: Delivered-To: freebsd-pkg@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 82B88AF022F for ; Mon, 13 Jun 2016 05:23:24 +0000 (UTC) (envelope-from trashcan@ellael.org) Received: from mx2.enfer-du-nord.net (mx2.enfer-du-nord.net [IPv6:2001:41d0:d:3049:1:1:0:1]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4DF9F2414; Mon, 13 Jun 2016 05:23:24 +0000 (UTC) (envelope-from trashcan@ellael.org) Received: from [IPv6:2003:8c:2e02:5601:7408:2ccb:7a0c:5429] (p2003008C2E02560174082CCB7A0C5429.dip0.t-ipconnect.de [IPv6:2003:8c:2e02:5601:7408:2ccb:7a0c:5429]) by mx2.enfer-du-nord.net (Postfix) with ESMTPSA id 3rSh564Nt0zQ4X; Mon, 13 Jun 2016 07:23:22 +0200 (CEST) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 9.3 \(3124\)) Subject: Re: pkg 1.8.4 seems to want to reintall lots of packages From: Michael Grimm In-Reply-To: <20160612224524.6ns3lgwt4ig47dym@ivaldir.etoilebsd.net> Date: Mon, 13 Jun 2016 07:23:20 +0200 Cc: freebsd-pkg@FreeBSD.org Content-Transfer-Encoding: 7bit Message-Id: <0703780C-C21B-40D8-AF23-61048F9CFD3A@ellael.org> References: <201606082233.u58MXHWk010878@gw.catspoiler.org> <20160610191504.mkkcufwes426pt7f@ivaldir.etoilebsd.net> <07EA9C70-C02D-4133-9BA5-F47CDAF61739@ellael.org> <20160612224524.6ns3lgwt4ig47dym@ivaldir.etoilebsd.net> To: Baptiste Daroussin X-Virus-Scanned: clamav-milter 0.99.2 at mail X-Virus-Status: Clean X-Mailer: Apple Mail (2.3124) X-BeenThere: freebsd-pkg@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Binary package management and package tools discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 13 Jun 2016 05:23:24 -0000 Baptiste Daroussin wrote: > > On Sun, Jun 12, 2016 at 02:58:55PM +0200, Michael Grimm wrote: >> Now, forced reinstallation of ports stopped working for me: > > Fixed, Confirmed. Thanks for fixing it, and with kind regards, Michael From owner-freebsd-pkg@freebsd.org Mon Jun 13 23:28:32 2016 Return-Path: Delivered-To: freebsd-pkg@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 966F8AF0C88 for ; Mon, 13 Jun 2016 23:28:32 +0000 (UTC) (envelope-from mjguzik@gmail.com) Received: from mail-wm0-x242.google.com (mail-wm0-x242.google.com [IPv6:2a00:1450:400c:c09::242]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 22E212BEE for ; Mon, 13 Jun 2016 23:28:32 +0000 (UTC) (envelope-from mjguzik@gmail.com) Received: by mail-wm0-x242.google.com with SMTP id n184so18353010wmn.1 for ; Mon, 13 Jun 2016 16:28:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=date:from:to:subject:message-id:mime-version:content-disposition :user-agent; bh=ymN0gdjveUeNciZal9C238i64BzAVmhbidoHyTN4O8o=; b=lpeLNktZIDBC7sgMqGbpQLo+SCmLqcO5PsBqJHPynSPz5MszRiz5lZUfwZVQ+uUZYa VLd3RMeGchSu6iA6t+cE/NUMVU1MFhsB14Qn+moHBYz+wPYKcf9lFA6WcrzShPRycO86 DTzpYQSuuJZrWv0ol5d3nSqwiJyC+vbKzqIkw5V9D9r9uW7LWLNMx69jhLCPZ9WvdMRw CRJ0OI4Tp84q5SR5Tbjw+vhzqOZJ+0jD3A4D/p2GwHDkUs5wbKw9hLgt8QI59D04zk6t FBcsoHOA3OtWgP7X40x4H841Qb5y+hL9A1G3QF9OZLqmtvok9hxohQEWjzVmxxjKUl92 nxfw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:date:from:to:subject:message-id:mime-version :content-disposition:user-agent; bh=ymN0gdjveUeNciZal9C238i64BzAVmhbidoHyTN4O8o=; b=S1bCuA1zc+A2TE3UbI2xRd+ZBxAXEFrYCkzzyfXP9TgwynXpZbBT5iqqA/8l4DTf+k XuF4kQMOL0nZ3ECTvW5qhkFgqHp+e+N4yqDdxAuSKO1Z0+XW11MdAhwrKleSV5Pv+QTE hWIthUPQW2ad9Ug4nYG3BO9dykQErQayJiPiuP5ZyV+D0wuif9E71ltzo3JytL5qRpXM 07u60hQ4dFLU2Axm+sroiVqBA8uB/Hie+xogvXQY0RvKsFG+C51Nmv9Ks5MS0IF5Q//+ rBknUq+aFeK8Lo6K6dihFi99roNZ7+lIf68ZGa6Otcaihwx1ypG8ZRhQyYfVlkOjWz9x Gq3Q== X-Gm-Message-State: ALyK8tJM7fez8QwCWWkke5+21nP0YA76zoa9hmfxnjg0N0FXoEnpFaBuR6cfUjjL3/eHOg== X-Received: by 10.194.109.232 with SMTP id hv8mr3177065wjb.115.1465860510129; Mon, 13 Jun 2016 16:28:30 -0700 (PDT) Received: from dft-labs.eu (n1x0n-1-pt.tunnel.tserv5.lon1.ipv6.he.net. [2001:470:1f08:1f7::2]) by smtp.gmail.com with ESMTPSA id kp9sm29805421wjb.28.2016.06.13.16.28.29 for (version=TLS1_2 cipher=AES128-SHA bits=128/128); Mon, 13 Jun 2016 16:28:29 -0700 (PDT) Date: Tue, 14 Jun 2016 01:28:27 +0200 From: Mateusz Guzik To: freebsd-pkg@freebsd.org Subject: pkg vs namecache Message-ID: <20160613232827.GA6841@dft-labs.eu> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline User-Agent: Mutt/1.5.21 (2010-09-15) X-BeenThere: freebsd-pkg@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Binary package management and package tools discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 13 Jun 2016 23:28:32 -0000 Hello, it is not a secret that poudriere builds expose a lot of contention in the kernel. I performed several tests few months back. On a 24-way machine, once all distfiles are downloaded, the top contending lock belongs to the namecache. The namecache is unfortunately protected with one big lock taken in shared or exclusive mode. I have a WIP patch which introduces fine grained locking and in effect pushes it way down. It is unlikely to hit 11.0, but 11.1 is an option. Regardless, if performance is a concern, userspace should not do unnecessary stuff (which in this case means hammering the namecache for no reason). Playing around with dtrace e.g. like this: dtrace -n 'vfs:namecache:: { @[execname] = count(); } tick-30s { exit(0); }' shows that pkg-static is amongst top namecache users and further inspection suggests that several (if not most) lookups are unnecessary. For any path-based operation, each component needs to be looked up. Lookups of non-cached files add entries, which blocks other consumers. Note that the startup procedure for dynamically linked binaries is wasteful to an extent and it may beget fixed later. With this out of the way, let's have a look what pkg 1.8.4_1 does when prompted to install rexima-1.4. I only left entries I consider suspicious + enough context to see what's up. Would be nice if this stuff got fixed. Thaks. 63984: __getcwd("/root",1024) = 0 (0x0) Why? 63984: open("/etc/pkg/",O_NONBLOCK|O_DIRECTORY|O_CLOEXEC,041673240) = 4 (0x4) 63984: fstatfs(4,{ fstypename=zfs,mntonname=/,mntfromname=zrootmjg,fsid=a294c20dde42cb7d }) = 0 (0x0) 63984: getdirentries(0x4,0x80305f000,0x1000,0x80305c268) = 72 (0x48) 63984: getdirentries(0x4,0x80305f000,0x1000,0x80305c268) = 0 (0x0) 63984: close(4) = 0 (0x0) 63984: __getcwd("/root",1024) = 0 (0x0) Another __getcwd. If the first one was really needed (was it?), the result should be remembered. This particular run got 7 getcwds in total, bigger packages take more. 63984: lstat("/etc",{ mode=drwxr-xr-x ,inode=17,size=113,blksize=7680 }) = 0 (0x0) 63984: lstat("/etc/pkg",{ mode=drwxr-xr-x ,inode=36,size=4,blksize=4096 }) = 0 (0x0) 63984: lstat("/etc/pkg/FreeBSD.conf",{ mode=-rw-r--r-- ,inode=321637,size=436,blksize=4096 }) = 0 (0x0) 63984: stat("/etc/pkg/FreeBSD.conf",{ mode=-rw-r--r-- ,inode=321637,size=436,blksize=4096 }) = 0 (0x0) 63984: openat(AT_FDCWD,"/etc/pkg/FreeBSD.conf",O_RDONLY,00) = 4 (0x4) Why not just open the file. If some kind of includes are supported, you can do the last stat and then compare inode + device id pairs. 63984: munmap(0x80064f000,436) = 0 (0x0) 63984: open("/usr/local/etc/pkg/repos/",O_NONBLOCK|O_DIRECTORY|O_CLOEXEC,041673240) ERR#2 'No such file or directory' 63984: stat("/var/db/pkg",{ mode=drwxr-xr-x ,inode=78,size=5,blksize=4096 }) = 0 (0x0) 63984: eaccess("/var/db/pkg",R_OK|W_OK) = 0 (0x0) 63984: stat("/var/db/pkg/local.sqlite",{ mode=-rw-r--r-- ,inode=4755,size=11584512,blksize=131072 }) = 0 (0x0) 63984: eaccess("/var/db/pkg/local.sqlite",R_OK|W_OK) = 0 (0x0) 63984: stat("/var/db/pkg/repo-FreeBSD.sqlite",{ mode=-rw-r--r-- ,inode=323679,size=46235648,blksize=131072 }) = 0 (0x0) 63984: eaccess("/var/db/pkg/repo-FreeBSD.sqlite",R_OK|W_OK) = 0 (0x0) 63984: stat("/var/db/pkg",{ mode=drwxr-xr-x ,inode=78,size=5,blksize=4096 }) = 0 (0x0) 63984: eaccess("/var/db/pkg",R_OK|W_OK) = 0 (0x0) 63984: stat("/var/db/pkg/repo-FreeBSD.sqlite",{ mode=-rw-r--r-- ,inode=323679,size=46235648,blksize=131072 }) = 0 (0x0) 63984: eaccess("/var/db/pkg/repo-FreeBSD.sqlite",R_OK|W_OK) = 0 (0x0) 63984: write(1,"Updating FreeBSD repository cata"...,41) = 41 (0x29) 63984: statfs("/var/db/pkg",{ fstypename=zfs,mntonname=/,mntfromname=zrootmjg,fsid=a294c20dde42cb7d }) = 0 (0x0) 63984: access("/var/db/pkg/FreeBSD.meta",R_OK) = 0 (0x0) 63984: __getcwd("/root",1024) = 0 (0x0) 63984: lstat("/var",{ mode=drwxr-xr-x ,inode=58,size=25,blksize=4096 }) = 0 (0x0) 63984: lstat("/var/db",{ mode=drwxr-xr-x ,inode=72,size=22,blksize=4096 }) = 0 (0x0) 63984: lstat("/var/db/pkg",{ mode=drwxr-xr-x ,inode=78,size=5,blksize=4096 }) = 0 (0x0) 63984: lstat("/var/db/pkg/FreeBSD.meta",{ mode=-rw-r--r-- ,inode=323678,size=246,blksize=4096 }) = 0 (0x0) 63984: stat("/var/db/pkg/FreeBSD.meta",{ mode=-rw-r--r-- ,inode=323678,size=246,blksize=4096 }) = 0 (0x0) 63984: openat(AT_FDCWD,"/var/db/pkg/FreeBSD.meta",O_RDONLY,00) = 4 (0x4) Again, several passes over /var/db/pkg. 63984: stat("/etc/nsswitch.conf",{ mode=-rw-r--r-- ,inode=14316,size=255,blksize=4096 }) = 0 (0x0) 63984: open("/etc/nsswitch.conf",O_CLOEXEC,0666) = 7 (0x7) 63984: ioctl(7,TIOCGETA,0xffff90f0) ERR#25 'Inappropriate ioctl for device' 63984: fstat(7,{ mode=-rw-r--r-- ,inode=14316,size=255,blksize=4096 }) = 0 (0x0) 63984: read(7,"#\n# nsswitch.conf(5) - name ser"...,4096) = 255 (0xff) 63984: read(7,0x80305f000,4096) = 0 (0x0) 63984: access("/usr/local/lib/nss_compat.so.1",F_OK) ERR#2 'No such file or directory' 63984: access("/lib/nss_compat.so.1",F_OK) ERR#2 'No such file or directory' 63984: access("/usr/lib/nss_compat.so.1",F_OK) ERR#2 'No such file or directory' 63984: access("/usr/lib/compat/nss_compat.so.1",F_OK) ERR#2 'No such file or directory' 63984: access("/usr/local/lib/nss_compat.so.1",F_OK) ERR#2 'No such file or directory' 63984: access("/usr/local/lib/gcc5/nss_compat.so.1",F_OK) ERR#2 'No such file or directory' 63984: access("/usr/local/lib/perl5/5.20/mach/CORE/nss_compat.so.1",F_OK) ERR#2 'No such file or directory' 63984: access("/usr/local/llvm36/lib/nss_compat.so.1",F_OK) ERR#2 'No such file or directory' 63984: access("/usr/local/llvm37/lib/nss_compat.so.1",F_OK) ERR#2 'No such file or directory' 63984: access("/lib/casper/nss_compat.so.1",F_OK) ERR#2 'No such file or directory' 63984: access("/lib/nss_compat.so.1",F_OK) ERR#2 'No such file or directory' 63984: access("/usr/lib/nss_compat.so.1",F_OK) ERR#2 'No such file or directory' 63984: access("/usr/local/lib/nss_nis.so.1",F_OK) ERR#2 'No such file or directory' 63984: access("/lib/nss_nis.so.1",F_OK) ERR#2 'No such file or directory' 63984: access("/usr/lib/nss_nis.so.1",F_OK) ERR#2 'No such file or directory' 63984: access("/usr/lib/compat/nss_nis.so.1",F_OK) ERR#2 'No such file or directory' 63984: access("/usr/local/lib/nss_nis.so.1",F_OK) ERR#2 'No such file or directory' 63984: access("/usr/local/lib/gcc5/nss_nis.so.1",F_OK) ERR#2 'No such file or directory' 63984: access("/usr/local/lib/perl5/5.20/mach/CORE/nss_nis.so.1",F_OK) ERR#2 'No such file or directory' 63984: access("/usr/local/llvm36/lib/nss_nis.so.1",F_OK) ERR#2 'No such file or directory' 63984: access("/usr/local/llvm37/lib/nss_nis.so.1",F_OK) ERR#2 'No such file or directory' 63984: access("/lib/casper/nss_nis.so.1",F_OK) ERR#2 'No such file or directory' 63984: access("/lib/nss_nis.so.1",F_OK) ERR#2 'No such file or directory' 63984: access("/usr/lib/nss_nis.so.1",F_OK) ERR#2 'No such file or directory' 63984: access("/usr/local/lib/nss_files.so.1",F_OK) ERR#2 'No such file or directory' 63984: access("/lib/nss_files.so.1",F_OK) ERR#2 'No such file or directory' 63984: access("/usr/lib/nss_files.so.1",F_OK) ERR#2 'No such file or directory' 63984: access("/usr/lib/compat/nss_files.so.1",F_OK) ERR#2 'No such file or directory' 63984: access("/usr/local/lib/nss_files.so.1",F_OK) ERR#2 'No such file or directory' 63984: access("/usr/local/lib/gcc5/nss_files.so.1",F_OK) ERR#2 'No such file or directory' 63984: access("/usr/local/lib/perl5/5.20/mach/CORE/nss_files.so.1",F_OK) ERR#2 'No such file or directory' 63984: access("/usr/local/llvm36/lib/nss_files.so.1",F_OK) ERR#2 'No such file or directory' 63984: access("/usr/local/llvm37/lib/nss_files.so.1",F_OK) ERR#2 'No such file or directory' 63984: access("/lib/casper/nss_files.so.1",F_OK) ERR#2 'No such file or directory' 63984: access("/lib/nss_files.so.1",F_OK) ERR#2 'No such file or directory' 63984: access("/usr/lib/nss_files.so.1",F_OK) ERR#2 'No such file or directory' 63984: access("/usr/local/lib/nss_dns.so.1",F_OK) ERR#2 'No such file or directory' 63984: access("/lib/nss_dns.so.1",F_OK) ERR#2 'No such file or directory' 63984: access("/usr/lib/nss_dns.so.1",F_OK) ERR#2 'No such file or directory' 63984: access("/usr/lib/compat/nss_dns.so.1",F_OK) ERR#2 'No such file or directory' 63984: access("/usr/local/lib/nss_dns.so.1",F_OK) ERR#2 'No such file or directory' 63984: access("/usr/local/lib/gcc5/nss_dns.so.1",F_OK) ERR#2 'No such file or directory' 63984: access("/usr/local/lib/perl5/5.20/mach/CORE/nss_dns.so.1",F_OK) ERR#2 'No such file or directory' 63984: access("/usr/local/llvm36/lib/nss_dns.so.1",F_OK) ERR#2 'No such file or directory' 63984: access("/usr/local/llvm37/lib/nss_dns.so.1",F_OK) ERR#2 'No such file or directory' 63984: access("/lib/casper/nss_dns.so.1",F_OK) ERR#2 'No such file or directory' 63984: access("/lib/nss_dns.so.1",F_OK) ERR#2 'No such file or directory' 63984: access("/usr/lib/nss_dns.so.1",F_OK) ERR#2 'No such file or directory' Now that's a bummer, but likely not a pkg issue. 63984: mkdir("/tmp",0777) ERR#17 'File exists' Why? 63984: getpid() = 63984 (0xf9f0) 63984: stat("/tmp",{ mode=drwxrwxrwt ,inode=4,size=76,blksize=16384 }) = 0 (0x0) 63984: open("/tmp/packagesite.yaml.txz.DlhH4v",O_RDWR|O_CREAT|O_EXCL,0600) = 5 (0x5) 63984: unlink("/tmp/packagesite.yaml.txz.DlhH4v") = 0 (0x0) 63984: unlink("") ERR#2 'No such file or directory' ? 3984: stat("/etc/nsswitch.conf",{ mode=-rw-r--r-- ,inode=14316,size=255,blksize=4096 }) = 0 (0x0) 63984: geteuid() = 0 (0x0) 63984: open("/etc/spwd.db",O_CLOEXEC,00) = 12 (0xc) 63984: fstat(12,{ mode=-rw------- ,inode=319614,size=40960,blksize=40960 }) = 0 (0x0) 63984: read(12,"\0\^F\^Ua\0\0\0\^B\0\0\^D\M-R\0"...,260) = 260 (0x104) 63984: pread(0xc,0x803064000,0x1000,0x6000) = 4096 (0x1000) 63984: pread(0xc,0x803063000,0x1000,0x4000) = 4096 (0x1000) 63984: close(12) = 0 (0x0) 63984: stat("/etc/nsswitch.conf",{ mode=-rw-r--r-- ,inode=14316,size=255,blksize=4096 }) = 0 (0x0) 63984: open("/etc/group",O_CLOEXEC,0666) = 12 (0xc) 63984: fstat(12,{ mode=-rw-r--r-- ,inode=319611,size=500,blksize=4096 }) = 0 (0x0) 63984: lseek(12,0x0,SEEK_CUR) = 0 (0x0) 63984: lseek(12,0x0,SEEK_SET) = 0 (0x0) 63984: read(12,"# $FreeBSD$\n#\nwheel:*:0:root\n"...,4096) = 500 (0x1f4) 63984: close(12) = 0 (0x0) This repeats itself a lot. I wonder if that's an iffeciency in libc or pkg. 63984: ioctl(1,TIOCGETA,0xffffced0) = 0 (0x0) 63984: write(1,"[1/1] Extracting rexima-1.4: 0"...,33) = 33 (0x21) 63984: mkdirat(11,"usr",0755) ERR#17 'File exists' 63984: mkdirat(11,"usr/local",0755) ERR#17 'File exists' 63984: mkdirat(11,"usr/local/bin",0755) ERR#17 'File exists' 63984: openat(11,"usr/local/bin/rexima.DcozrktiDx8Y",O_WRONLY|O_CREAT|O_EXCL,0755) = 12 (0xc) 63984: fstat(12,{ mode=-rwxr-xr-x ,inode=4182,size=0,blksize=131072 }) = 0 (0x0) why fstat? In the common case you can expect these directories to already be there. So how about instead you try the open first and only if it fails you try to create the tree. But you start trying from the top (here: "bin"). 63984: fstatat(11,"usr/local/bin/rexima",0x7fffffffd5f0,AT_SYMLINK_NOFOLLOW) ERR#2 'No such file or directory' 63984: renameat(11,"usr/local/bin/rexima.DcozrktiDx8Y",11,"usr/local/bin/rexima") = 0 (0x0) Why not just replace? -- Mateusz Guzik From owner-freebsd-pkg@freebsd.org Wed Jun 15 20:07:27 2016 Return-Path: Delivered-To: freebsd-pkg@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 39354A44E74 for ; Wed, 15 Jun 2016 20:07:27 +0000 (UTC) (envelope-from bdrewery@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:1900:2254:206c::16:87]) by mx1.freebsd.org (Postfix) with ESMTP id 298701957; Wed, 15 Jun 2016 20:07:27 +0000 (UTC) (envelope-from bdrewery@FreeBSD.org) Received: from mail.xzibition.com (localhost [IPv6:::1]) by freefall.freebsd.org (Postfix) with ESMTP id 1890B13BB; Wed, 15 Jun 2016 20:07:27 +0000 (UTC) (envelope-from bdrewery@FreeBSD.org) Received: from mail.xzibition.com (localhost [172.31.3.2]) by mail.xzibition.com (Postfix) with ESMTP id 61CF01D759; Wed, 15 Jun 2016 20:07:20 +0000 (UTC) X-Virus-Scanned: amavisd-new at mail.xzibition.com Received: from mail.xzibition.com ([172.31.3.2]) by mail.xzibition.com (mail.xzibition.com [172.31.3.2]) (amavisd-new, port 10026) with LMTP id XiOaPOCEgBDk; Wed, 15 Jun 2016 20:07:16 +0000 (UTC) Subject: Re: pkg vs namecache DKIM-Filter: OpenDKIM Filter v2.9.2 mail.xzibition.com F35931D754 To: Mateusz Guzik , freebsd-pkg@freebsd.org References: <20160613232827.GA6841@dft-labs.eu> From: Bryan Drewery Organization: FreeBSD Message-ID: <73fee5f2-5a18-7eb1-be66-c0ea981f15de@FreeBSD.org> Date: Wed, 15 Jun 2016 13:07:15 -0700 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:45.0) Gecko/20100101 Thunderbird/45.1.1 MIME-Version: 1.0 In-Reply-To: <20160613232827.GA6841@dft-labs.eu> Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: quoted-printable X-BeenThere: freebsd-pkg@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Binary package management and package tools discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 15 Jun 2016 20:07:27 -0000 On 6/13/16 4:28 PM, Mateusz Guzik wrote: > Hello, >=20 > it is not a secret that poudriere builds expose a lot of contention in > the kernel. >=20 > I performed several tests few months back. On a 24-way machine, once al= l > distfiles are downloaded, the top contending lock belongs to the > namecache. The namecache is unfortunately protected with one big lock > taken in shared or exclusive mode. I have a WIP patch which introduces > fine grained locking and in effect pushes it way down. It is unlikely t= o > hit 11.0, but 11.1 is an option. >=20 > Regardless, if performance is a concern, userspace should not do > unnecessary stuff (which in this case means hammering the namecache for > no reason). >=20 > Playing around with dtrace e.g. like this: > dtrace -n 'vfs:namecache:: { @[execname] =3D count(); } tick-30s { > exit(0); }' >=20 > shows that pkg-static is amongst top namecache users and further > inspection suggests that several (if not most) lookups are unnecessary. >=20 > For any path-based operation, each component needs to be looked up. > Lookups of non-cached files add entries, which blocks other consumers. >=20 > Note that the startup procedure for dynamically linked binaries is > wasteful to an extent and it may beget fixed later. >=20 > With this out of the way, let's have a look what pkg 1.8.4_1 does when > prompted to install rexima-1.4. I only left entries I consider > suspicious + enough context to see what's up.=20 >=20 > Would be nice if this stuff got fixed. Thaks. >=20 > 63984: __getcwd("/root",1024) =3D 0 (0x0) >=20 > Why? >=20 > 63984: open("/etc/pkg/",O_NONBLOCK|O_DIRECTORY|O_CLOEXEC,041673240) =3D= 4 (0x4) > 63984: fstatfs(4,{ fstypename=3Dzfs,mntonname=3D/,mntfromname=3Dzrootmj= g,fsid=3Da294c20dde42cb7d }) =3D 0 (0x0) > 63984: getdirentries(0x4,0x80305f000,0x1000,0x80305c268) =3D 72 (0x48) > 63984: getdirentries(0x4,0x80305f000,0x1000,0x80305c268) =3D 0 (0x0) > 63984: close(4) =3D 0 (0x0) > 63984: __getcwd("/root",1024) =3D 0 (0x0) >=20 > Another __getcwd. If the first one was really needed (was it?), the I think an important point to remember here is that pkg itself uses a lot of other libraries that do redundant things. Libc and the kernel should be optimized more for doing such things. getcwd() should in theory not be expensive to call and should cache the value somewhere in userspace and change it on [f]chdir(2). Of course this breaks ENOENT, but tradeoffs are needed to fix our expensive syscalls IMO. getpid() is another that should be a free lunch every time IMO. > result should be remembered. This particular run got 7 getcwds in total= , > bigger packages take more. >=20 > 63984: lstat("/etc",{ mode=3Ddrwxr-xr-x ,inode=3D17,size=3D113,blksize=3D= 7680 }) =3D 0 (0x0) > 63984: lstat("/etc/pkg",{ mode=3Ddrwxr-xr-x ,inode=3D36,size=3D4,blksiz= e=3D4096 }) =3D 0 (0x0) > 63984: lstat("/etc/pkg/FreeBSD.conf",{ mode=3D-rw-r--r-- ,inode=3D32163= 7,size=3D436,blksize=3D4096 }) =3D 0 (0x0) > 63984: stat("/etc/pkg/FreeBSD.conf",{ mode=3D-rw-r--r-- ,inode=3D321637= ,size=3D436,blksize=3D4096 }) =3D 0 (0x0) > 63984: openat(AT_FDCWD,"/etc/pkg/FreeBSD.conf",O_RDONLY,00) =3D 4 (0x4) >=20 This particular pattern looks like realpath(3). It was hugely problematic for buildworld and bmake's processing of .meta files with filemon data. It would spam a ton of realpath(3) that led to a 48 minutes NOP build. We added in a simple realpath(3) cache in bmake which dropped it down to 8 minutes. The caching fix was tested on Linux and saw no improvement there since their realpath(3) was already fast. For pkg, indeed it is from UCL's ucl_parser_add_file() -> ucl_parser_add_file_priority() which does a realpath(3), stat(2) and then open(2) on the file provided. It seems odd that UCL would need to realpath(3) the file being opened. > Why not just open the file. If some kind of includes are supported, you > can do the last stat and then compare inode + device id pairs. >=20 > 63984: munmap(0x80064f000,436) =3D 0 (0x0) > 63984: open("/usr/local/etc/pkg/repos/",O_NONBLOCK|O_DIRECTORY|O_CLOEXE= C,041673240) ERR#2 'No such file or directory' > 63984: stat("/var/db/pkg",{ mode=3Ddrwxr-xr-x ,inode=3D78,size=3D5,blks= ize=3D4096 }) =3D 0 (0x0) > 63984: eaccess("/var/db/pkg",R_OK|W_OK) =3D 0 (0x0) > 63984: stat("/var/db/pkg/local.sqlite",{ mode=3D-rw-r--r-- ,inode=3D475= 5,size=3D11584512,blksize=3D131072 }) =3D 0 (0x0) > 63984: eaccess("/var/db/pkg/local.sqlite",R_OK|W_OK) =3D 0 (0x0) > 63984: stat("/var/db/pkg/repo-FreeBSD.sqlite",{ mode=3D-rw-r--r-- ,inod= e=3D323679,size=3D46235648,blksize=3D131072 }) =3D 0 (0x0) > 63984: eaccess("/var/db/pkg/repo-FreeBSD.sqlite",R_OK|W_OK) =3D 0 (0x0) > 63984: stat("/var/db/pkg",{ mode=3Ddrwxr-xr-x ,inode=3D78,size=3D5,blks= ize=3D4096 }) =3D 0 (0x0) > 63984: eaccess("/var/db/pkg",R_OK|W_OK) =3D 0 (0x0) > 63984: stat("/var/db/pkg/repo-FreeBSD.sqlite",{ mode=3D-rw-r--r-- ,inod= e=3D323679,size=3D46235648,blksize=3D131072 }) =3D 0 (0x0) > 63984: eaccess("/var/db/pkg/repo-FreeBSD.sqlite",R_OK|W_OK) =3D 0 (0x0) > 63984: write(1,"Updating FreeBSD repository cata"...,41) =3D 41 (0x29) > 63984: statfs("/var/db/pkg",{ fstypename=3Dzfs,mntonname=3D/,mntfromnam= e=3Dzrootmjg,fsid=3Da294c20dde42cb7d }) =3D 0 (0x0) > 63984: access("/var/db/pkg/FreeBSD.meta",R_OK) =3D 0 (0x0) > 63984: __getcwd("/root",1024) =3D 0 (0x0) > 63984: lstat("/var",{ mode=3Ddrwxr-xr-x ,inode=3D58,size=3D25,blksize=3D= 4096 }) =3D 0 (0x0) > 63984: lstat("/var/db",{ mode=3Ddrwxr-xr-x ,inode=3D72,size=3D22,blksiz= e=3D4096 }) =3D 0 (0x0) > 63984: lstat("/var/db/pkg",{ mode=3Ddrwxr-xr-x ,inode=3D78,size=3D5,blk= size=3D4096 }) =3D 0 (0x0) > 63984: lstat("/var/db/pkg/FreeBSD.meta",{ mode=3D-rw-r--r-- ,inode=3D32= 3678,size=3D246,blksize=3D4096 }) =3D 0 (0x0) > 63984: stat("/var/db/pkg/FreeBSD.meta",{ mode=3D-rw-r--r-- ,inode=3D323= 678,size=3D246,blksize=3D4096 }) =3D 0 (0x0) > 63984: openat(AT_FDCWD,"/var/db/pkg/FreeBSD.meta",O_RDONLY,00) =3D 4 (0= x4) >=20 > Again, several passes over /var/db/pkg. >=20 > 63984: stat("/etc/nsswitch.conf",{ mode=3D-rw-r--r-- ,inode=3D14316,siz= e=3D255,blksize=3D4096 }) =3D 0 (0x0) > 63984: open("/etc/nsswitch.conf",O_CLOEXEC,0666) =3D 7 (0x7) > 63984: ioctl(7,TIOCGETA,0xffff90f0) ERR#25 'Inappropriate ioctl for d= evice' > 63984: fstat(7,{ mode=3D-rw-r--r-- ,inode=3D14316,size=3D255,blksize=3D= 4096 }) =3D 0 (0x0) > 63984: read(7,"#\n# nsswitch.conf(5) - name ser"...,4096) =3D 255 (0xff= ) > 63984: read(7,0x80305f000,4096) =3D 0 (0x0) > 63984: access("/usr/local/lib/nss_compat.so.1",F_OK) ERR#2 'No such fil= e or directory' > 63984: access("/lib/nss_compat.so.1",F_OK) ERR#2 'No such file or dire= ctory' > 63984: access("/usr/lib/nss_compat.so.1",F_OK) ERR#2 'No such file or = directory' > 63984: access("/usr/lib/compat/nss_compat.so.1",F_OK) ERR#2 'No such fi= le or directory' > 63984: access("/usr/local/lib/nss_compat.so.1",F_OK) ERR#2 'No such fil= e or directory' > 63984: access("/usr/local/lib/gcc5/nss_compat.so.1",F_OK) ERR#2 'No suc= h file or directory' > 63984: access("/usr/local/lib/perl5/5.20/mach/CORE/nss_compat.so.1",F_O= K) ERR#2 'No such file or directory' > 63984: access("/usr/local/llvm36/lib/nss_compat.so.1",F_OK) ERR#2 'No s= uch file or directory' > 63984: access("/usr/local/llvm37/lib/nss_compat.so.1",F_OK) ERR#2 'No s= uch file or directory' > 63984: access("/lib/casper/nss_compat.so.1",F_OK) ERR#2 'No such file o= r directory' > 63984: access("/lib/nss_compat.so.1",F_OK) ERR#2 'No such file or dire= ctory' > 63984: access("/usr/lib/nss_compat.so.1",F_OK) ERR#2 'No such file or = directory' > 63984: access("/usr/local/lib/nss_nis.so.1",F_OK) ERR#2 'No such file o= r directory' > 63984: access("/lib/nss_nis.so.1",F_OK) ERR#2 'No such file or direct= ory' > 63984: access("/usr/lib/nss_nis.so.1",F_OK) ERR#2 'No such file or dir= ectory' > 63984: access("/usr/lib/compat/nss_nis.so.1",F_OK) ERR#2 'No such file = or directory' > 63984: access("/usr/local/lib/nss_nis.so.1",F_OK) ERR#2 'No such file o= r directory' > 63984: access("/usr/local/lib/gcc5/nss_nis.so.1",F_OK) ERR#2 'No such f= ile or directory' > 63984: access("/usr/local/lib/perl5/5.20/mach/CORE/nss_nis.so.1",F_OK) = ERR#2 'No such file or directory' > 63984: access("/usr/local/llvm36/lib/nss_nis.so.1",F_OK) ERR#2 'No such= file or directory' > 63984: access("/usr/local/llvm37/lib/nss_nis.so.1",F_OK) ERR#2 'No such= file or directory' > 63984: access("/lib/casper/nss_nis.so.1",F_OK) ERR#2 'No such file or = directory' > 63984: access("/lib/nss_nis.so.1",F_OK) ERR#2 'No such file or direct= ory' > 63984: access("/usr/lib/nss_nis.so.1",F_OK) ERR#2 'No such file or dir= ectory' > 63984: access("/usr/local/lib/nss_files.so.1",F_OK) ERR#2 'No such file= or directory' > 63984: access("/lib/nss_files.so.1",F_OK) ERR#2 'No such file or direc= tory' > 63984: access("/usr/lib/nss_files.so.1",F_OK) ERR#2 'No such file or d= irectory' > 63984: access("/usr/lib/compat/nss_files.so.1",F_OK) ERR#2 'No such fil= e or directory' > 63984: access("/usr/local/lib/nss_files.so.1",F_OK) ERR#2 'No such file= or directory' > 63984: access("/usr/local/lib/gcc5/nss_files.so.1",F_OK) ERR#2 'No such= file or directory' > 63984: access("/usr/local/lib/perl5/5.20/mach/CORE/nss_files.so.1",F_OK= ) ERR#2 'No such file or directory' > 63984: access("/usr/local/llvm36/lib/nss_files.so.1",F_OK) ERR#2 'No su= ch file or directory' > 63984: access("/usr/local/llvm37/lib/nss_files.so.1",F_OK) ERR#2 'No su= ch file or directory' > 63984: access("/lib/casper/nss_files.so.1",F_OK) ERR#2 'No such file or= directory' > 63984: access("/lib/nss_files.so.1",F_OK) ERR#2 'No such file or direc= tory' > 63984: access("/usr/lib/nss_files.so.1",F_OK) ERR#2 'No such file or d= irectory' > 63984: access("/usr/local/lib/nss_dns.so.1",F_OK) ERR#2 'No such file o= r directory' > 63984: access("/lib/nss_dns.so.1",F_OK) ERR#2 'No such file or direct= ory' > 63984: access("/usr/lib/nss_dns.so.1",F_OK) ERR#2 'No such file or dir= ectory' > 63984: access("/usr/lib/compat/nss_dns.so.1",F_OK) ERR#2 'No such file = or directory' > 63984: access("/usr/local/lib/nss_dns.so.1",F_OK) ERR#2 'No such file o= r directory' > 63984: access("/usr/local/lib/gcc5/nss_dns.so.1",F_OK) ERR#2 'No such f= ile or directory' > 63984: access("/usr/local/lib/perl5/5.20/mach/CORE/nss_dns.so.1",F_OK) = ERR#2 'No such file or directory' > 63984: access("/usr/local/llvm36/lib/nss_dns.so.1",F_OK) ERR#2 'No such= file or directory' > 63984: access("/usr/local/llvm37/lib/nss_dns.so.1",F_OK) ERR#2 'No such= file or directory' > 63984: access("/lib/casper/nss_dns.so.1",F_OK) ERR#2 'No such file or = directory' > 63984: access("/lib/nss_dns.so.1",F_OK) ERR#2 'No such file or direct= ory' > 63984: access("/usr/lib/nss_dns.so.1",F_OK) ERR#2 'No such file or dir= ectory' >=20 > Now that's a bummer, but likely not a pkg issue. >=20 > 63984: mkdir("/tmp",0777) ERR#17 'File exists' >=20 > Why? >=20 > 63984: getpid() =3D 63984 (0xf9f0) > 63984: stat("/tmp",{ mode=3Ddrwxrwxrwt ,inode=3D4,size=3D76,blksize=3D1= 6384 }) =3D 0 (0x0) > 63984: open("/tmp/packagesite.yaml.txz.DlhH4v",O_RDWR|O_CREAT|O_EXCL,06= 00) =3D 5 (0x5) > 63984: unlink("/tmp/packagesite.yaml.txz.DlhH4v") =3D 0 (0x0) > 63984: unlink("") ERR#2 'No such file or directory' >=20 > ? >=20 > 3984: stat("/etc/nsswitch.conf",{ mode=3D-rw-r--r-- > ,inode=3D14316,size=3D255,blksize=3D4096 }) =3D 0 (0x0) > 63984: geteuid() =3D 0 (0x0) > 63984: open("/etc/spwd.db",O_CLOEXEC,00) =3D 12 (0xc) > 63984: fstat(12,{ mode=3D-rw------- ,inode=3D319614,size=3D40960,blksiz= e=3D40960 > }) =3D 0 (0x0) > 63984: read(12,"\0\^F\^Ua\0\0\0\^B\0\0\^D\M-R\0"...,260) =3D 260 (0x104= ) > 63984: pread(0xc,0x803064000,0x1000,0x6000) =3D 4096 (0x1000) > 63984: pread(0xc,0x803063000,0x1000,0x4000) =3D 4096 (0x1000) > 63984: close(12) =3D 0 (0x0) > 63984: stat("/etc/nsswitch.conf",{ mode=3D-rw-r--r-- > ,inode=3D14316,size=3D255,blksize=3D4096 }) =3D 0 (0x0) > 63984: open("/etc/group",O_CLOEXEC,0666) =3D 12 (0xc) > 63984: fstat(12,{ mode=3D-rw-r--r-- ,inode=3D319611,size=3D500,blksize=3D= 4096 }) > =3D 0 (0x0) > 63984: lseek(12,0x0,SEEK_CUR) =3D 0 (0x0) > 63984: lseek(12,0x0,SEEK_SET) =3D 0 (0x0) > 63984: read(12,"# $FreeBSD$\n#\nwheel:*:0:root\n"...,4096) =3D 500 (0x1= f4) > 63984: close(12) =3D 0 (0x0) >=20 > This repeats itself a lot. I wonder if that's an iffeciency in libc or > pkg. I think some of this and the nss stuff earlier is purely libc. There's also timezone lookups in libc that are done quite often that can be optimized. I did some work on this at Isilon to help lookups on the timezone data but it's not really safe for upstream libc since it is dangerous for long-living processes. For NSS perhaps we could use a database to avoid all of the module lookup syscall spam by only doing it when updating the nssswitch.conf fil= e. >=20 > 63984: ioctl(1,TIOCGETA,0xffffced0) =3D 0 (0x0) > 63984: write(1,"[1/1] Extracting rexima-1.4: 0"...,33) =3D 33 (0x21) > 63984: mkdirat(11,"usr",0755) ERR#17 'File exists' > 63984: mkdirat(11,"usr/local",0755) ERR#17 'File exists' > 63984: mkdirat(11,"usr/local/bin",0755) ERR#17 'File exists' It would be better if the do_extract pre-calculated a list of directories needed and made them all then rather than on every single file extraction trying mkdirat_p() again. And then also optimize mkdirs/mkdirat_p to check if the path already exists before walking the tree as you suggest. > 63984: openat(11,"usr/local/bin/rexima.DcozrktiDx8Y",O_WRONLY|O_CREAT|O= _EXCL,0755) =3D 12 (0xc) > 63984: fstat(12,{ mode=3D-rwxr-xr-x ,inode=3D4182,size=3D0,blksize=3D13= 1072 }) =3D 0 (0x0) >=20 > why fstat? It is archive_read_data_into_fd(). Pkg opens it then calls archive_read_data_into_fd: can_lseek =3D (fstat(fd, &st) =3D=3D 0) && S_ISREG(st.st_mode); if (!can_lseek) nulls =3D calloc(1, nulls_size); >=20 > In the common case you can expect these directories to already be there= . > So how about instead you try the open first and only if it fails you tr= y > to create the tree. But you start trying from the top (here: "bin"). >=20 > 63984: > fstatat(11,"usr/local/bin/rexima",0x7fffffffd5f0,AT_SYMLINK_NOFOLLOW) > ERR#2 'No such file or directory' > 63984: > renameat(11,"usr/local/bin/rexima.DcozrktiDx8Y",11,"usr/local/bin/rexim= a") > =3D 0 (0x0) >=20 > Why not just replace? >=20 The fstat is done to check for flag schg and remove if there. Worse though is that pkg is now unlinking before rename which invalidates half of the whole point of using rename (atomic and not leaving files missing). --=20 Regards, Bryan Drewery From owner-freebsd-pkg@freebsd.org Sat Jun 18 11:20:43 2016 Return-Path: Delivered-To: freebsd-pkg@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id E24A3A78B0A for ; Sat, 18 Jun 2016 11:20:43 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mailman.ysv.freebsd.org (unknown [127.0.1.3]) by mx1.freebsd.org (Postfix) with ESMTP id CC1371961 for ; Sat, 18 Jun 2016 11:20:43 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: by mailman.ysv.freebsd.org (Postfix) id C7A72A78B09; Sat, 18 Jun 2016 11:20:43 +0000 (UTC) Delivered-To: pkg@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id C7527A78B08 for ; Sat, 18 Jun 2016 11:20:43 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id B66CA1960 for ; Sat, 18 Jun 2016 11:20:43 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id u5IBKhWc004998 for ; Sat, 18 Jun 2016 11:20:43 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: pkg@FreeBSD.org Subject: [Bug 210370] graphics/ImageMagick7: `pkg install ImageMagick7` needs to be run multiple times Date: Sat, 18 Jun 2016 11:20:43 +0000 X-Bugzilla-Reason: CC X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Ports & Packages X-Bugzilla-Component: Individual Port(s) X-Bugzilla-Version: Latest X-Bugzilla-Keywords: needs-qa X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: vlad-fbsd@acheronmedia.com X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: kwm@FreeBSD.org X-Bugzilla-Flags: maintainer-feedback? X-Bugzilla-Changed-Fields: short_desc cc keywords Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-pkg@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Binary package management and package tools discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 18 Jun 2016 11:20:44 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D210370 Vladimir Krstulja changed: What |Removed |Added ---------------------------------------------------------------------------- Summary|graphics/ImageMagick7 - |graphics/ImageMagick7: `pkg |`pkg install ImageMagick7` |install ImageMagick7` needs |needs to be run twice |to be run multiple times CC| |pkg@FreeBSD.org Keywords| |needs-qa --- Comment #1 from Vladimir Krstulja --- Hm indeed. In my case it says 104 packages will be affected, but after the fetch stage it installs 22. The second run of 'pkg install imagemagick7' installs 79 packages. The third run installs ImageMagick7. CC'ing pkg@ as this looks like a pkg problem to me. --=20 You are receiving this mail because: You are on the CC list for the bug.= From owner-freebsd-pkg@freebsd.org Sat Jun 18 15:29:58 2016 Return-Path: Delivered-To: freebsd-pkg@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id A2A81A78AAC for ; Sat, 18 Jun 2016 15:29:58 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mailman.ysv.freebsd.org (unknown [127.0.1.3]) by mx1.freebsd.org (Postfix) with ESMTP id 8CD1018B4 for ; Sat, 18 Jun 2016 15:29:58 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: by mailman.ysv.freebsd.org (Postfix) id 8C2A4A78AAA; Sat, 18 Jun 2016 15:29:58 +0000 (UTC) Delivered-To: pkg@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 8BCD0A78AA9 for ; Sat, 18 Jun 2016 15:29:58 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 7BA5F18B3 for ; Sat, 18 Jun 2016 15:29:58 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id u5IFTw7U061261 for ; Sat, 18 Jun 2016 15:29:58 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: pkg@FreeBSD.org Subject: [Bug 210370] graphics/ImageMagick7: `pkg install ImageMagick7` needs to be run multiple times Date: Sat, 18 Jun 2016 15:29:58 +0000 X-Bugzilla-Reason: CC X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Ports & Packages X-Bugzilla-Component: Individual Port(s) X-Bugzilla-Version: Latest X-Bugzilla-Keywords: needs-qa X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: bapt@FreeBSD.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: kwm@FreeBSD.org X-Bugzilla-Flags: maintainer-feedback? X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-pkg@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Binary package management and package tools discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 18 Jun 2016 15:29:58 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D210370 --- Comment #2 from Baptiste Daroussin --- The reason it has happened is probably you have no run pkg upgrade before install imagemagick7 You may also have things installed that were conflicts with it and then the solver tries to handle that conflict. In next pkg release the conflicts will be explicitly shown to the users --=20 You are receiving this mail because: You are on the CC list for the bug.=