From owner-freebsd-current@FreeBSD.ORG Fri Oct 17 14:14:49 2014 Return-Path: Delivered-To: freebsd-current@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id BB0649A7; Fri, 17 Oct 2014 14:14:49 +0000 (UTC) Received: from mail-wi0-x234.google.com (mail-wi0-x234.google.com [IPv6:2a00:1450:400c:c05::234]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 2CF8CCB5; Fri, 17 Oct 2014 14:14:49 +0000 (UTC) Received: by mail-wi0-f180.google.com with SMTP id em10so1391026wid.1 for ; Fri, 17 Oct 2014 07:14:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=date:from:to:cc:subject:message-id:mail-followup-to:mime-version :content-type:content-disposition:user-agent; bh=OrlPgThDKp4yqUgxmtF4rmDACry4QLRbHcMzn2oHu5k=; b=cdSBkgqYMfBw24qly+HEsV6kUn0Kqkv/cl7KETYHlMFqaKaF/x9iRKOaVH4k8MTUiE igQ0hIjnKKeNLk+F65s6a/EHckk10yBINANYNL0g/P/GsyA9JtewCNXHUtN2IAjqXmeR Wwhb7t1KSrDgi1yGwXmp35q2RyHpb5t5XN6KkrCKhHRFB65jdhO/2+s31sJ0sbLIV7QO 9xcBryWhMogX2AqvAEPF1ANsmn1PeGP5p8skT5Qdzg/IJOBJKFFpOqNsYf41oiJrATf5 yL4AWWdlnHWUyMiTZK7QM8tOhdh2wm2eF+tF1ga7AFYRS2zDOvLkhXCzsrLUaUq8JSHG Bvaw== X-Received: by 10.180.76.41 with SMTP id h9mr14521907wiw.40.1413555287493; Fri, 17 Oct 2014 07:14:47 -0700 (PDT) Received: from dft-labs.eu (n1x0n-1-pt.tunnel.tserv5.lon1.ipv6.he.net. [2001:470:1f08:1f7::2]) by mx.google.com with ESMTPSA id w10sm1855466wje.10.2014.10.17.07.14.46 for (version=TLSv1.2 cipher=RC4-SHA bits=128/128); Fri, 17 Oct 2014 07:14:46 -0700 (PDT) Date: Fri, 17 Oct 2014 16:14:44 +0200 From: Mateusz Guzik To: freebsd-current@freebsd.org Subject: libmap.conf vs packages and whatnot Message-ID: <20141017141444.GA16591@dft-labs.eu> Mail-Followup-To: Mateusz Guzik , freebsd-current@freebsd.org, bapt@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline User-Agent: Mutt/1.5.21 (2010-09-15) Cc: bapt@freebsd.org X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 17 Oct 2014 14:14:49 -0000 Hello, turns out that we got some additional cost when executing binaries. I'm not declaring I'm going to do any work on this, but I'd like to state why I don't like current approach. It's just my $0,03. So we got libmap.conf + /usr/local/etc/libmap.d directory. This results in: lstat("/etc",{ mode=drwxr-xr-x ,inode=22,size=117,blksize=7680 }) = 0 (0x0) lstat("/etc/libmap.conf",{ mode=-rw-r--r-- ,inode=384351,size=47,blksize=4096 }) = 0 (0x0) open("/etc/libmap.conf",O_CLOEXEC,0146) = 3 (0x3) fstat(3,{ mode=-rw-r--r-- ,inode=384351,size=47,blksize=4096 }) = 0 (0x0) mmap(0x0,47,PROT_READ,MAP_PRIVATE,3,0x0) = 34366185472 (0x800626000) close(3) = 0 (0x0) lstat("/usr",{ mode=drwxr-xr-x ,inode=9,size=17,blksize=4096 }) = 0 (0x0) lstat("/usr/local",{ mode=drwxr-xr-x ,inode=563,size=17,blksize=4096 }) = 0 (0x0) lstat("/usr/local/etc",{ mode=drwxr-xr-x ,inode=16407,size=44,blksize=4096 }) = 0 (0x0) lstat("/usr/local/etc/libmap.d",{ mode=drwxr-xr-x ,inode=12994,size=2,blksize=4096 }) = 0 (0x0) open("/usr/local/etc/libmap.d",O_NONBLOCK|O_DIRECTORY|O_CLOEXEC,0165) = 3 (0x3) fstatfs(0x3,0x7fffffffcaa0,0x0,0x400,0x1000,0x8080808080808080) = 0 (0x0) getdirentries(0x3,0x800624000,0x1000,0x800623028,0x1000,0x8080808080808080) = 24 (0x18) getdirentries(0x3,0x800624000,0x1000,0x800623028,0x1000,0x8080808080808080) = 0 (0x0) close(3) = 0 (0x0) For each exec, even though in vast majority of cases there is no need for this. Also note the directory is empty, otherwise this is another round of lstats + open for each file. I'm told this will be needed for ports so that they can shuffle libraries for interested programs. I don't know problem area very well, so please correct me if this stuff cannot work. First off, if there is no way around needing to parse data, I would suggest generating a binary blob which can be mmapped and quickly traversed. 1st case: we want new lib for each program just replace a symlink (or better yet, a hardlink) to the old one. 2nd case: only selected programs should get new lib For each prog in need, one can create /usr/local/etc/libmap.d/${prog}. And there you go, one open and if it failed there is nothing to do. Path in question can be a symlink to a bigger file with a set of remmapped libraries, or possibly can 'include ${some_set}'. It is unclear to me whether ports tree guarantees unique names of binaries. If not, maybe there will be an easy way to guarantee lookups or unique stuff anyway (putting some crap in during compilation/linkage?). This way the cost for uninterested progs is greatly reduced. Interested progs also get this faster, as long as there are more than 3 files (as with current approach one would have to parse them all). -- Mateusz Guzik