From owner-freebsd-ports@FreeBSD.ORG Tue May 28 21:14:34 2013 Return-Path: Delivered-To: freebsd-ports@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 18580A3C for ; Tue, 28 May 2013 21:14:34 +0000 (UTC) (envelope-from matthias.andree@gmx.de) Received: from mout.gmx.net (mout.gmx.net [212.227.17.21]) by mx1.freebsd.org (Postfix) with ESMTP id B99699F1 for ; Tue, 28 May 2013 21:14:33 +0000 (UTC) Received: from mailout-de.gmx.net ([10.1.76.20]) by mrigmx.server.lan (mrigmx002) with ESMTP (Nemesis) id 0LqXa3-1UCUFf2tqj-00e5Ul for ; Tue, 28 May 2013 23:14:26 +0200 Received: (qmail invoked by alias); 28 May 2013 21:14:26 -0000 Received: from f049073108.adsl.alicedsl.de (EHLO mandree.no-ip.org) [78.49.73.108] by mail.gmx.net (mp020) with SMTP; 28 May 2013 23:14:26 +0200 X-Authenticated: #428038 X-Provags-ID: V01U2FsdGVkX19ZjdFyOTlZYfJ8/oT94rzUslALrZ2rjuqaIQVTLQ TJjf7Pe58uZrYB Received: from [IPv6:::1] (localhost6.localdomain6 [IPv6:::1]) by apollo.emma.line.org (Postfix) with ESMTP id 33FC023CEC5 for ; Tue, 28 May 2013 23:14:25 +0200 (CEST) Message-ID: <51A51E31.9040508@gmx.de> Date: Tue, 28 May 2013 23:14:25 +0200 From: Matthias Andree User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:17.0) Gecko/20130510 Thunderbird/17.0.6 MIME-Version: 1.0 To: freebsd-ports@freebsd.org Subject: Re: The vim port needs a refresh References: <20130524212318.B967FE6739@smtp.hushmail.com> <51A4ADCC.4070204@marino.st> <20130528151600.4eb6f028@gumby.homeunix.com> <20130528195137.312c58d9@bsd64.grem.de> <20130528202415.GC9788@e-new.0x20.net> In-Reply-To: <20130528202415.GC9788@e-new.0x20.net> X-Enigmail-Version: 1.4.6 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-Y-GMX-Trusted: 0 X-BeenThere: freebsd-ports@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Porting software to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 28 May 2013 21:14:34 -0000 -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Am 28.05.2013 22:24, schrieb Lars Engels: > Someone in this thread proposed to change the port to use phttpget, so I > gave it a try using a German mirror nearby with 6 Mbit/s downlink: > > $ time /usr/libexec/phttpget ftp.vim.ossmirror.de $(eval echo /pub/vim/patches/7.3/{$(make -C /usr/ports/editors/vim -VPATCHFILES | sed 's/\ /,/g')}) > http://ftp.vim.ossmirror.de//pub/vim/patches/7.3/7.3.001: 200 OK > http://ftp.vim.ossmirror.de//pub/vim/patches/7.3/7.3.002: 200 OK > http://ftp.vim.ossmirror.de//pub/vim/patches/7.3/7.3.003: 200 OK > http://ftp.vim.ossmirror.de//pub/vim/patches/7.3/7.3.004: 200 OK > http://ftp.vim.ossmirror.de//pub/vim/patches/7.3/7.3.005: 200 OK > [...] > http://ftp.vim.ossmirror.de//pub/vim/patches/7.3/7.3.974: 200 OK > http://ftp.vim.ossmirror.de//pub/vim/patches/7.3/7.3.984: 200 OK > http://ftp.vim.ossmirror.de//pub/vim/patches/7.3/7.3.985: 200 OK > http://ftp.vim.ossmirror.de//pub/vim/patches/7.3/7.3.1000: 200 OK > > real 0m12.509s > user 0m0.154s > sys 0m0.089s > > > That's really nice! > > Compare this to the current version using fetch(1): This incurs massive overhead from the generation of the fetch URLs, on top of "fetch" being slower. You would have to come up with something similar to your phttpget line, only that it runs fetch(1) instead. Try: $ make -C /usr/ports/editors/vim fetch FETCH_CMD=true to see how slow URL generation is. On one of my reasonably fast amd64 machines (dual-core 2.1 GHz AMD BE-2350), residing on the T-Home domestic backbone near the Ruhr area in Germany with a 3000/384 kBit/s ADSL that has roughly 50 ms ping round trips, it generates approximately 1.7 fetch commands in 1 s, and see below for fetch speed: > time make PATCH_SITES=http://ftp.vim.ossmirror.de/pub/vim/patches/7.3/ fetch > ===> Found saved configuration for vim-7.3.669_1 > ===> vim-7.3.1014 depends on file: /usr/local/sbin/pkg - found > => 7.3.002 doesn't seem to exist in /usr/ports/distfiles/vim. > => Attempting to fetch http://ftp.vim.ossmirror.de/pub/vim/patches/7.3/7.3.002 > 7.3.002 100% of 1610 B 16 MBps 00m00s > => 7.3.003 doesn't seem to exist in /usr/ports/distfiles/vim. > => Attempting to fetch http://ftp.vim.ossmirror.de/pub/vim/patches/7.3/7.3.003 > 7.3.003 100% of 1299 B 1281 kBps 00m00s > => 7.3.004 doesn't seem to exist in /usr/ports/distfiles/vim. > [...] > => 7.3.984 doesn't seem to exist in /usr/ports/distfiles/vim. > => Attempting to fetch http://ftp.vim.ossmirror.de/pub/vim/patches/7.3/7.3.984 > 7.3.984 100% of 1706 B 2852 kBps 00m00s > => 7.3.985 doesn't seem to exist in /usr/ports/distfiles/vim. > => Attempting to fetch http://ftp.vim.ossmirror.de/pub/vim/patches/7.3/7.3.985 > 7.3.985 100% of 1691 B 14 MBps 00m00s > => 7.3.1000 doesn't seem to exist in /usr/ports/distfiles/vim. > => Attempting to fetch http://ftp.vim.ossmirror.de/pub/vim/patches/7.3/7.3.1000 > 7.3.1000 100% of 1637 B 1715 kBps 00m00s > ===> Fetching all distfiles required by vim-7.3.1014 for building > > Total time : 3:48.55s > CPU utilisation (percentage) : 54.5% Now, using a similar approach as yours, I get, with fetch: $ time fetch $(eval echo -O http://ftp.vim.ossmirror.de/pub/vim/patches/7.3/{$(make -C /usr/ports/editors/vim -VPATCHFILES | tr ' ' ',')}) ... real 2m2.841s user 0m1.083s sys 0m5.238s Which takes some 2 min of wasted CPU time out of your calculation. Not that it helps overly in terms of wallclock time. Using curl with -O cuts this down to: real 1m5.195s user 0m0.668s sys 0m1.479s And wget -nv to: real 1m15.426s user 0m0.845s sys 0m1.536s Finally, so you can compare gains whilst taking line speed out of the picture, phttpget: real 0m22.673s user 0m0.530s sys 0m0.659s The computer on its link should be able to fetch a tarball in c. 11 s wallclock time, we have 3.7 MB of payload in patches and the link manages roughly 355 kB/s. Now, any takers for "make make fetch more efficient" jobs? -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with undefined - http://www.enigmail.net/ iEYEARECAAYFAlGlHjAACgkQvmGDOQUufZWaNgCgjERowRAqRo2Rgv8rQOZQiA7f ahcAoMkjJxeFHGBe7hcZIjZb8rrLh+0/ =XoQd -----END PGP SIGNATURE-----