Date: Tue, 28 May 2013 22:24:15 +0200 From: Lars Engels <lars.engels@0x20.net> To: Michael Gmelin <freebsd@grem.de> Cc: ports@freebsd.org, RW <rwmaillists@googlemail.com> Subject: Re: The vim port needs a refresh Message-ID: <20130528202415.GC9788@e-new.0x20.net> In-Reply-To: <20130528195137.312c58d9@bsd64.grem.de> References: <20130524212318.B967FE6739@smtp.hushmail.com> <op.wxsmqsh834t2sn@markf.office.supranet.net> <51A4ADCC.4070204@marino.st> <20130528151600.4eb6f028@gumby.homeunix.com> <20130528195137.312c58d9@bsd64.grem.de>
next in thread | previous in thread | raw e-mail | index | archive | help
--nmemrqcdn5VTmUEE Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Tue, May 28, 2013 at 07:51:37PM +0200, Michael Gmelin wrote: > On Tue, 28 May 2013 15:16:00 +0100 > RW <rwmaillists@googlemail.com> wrote: >=20 > > On Tue, 28 May 2013 15:14:52 +0200 > > John Marino wrote: > >=20 > > > All > > > patches only take 74 seconds to download[2] so there is no sympathy > > > for your obviously single data point anecdote,=20 > >=20 > > Well at the point you provided one data-point there was only one data > > point. And it was like pulling teeth to get you to eliminate the > > alternative explanations. Was it really too much to ask that you > > provided some actual evidence.=20 > >=20 > > > you're clearly doing > > > something wrong. You need to stop complaining and start think about > > > folks with slow connections[3] who also rebuild Vim frequently.=20 > >=20 > > Don't make things up. I never said anything about frequent rebuilds, > > the patches all get redownloaded on the next rebuild.=20 >=20 > The real issue is not the number of patches, but the fact that every > single patch is downloaded by invoking the fetch(1) command, creating > lots of overhead not limited to the fetch command itself. The ports > system wasn't designed for such an amount of distfiles in a single > port I guess. >=20 > I just timed fetching the patches through ports vs. fetching over > HTTP/1.1 using ftp/curl vs calling fetch directly. The VIM tarball was > already downloaded, so this is really just the patches (plus > downloading 6mb is barely noticeable on a fast line). It's a slow > machine on a fast line. >=20 > Fetch: > [user@server /usr/ports/editors/vim]$ time sudo make fetch > .... > real 4m57.327s > user 0m17.010s > sys 0m39.588s >=20 > Curl: > [user@server /tmp]$ longcurlcommandline=20 > .... > real 0m15.291s > user 0m0.026s > sys 0m0.272s >=20 > Fetch on the command line (after initial make fetch, so this is only > measuring transmission of the files): > cd /usr/ports/editors/distfiles > time for name in 7.3.*; do > fetch http://artfiles.org/vim.org/patches/7.3/$name > done > .... > real 1m25.329s > user 0m0.660s > sys 0m3.174s >=20 > So just the fact we're invoking fetch for every file costs us about one > minute - I assume the time lost is much bigger on a slow line with > long latency. The remaining 3.5 minutes are spent somewhere in the > ports infrastructure and clearly depend on the performance of the > machine used. For comparison I timed "make fetch" on a reasonably fast > server (good IO, fast datacenter connection), make fetch still took > about 120 seconds(!). >=20 > So the bottomline is: > - Using HTTP/1.1 and keepalive could safe a lot of time > - The ports infrastructure creates a lot of overhead per patch file >=20 > Maybe there's something we can do to improve the situation. >=20 > Cheers, > Michael >=20 > PS: I don't use vim myself and have no stake in this discussion > whatsoever. Someone in this thread proposed to change the port to use phttpget, so I gave it a try using a German mirror nearby with 6 Mbit/s downlink: $ time /usr/libexec/phttpget ftp.vim.ossmirror.de $(eval echo /pub/vim/patc= hes/7.3/{$(make -C /usr/ports/editors/vim -VPATCHFILES | sed 's/\ /,/g')}) http://ftp.vim.ossmirror.de//pub/vim/patches/7.3/7.3.001: 200 OK http://ftp.vim.ossmirror.de//pub/vim/patches/7.3/7.3.002: 200 OK http://ftp.vim.ossmirror.de//pub/vim/patches/7.3/7.3.003: 200 OK http://ftp.vim.ossmirror.de//pub/vim/patches/7.3/7.3.004: 200 OK http://ftp.vim.ossmirror.de//pub/vim/patches/7.3/7.3.005: 200 OK [...] http://ftp.vim.ossmirror.de//pub/vim/patches/7.3/7.3.974: 200 OK http://ftp.vim.ossmirror.de//pub/vim/patches/7.3/7.3.984: 200 OK http://ftp.vim.ossmirror.de//pub/vim/patches/7.3/7.3.985: 200 OK http://ftp.vim.ossmirror.de//pub/vim/patches/7.3/7.3.1000: 200 OK real 0m12.509s user 0m0.154s sys 0m0.089s That's really nice!=20 Compare this to the current version using fetch(1): time make PATCH_SITES=3Dhttp://ftp.vim.ossmirror.de/pub/vim/patches/7.3/ f= etch =3D=3D=3D> Found saved configuration for vim-7.3.669_1 =3D=3D=3D> vim-7.3.1014 depends on file: /usr/local/sbin/pkg - found =3D> 7.3.002 doesn't seem to exist in /usr/ports/distfiles/vim. =3D> Attempting to fetch http://ftp.vim.ossmirror.de/pub/vim/patches/7.3/7.= 3.002 7.3.002 100% of 1610 B 16 MBps 00m= 00s =3D> 7.3.003 doesn't seem to exist in /usr/ports/distfiles/vim. =3D> Attempting to fetch http://ftp.vim.ossmirror.de/pub/vim/patches/7.3/7.= 3.003 7.3.003 100% of 1299 B 1281 kBps 00m= 00s =3D> 7.3.004 doesn't seem to exist in /usr/ports/distfiles/vim. [...] =3D> 7.3.984 doesn't seem to exist in /usr/ports/distfiles/vim. =3D> Attempting to fetch http://ftp.vim.ossmirror.de/pub/vim/patches/7.3/7.= 3.984 7.3.984 100% of 1706 B 2852 kBps 00m= 00s =3D> 7.3.985 doesn't seem to exist in /usr/ports/distfiles/vim. =3D> Attempting to fetch http://ftp.vim.ossmirror.de/pub/vim/patches/7.3/7.= 3.985 7.3.985 100% of 1691 B 14 MBps 00m= 00s =3D> 7.3.1000 doesn't seem to exist in /usr/ports/distfiles/vim. =3D> Attempting to fetch http://ftp.vim.ossmirror.de/pub/vim/patches/7.3/7.= 3.1000 7.3.1000 100% of 1637 B 1715 kBps 00m= 00s =3D=3D=3D> Fetching all distfiles required by vim-7.3.1014 for building Total time : 3:48.55s CPU utilisation (percentage) : 54.5% That's much slower using the same mirror. --nmemrqcdn5VTmUEE Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.19 (FreeBSD) iEYEARECAAYFAlGlEm8ACgkQKc512sD3afhN9QCgllxb6HHneys+65No3FHoDFe6 vhAAoIp6ciekLeaSIdArhC4Gxr+QwbT5 =+Xvf -----END PGP SIGNATURE----- --nmemrqcdn5VTmUEE--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20130528202415.GC9788>