Date: Sat, 16 Mar 2013 13:51:48 -0700 From: Kevin Oberman <rkoberman@gmail.com> To: Ian Smith <smithi@nimnet.asn.au> Cc: freebsd-stable@freebsd.org, John Mehr <jcm@visi.com> Subject: Re: svn - but smaller? Message-ID: <CAN6yY1sd3rsmK_5JLYDrzLwXLCKUafZoDxr512str9n7w2C6mg@mail.gmail.com> In-Reply-To: <20130315001321.K32142@sola.nimnet.asn.au> References: <web-11636850@mailback4.g2host.com> <513E2DA5.70200@mac.com> <web-12282796@mailback4.g2host.com> <op.wts7cnaeg7njmm@michael-think> <web-11149903@mailback3.g2host.com> <20130313152150.E32142@sola.nimnet.asn.au> <web-11167474@mailback3.g2host.com> <20130315001321.K32142@sola.nimnet.asn.au>
next in thread | previous in thread | raw e-mail | index | archive | help
On Sat, Mar 16, 2013 at 8:14 AM, Ian Smith <smithi@nimnet.asn.au> wrote:
> On Wed, 13 Mar 2013 21:11:28 -0500, John Mehr wrote:
> > On Wed, 13 Mar 2013 16:29:45 +1100 (EST)
> > Ian Smith <smithi@nimnet.asn.au> wrote:
> [..]
> > > I have a small test system on which I'd installed (two instances of)
> 9.1 so
> > > a couple of days ago I fetched ports with portsnap, installed svnup,
> and
> > > ran it using the (just what I needed) example command in svnup(1).
>
> Just to reiterate: this is vanilla releng/9.1, no ports but docs and
> svnup installed, no X, 512MB RAM, nothing happening but occasional ARP.
> For completeness, it's running powerd and acpi_ibm but not cam.ctl.
>
> > > I get about 700KB/s here, and svnup took about 15 minutes to update
> 9.1
> > > sources to 9-stable. This is fine. Last night I ran it again, but
> it took
> > > 12:42 to make no changes. This seemed puzzling, as you'd said only a
> few
> > > minutes for subsequent updates, but the reason appears to be that in
> both
> > > cases, I ran it in script(1), and the default verbosity of 1 includes
> a
> > > listing of every directory and file examined, followed by <CR> then
> <erase
> > > to eol> codes. Even in less -r (raw) mode it still has to display
> and skip
> > > through all the (now invisible) lines; bit messy.
> > >
> > > Even the second do-nothing run made a 2MB script file, the original
> with
> > > all 9.1 to -stable updates being 3.4MB. So I'd love the option to
> only
> > > list the changes (- and +) and simply ignore unchanged dirs/files
> without
> > > any display for use in script(1). Apart from that, I'm happy.
> >
> > Which mirror are you using? I ran several tests tonight repeatedly
> fetching
> > 9/stable from svn0.us-west (so they would all be do-nothing runs) both
> inside
> > and outside of a script(1) capture and on both an old SSD and on a ZFS
> > mirrored array (to see if the target media made any difference) and
> they all
> > completed in 2 minutes, 43 seconds +/- 2 seconds on my 350 KB/s
> connection.
>
> svn0.us-west.freebsd.org as per example, but it's the closest to here
> anyway; pings avg 217.5ms stddev 0.4ms, us-east avg 269.3ms sd 9.5ms.
> fastest_cvsup used to find cvs mirrors in Australia at around 55-70ms,
> so for another while I thought RTT might be the issue - but it's not.
>
> > I'll definitely put in a verbosity level that does exactly what you
> suggest.
> > Sorry about that.
>
> Not at all, and thanks! But neither, I now find, is logging the extra
> time issue; as mentioned elsewhere, running at -v0 makes next to no
> difference, 12:48 west vs 12:54 east on 4 do-nothing runs, 700KB/s link,
> not running in script(1) - though I think that matters not, or little.
>
> So I began watching top more closely, also running iostat -w10, noticing
> that svnup was pegging CPU at 100% with ~75% _system_? time for nearly 9
> minutes of the run, doing little or no disk I/O and a max of 130KB/s in,
> 22.6KB/s out, no stress there, growing to ~20MB resident, then doing a
> flurry of disk I/O at about 3-5MB/s for the last 3m40s odd. IOW, svnup
> here is mostly heavily CPU-bound, which I don't understand, but then I
> know nothing about what level of computation the protocol demands.
>
> I've done some more controlled tests since, running both iostat -w10 and
> netstat -w10 -dn before, during and until a bit after the run, attached,
> hoping these might help pinpoint any bottlenecks? This was running -v1
> default verbosity, hence the (not at all heavy) tty output numbers.
>
> Just sometimes, running slower hardware can have advantages! On your
> times, yours likely has more than 500% of my single-core P3-M's 1133MHz
> performance, which would tend to mask this issue pretty well for you :)
>
> John Baldwin has best described when c{,v}sup deletes (only in-tree)
> files, covered in section 'delete' in csup(1). I think this should
> probably be default behaviour, even if svn proper would do otherwise.
>
> cheers, Ian
>
Ian,
Have you looked with either truss or ktrace? This looks a lot like some
silliness like repeated calls to expensive system service (e.g.
gettimeofday(3) or stat(3)). If it's something like that, It can possibly
be improved a lot fairly easily. My best guess is some sort of graph of the
ports tree is going on, but that is just a wild guess as I have never
looked at or installed the code.
--
R. Kevin Oberman, Network Engineer
E-mail: rkoberman@gmail.com
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAN6yY1sd3rsmK_5JLYDrzLwXLCKUafZoDxr512str9n7w2C6mg>