From owner-freebsd-stable@FreeBSD.ORG Sun Mar 17 01:25:37 2013 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 19DBEBC2 for ; Sun, 17 Mar 2013 01:25:37 +0000 (UTC) (envelope-from jcm@visi.com) Received: from g2host.com (mailback4.g2host.com [208.42.184.244]) by mx1.freebsd.org (Postfix) with ESMTP id AF393AFF for ; Sun, 17 Mar 2013 01:25:35 +0000 (UTC) Received: from [208.42.90.57] (account jcm@visi.com) by mailback4.g2host.com (CommuniGate Pro WEBUSER 5.3.11) with HTTP id 12358340 for freebsd-stable@freebsd.org; Sat, 16 Mar 2013 20:25:29 -0500 From: "John Mehr" Subject: Re: svn - but smaller? To: X-Mailer: CommuniGate Pro WebUser v5.3.11 Date: Sat, 16 Mar 2013 20:25:29 -0500 Message-ID: In-Reply-To: <20130315001321.K32142@sola.nimnet.asn.au> References: <513E2DA5.70200@mac.com> <20130313152150.E32142@sola.nimnet.asn.au> <20130315001321.K32142@sola.nimnet.asn.au> MIME-Version: 1.0 Content-Type: text/plain;charset=iso-8859-1; format="flowed" Content-Transfer-Encoding: 8bit X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 17 Mar 2013 01:25:37 -0000 On Sun, 17 Mar 2013 02:14:30 +1100 (EST)  Ian Smith wrote: > On Wed, 13 Mar 2013 21:11:28 -0500, John Mehr wrote: > > On Wed, 13 Mar 2013 16:29:45 +1100 (EST) > >  Ian Smith wrote: > svn0.us-west.freebsd.org as per example, but it's the >closest to here > anyway; pings avg 217.5ms stddev 0.4ms, us-east avg >269.3ms sd 9.5ms. > fastest_cvsup used to find cvs mirrors in Australia at >around 55-70ms, > so for another while I thought RTT might be the issue - >but it's not. I'm seeing ~110ms pings to svn0.us-west.freebsd.org. > > > I'll definitely put in a verbosity level that does >exactly what you suggest.  > > Sorry about that. > > Not at all, and thanks!  But neither, I now find, is >logging the extra > time issue; as mentioned elsewhere, running at -v0 makes >next to no > difference, 12:48 west vs 12:54 east on 4 do-nothing >runs, 700KB/s link, > not running in script(1) - though I think that matters >not, or little. > > So I began watching top more closely, also running >iostat -w10, noticing > that svnup was pegging CPU at 100% with ~75% _system_? >time for nearly 9 > minutes of the run, doing little or no disk I/O and a >max of 130KB/s in, > 22.6KB/s out, no stress there, growing to ~20MB >resident, then doing a > flurry of disk I/O at about 3-5MB/s for the last 3m40s >odd.  IOW, svnup > here is mostly heavily CPU-bound, which I don't >understand, but then I > know nothing about what level of computation the >protocol demands. Except for the time duration of the run, the behavior you're describing is normal.  Using the svn protocol, the program runs in three phases: 1) It issues a series of get-dir commands to the server to get a list of all the files/directories in the repository and stores the structure in memory.  There is some disk activity during this as it creates the directory tree (it uses one lstat per directory and creates the directory if it doesn't exist).  On my development machine, this phase takes about 25% of the total time of a do-nothing run. 2) Because the svn protocol doesn't include MD5 signatures and the last-author/committed date/committed revision fields in the results of the get-dir commands although they *are* included in the http protocol responses it has to issue a get-file request for each file to get this information.  This phase takes about 50% of the total time of a do-nothing run. 3) The downloaded MD5 signatures are for versions of the files that do not have their $FreeBSD$ tags expanded, so each file on the local machine must be loaded into memory and have their $FreeBSD: ... $ tags collapsed and MD5 summed to see if there's a match.  If there's no match, the file on the server is downloaded.  This phase takes up the remaining 25% and is I/O intensive. This three step process is further complicated because of the huge time penalty that occurs when sending commands one-at-a-time.  To get around this, as many commands that can fit in 32KB are grouped together and sent in bulk. > I've done some more controlled tests since, running both >iostat -w10 and > netstat -w10 -dn before, during and until a bit after >the run, attached, > hoping these might help pinpoint any bottlenecks?  This >was running -v1 > default verbosity, hence the (not at all heavy) tty >output numbers. > > Just sometimes, running slower hardware can have >advantages!  On your > times, yours likely has more than 500% of my single-core >P3-M's 1133MHz > performance, which would tend to mask this issue pretty >well for you :) I do all of my development and test runs on a nettop with an AMD E-350 (dual core, 1.6GHz) processor, and with the program only utilizing one core I'd be willing to bet the two chips are in roughly the same league.  Memory and hard drives probably aren't, and with 512MB of memory, how does your swap utilization look? As soon as I can get a replacement CMOS battery, I'll be dusting off my old Athlon 2700+ system to run tests on it. > John Baldwin has best described when c{,v}sup deletes >(only in-tree) > files, covered in section 'delete' in csup(1).  I think >this should > probably be default behaviour, even if svn proper would >do otherwise. Will do.  Setting up both a default of deleting only files that appear in previous checkouts plus giving each user the option of skipping specific directories from any pruning is on the to-do list.  Thanks! > > cheers, Ian