From owner-freebsd-stable@FreeBSD.ORG Sun Feb 24 16:42:46 2013 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id CF5BD2BF for ; Sun, 24 Feb 2013 16:42:46 +0000 (UTC) (envelope-from jcm@visi.com) Received: from g2host.com (mailback3.g2host.com [208.42.184.243]) by mx1.freebsd.org (Postfix) with ESMTP id 7DCDB90D for ; Sun, 24 Feb 2013 16:42:45 +0000 (UTC) Received: from [208.42.90.57] (account jcm@visi.com) by mailback3.g2host.com (CommuniGate Pro WEBUSER 5.3.11) with HTTP id 10871226 for freebsd-stable@freebsd.org; Sun, 24 Feb 2013 10:42:44 -0600 From: "John Mehr" Subject: Re: svn - but smaller? To: X-Mailer: CommuniGate Pro WebUser v5.3.11 Date: Sun, 24 Feb 2013 10:42:44 -0600 Message-ID: In-Reply-To: <20130224063110.GA53348@icarus.home.lan> References: <1359320641-6493504.60501067.fr0RL3aYw027137@rs149.luxsci.com> <1359380582-6256705.77592125.fr0SDgrYH000991@rs149.luxsci.com> <20130224031509.GA47838@icarus.home.lan> <20130224041638.GA51493@icarus.home.lan> <20130224063110.GA53348@icarus.home.lan> MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" X-Content-Filtered-By: Mailman/MimeDel 2.1.14 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 24 Feb 2013 16:42:46 -0000 On Sat, 23 Feb 2013 22:31:10 -0800 Jeremy Chadwick wrote: > > I downloaded it and looked at the source. > > svnup.c:1002: warning: zero-length printf format string > svnup.c:1020: warning: zero-length printf format string > svnup.c:1027: warning: zero-length printf format string > svnup.c:1065: warning: zero-length printf format string > > 1002 sprintf(command, ""); > 1020 sprintf(command, ""); > 1027 sprintf(command, ""); > 1065 sprintf(command, >""); > > I'm not sure what the intention is here. If it's to >terminate the > buffer, then all of these should be: > > command[0] = '\0'; Correct. Clang didn't complain when I went the sprintf route... :( > Also, John, please consider using malloc(3) instead of >heap-allocated > buffers like file_buffer[6][] (196608 bytes) and >command[] (32769 > bytes). I'm referring to this: > > 47 #define COMMAND_BUFFER 32768 > 386 char new_path_target[BUFFER_UNIT], >file_buffer[6][COMMAND_BUFFER], *path_source; > 836 char *start, *value, *addr, *branch, >*path_target, temp_file[BUFFER_UNIT], >command[COMMAND_BUFFER + 1]; These were leftovers from a malloc debug session that I forgot to undo. Processing the ports tree needs way more that six buffers... > I should also point out that watching this thing in >top(1) shows RES/RSS > increasing until the segfault (for me dies out around >4.5MBytes). I > don't know if that's by design or not, but I don't see >why this thing > would need so much memory given what it's doing. You'd >know better than > I though. The code is definitely going to be memory intensive. The initial, naive implementation I wrote sent get-file and get-dir requests one at a time and the time penalty was atrocious -- it took four hours to process base/releng/8.3. To get around this, since the subversion server can handle multiple requests at a time, I have to cram as many get-file and get-dir requests together as I can to minimize the number of interactions with the server. > You may want to consider running this under valgrind. > It's remarkable > what you can find with that during debugging. Will do. > > -- > | Jeremy Chadwick > jdc@koitsu.org | > | UNIX Systems Administrator > http://jdc.koitsu.org/ | > | Mountain View, CA, US > | > | Making life hard for others since 1977. > PGP 4BD6C0CB | > I've got a rough fix in place and the ports tree is chugging along now (I only tested my code against the base/releng and base/stable branches -- oops). I should be able to post revised code tonight. Thanks for taking a look at this. As a first-timer here, I definitely appreciate it!