From owner-freebsd-stable@FreeBSD.ORG Sun Feb 24 06:31:13 2013 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 5C99A7F0 for ; Sun, 24 Feb 2013 06:31:13 +0000 (UTC) (envelope-from jdc@koitsu.org) Received: from qmta01.emeryville.ca.mail.comcast.net (qmta01.emeryville.ca.mail.comcast.net [IPv6:2001:558:fe2d:43:76:96:30:16]) by mx1.freebsd.org (Postfix) with ESMTP id 18D9C16C9 for ; Sun, 24 Feb 2013 06:31:13 +0000 (UTC) Received: from omta20.emeryville.ca.mail.comcast.net ([76.96.30.87]) by qmta01.emeryville.ca.mail.comcast.net with comcast id 44ec1l0051smiN4A16XCdQ; Sun, 24 Feb 2013 06:31:12 +0000 Received: from koitsu.strangled.net ([67.180.84.87]) by omta20.emeryville.ca.mail.comcast.net with comcast id 46XB1l0011t3BNj8g6XBDF; Sun, 24 Feb 2013 06:31:11 +0000 Received: by icarus.home.lan (Postfix, from userid 1000) id F071173A31; Sat, 23 Feb 2013 22:31:10 -0800 (PST) Date: Sat, 23 Feb 2013 22:31:10 -0800 From: Jeremy Chadwick To: Michael Ross Subject: Re: svn - but smaller? Message-ID: <20130224063110.GA53348@icarus.home.lan> References: <1359320641-6493504.60501067.fr0RL3aYw027137@rs149.luxsci.com> <1359380582-6256705.77592125.fr0SDgrYH000991@rs149.luxsci.com> <20130224031509.GA47838@icarus.home.lan> <20130224041638.GA51493@icarus.home.lan> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: attachment In-Reply-To: User-Agent: Mutt/1.5.21 (2010-09-15) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=comcast.net; s=q20121106; t=1361687472; bh=dvi27qLxjGrKb/1U8RvbaYefWgqxtSyS0d6rwQ0z4ow=; h=Received:Received:Received:Date:From:To:Subject:Message-ID: MIME-Version:Content-Type; b=V9pngpZ5bDlvt+jsJ/NkvkCdRwj2ebvfZPsNVSCcF60U+vaRcQGLUSrmshST0SUpv s7mM8RuuYwAvOqEd+wzrk+3RcWICHo8UFJQrliyvNuZr5HdVwIxW59E1/vq81FYfEg pavR5aGLSX1FjwuY4Gee9vPeLEC/hGWO8jHGTPFwxedB23AR6s0uLfGeTvowECtu3p /H9HxVJf2SzCULu8NW51NkvqmMxAv/4i0urhR8IEpVQeQ5W6T5LK5PWA3+ThP/7GKR 98h9qnouDidRIglGcYc/Boq25+eNOsGB45ZYLY4WjcZWrXIeysI9kcfryx4oNVeCXe ZHH9Hi1o2R8cw== Cc: freebsd-stable@freebsd.org, John Mehr X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 24 Feb 2013 06:31:13 -0000 On Sun, Feb 24, 2013 at 05:44:10AM +0100, Michael Ross wrote: > On Sun, 24 Feb 2013 05:16:38 +0100, Jeremy Chadwick wrote: > > >On Sun, Feb 24, 2013 at 04:56:23AM +0100, Michael Ross wrote: > >>On Sun, 24 Feb 2013 04:15:09 +0100, Jeremy Chadwick > >> wrote: > >> > >>>On Sun, Feb 24, 2013 at 03:45:57AM +0100, Michael Ross wrote: > >>>>On Sun, 24 Feb 2013 01:36:36 +0100, John Mehr wrote: > >>>> > >>>>> Hello all, > >>>>> I've believe I've made just about all of the progress > >>>>optimizing svnup > >>>>> as I can and I've just submitted it as a new port. With my > >>>>~ 350kb/s > >>>>> DSL connection, it now takes just under 30 minutes to > >>>>download a fresh > >>>>> base/releng/8.3 tree using svnup (Subversion's svn takes > >>>>approximately > >>>>> 12 minutes). Incremental updates, such as tracking one of > >>>>the stable > >>>>> branches takes only 2-3 minutes. > >>>>> For anyone that wants to preview the port before it gets > >>>>added to the > >>>>> ports tree (assuming I got the send-pr correct), the tarball is > >>>>>located > >>>>> at: > >>>>> http://jcm.dsl.visi.com/freebsd/svnup/svnup-0.5.tar.xz > >>>>> Please let me know if you find any issues. > >>>> > >>>> > >>>>Maybe it's me, but: > >>>> > >>>>No Makefile. > >>>>So I try manually: > >>>> > >>>>gurder> cc svnup.c > >>>>/tmp//cconKEOv.o: In function `compare_md5': > >>>>svnup.c:(.text+0x175e): undefined reference to `MD5Init' > >>>>svnup.c:(.text+0x1774): undefined reference to `MD5Update' > >>>>svnup.c:(.text+0x1785): undefined reference to `MD5End' > >>>>/tmp//cconKEOv.o: In function `get_files': > >>>>svnup.c:(.text+0x1f20): undefined reference to `MD5Init' > >>>>svnup.c:(.text+0x1f4e): undefined reference to `MD5Update' > >>>>svnup.c:(.text+0x1f5f): undefined reference to `MD5End' > >>>> > >>>> > >>>>9.0-STABLE FreeBSD 9.0-STABLE #17: Fri May 4 02:53:49 CEST 2012 > >>> > >>>Those are all defined in libmd (see MD5Init(3) man page). Thus: > >>> > >>>cc -o svnup svnup.c -lmd > >>> > >> > >>Thanks. > >> > >>gurder> ./svnup -h svn0.us-west.FreeBSD.org -b ports/head -l test > >> > >>Dumps core: http://gurder.ross.cx/misc/svnup.core > > > >Should be easy enough to debug: > > > >$ cc -g3 -ggdb -o svnup svnup.c -lmd > >$ gdb svnup > >(gdb) run -h svn0.us-west.FreeBSD.org -b ports/head -l test > > > >Then once it cores, provide the output from "bt" and/or "bt full". > > > >Might also be useful to compile with -Wall to see if there are any > >compile-time warnings. > > > > gurder> cc -Wall -g3 -ggdb -o svnup svnup.c -lmd > svnup.c: In function 'main': > svnup.c:1002: warning: zero-length printf format string > svnup.c:1020: warning: zero-length printf format string > svnup.c:1027: warning: zero-length printf format string > svnup.c:1065: warning: zero-length printf format string > > > (gdb) run -h svn0.us-west.FreeBSD.org -b ports/head -l test > Starting program: /usr/ports/sysutils/svnup-0.5/svnup -h > svn0.us-west.FreeBSD.org -b ports/head -l test > ####### Fetching revision: 312860 > ? test/archivers > Program received signal SIGSEGV, Segmentation fault. > 0x0000000800b22950 in strchr () from /lib/libc.so.7 > > Backtrace: http://gurder.ross.cx/misc/backtrace.txt I downloaded it and looked at the source. svnup.c:1002: warning: zero-length printf format string svnup.c:1020: warning: zero-length printf format string svnup.c:1027: warning: zero-length printf format string svnup.c:1065: warning: zero-length printf format string 1002 sprintf(command, ""); 1020 sprintf(command, ""); 1027 sprintf(command, ""); 1065 sprintf(command, ""); I'm not sure what the intention is here. If it's to terminate the buffer, then all of these should be: command[0] = '\0'; Or if the entire command[] buffer needs to be zeroed, use memset(3). Doing this does not fix the segfault. The segfault experienced is caused by the following problem: Program received signal SIGSEGV, Segmentation fault. 0x00000000a0b2eb10 in strchr () from /lib/libc.so.7 (gdb) bt #0 0x00000000a0b2eb10 in strchr () from /lib/libc.so.7 #1 0x00000000004021e8 in build_source_directory_tree (connection=0x7fffffff4820, command=0x7ffffffc3650 "", file=0xa1135000, file_count=0x7fffffff48f8, max_file=0x7fffffff48fc, path_target=0xa1009058 "test", revision=0) at svnup.c:412 #2 0x0000000000000000 in ?? () (gdb) f 1 #1 0x00000000004021e8 in build_source_directory_tree (connection=0x7fffffff4820, command=0x7ffffffc3650 "", file=0xa1135000, file_count=0x7fffffff48f8, max_file=0x7fffffff48fc, path_target=0xa1009058 "test", revision=0) at svnup.c:412 412 end = strchr(directory, '\n'); (gdb) p directory $1 = 0x0 It's a wee bit hard to do strchr() on something that points to NULL. At line 412 I inserted this, which is almost certainly not sufficient/correct: if (directory == NULL) { continue; } This allows the program to get a bit further than before (same goes if I use break instead of continue), but not before segfaulting again. And segfaults after this point contain a completely smashed stack: Program received signal SIGSEGV, Segmentation fault. 0x0000000000000000 in ?? () (gdb) bt #0 0x0000000000000000 in ?? () #1 0x0000000000000000 in ?? () John will need to look into these. They all reek of string parsing anomalies and so on. I would need to sit down and read/understand the code in full to know how to fix this one. Also, John, please consider using malloc(3) instead of heap-allocated buffers like file_buffer[6][] (196608 bytes) and command[] (32769 bytes). I'm referring to this: 47 #define COMMAND_BUFFER 32768 386 char new_path_target[BUFFER_UNIT], file_buffer[6][COMMAND_BUFFER], *path_source; 836 char *start, *value, *addr, *branch, *path_target, temp_file[BUFFER_UNIT], command[COMMAND_BUFFER + 1]; I should also point out that watching this thing in top(1) shows RES/RSS increasing until the segfault (for me dies out around 4.5MBytes). I don't know if that's by design or not, but I don't see why this thing would need so much memory given what it's doing. You'd know better than I though. You may want to consider running this under valgrind. It's remarkable what you can find with that during debugging. -- | Jeremy Chadwick jdc@koitsu.org | | UNIX Systems Administrator http://jdc.koitsu.org/ | | Mountain View, CA, US | | Making life hard for others since 1977. PGP 4BD6C0CB |