From owner-freebsd-current@FreeBSD.ORG Sat Sep 22 00:38:24 2007 Return-Path: Delivered-To: current@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 64F8516A417 for ; Sat, 22 Sep 2007 00:38:24 +0000 (UTC) (envelope-from jasone@freebsd.org) Received: from canonware.com (canonware.com [64.183.146.166]) by mx1.freebsd.org (Postfix) with ESMTP id 5624013C459 for ; Sat, 22 Sep 2007 00:38:24 +0000 (UTC) (envelope-from jasone@freebsd.org) Received: from [192.168.168.208] (canonware.com [64.183.146.166]) by canonware.com (Postfix) with ESMTP id C95DB1298C8; Fri, 21 Sep 2007 17:14:31 -0700 (PDT) Message-ID: <46F45CE2.9080204@freebsd.org> Date: Fri, 21 Sep 2007 17:08:02 -0700 From: Jason Evans User-Agent: Thunderbird 1.5.0.13 (X11/20070824) MIME-Version: 1.0 To: Alexandre Biancalana References: <8e10486b0709211457v7771bd66kf3df5ea45ab0d325@mail.gmail.com> In-Reply-To: <8e10486b0709211457v7771bd66kf3df5ea45ab0d325@mail.gmail.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: current@freebsd.org Subject: Re: process size X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 22 Sep 2007 00:38:24 -0000 Alexandre Biancalana wrote: > I'm trying to copy ~800GB to my new backup server via rsync, but the rsync > process die when it's vsz is close to 1G with the following message: > > # rsync -av 192.168.0.13::root/backup/* . > receiving file list ... ERROR: out of memory in make_file > rsync error: error allocating core memory buffers (code 22) at util.c(120) > [sender=2.6.8] > rsync: connection unexpectedly closed (155637057 bytes received so far) > [receiver] > rsync error: error in rsync protocol data stream (code 12) at io.c(462) > [receiver=2.6.9] > > > Here is the last ps -aux line of the process before die: > > root 856 0.4 68.1 915248 707048 p0 S+ 6:32PM 0:12.51 rsync -av > 192.168.0.13::root/backup/* . You're running your system as i386 rather than amd64, right? It looks like rsync is exhausting its address space while trying to reallocate the growing (and apparently very large) file list. There's nothing surprising here to me. If you use amd64 rather than i386 you won't have this problem, though you will still see poor performance due to swapping. The most prudent solution is probably to use multiple rsync calls to copy portions of your data at a time. Jason