From owner-freebsd-amd64@FreeBSD.ORG Tue Jan 1 15:08:11 2008 Return-Path: Delivered-To: amd64@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id F0DAC16A474 for ; Tue, 1 Jan 2008 15:08:11 +0000 (UTC) (envelope-from kris@FreeBSD.org) Received: from weak.local (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 3B74813C467; Tue, 1 Jan 2008 15:08:11 +0000 (UTC) (envelope-from kris@FreeBSD.org) Message-ID: <477A5759.6080906@FreeBSD.org> Date: Tue, 01 Jan 2008 16:08:09 +0100 From: Kris Kennaway User-Agent: Thunderbird 2.0.0.9 (Macintosh/20071031) MIME-Version: 1.0 To: =?ISO-8859-1?Q?Jo=E3o_Carlos_Mendes_Lu=EDs?= References: <477A4EBB.8090403@jonny.eng.br> In-Reply-To: <477A4EBB.8090403@jonny.eng.br> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 8bit Cc: amd64@freebsd.org Subject: Re: Anybody seeing problems with large TCP streams over rsync? X-BeenThere: freebsd-amd64@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to the AMD64 platform List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 01 Jan 2008 15:08:12 -0000 João Carlos Mendes Luís wrote: > Hi folks, > > I am migrating a backup server from a server running FreeBSD 6.2/i386 > to another server running FreeBSD 7/amd64. It is currently running > 7.0-BETA4, as distributed from the CD image. > > Both machines have SATA discs and some mix of gmirror and gstripe of > its partitions. > > Since it is a Backup server, I am in the process of transferring > large files from one to another. But I always have trouble when > transferring large files (75GB). > > The first trouble I had was in the back-to-back Gigabit Ethernet > connection, to speed-up transfers. Somehow, data got quickly screwed, > even above TCP layer! > > One side is an onboard nVidia nfe(4), and the other a generic Realtek > re(4). The troubles went on, even after I disable TSO (on fbsd 7), > RXCSUM and TXCSUM on both sides. They went on even after I reduced the > speed to 100Mbps. > > The copy process is basically rsync+rsh, but I have also tried > rsync+ssh, scp and rcp. If ssh does not detect an error in its stream > first, rsync does. > > Assuming it was a problem with the re(4) interface, being it a low > end adapter, I got back to the main network. The new machine is on > (another) onboard nfe(4), and the old one is on a old xl(4) (3Com 905). > Just to be sure, I've also disabled TSO, RXCSUM and TXCSUM on these > interfaces. Also, the copy process is now a raw rsyncd. > > I can now copy the files, but the big ones still get screwed. > (Verification by rsync -c and/or md5) > > After some days experimenting, I finally could make the copy by > split(1)ing them first and cat(1)ing ater network copy. > > This makes me wonder: Where is (are?) the problem(s)? > > It's could be rsync, but not only, since scp got screwed too, when > copying over gigabit. But even when working over FastEthernet, there > were errors. These could be at TCP layer and/or at rsync layer. > > I have found some reports with rsync failing for large files, but > they were dated from 2004, and fixed since then. Looks like you didn't rule out nfe. Kris