From owner-freebsd-hackers@FreeBSD.ORG Thu Nov 4 14:47:10 2010 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id A5F0F1065675 for ; Thu, 4 Nov 2010 14:47:10 +0000 (UTC) (envelope-from kraduk@gmail.com) Received: from mail-wy0-f182.google.com (mail-wy0-f182.google.com [74.125.82.182]) by mx1.freebsd.org (Postfix) with ESMTP id 30E078FC21 for ; Thu, 4 Nov 2010 14:47:09 +0000 (UTC) Received: by wyb34 with SMTP id 34so54938wyb.13 for ; Thu, 04 Nov 2010 07:47:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:received:in-reply-to :references:date:message-id:subject:from:to:cc:content-type; bh=4SkO1OMiom2t2lFDIxyUl9EJyGsyuwJaPxVWBaStl3k=; b=XZ5Oh3MFbTqxa9w79KjVrCRRTy8pZiGu+fnWOLal+Ga5KBQOkxCpvEXw7toHI6d7oQ gHysVHi+tJixtOnX87vdkIKuEuWEJZC2W2Z0/xEMT/n+8gwyIWR6lBpt7y8nPVlnFMuK dyCds2LM7km+AVO5iLTrBEzImiruX1H9aRnGk= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; b=hlCCK20kQZwbCTJcFADXjtoheLLq5H+9KIoquM67isKHQVPpk5sfXbUE9y78uztUvf UPWE1w4ngOv0lm/VSyt1C/ukeZbeRQcGrosKvHTeD6KOcvGkGCPZaOVmEOuRjqxEQCaj mGReIFXLWZUvFx/pPCa4ghHBOyRy81bwmPRho= MIME-Version: 1.0 Received: by 10.216.5.21 with SMTP id 21mr1990658wek.20.1288880705550; Thu, 04 Nov 2010 07:25:05 -0700 (PDT) Received: by 10.216.25.85 with HTTP; Thu, 4 Nov 2010 07:25:05 -0700 (PDT) In-Reply-To: <201011011511.38244.Daan@vehosting.nl> References: <201011011511.38244.Daan@vehosting.nl> Date: Thu, 4 Nov 2010 14:25:05 +0000 Message-ID: From: krad To: Daan Vreeken X-Mailman-Approved-At: Thu, 04 Nov 2010 16:40:03 +0000 Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: cronfy , freebsd-hackers@freebsd.org Subject: Re: Slow disk access while rsync - what should I tune? X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 04 Nov 2010 14:47:10 -0000 On 1 November 2010 14:11, Daan Vreeken wrote: > Hi Cronfy, > > On Saturday 30 October 2010 23:48:45 cronfy wrote: > > Hello. > > > > > Every time backup starts server slows down significantly, disk > > > operations become very slow. It may take up to 10 seconds to stat() a > > > file that is not in filesystem cache. At the same time, rsync on > > > remote server does not affect disk load much, server works without > > > slowdown. > > > > Thank you all for the answers. > ... > > Also, is it possible to limit disk operations for rm -rf somehow? The > > only idea I have at the moment is to replace rm -rf with 'find | > > slow_down_script | xargs rm' (or use similar patch as for rsync)... > > Yes there is. You could use the same 'trick' I've added to rsync and limit > the > amount of I/O-creating system calls an application creates. > You could even create a small wrapper library that does this for a specific > application, without having to recompile or change the application. > > You can find a working proof of concept in "slowdown.c" here : > http://vehosting.nl/pub_diffs/ > > The library can be compiled with : > gcc -Wall -fPIC -shared -o slowdown.so slowdown.c > > Then start the application you want to I/O-limit with something like : > ( > export LD_PRELOAD=slowdown.so > export LD_LIBRARY_PATH=.:${LD_LIBRARY_PATH} > ls -R /a/random/huge/directory/ > ) > > (Assuming you start the application from withing the directory > where "slowdown.so" resides.) > This should work with rsync, ls and rm "out of the box", without changing > the > source of the applications. > > > Regards, > -- > Daan Vreeken > VEHosting > http://VEHosting.nl > tel: +31-(0)40-7113050 / +31-(0)6-46210825 > KvK nr: 17174380 > _______________________________________________ > freebsd-hackers@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-hackers > To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org" > Just a wild punt here, but are you using zfs on both systems? If you are look at doing incremental zfs sends as an alternative.