From owner-freebsd-hackers@FreeBSD.ORG Mon Nov 1 14:12:55 2010 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id BDCCD1065693 for ; Mon, 1 Nov 2010 14:12:55 +0000 (UTC) (envelope-from Daan@vehosting.nl) Received: from VM01.VEHosting.nl (vm01.vehosting.nl [85.17.51.140]) by mx1.freebsd.org (Postfix) with ESMTP id 5C7BD8FC12 for ; Mon, 1 Nov 2010 14:12:55 +0000 (UTC) Received: from [192.168.72.11] (124-54.bbned.dsl.internl.net [92.254.54.124]) (authenticated bits=0) by VM01.VEHosting.nl (8.14.3/8.13.8) with ESMTP id oA1EBfZJ092605; Mon, 1 Nov 2010 15:11:41 +0100 (CET) (envelope-from Daan@vehosting.nl) From: Daan Vreeken Organization: http://VEHosting.nl/ To: cronfy Date: Mon, 1 Nov 2010 15:11:38 +0100 User-Agent: KMail/1.9.10 References: In-Reply-To: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit Content-Disposition: inline Message-Id: <201011011511.38244.Daan@vehosting.nl> x-ve-auth-version: mi-1.0.3 2008-05-30 - Copyright (c) 2008 - Daan Vreeken - VEHosting x-ve-auth: authenticated as 'pa4dan' on VM01.VEHosting.nl X-Mailman-Approved-At: Mon, 01 Nov 2010 15:16:05 +0000 Cc: freebsd-hackers@freebsd.org Subject: Re: Slow disk access while rsync - what should I tune? X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 01 Nov 2010 14:12:55 -0000 Hi Cronfy, On Saturday 30 October 2010 23:48:45 cronfy wrote: > Hello. > > > Every time backup starts server slows down significantly, disk > > operations become very slow. It may take up to 10 seconds to stat() a > > file that is not in filesystem cache. At the same time, rsync on > > remote server does not affect disk load much, server works without > > slowdown. > > Thank you all for the answers. ... > Also, is it possible to limit disk operations for rm -rf somehow? The > only idea I have at the moment is to replace rm -rf with 'find | > slow_down_script | xargs rm' (or use similar patch as for rsync)... Yes there is. You could use the same 'trick' I've added to rsync and limit the amount of I/O-creating system calls an application creates. You could even create a small wrapper library that does this for a specific application, without having to recompile or change the application. You can find a working proof of concept in "slowdown.c" here : http://vehosting.nl/pub_diffs/ The library can be compiled with : gcc -Wall -fPIC -shared -o slowdown.so slowdown.c Then start the application you want to I/O-limit with something like : ( export LD_PRELOAD=slowdown.so export LD_LIBRARY_PATH=.:${LD_LIBRARY_PATH} ls -R /a/random/huge/directory/ ) (Assuming you start the application from withing the directory where "slowdown.so" resides.) This should work with rsync, ls and rm "out of the box", without changing the source of the applications. Regards, -- Daan Vreeken VEHosting http://VEHosting.nl tel: +31-(0)40-7113050 / +31-(0)6-46210825 KvK nr: 17174380