From owner-freebsd-hackers@FreeBSD.ORG Sat Oct 30 21:49:17 2010 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id DE0B1106566B for ; Sat, 30 Oct 2010 21:49:16 +0000 (UTC) (envelope-from cronfy@gmail.com) Received: from mail-bw0-f54.google.com (mail-bw0-f54.google.com [209.85.214.54]) by mx1.freebsd.org (Postfix) with ESMTP id 6B5C08FC13 for ; Sat, 30 Oct 2010 21:49:16 +0000 (UTC) Received: by bwz3 with SMTP id 3so3568201bwz.13 for ; Sat, 30 Oct 2010 14:49:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:mime-version:received:in-reply-to :references:from:date:message-id:subject:to:content-type; bh=8DLMZpKT40AQ+dMoDFwi3597TNrJH60sVlMMDfMkn5c=; b=EuNowpBf94mlDGeCV/iXvW2UgTOHFNJMaeoTDc/5p+gdmGo0Q9wnxudKzqlVIC1bRM qhIIGQorXZiyXfCDbKzmbwiaRIsDjjod2JfDxr2wwppc427xurRw9WQp1CF/B6JlKGUn J+LWoafOuTIGX4R27GzCuSREQqXcrQfujDyZo= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :content-type; b=GCG+Rgd/jrcxtpUZbq4SFTcZOKvLO6F+ASfJztTqqgXwcsBu9NbdSh8xgFkGT9lC8I bXsZCHyUwqmfULJdEAUNoWyz/IYfi3PqiigQV7DZd91mNMu3zMdXvqUfbpa3K4Ex5fg4 hC6uWgCHj9q9WcScwIJAgv5XF4k2+aZP5nC7c= Received: by 10.204.137.80 with SMTP id v16mr10624517bkt.46.1288475355310; Sat, 30 Oct 2010 14:49:15 -0700 (PDT) MIME-Version: 1.0 Received: by 10.204.73.135 with HTTP; Sat, 30 Oct 2010 14:48:45 -0700 (PDT) In-Reply-To: References: From: cronfy Date: Sun, 31 Oct 2010 01:48:45 +0400 Message-ID: To: freebsd-hackers@freebsd.org Content-Type: text/plain; charset=ISO-8859-1 Subject: Re: Slow disk access while rsync - what should I tune? X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 30 Oct 2010 21:49:17 -0000 Hello. > Every time backup starts server slows down significantly, disk > operations become very slow. It may take up to 10 seconds to stat() a > file that is not in filesystem cache. At the same time, rsync on > remote server does not affect disk load much, server works without > slowdown. Thank you all for the answers. Matthew, yes, I know about --bwlimit in rsync, but this will mostly slow down the transfer of data and not stat() operations that are used for comparing files. I afraid bwlimiting will not make any better (I tried it some time ago without success). Daan, thanks for the patch! I will try it. A lot of impact also produced by rm -rf of old backups. I assume that low performance is also related to a large numbers of hardlinks. There was a moment when I had ~15 backups hardlinked by rsync, and rm -rf of single backup was VERY slow and slowed down the server dramatically. I had no choice except installing a new clean disk for backups, limiting number of future backups and issuing a newfs over old backup disk. When there is less number of hardlinked copies, backups cleanup works much better. Can large number of hardlinks produce such an impact on filesystem operations? May be it is possible to increase disk performance somehow? Server has a lot of memory. At this time vfs.ufs.dirhash_maxmem = 67108864 (max monitored value for vfs.ufs.dirhash_mem was 52290119) and kern.maxvnodes = 500000 (max monitored value for vfs.numvnodes was 450567). Can increasing of these (or other) sysctls help? I ask because (as you can see) these tunables are already incremented, and I am not sure further increment really makes sense. Also, is it possible to limit disk operations for rm -rf somehow? The only idea I have at the moment is to replace rm -rf with 'find | slow_down_script | xargs rm' (or use similar patch as for rsync)... Or, probably, it is possible to limit the IO bandwith for a particular device somehow? I would then limit disk operations for SATA backup disk to prevent it affecting the rest of services that work on SAS mirror. And also, maybe there are other ways to create incremental backups instead of using rsync/hardlinks? I was thinking about generating list of changed files with own script and packing it with tar, but I did not find a way to remove old backups with such an easy way as it is with hardlnks.. Thanks in advance! -- // cronfy