From owner-freebsd-hackers@FreeBSD.ORG Sun Oct 24 13:47:50 2010 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id A5B4D1065672 for ; Sun, 24 Oct 2010 13:47:50 +0000 (UTC) (envelope-from cronfy@gmail.com) Received: from mail-bw0-f54.google.com (mail-bw0-f54.google.com [209.85.214.54]) by mx1.freebsd.org (Postfix) with ESMTP id 376588FC08 for ; Sun, 24 Oct 2010 13:47:49 +0000 (UTC) Received: by bwz3 with SMTP id 3so2340615bwz.13 for ; Sun, 24 Oct 2010 06:47:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:mime-version:received:from:date :message-id:subject:to:content-type; bh=H5ixNfCrVKwTykwO2RRVp31/8blbuWJvq1GLRj98IE8=; b=g/PvhRFDA7JrX5CPX+LnDNEwEhYWiFM0jivAYGK7dY/Sznj8ACWD+lL2NCp39yRUhG qDzEcQ4MMAz6UGD+bXlQT7hh4ikW/SP6GMa0jhVh0vv/ILo432l/Iib93sTcgZJXaxWe 5x2WbsOlWvKweKfnAO0DM8lG8QZdFQIbhUp/c= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:from:date:message-id:subject:to:content-type; b=qmsNfu4Ux+NemRk5p3rWG27qysqau1yGvUm0xj9nc5Ee3PjcyJNrmTEOCu5VTi/Kq0 thWDH4th4P3n9zo3cm6tYix9/q8diKyAnU+9ZXMYLXBKPnjFg8tVQ3JkVoS0PRoQKRz4 lr+r/3Qyn7xlxqJITKoFFA1qgU6E81pRPx230= Received: by 10.204.122.8 with SMTP id j8mr3995037bkr.135.1287926184104; Sun, 24 Oct 2010 06:16:24 -0700 (PDT) MIME-Version: 1.0 Received: by 10.204.73.135 with HTTP; Sun, 24 Oct 2010 06:15:53 -0700 (PDT) From: cronfy Date: Sun, 24 Oct 2010 17:15:53 +0400 Message-ID: To: freebsd-hackers@freebsd.org Content-Type: text/plain; charset=ISO-8859-1 Subject: Slow disk access while rsync - what should I tune? X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 24 Oct 2010 13:47:50 -0000 Hello, I have a web-server (nginx + apache + mysql, FreeBSD 7.3) with many sites. Every night it creates a backup of /home on a separate disk. /home is a RAID1 mirror on Adaptec 3405 (128M write cache) with SAS drives; /backup is a single SATA drive on the same controller. Rsync creates backups using hardlinks, it stores 7 daily and 4 weekly copies. Total amount of data is ~300G and 11M of files. The server is under heavy web load every time (appox 100 queries/sec). Every time backup starts server slows down significantly, disk operations become very slow. It may take up to 10 seconds to stat() a file that is not in filesystem cache. At the same time, rsync on remote server does not affect disk load much, server works without slowdown. I think that problem can be caused by two reasons: * either bulk of reads on SATA /backup drive, that fills OS filesystem cache and many file access operations require real disk read. * or bulk of writes on /backup fills controller write cache and geom disk operations queue grown, causing all disk operations to wait. This is only my assumption of course, I may be wrong. How can I find a real reason of these slowdowns, to either conclude that it is not possible to solve this because of hardware/software limits, or tune my software/hardware system to make this all work at an acceptable speed? Here is my current sysctl setup (what should I tune?): kern.maxvnodes=500000 vfs.ufs.dirhash_maxmem=67108864 vfs.lookup_shared=1 kern.dirdelay=6 kern.metadelay=5 kern.filedelay=7 sysctl's counters (which others should I monitor?): vfs.numvnodes: 407690 vfs.ufs.dirhash_mem: 27158118 I tried to enable async (in hope it will make rsync faster) or even disable softupdates on /backup partition (in hope it will make rsync slower and OS filesytem cache will not be flushed by backups), it did not help. I also want to try to upgrate to Adaptec 5405 (it has 256M of write cache) or move mysql databases on a separate SAS disk, but I just not quite sure what will help better. What would I start from to diagnose the issue? Thanks in advance! -- // cronfy