From owner-freebsd-fs@FreeBSD.ORG Tue Mar 31 01:24:19 2015 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 90C43CAF for ; Tue, 31 Mar 2015 01:24:19 +0000 (UTC) Received: from smtp27.mail.ru (smtp27.mail.ru [94.100.181.182]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 0B85D8AF for ; Tue, 31 Mar 2015 01:24:17 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=mail.ru; s=mail2; h=Content-Transfer-Encoding:Content-Type:In-Reply-To:References:Subject:To:MIME-Version:From:Date:Message-ID; bh=tni1I9gKS/2WvXXoCbdZTOBaRmSY0gkk4bJikUbZhYA=; b=mfFOxtOttg8vaTcWx4qu+tTGU7olZNp4N1ImP0uLUrs3tvcMkJDQLWi9Rqqm7v+16oagcn4IALmxj2ZHyqWc9VzicBEJ8Tabf7K6WeSBguAJrtsHxtNXu9cRYz7bnpvz+Y1Otg+HrsVonlXIG73lt0iSuLbfHtVnC396bHchUK8=; Received: from [109.188.127.13] (port=55067 helo=[192.168.0.12]) by smtp27.mail.ru with esmtpa (envelope-from ) id 1YckuR-0003N5-4x for freebsd-fs@freebsd.org; Tue, 31 Mar 2015 04:24:08 +0300 Message-ID: <5519F74C.1040308@artem.ru> Date: Tue, 31 Mar 2015 04:24:28 +0300 From: Artem Kuchin User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:31.0) Gecko/20100101 Thunderbird/31.5.0 MIME-Version: 1.0 To: freebsd-fs@freebsd.org Subject: Re: Little research how rm -rf and tar kill server References: <55170D9C.1070107@artem.ru> <1427727936.293597.247070269.5CE0D411@webmail.messagingengine.com> <55196FC7.8090107@artem.ru> <1427730597.303984.247097389.165D5AAB@webmail.messagingengine.com> <5519716F.6060007@artem.ru> <1427731061.306961.247099633.0A421E90@webmail.messagingengine.com> <5519740A.1070902@artem.ru> <1427731759.309823.247107417.308CD298@webmail.messagingengine.com> In-Reply-To: <1427731759.309823.247107417.308CD298@webmail.messagingengine.com> Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit X-Spam: Not detected X-Mras: Ok X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 31 Mar 2015 01:24:19 -0000 30.03.2015 19:09, Mark Felder пишет: > > On Mon, Mar 30, 2015, at 11:04, Artem Kuchin wrote: >> 30.03.2015 18:57, Mark Felder пишет: >>> On Mon, Mar 30, 2015, at 10:53, Artem Kuchin wrote: >>>> This is normal state, not under rm -rf >>>> Do you need it during rm -rf ? >>>> >>> No, but I wonder if changing the timer from LAPIC to HPET or possibly >>> one of the other timers makes the system more responsive under that >>> load. Would you mind testing that? >>> >>> You can switch the timer like this: >>> >>> sysctl kern.eventtimer.timer=HPET >>> >>> And then run some of your I/O tests >>> >> I see. I will test at night, when load goes down. >> I cannot say sure that's a right way to dig, but i will test anything :) >> >> Just to remind: untar overloads the system, but untar + sync every 120s >> does not. >> That seems very strange to me. I think the problem might be somewhere >> here. >> > I just heard from mav that there was a bottleneck in gmirror/graid with > regards to BIO_DELETE requests > > https://svnweb.freebsd.org/base?view=revision&revision=280757 > I applied this patch manually and rebuilt the kernel. Hit this bug https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=195458 on reboot, wasted 1 hour fsck-ing 2 times (was dirty after first fsck) and after boot tried doing rm -rf test1 I coult not test anything, because it complete after 1 minute, instead 15 minutes before. I copier the dir 4 times into subdirs and rm -rf full tree (4x larger) - fast and smooth, mariadb did not tonice this, server were working fine. However, i also noticed another thing: cp -Rp test test1 also work a lot faster now, probably 3-5 times faster Maybe it is because fs is free of tons BIO_DELETE from other processes Then i did the untar test at maximum speed ( no pv to limit bandwidth): i see that mysql request became slower, but mysql sql request queue built up slower now. However, when it reached 70 i stopped untar and mariadb could not recover from condition until i executed sync. However, this time sync took only a second. I see big improvement, but i still don't understand why i need to issue sync manually to push everything to recover from overload. # man 2 sync a sync() system call is issued frequently by the user process syncer(4) (about every 30 seconds). it does not seem to be true I checked syncer sysctl # sysctl kern.filedelay kern.filedelay: 30 # sysctl kern.dirdelay kern.dirdelay: 29 # sysctl kern.metadelay kern.metadelay: 28 # ps ax | grep sync 23 - DL 0:03.82 [syncer] no clue why need manual sync By the way: is there way to make sure that SU+J is really working? Maybe it is disabled for some reason and i don't know it. tunefs just shows stored setting, but, for example, with dirty fs, journaling is not working in reality. Any way to get current status of SU journaling? off topic: suggestion to move to ZFA was not so good, i see a "All available memory used when deleting files from ZFS" topic. I'd rather have slow server when i can login and fix than halted on panic. Just to point that ZFS still have plenty of unpredictable issues. Artem