From owner-freebsd-fs@FreeBSD.ORG Fri Apr 3 21:59:35 2015 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id E3EBF629 for ; Fri, 3 Apr 2015 21:59:35 +0000 (UTC) Received: from smtp21.mail.ru (smtp21.mail.ru [94.100.179.250]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 5B72E91F for ; Fri, 3 Apr 2015 21:59:35 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=mail.ru; s=mail2; h=Content-Transfer-Encoding:Content-Type:In-Reply-To:References:Subject:CC:To:MIME-Version:From:Date:Message-ID; bh=ytW8LL25wCqeNUw7sXG2VZ/5y9s9DwqlYU9HThYYcok=; b=kDpL7O8moO+xxc3GfI3upK56OiK4g+b9s5I9aLGg2QWgaNb6J6MvQfdba9q+wlxAiz5TVj1wPgg1mgsvmD2qFomqgGBv0YUX4ZoaO5XDuzTSQqT8L5xQZgpxX5YFYl0MMlhqMOPEp1BgkVkJ2HnubHEXtizsnjoXH6uiwwnSNz8=; Received: from [109.188.125.8] (port=13271 helo=[192.168.0.12]) by smtp21.mail.ru with esmtpa (envelope-from ) id 1Ye9cb-0003wH-KT; Sat, 04 Apr 2015 00:59:26 +0300 Message-ID: <551F0D4A.5040007@artem.ru> Date: Sat, 04 Apr 2015 00:59:38 +0300 From: Artem Kuchin User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:31.0) Gecko/20100101 Thunderbird/31.5.0 MIME-Version: 1.0 To: Konstantin Belousov Subject: Re: Little research how rm -rf and tar kill server References: <1427727936.293597.247070269.5CE0D411@webmail.messagingengine.com> <55196FC7.8090107@artem.ru> <1427730597.303984.247097389.165D5AAB@webmail.messagingengine.com> <5519716F.6060007@artem.ru> <1427731061.306961.247099633.0A421E90@webmail.messagingengine.com> <5519740A.1070902@artem.ru> <1427731759.309823.247107417.308CD298@webmail.messagingengine.com> <5519F74C.1040308@artem.ru> <20150331164202.GN2379@kib.kiev.ua> <551C6D9F.8010506@artem.ru> <20150402210241.GD2379@kib.kiev.ua> In-Reply-To: <20150402210241.GD2379@kib.kiev.ua> Content-Type: text/plain; charset=koi8-r; format=flowed Content-Transfer-Encoding: 8bit X-Spam: Not detected X-Mras: Ok Cc: freebsd-fs@freebsd.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 03 Apr 2015 21:59:36 -0000 03.04.2015 0:02, Konstantin Belousov пишет: > On Thu, Apr 02, 2015 at 01:13:51AM +0300, Artem Kuchin wrote: >> 31.03.2015 19:42, Konstantin Belousov пишет: >>> Syncer and sync(2) perform different kind of syncs. Take the snapshot of >>> sysctl debug.softdep before and after the situation occur to have some >>> hints what is going on. >>> >>> >> Okay. Here is the sysctl data > Try this. It may be not enough, I will provide some update in this case. > No need to resend the sysctl data. Just test whether explicit sync(2) is > needed in your situation after the patch. > > Okay, patches, recompiled and installed new kernel. The behaviour changed a bit. Now when i start untar mysql quickly rises to 40 queries in the queue in opening table state. (before the rise was slower) BUT after a while (20-30 seconds) all queries are executed. This cycle repeated 4 times and then situation aggravated quickly. It happened when untar reached big subtree with tons of small files. Queue grew to 70 queries, processes went to 600 (from 450). I stopped untar. Waited 3 minutes. Everything was becoming even worse (700 process, over 100 queries). Issued sync. It executed for 3 seconds and voila! 20 idle connections, 450 processes. So, manual sync is still need. Also it seems like during untar shell was less responsive than before. Also, when the system managed to flush query queue systat -io shows over 1000 tps, but when they got stuck it showed only about 200 tps. Artem