From owner-freebsd-fs@FreeBSD.ORG Fri Apr 3 23:15:43 2015 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 838A2D65 for ; Fri, 3 Apr 2015 23:15:43 +0000 (UTC) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 25DC5FA9 for ; Fri, 3 Apr 2015 23:15:42 +0000 (UTC) Received: from tom.home (kostik@localhost [127.0.0.1]) by kib.kiev.ua (8.14.9/8.14.9) with ESMTP id t33NFUBR031021 (version=TLSv1/SSLv3 cipher=DHE-RSA-CAMELLIA256-SHA bits=256 verify=NO); Sat, 4 Apr 2015 02:15:30 +0300 (EEST) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.9.2 kib.kiev.ua t33NFUBR031021 Received: (from kostik@localhost) by tom.home (8.14.9/8.14.9/Submit) id t33NFUO5031019; Sat, 4 Apr 2015 02:15:30 +0300 (EEST) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Sat, 4 Apr 2015 02:15:30 +0300 From: Konstantin Belousov To: Artem Kuchin Subject: Re: Little research how rm -rf and tar kill server Message-ID: <20150403231530.GH2379@kib.kiev.ua> References: <1427730597.303984.247097389.165D5AAB@webmail.messagingengine.com> <5519716F.6060007@artem.ru> <1427731061.306961.247099633.0A421E90@webmail.messagingengine.com> <5519740A.1070902@artem.ru> <1427731759.309823.247107417.308CD298@webmail.messagingengine.com> <5519F74C.1040308@artem.ru> <20150331164202.GN2379@kib.kiev.ua> <551C6D9F.8010506@artem.ru> <20150402210241.GD2379@kib.kiev.ua> <551F0D4A.5040007@artem.ru> MIME-Version: 1.0 Content-Type: text/plain; charset=koi8-r Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <551F0D4A.5040007@artem.ru> User-Agent: Mutt/1.5.23 (2014-03-12) X-Spam-Status: No, score=-2.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FREEMAIL_FROM,NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=3.4.0 X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on tom.home Cc: freebsd-fs@freebsd.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 03 Apr 2015 23:15:43 -0000 On Sat, Apr 04, 2015 at 12:59:38AM +0300, Artem Kuchin wrote: > 03.04.2015 0:02, Konstantin Belousov пишет: > > On Thu, Apr 02, 2015 at 01:13:51AM +0300, Artem Kuchin wrote: > >> 31.03.2015 19:42, Konstantin Belousov пишет: > >>> Syncer and sync(2) perform different kind of syncs. Take the snapshot of > >>> sysctl debug.softdep before and after the situation occur to have some > >>> hints what is going on. > >>> > >>> > >> Okay. Here is the sysctl data > > Try this. It may be not enough, I will provide some update in this case. > > No need to resend the sysctl data. Just test whether explicit sync(2) is > > needed in your situation after the patch. > > > > > > Okay, patches, recompiled and installed new kernel. > > The behaviour changed a bit. > > Now when i start untar mysql quickly rises to 40 queries in the queue in > opening table state. > (before the rise was slower) > BUT after a while (20-30 seconds) all queries are executed. > This cycle repeated 4 times and then situation aggravated quickly. It > happened when untar > reached big subtree with tons of small files. > Queue grew to 70 queries, processes went to 600 (from 450). > I stopped untar. Waited 3 minutes. Everything was becoming even worse > (700 process, over 100 > queries). Issued sync. It executed for 3 seconds and voila! 20 idle > connections, 450 processes. > So, manual sync is still need. > > Also it seems like during untar shell was less responsive than before. > > Also, when the system managed to flush query queue systat -io shows over > 1000 tps, but when > they got stuck it showed only about 200 tps. So there were the i/o ops during the stall period ? I.e., a situation where there is clogged queue and hung processes, but no disk activity, does not occur, even temporary ? In what state the hung processes are blocked ? Look at the wchan name either in top or ps output. Are there processes in "suspfs" state ? Try the following patch. diff --git a/sys/ufs/ffs/ffs_extern.h b/sys/ufs/ffs/ffs_extern.h index c29e5d5..8494223 100644 --- a/sys/ufs/ffs/ffs_extern.h +++ b/sys/ufs/ffs/ffs_extern.h @@ -160,7 +160,7 @@ void softdep_journal_fsync(struct inode *); void softdep_buf_append(struct buf *, struct workhead *); void softdep_inode_append(struct inode *, struct ucred *, struct workhead *); void softdep_freework(struct workhead *); - +int softdep_need_sbupdate(struct ufsmount *ump); /* * Things to request flushing in softdep_request_cleanup() diff --git a/sys/ufs/ffs/ffs_softdep.c b/sys/ufs/ffs/ffs_softdep.c index ab2bd41..da7a34f 100644 --- a/sys/ufs/ffs/ffs_softdep.c +++ b/sys/ufs/ffs/ffs_softdep.c @@ -612,6 +612,13 @@ softdep_freework(wkhd) panic("softdep_freework called"); } +int +softdep_need_sbupdate(ump) + struct ufsmount *ump; +{ + + panic("softdep_need_sbupdate called"); +} #else FEATURE(softupdates, "FFS soft-updates support"); @@ -3560,8 +3567,10 @@ softdep_process_journal(mp, needwk, flags) * unsuspend it if we already have. */ if (flags == 0 && jblocks->jb_suspended) { +#if 0 if (journal_unsuspend(ump)) return; +#endif FREE_LOCK(ump); VFS_SYNC(mp, MNT_NOWAIT); ffs_sbupdate(ump, MNT_WAIT, 0); @@ -9479,6 +9488,18 @@ first_unlinked_inodedep(ump) return (inodedep); } +int +softdep_need_sbupdate(ump) + struct ufsmount *ump; +{ + struct inodedep *inodedep; + + ACQUIRE_LOCK(ump); + inodedep = first_unlinked_inodedep(ump); + FREE_LOCK(ump); + return (inodedep != NULL); +} + /* * Set the sujfree unlinked head pointer prior to writing a superblock. */ diff --git a/sys/ufs/ffs/ffs_vfsops.c b/sys/ufs/ffs/ffs_vfsops.c index 6e2e556..274c0f9 100644 --- a/sys/ufs/ffs/ffs_vfsops.c +++ b/sys/ufs/ffs/ffs_vfsops.c @@ -1419,7 +1419,8 @@ static int ffs_sync_lazy(mp) struct mount *mp; { - struct vnode *mvp, *vp; + struct ufsmount *ump; + struct vnode *devvp, *mvp, *vp; struct inode *ip; struct thread *td; int allerror, error; @@ -1461,9 +1462,21 @@ qupdate: qsync(mp); #endif - if (VFSTOUFS(mp)->um_fs->fs_fmod != 0 && - (error = ffs_sbupdate(VFSTOUFS(mp), MNT_LAZY, 0)) != 0) - allerror = error; + ump = VFSTOUFS(mp); + if (MOUNTEDSUJ(mp)) { + devvp = ump->um_devvp; + vn_lock(devvp, LK_EXCLUSIVE | LK_RETRY); + error = VOP_FSYNC(devvp, MNT_WAIT, td); + VOP_UNLOCK(devvp, 0); + if (error != 0) + allerror = error; + } + if (ump->um_fs->fs_fmod != 0 || (MOUNTEDSUJ(mp) && + softdep_need_sbupdate(ump))) { + error = ffs_sbupdate(ump, MNT_LAZY, 0); + if (error != 0) + allerror = error; + } return (allerror); }