From owner-freebsd-stable@FreeBSD.ORG Wed Nov 12 21:05:21 2008 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 139EE1065676 for ; Wed, 12 Nov 2008 21:05:21 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from mail.terabit.net.ua (mail.terabit.net.ua [195.137.202.147]) by mx1.freebsd.org (Postfix) with ESMTP id A6F538FC1D for ; Wed, 12 Nov 2008 21:05:20 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from skuns.zoral.com.ua ([91.193.166.194] helo=mail.zoral.com.ua) by mail.terabit.net.ua with esmtps (TLSv1:AES256-SHA:256) (Exim 4.63 (FreeBSD)) (envelope-from ) id 1L0Mtq-0008Rz-GU; Wed, 12 Nov 2008 23:05:18 +0200 Received: from deviant.kiev.zoral.com.ua (root@deviant.kiev.zoral.com.ua [10.1.1.148]) by mail.zoral.com.ua (8.14.2/8.14.2) with ESMTP id mACL5FbL048992 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Wed, 12 Nov 2008 23:05:15 +0200 (EET) (envelope-from kostikbel@gmail.com) Received: from deviant.kiev.zoral.com.ua (kostik@localhost [127.0.0.1]) by deviant.kiev.zoral.com.ua (8.14.3/8.14.3) with ESMTP id mACL5FkB095168; Wed, 12 Nov 2008 23:05:15 +0200 (EET) (envelope-from kostikbel@gmail.com) Received: (from kostik@localhost) by deviant.kiev.zoral.com.ua (8.14.3/8.14.3/Submit) id mACL5DM6095167; Wed, 12 Nov 2008 23:05:13 +0200 (EET) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: deviant.kiev.zoral.com.ua: kostik set sender to kostikbel@gmail.com using -f Date: Wed, 12 Nov 2008 23:05:13 +0200 From: Kostik Belousov To: Tim Bishop Message-ID: <20081112210513.GM47073@deviant.kiev.zoral.com.ua> References: <20081112175826.GD26195@carrick.bishnet.net> <20081112194928.GA19539@carrick.bishnet.net> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="4ybNbZnZ8tziJ7D6" Content-Disposition: inline In-Reply-To: <20081112194928.GA19539@carrick.bishnet.net> User-Agent: Mutt/1.4.2.3i X-Virus-Scanned: ClamAV version 0.93.3, clamav-milter version 0.93.3 on skuns.kiev.zoral.com.ua X-Virus-Status: Clean X-Spam-Status: No, score=-4.4 required=5.0 tests=ALL_TRUSTED,AWL,BAYES_00 autolearn=ham version=3.2.5 X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on skuns.kiev.zoral.com.ua X-Virus-Scanned: mail.terabit.net.ua 1L0Mtq-0008Rz-GU 2b2ec70661d736076d77d7f5ac614eca X-Terabit: YES Cc: Tim Bishop , Jeremy Chadwick , freebsd-stable@freebsd.org, David Peall Subject: Re: System deadlock when using mksnap_ffs X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 12 Nov 2008 21:05:21 -0000 --4ybNbZnZ8tziJ7D6 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Wed, Nov 12, 2008 at 07:49:28PM +0000, Tim Bishop wrote: > On Wed, Nov 12, 2008 at 05:58:26PM +0000, Tim Bishop wrote: > > I run the mksnap_ffs command to take the snapshot and some time later > > the system completely freezes up: > >=20 > > paladin# cd /u2/.snap/ > > paladin# mksnap_ffs /u2 test.1 >=20 > Someone (not named because they choose not to reply to the list) gave me > the following patch: >=20 > --- sys/ufs/ffs/ffs_snapshot.c.orig Wed Mar 22 09:42:31 2006 > +++ sys/ufs/ffs/ffs_snapshot.c Mon Nov 20 14:59:13 2006 > @@ -282,6 +282,8 @@ restart: > if (error) > goto out; > bawrite(nbp); > + if (cg % 10 =3D=3D 0) > + ffs_syncvnode(vp, MNT_WAIT); > } > /* > * Copy all the cylinder group maps. Although the > @@ -303,6 +305,8 @@ restart: > goto out; > error =3D cgaccount(cg, vp, nbp, 1); > bawrite(nbp); > + if (cg % 10 =3D=3D 0) > + ffs_syncvnode(vp, MNT_WAIT); > if (error) > goto out; > } >=20 > With the description: >=20 > "What can happen is on a big file system it will fill up the buffer > cache with I/O and then run out. When the buffer cache fills up then no > more disk I/O can happen :-( When you do a sync, it flushes that out to > disk so things don't hang." >=20 > It seems to work too. But it seems more like a workaround than a fix? It looks hackish, but in fact it is not that wrong, and I even say that it provides reasonable workaround. The usual way to prevent wdrain deadlock is to issue bwillwrite() call before any vnode lock is taken. This is sufficient for most VFS syscalls that typically put dozen or less dirty buffers into delayed write queue. Snapshot creation does not call bwillwrite() at all, and then does a lot of async writes, completely saturating buffer cache with dirty buffers. bwillwrite cannot be called after the vnode is locked, and just forcing a sync for the embrionic snapshot vnode is good enough. The 10 counter is debatable, but debate shall be postponed until the patch goes into tree. I ask an anonymous submitter to commit it. Thanks ! --4ybNbZnZ8tziJ7D6 Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (FreeBSD) iEYEARECAAYFAkkbRQkACgkQC3+MBN1Mb4iecwCg6wKWEMcVZ30HMLCc9kfE1J9h 6gcAnRyVqnhKYTph1pEUD4ao1OyNOkR6 =Uvcf -----END PGP SIGNATURE----- --4ybNbZnZ8tziJ7D6--