From owner-freebsd-fs@FreeBSD.ORG Wed Sep 26 11:38:43 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id E54E21065670 for ; Wed, 26 Sep 2012 11:38:43 +0000 (UTC) (envelope-from bf1783@googlemail.com) Received: from mail-vb0-f54.google.com (mail-vb0-f54.google.com [209.85.212.54]) by mx1.freebsd.org (Postfix) with ESMTP id 93AD78FC0C for ; Wed, 26 Sep 2012 11:38:43 +0000 (UTC) Received: by vbmv11 with SMTP id v11so623627vbm.13 for ; Wed, 26 Sep 2012 04:38:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlemail.com; s=20120113; h=mime-version:reply-to:in-reply-to:references:date:message-id :subject:from:to:cc:content-type; bh=nKrcLzjHJHbfwCTm9C+WrUuB3L0JlIB4ka71QC1lVdc=; b=Pq4rJIxEgKyOi4ejTc0yVqsYzM8wYVph1ThMrwF4MajopP2rxWmpQ/hdiHwJlMH4As znKd+9OX5o63fthwFvxn03rJyMFuLp+JhSTtDPsEMBAwH85BKIv52T+Pd98W8mETW8wB uJk/mR0gN3D641lgSpkJeMsErJ+l3MEMjEufQnUdieTjMnSgcWwgNL853Kd3Zy9bAVg5 aBJzk3UC0htD6Vct7/uM7z6Ikip9fcB1PoHOuyGwRUjwtd5rmIlcBVziziAoxc1vGzk6 cPx8ipjaQTCa/Xw4ZLIved6BcoCW5XLsPbecv6pSsgw8hMFDvStFk9mzH7rwtC6mzd60 2Qig== MIME-Version: 1.0 Received: by 10.220.149.6 with SMTP id r6mr85582vcv.53.1348659522273; Wed, 26 Sep 2012 04:38:42 -0700 (PDT) Received: by 10.58.54.137 with HTTP; Wed, 26 Sep 2012 04:38:42 -0700 (PDT) In-Reply-To: References: <20120917121925.GQ37286@deviant.kiev.zoral.com.ua> <20120917183654.GA13273@x2.osted.lan> <20120918084924.GY37286@deviant.kiev.zoral.com.ua> Date: Wed, 26 Sep 2012 07:38:42 -0400 Message-ID: From: "b. f." To: freebsd-fs@freebsd.org Content-Type: text/plain; charset=ISO-8859-1 Cc: Subject: Re: Problems after recent nullfs,vfs changes in 10.0-CURRENT X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: bf1783@gmail.com List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 26 Sep 2012 11:38:44 -0000 On 9/18/12, b. f. wrote: > The following deals with some problems exposed by r240283-5, > particularly (but not only) when used with changes to tmpfs that were > first proposed by kib@ on 21 June 2010 on this list, in a thread > entitled "Tmpfs elimination of double copy": > > http://docs.freebsd.org/cgi/getmsg.cgi?fetch=20463+0+archive/2010/freebsd-fs/20100627.freebsd-fs > > On 9/18/12, Konstantin Belousov wrote: >> On Mon, Sep 17, 2012 at 08:36:54PM +0200, Peter Holm wrote: >>> On Mon, Sep 17, 2012 at 03:19:25PM +0300, Konstantin Belousov wrote: >>> > Please mail fs@, possibly Cc:-ing me. >>> > >>> > On Mon, Sep 17, 2012 at 03:04:46AM -0400, b. f. wrote: >>> > > The recent nullfs or vfs changes (r240283-5) have exposed some >>> > > problems with my tinderbox. In this tinderbox, I've been using >>> > > recent >>> > > versions of -CURRENT with Gleb's tmpfs rbtree patch: >>> > > >>> > > http://people.freebsd.org/~gleb/tmpfs-nrbtree.1.patch >>> > > >>> > > and a merged version of your tmpfs single-buffer patch: >>> > > >>> > > http://people.freebsd.org/~kib/misc/tmpfs.12.patch >>> > > >>> > > The tinderbox performs builds in a tmpfs filesystem that is nullfs >>> > > grafted to a ufs filesystem. After r240283-5, builds of >>> > > ports/lang/ocaml failed when a cp(1) of an executable failed with >>> > > ETXTBSY. After reverting r240285, the builds of ocaml succeeded. >>> > > >>> > > I've attached logs of the failed and successful builds. Can you >>> > > guess >>> > > whether the problem is solely due to the recent nullfs and vfs >>> > > changes, or to some defect in Gleb's proposed changes, or to a >>> > > problem >>> > > with your proposed tmpfs change, or my merging of it? What further >>> > > changes or tests would you suggest to help find the source of the >>> > > problem? >>> > > >>> > > I've attached a diff of the relevant changes to the system sources >>> > > used in the tinderbox, and logs of the successful (*.log) and >>> > > unsuccessful (*.log.error) ocaml builds. >>> > >>> > Please show me the mount -v output, and specify which filesystems >>> > are used where. > > The following is a typical layout for one run of the tinderbox (which > is in /home/shared/freebsd/tinderbox): > > /dev/ufs/d1root on / (ufs, local, noatime, writes: sync 13 async 25, > reads: sync 553 async 42, fsid 8aabfa4d68614a9f) > devfs on /dev (devfs, local, fsid 00ff007171000000) > tmpfs on /tmp (tmpfs, local, nosuid, fsid 01ff008787000000) > /dev/ufs/d1var on /var (ufs, local, noatime, journaled soft-updates, > writes: sync 15 async 269, reads: sync 664 async 12, fsid > a5abfa4d331091c9) > /dev/ufs/d1usr on /usr (ufs, local, noatime, journaled soft-updates, > writes: sync 2 async 0, reads: sync 765 async 12, fsid > b4abfa4d94c0f782) > /dev/ufs/d1usrlocal on /usr/local (ufs, local, noatime, journaled > soft-updates, writes: sync 32 async 298, reads: sync 2867 async 106, > fsid c4abfa4d96ab4351) > /dev/ufs/d1home on /home (ufs, local, noatime, journaled soft-updates, > writes: sync 16 async 123, reads: sync 2065 async 268, fsid > ceabfa4d9bb85870) > > the filesystem used for the port builds: > > /tmp/tinderbox/7.4-amd64-u1 on > /home/shared/freebsd/tinderbox/7.4-amd64-u1 (nullfs, local, fsid > 03ff002929000000) > /home/shared/freebsd/ports/head on > /home/shared/freebsd/tinderbox/7.4-amd64-u1/a/ports (nullfs, local, > read-only, fsid 04ff002929000000) > /home/shared/freebsd/tinderbox/jails/7.4-amd64/src on > /home/shared/freebsd/tinderbox/7.4-amd64-u1/usr/src (nullfs, local, > read-only, fsid 05ff002929000000) > devfs on /home/shared/freebsd/tinderbox/7.4-amd64-u1/dev (devfs, > local, fsid 06ff007171000000) > /home/shared/freebsd/distfiles on > /home/shared/freebsd/tinderbox/7.4-amd64-u1/distcache (nullfs, local, > fsid 07ff002929000000) > linprocfs on /home/shared/freebsd/tinderbox/7.4-amd64-u1/compat/linux/proc > (linprocfs, local, fsid 08ff00b5b5000000) > procfs on /home/shared/freebsd/tinderbox/7.4-amd64-u1/proc (procfs, > local, fsid 09ff000202000000) > >>> > >>> > The issue almost definitely is the held reference on the vm object. >>> > Lets remove Gleb' patches from the picture at all. >>> > >>> > After rethinking VV_TEXT handling both for nullfs and tmpfs (patched), >>> > I see two issues ATM: >>> > >>> > 1. VV_TEXT may be set either on the lower vnode, or on the nullfs >>> > vnode. >>> > So if you executed a file from nullfs alias, lower vnode does not get >>> > VV_TEXT set, and executable can still be opened for write. >>> > >>> > 2. For tmpfs, the hack I added to clear VV_TEXT if swap vm object >>> > reference >>> > count == 1, is not called often enough. This allows to VV_TEXT to >>> > leak, >>> > esp. >>> > because nullfs after r240283 is not eager to reclaim its vnodes. >>> > >>> > I updated my branch with tmpfs patches with the following changes: >>> > >>> > 1. nullfs now bypasses the VV_TEXT set and clear operations to the >>> > lower >>> > vnode. >>> > >>> > 2. the tmpfs_clear_text() hack is removed, instead >>> > vm_object_deallocate() >>> > clears VV_TEXT on the tmpfs vnode if reference count goes to 1. >>> > >>> > Updated patch is at >>> > http://people.freebsd.org/~kib/misc/tmpfs.13.patch >>> > I tested it very lightly, so to say. >>> >>> I see the problem on a pristine r240611. Test scenario included. >>> >>> + mdconfig -a -t swap -s 1g -u 5 >>> + bsdlabel -w md5 auto >>> + newfs -U md5a >>> + mount /dev/md5a /mnt2 >>> + chmod 777 /mnt2 >>> + mount >>> + grep /mnt >>> + grep -q tmpfs >>> + mount -t tmpfs tmpfs /mnt >>> + chmod 777 /mnt >>> + mkdir /mnt2/mp >>> + mount -t nullfs /mnt /mnt2/mp >>> + cp /usr/bin/true /mnt2/mp/true >>> + /mnt/true >>> + >>> + rm -f /mnt/true >>> + cp /usr/bin/true /mnt2/mp/true >>> + /mnt2/mp/true >>> + >>> ./nullfs12.sh: cannot create /mnt2/mp/true: Text file busy >>> + echo FAIL 2 >>> FAIL 2 >>> + mount >>> + egrep 'tmpfs|nullfs|/mnt |/mnt2 ' >>> /dev/md5a on /mnt2 (ufs, local, soft-updates) >>> tmpfs on /mnt (tmpfs, NFS exported, local) >>> /mnt on /mnt2/mp (nullfs, local) >>> + rm -f /mnt2/mp/true >> >> Yes, this is very close if not identical to the only test which I >> performed >> with the tmpfs.13.patch. >> > > I can no longer reproduce the port build failures on r240651 amd64 > after applying your tmpfs.13.patch, and I haven't encountered any > other obvious problems in the short time that I've been using it. I > did not rerun Peter Holm's nullfs12.sh test, since you had already > subjected your patch to a similar test. > After further experiments, it appears that there are still some problems with tmpfs.13.patch and the recent vfs/nullfs changes. If I use: sysctl debug.iosize_max_clamp=0 I observe build failures with some ports, when various utilities fail with EIO. For example textproc/libxml2: "... /bin/sh ../../libtool --tag=CC --mode=link cc -O2 -pipe -fno-strict-aliasing -std=gnu89 -pedantic -W -Wformat -Wunused -Wimplicit -Wreturn-type -Wswitch -Wcomment -Wtrigraphs -Wformat -Wchar-subscripts -Wuninitialized -Wparentheses -Wshadow -Wpointer-arith -Wcast-align -Wwrite-strings -Waggregate-return -Wstrict-prototypes -Wmissing-prototypes -Wnested-externs -Winline -Wredundant-decls -L/usr/local/lib -pthread -o tree1 tree1.o ../../libxml2.la -lz -L/usr/local/lib -liconv -lm eval: printf: Input/output error ../../libtool: printf: Input/output error ../../libtool: printf: Input/output error ../../libtool: printf: Input/output error ../../libtool: printf: Input/output error ../../libtool: printf: Input/output error ../../libtool: printf: Input/output error ../../libtool: printf: Input/output error ../../libtool: printf: Input/output error ../../libtool: printf: Input/output error ../../libtool: printf: Input/output error ../../libtool: printf: Input/output error ../../libtool: printf: Input/output error ../../libtool: printf: Input/output error ../../libtool: printf: Input/output error eval: printf: Input/output error ../../libtool: printf: Input/output error ../../libtool: printf: Input/output error ../../libtool: printf: Input/output error ../../libtool: printf: Input/output error ../../libtool: printf: Input/output error ../../libtool: printf: Input/output error ../../libtool: printf: Input/output error ../../libtool: printf: Input/output error ../../libtool: printf: Input/output error gmake[3]: Leaving directory `/work/a/ports/textproc/libxml2/work/libxml2-2.7.8/doc/examples' gmake[3]: Entering directory `/work/a/ports/textproc/libxml2/work/libxml2-2.7.8/doc' gmake[3]: Nothing to be done for `all-am'. gmake[3]: Leaving directory `/work/a/ports/textproc/libxml2/work/libxml2-2.7.8/doc' gmake[2]: Leaving directory `/work/a/ports/textproc/libxml2/work/libxml2-2.7.8/doc' Making all in example gmake[2]: Entering directory `/work/a/ports/textproc/libxml2/work/libxml2-2.7.8/example' cc -DHAVE_CONFIG_H -I. -I.. -I../include -I../include -I./include -D_REENTRANT -I/usr/local/include -I/usr/local/include -O2 -pipe -fno-strict-aliasing -std=gnu89 -pedantic -W -Wformat -Wunused -Wimplicit -Wreturn-type -Wswitch -Wcomment -Wtrigraphs -Wformat -Wchar-subscripts -Wuninitialized -Wparentheses -Wshadow -Wpointer-arith -Wcast-align -Wwrite-strings -Waggregate-return -Wstrict-prototypes -Wmissing-prototypes -Wnested-externs -Winline -Wredundant-decls -MT gjobread.o -MD -MP -MF .deps/gjobread.Tpo -c -o gjobread.o gjobread.c mv -f .deps/gjobread.Tpo .deps/gjobread.Po /bin/sh ../libtool --tag=CC --mode=link cc -O2 -pipe -fno-strict-aliasing -std=gnu89 -pedantic -W -Wformat -Wunused -Wimplicit -Wreturn-type -Wswitch -Wcomment -Wtrigraphs -Wformat -Wchar-subscripts -Wuninitialized -Wparentheses -Wshadow -Wpointer-arith -Wcast-align -Wwrite-strings -Waggregate-return -Wstrict-prototypes -Wmissing-prototypes -Wnested-externs -Winline -Wredundant-decls -L/usr/local/lib -pthread -o gjobread gjobread.o ../libxml2.la -lz -L/usr/local/lib -liconv -lm eval: printf: Input/output error ../libtool: printf: Input/output error ../libtool: printf: Input/output error gmake[2]: *** [gjobread] Error 1 gmake[2]: Leaving directory `/work/a/ports/textproc/libxml2/work/libxml2-2.7.8/example' gmake[1]: *** [all-recursive] Error 1 gmake[1]: Leaving directory `/work/a/ports/textproc/libxml2/work/libxml2-2.7.8' gmake: *** [all] Error 2 *** Error code 1 Stop in /a/ports/textproc/libxml2." or x11/libxcb: "... ===> Registering installation for libxcb-1.7 /usr/bin/tr: Input/output error /usr/bin/tr: Input/output error /usr/bin/tr: Input/output error ================================================================ ======================================== ===> Building package for libxcb-1.7 Deleting libxcb-1.7 === Checking filesystem state /buildscript: tr: Input/output error /buildscript: tr: Input/output error ..." There seems to be a temporal element to these failures, as they occur often, but not always. So far I have not been able to produce them with the default debug.iosize_max_clamp=1, or with debug.iosize_max_clamp=0 but without tmpfs.13.patch. And they didn't occur before the recent vfs/nullfs changes with timpfs.12.patch, regardless of the debug.iosize_max_clamp setting. Regards, b.