From owner-freebsd-fs@FreeBSD.ORG Wed May 15 16:28:37 2013 Return-Path: <owner-freebsd-fs@FreeBSD.ORG> Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 24F6B4A7 for <freebsd-fs@freebsd.org>; Wed, 15 May 2013 16:28:37 +0000 (UTC) (envelope-from martin@lispworks.com) Received: from lwfs1-cam.cam.lispworks.com (mail.lispworks.com [193.34.186.230]) by mx1.freebsd.org (Postfix) with ESMTP id 9F06EA9A for <freebsd-fs@freebsd.org>; Wed, 15 May 2013 16:28:36 +0000 (UTC) Received: from higson.cam.lispworks.com (higson.cam.lispworks.com [192.168.1.7]) by lwfs1-cam.cam.lispworks.com (8.14.5/8.14.5) with ESMTP id r4FGHRJM092927; Wed, 15 May 2013 17:17:27 +0100 (BST) (envelope-from martin@lispworks.com) Received: from higson.cam.lispworks.com (localhost.localdomain [127.0.0.1]) by higson.cam.lispworks.com (8.14.4) id r4FGHRDH032168; Wed, 15 May 2013 17:17:27 +0100 Received: (from martin@localhost) by higson.cam.lispworks.com (8.14.4/8.14.4/Submit) id r4FGHRGo031542; Wed, 15 May 2013 17:17:27 +0100 Date: Wed, 15 May 2013 17:17:27 +0100 Message-Id: <201305151617.r4FGHRGo031542@higson.cam.lispworks.com> From: Martin Simmons <martin@lispworks.com> To: Outback Dingo <outbackdingo@gmail.com> In-reply-to: <CAKYr3zwhcyto4ZPOs9_1EJx1N6jRF35E=RJvfw-3FsCesbdhkA@mail.gmail.com> (message from Outback Dingo on Wed, 15 May 2013 11:56:50 -0400) Subject: Re: FreeBSD 9-RELEASE zpool Out of File Descriptors ?? References: <CAKYr3zwer=P=GhW9WyxrMpLv4+AkxBfJkuZk99RTHtRW+-+24g@mail.gmail.com> <op.ww4kn4wx8527sy@ronaldradial.versatec.local> <CAKYr3zznxT7iL01ts1Cs9Eur-QkkK5mkKnb=qKPG-2UX+raqEw@mail.gmail.com> <op.ww4k7gmr8527sy@ronaldradial.versatec.local> <CAKYr3zznHFWJTiM5B980s7-2JDU99Dk2svf_+aUhhMZngFkayA@mail.gmail.com> <1CE05B33-7639-4F7F-9FD6-78B27D31E186@my.gd> <CAKYr3zwhcyto4ZPOs9_1EJx1N6jRF35E=RJvfw-3FsCesbdhkA@mail.gmail.com> Cc: freebsd-fs@freebsd.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems <freebsd-fs.freebsd.org> List-Unsubscribe: <http://lists.freebsd.org/mailman/options/freebsd-fs>, <mailto:freebsd-fs-request@freebsd.org?subject=unsubscribe> List-Archive: <http://lists.freebsd.org/pipermail/freebsd-fs> List-Post: <mailto:freebsd-fs@freebsd.org> List-Help: <mailto:freebsd-fs-request@freebsd.org?subject=help> List-Subscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-fs>, <mailto:freebsd-fs-request@freebsd.org?subject=subscribe> X-List-Received-Date: Wed, 15 May 2013 16:28:37 -0000 >>>>> On Wed, 15 May 2013 11:56:50 -0400, Outback Dingo said: > > On Wed, May 15, 2013 at 11:49 AM, Fleuriot Damien <ml@my.gd> wrote: > > > > > On May 15, 2013, at 2:46 PM, Outback Dingo <outbackdingo@gmail.com> wrote: > > > > > On Wed, May 15, 2013 at 8:34 AM, Ronald Klop < > > ronald-freebsd8@klop.yi.org>wrote: > > > > > >> On Wed, 15 May 2013 14:27:10 +0200, Outback Dingo < > > outbackdingo@gmail.com> > > >> wrote: > > >> > > >> On Wed, May 15, 2013 at 8:22 AM, Ronald Klop < > > ronald-freebsd8@klop.yi.org > > >>>> **wrote: > > >>> > > >>> On Wed, 15 May 2013 14:13:32 +0200, Outback Dingo < > > >>>> outbackdingo@gmail.com> > > >>>> wrote: > > >>>> > > >>>> So it seems a new deployment we just built with zfsonroot mirror and a > > >>>> > > >>>>> 48TB > > >>>>> master pool is already out of File Descriptors??? > > >>>>> > > >>>>> pool: master > > >>>>> state: ONLINE > > >>>>> status: The pool is formatted using a legacy on-disk format. The > > pool > > >>>>> can > > >>>>> still be used, but some features are unavailable. > > >>>>> action: Upgrade the pool using 'zpool upgrade'. Once this is done, > > the > > >>>>> pool will no longer be accessible on software that does not > > >>>>> support > > >>>>> feature > > >>>>> flags. > > >>>>> scan: none requested > > >>>>> config: > > >>>>> > > >>>>> NAME STATE READ WRITE CKSUM > > >>>>> master ONLINE 0 0 0 > > >>>>> raidz3-0 ONLINE 0 0 0 > > >>>>> multipath/SATA_LUN01 ONLINE 0 0 0 > > >>>>> multipath/SATA_LUN03 ONLINE 0 0 0 > > >>>>> multipath/SATA_LUN04 ONLINE 0 0 0 > > >>>>> multipath/SATA_LUN05 ONLINE 0 0 0 > > >>>>> multipath/SATA_LUN07 ONLINE 0 0 0 > > >>>>> multipath/SATA_LUN08 ONLINE 0 0 0 > > >>>>> multipath/SATA_LUN09 ONLINE 0 0 0 > > >>>>> multipath/SATA_LUN10 ONLINE 0 0 0 > > >>>>> raidz3-1 ONLINE 0 0 0 > > >>>>> multipath/SATA_LUN11 ONLINE 0 0 0 > > >>>>> multipath/SATA_LUN12 ONLINE 0 0 0 > > >>>>> multipath/SATA_LUN13 ONLINE 0 0 0 > > >>>>> multipath/SATA_LUN14 ONLINE 0 0 0 > > >>>>> multipath/SATA_LUN15 ONLINE 0 0 0 > > >>>>> multipath/SATA_LUN16 ONLINE 0 0 0 > > >>>>> multipath/SATA_LUN17 ONLINE 0 0 0 > > >>>>> multipath/SATA_LUN18 ONLINE 0 0 0 > > >>>>> raidz3-2 ONLINE 0 0 0 > > >>>>> multipath/SATA_LUN19 ONLINE 0 0 0 > > >>>>> multipath/SATA_LUN20 ONLINE 0 0 0 > > >>>>> multipath/SATA_LUN21 ONLINE 0 0 0 > > >>>>> multipath/SATA_LUN22 ONLINE 0 0 0 > > >>>>> multipath/SATA_LUN23 ONLINE 0 0 0 > > >>>>> multipath/SATA_LUN24 ONLINE 0 0 0 > > >>>>> multipath/SATA_LUN26 ONLINE 0 0 0 > > >>>>> multipath/SATA_LUN27 ONLINE 0 0 0 > > >>>>> raidz3-3 ONLINE 0 0 0 > > >>>>> multipath/SATA_LUN29 ONLINE 0 0 0 > > >>>>> multipath/SATA_LUN30 ONLINE 0 0 0 > > >>>>> multipath/SATA_LUN32 ONLINE 0 0 0 > > >>>>> multipath/SATA_LUN33 ONLINE 0 0 0 > > >>>>> multipath/SATA_LUN35 ONLINE 0 0 0 > > >>>>> multipath/SATA_LUN36 ONLINE 0 0 0 > > >>>>> multipath/SATA_LUN37 ONLINE 0 0 0 > > >>>>> multipath/SATA_LUN38 ONLINE 0 0 0 > > >>>>> logs > > >>>>> multipath/SATA_LUN06 ONLINE 0 0 0 > > >>>>> cache > > >>>>> multipath/SATA_LUN02 ONLINE 0 0 0 > > >>>>> > > >>>>> errors: No known data errors > > >>>>> pool: tank > > >>>>> state: ONLINE > > >>>>> status: The pool is formatted using a legacy on-disk format. The > > pool > > >>>>> can > > >>>>> still be used, but some features are unavailable. > > >>>>> action: Upgrade the pool using 'zpool upgrade'. Once this is done, > > the > > >>>>> pool will no longer be accessible on software that does not > > >>>>> support > > >>>>> feature > > >>>>> flags. > > >>>>> scan: none requested > > >>>>> config: > > >>>>> > > >>>>> NAME STATE READ WRITE CKSUM > > >>>>> tank ONLINE 0 0 0 > > >>>>> mirror-0 ONLINE 0 0 0 > > >>>>> da34p3 ONLINE 0 0 0 > > >>>>> da35p3 ONLINE 0 0 0 > > >>>>> > > >>>>> errors: No known data errors > > >>>>> > > >>>>> while compiling a kernel, buildworld worked and installed ok > > >>>>> > > >>>>> lex -t > > >>>>> /usr/src/sys/modules/aic7xxx/****aicasm/../../../dev/aic7xxx/**** > > >>>>> aicasm/aicasm_scan.l > > >>>>> > > >>>>> aicasm_scan.c > > >>>>>> > > >>>>>> lex -t -Pmm > > >>>>> /usr/src/sys/modules/aic7xxx/****aicasm/../../../dev/aic7xxx/**** > > >>>>> > > >>>>> aicasm/aicasm_macro_scan.l > > >>>>> > > >>>>> aicasm_macro_scan.c > > >>>>>> > > >>>>>> rm -f .depend_aicasm > > >>>>> mkdep -f .depend_aicasm -a -I. > > >>>>> -I/usr/src/sys/modules/****aic7xxx/aicasm/../../../dev/**** > > >>>>> aic7xxx/aicasm > > >>>>> -std=gnu99 > > >>>>> /usr/src/sys/modules/aic7xxx/****aicasm/../../../dev/aic7xxx/**** > > >>>>> aicasm/aicasm.c > > >>>>> /usr/src/sys/modules/aic7xxx/****aicasm/../../../dev/aic7xxx/**** > > >>>>> > > >>>>> aicasm/aicasm_symbol.c > > >>>>> aicasm_gram.c aicasm_macro_gram.c aicasm_scan.c aicasm_macro_scan.c > > >>>>> Out of file descriptors > > >>>>> *** [.depend_aicasm] Error code 2 > > >>>>> > > >>>>> Stop in /usr/src/sys/modules/aic7xxx/****aicasm. > > >>>>> > > >>>>> *** [buildkernel] Error code 1 > > >>>>> > > >>>>> Stop in /usr/src. > > >>>>> *** [buildkernel] Error code 1 > > >>>>> > > >>>>> > > >>>> > > >>>> Do you know what a file descriptor is? > > >>>> > > >>>> There is something about it here: http://www.freebsd.org/doc/en/****< > > http://www.freebsd.org/doc/en/**> > > >>>> books/handbook/configtuning-****kernel-limits.html<http://www.** > > >>>> freebsd.org/doc/en/books/**handbook/configtuning-kernel-**limits.html > > < > > http://www.freebsd.org/doc/en/books/handbook/configtuning-kernel-limits.html > > >>(search > > >>>> for 'file descriptor') > > >>>> Or more info in here: https://www.google.nl/search?****< > > https://www.google.nl/search?**> > > >>>> q=freebsd+Out+of+file+****descriptors<https://www.** > > >>>> google.nl/search?q=freebsd+**Out+of+file+descriptors< > > https://www.google.nl/search?q=freebsd+Out+of+file+descriptors> > > >>>>> > > >>>> > > >>>> > > >>>> It mainly says that you have more files open than your system is > > >>>> configured to allow. This is often produced by a bug in a program > > which > > >>>> does not close some files properly. > > >>>> > > >>>> > > >>>> Yes I do.... but this is a brand new zfsonroot with barely any data on > > >>> it > > >>> and > > >>> > > >>> sysctl -a | grep kern.openfiles > > >>> kern.openfiles: 68 > > >>> root@:/master/builder # sysctl -a | grep kern.maxfiles > > >>> kern.maxfiles: 24600 > > >>> kern.maxfilesperproc: 11095 > > >>> > > >>> the openfiles and maxfiles seems to be plenty, but im out of > > descriptors, > > >>> i > > >>> used to see this back in the 4.x days when you could format > > >>> ufs with larger inodes, but zfs ??? really? > > >>> > > >> > > >> > > >> Is this kern.openfiles when idle or during your buildkernel? > > >> Did you check 'ulimit -a'? > > >> You say installworld worked ok while you are just now doing buildkernel. > > >> Are your kernel and world out of sync? > > >> > > >> > > > well, i was upgrading FreeBSD 9-RELEASE to CURRENT, did a make > > buildworld, > > > then a make installworld, when i went to do a make buildkernel > > > thats the error i got, so id say, yes it is "out of sync" > > > > > > Wow wow wow, hold on a sec here. > > > > You're installing your new world before the kernel ? > > > > > > I've had no trouble upgrading several boxes from 8-STABLE to 10.0-CURRENT, > > following the regular procedure as described at: > > http://www.freebsd.org/doc/handbook/makeworld.html > > > > > > Note that the correct order is: > > - buildworld (or kernel-toolchain if you only want to test the kernel and > > not do the whole world) > > - buildkernel > > - installkernel > > - reboot > > - mergemaster -p (preferably in single user, I've always run it multiuser > > w/o problems) > > - installworld > > - mergemaster > > - reboot > > > > You don't want to run the new world on the old kernel. > > > > yeah i know better, seems when i went to install the old kernel the build > had failed, i can get through it from here > > it was odd because the CURRENT kernel failed to build with a yyparse error, > until after i had completed a buildworld Taking a zfs snapshot of the system file systems before installing anything is a good idea. __Martin