From owner-freebsd-fs@FreeBSD.ORG Wed May 15 13:08:23 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 354D7A8B for ; Wed, 15 May 2013 13:08:23 +0000 (UTC) (envelope-from ronald-freebsd8@klop.yi.org) Received: from smarthost1.greenhost.nl (smarthost1.greenhost.nl [195.190.28.78]) by mx1.freebsd.org (Postfix) with ESMTP id AB1F8B68 for ; Wed, 15 May 2013 13:08:22 +0000 (UTC) Received: from smtp.greenhost.nl ([213.108.104.138]) by smarthost1.greenhost.nl with esmtps (TLS1.0:RSA_AES_256_CBC_SHA1:32) (Exim 4.69) (envelope-from ) id 1UcbRM-0000ay-3n; Wed, 15 May 2013 15:08:21 +0200 Received: from [81.21.138.17] (helo=ronaldradial.versatec.local) by smtp.greenhost.nl with esmtpsa (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.72) (envelope-from ) id 1UcbRK-0000nq-DR; Wed, 15 May 2013 15:08:18 +0200 Content-Type: text/plain; charset=us-ascii; format=flowed; delsp=yes To: "Outback Dingo" Subject: Re: FreeBSD 9-RELEASE zpool Out of File Descriptors ?? References: Date: Wed, 15 May 2013 15:08:17 +0200 MIME-Version: 1.0 Content-Transfer-Encoding: 8bit From: "Ronald Klop" Message-ID: In-Reply-To: User-Agent: Opera Mail/12.15 (Win32) X-Virus-Scanned: by clamav at smarthost1.samage.net X-Spam-Level: / X-Spam-Score: 0.8 X-Spam-Status: No, score=0.8 required=5.0 tests=BAYES_50 autolearn=disabled version=3.3.1 X-Scan-Signature: 0b90d2ac815c9ef578df8ccc1b50cfb0 Cc: freebsd-fs@freebsd.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 15 May 2013 13:08:23 -0000 On Wed, 15 May 2013 14:46:32 +0200, Outback Dingo wrote: > On Wed, May 15, 2013 at 8:34 AM, Ronald Klop > wrote: > >> On Wed, 15 May 2013 14:27:10 +0200, Outback Dingo >> >> wrote: >> >> On Wed, May 15, 2013 at 8:22 AM, Ronald Klop >> >> >**wrote: >>> >>> On Wed, 15 May 2013 14:13:32 +0200, Outback Dingo < >>>> outbackdingo@gmail.com> >>>> wrote: >>>> >>>> So it seems a new deployment we just built with zfsonroot mirror and >>>> a >>>> >>>>> 48TB >>>>> master pool is already out of File Descriptors??? >>>>> >>>>> pool: master >>>>> state: ONLINE >>>>> status: The pool is formatted using a legacy on-disk format. The >>>>> pool >>>>> can >>>>> still be used, but some features are unavailable. >>>>> action: Upgrade the pool using 'zpool upgrade'. Once this is done, >>>>> the >>>>> pool will no longer be accessible on software that does not >>>>> support >>>>> feature >>>>> flags. >>>>> scan: none requested >>>>> config: >>>>> >>>>> NAME STATE READ WRITE CKSUM >>>>> master ONLINE 0 0 0 >>>>> raidz3-0 ONLINE 0 0 0 >>>>> multipath/SATA_LUN01 ONLINE 0 0 0 >>>>> multipath/SATA_LUN03 ONLINE 0 0 0 >>>>> multipath/SATA_LUN04 ONLINE 0 0 0 >>>>> multipath/SATA_LUN05 ONLINE 0 0 0 >>>>> multipath/SATA_LUN07 ONLINE 0 0 0 >>>>> multipath/SATA_LUN08 ONLINE 0 0 0 >>>>> multipath/SATA_LUN09 ONLINE 0 0 0 >>>>> multipath/SATA_LUN10 ONLINE 0 0 0 >>>>> raidz3-1 ONLINE 0 0 0 >>>>> multipath/SATA_LUN11 ONLINE 0 0 0 >>>>> multipath/SATA_LUN12 ONLINE 0 0 0 >>>>> multipath/SATA_LUN13 ONLINE 0 0 0 >>>>> multipath/SATA_LUN14 ONLINE 0 0 0 >>>>> multipath/SATA_LUN15 ONLINE 0 0 0 >>>>> multipath/SATA_LUN16 ONLINE 0 0 0 >>>>> multipath/SATA_LUN17 ONLINE 0 0 0 >>>>> multipath/SATA_LUN18 ONLINE 0 0 0 >>>>> raidz3-2 ONLINE 0 0 0 >>>>> multipath/SATA_LUN19 ONLINE 0 0 0 >>>>> multipath/SATA_LUN20 ONLINE 0 0 0 >>>>> multipath/SATA_LUN21 ONLINE 0 0 0 >>>>> multipath/SATA_LUN22 ONLINE 0 0 0 >>>>> multipath/SATA_LUN23 ONLINE 0 0 0 >>>>> multipath/SATA_LUN24 ONLINE 0 0 0 >>>>> multipath/SATA_LUN26 ONLINE 0 0 0 >>>>> multipath/SATA_LUN27 ONLINE 0 0 0 >>>>> raidz3-3 ONLINE 0 0 0 >>>>> multipath/SATA_LUN29 ONLINE 0 0 0 >>>>> multipath/SATA_LUN30 ONLINE 0 0 0 >>>>> multipath/SATA_LUN32 ONLINE 0 0 0 >>>>> multipath/SATA_LUN33 ONLINE 0 0 0 >>>>> multipath/SATA_LUN35 ONLINE 0 0 0 >>>>> multipath/SATA_LUN36 ONLINE 0 0 0 >>>>> multipath/SATA_LUN37 ONLINE 0 0 0 >>>>> multipath/SATA_LUN38 ONLINE 0 0 0 >>>>> logs >>>>> multipath/SATA_LUN06 ONLINE 0 0 0 >>>>> cache >>>>> multipath/SATA_LUN02 ONLINE 0 0 0 >>>>> >>>>> errors: No known data errors >>>>> pool: tank >>>>> state: ONLINE >>>>> status: The pool is formatted using a legacy on-disk format. The >>>>> pool >>>>> can >>>>> still be used, but some features are unavailable. >>>>> action: Upgrade the pool using 'zpool upgrade'. Once this is done, >>>>> the >>>>> pool will no longer be accessible on software that does not >>>>> support >>>>> feature >>>>> flags. >>>>> scan: none requested >>>>> config: >>>>> >>>>> NAME STATE READ WRITE CKSUM >>>>> tank ONLINE 0 0 0 >>>>> mirror-0 ONLINE 0 0 0 >>>>> da34p3 ONLINE 0 0 0 >>>>> da35p3 ONLINE 0 0 0 >>>>> >>>>> errors: No known data errors >>>>> >>>>> while compiling a kernel, buildworld worked and installed ok >>>>> >>>>> lex -t >>>>> /usr/src/sys/modules/aic7xxx/****aicasm/../../../dev/aic7xxx/**** >>>>> aicasm/aicasm_scan.l >>>>> >>>>> aicasm_scan.c >>>>>> >>>>>> lex -t -Pmm >>>>> /usr/src/sys/modules/aic7xxx/****aicasm/../../../dev/aic7xxx/**** >>>>> >>>>> aicasm/aicasm_macro_scan.l >>>>> >>>>> aicasm_macro_scan.c >>>>>> >>>>>> rm -f .depend_aicasm >>>>> mkdep -f .depend_aicasm -a -I. >>>>> -I/usr/src/sys/modules/****aic7xxx/aicasm/../../../dev/**** >>>>> aic7xxx/aicasm >>>>> -std=gnu99 >>>>> /usr/src/sys/modules/aic7xxx/****aicasm/../../../dev/aic7xxx/**** >>>>> aicasm/aicasm.c >>>>> /usr/src/sys/modules/aic7xxx/****aicasm/../../../dev/aic7xxx/**** >>>>> >>>>> aicasm/aicasm_symbol.c >>>>> aicasm_gram.c aicasm_macro_gram.c aicasm_scan.c aicasm_macro_scan.c >>>>> Out of file descriptors >>>>> *** [.depend_aicasm] Error code 2 >>>>> >>>>> Stop in /usr/src/sys/modules/aic7xxx/****aicasm. >>>>> >>>>> *** [buildkernel] Error code 1 >>>>> >>>>> Stop in /usr/src. >>>>> *** [buildkernel] Error code 1 >>>>> >>>>> >>>> >>>> Do you know what a file descriptor is? >>>> >>>> There is something about it here: >>>> http://www.freebsd.org/doc/en/**** >>>> books/handbook/configtuning-****kernel-limits.html>>> freebsd.org/doc/en/books/**handbook/configtuning-kernel-**limits.html>(search >>>> for 'file descriptor') >>>> Or more info in here: >>>> https://www.google.nl/search?**** >>>> q=freebsd+Out+of+file+****descriptors>>> google.nl/search?q=freebsd+**Out+of+file+descriptors >>>> > >>>> >>>> >>>> It mainly says that you have more files open than your system is >>>> configured to allow. This is often produced by a bug in a program >>>> which >>>> does not close some files properly. >>>> >>>> >>>> Yes I do.... but this is a brand new zfsonroot with barely any data >>>> on >>> it >>> and >>> >>> sysctl -a | grep kern.openfiles >>> kern.openfiles: 68 >>> root@:/master/builder # sysctl -a | grep kern.maxfiles >>> kern.maxfiles: 24600 >>> kern.maxfilesperproc: 11095 >>> >>> the openfiles and maxfiles seems to be plenty, but im out of >>> descriptors, >>> i >>> used to see this back in the 4.x days when you could format >>> ufs with larger inodes, but zfs ??? really? >>> >> >> >> Is this kern.openfiles when idle or during your buildkernel? >> Did you check 'ulimit -a'? >> You say installworld worked ok while you are just now doing buildkernel. >> Are your kernel and world out of sync? >> >> > well, i was upgrading FreeBSD 9-RELEASE to CURRENT, did a make > buildworld, > then a make installworld, when i went to do a make buildkernel > thats the error i got, so id say, yes it is "out of sync" Read /usr/src/UPDATING from the line 'COMMON ITEMS:'. Especially interesting is 'To upgrade in-place from stable to current'. The order is important. See also http://www.freebsd.org/doc/en/books/handbook/updating-upgrading.html for a lot of text about the same. To fix your current situation. I'm not familiar with that situation but can imagine to try to put back a 9-RELEASE world. Do you have a snapshot of your current system with world from 9-RELEASE? Or another backup? Ronald.