From owner-freebsd-fs@FreeBSD.ORG Wed May 15 15:49:19 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 5C40799A for ; Wed, 15 May 2013 15:49:19 +0000 (UTC) (envelope-from ml@my.gd) Received: from mail-wi0-x231.google.com (mail-wi0-x231.google.com [IPv6:2a00:1450:400c:c05::231]) by mx1.freebsd.org (Postfix) with ESMTP id E5B8588F for ; Wed, 15 May 2013 15:49:18 +0000 (UTC) Received: by mail-wi0-f177.google.com with SMTP id hr14so1784191wib.16 for ; Wed, 15 May 2013 08:49:18 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=x-received:content-type:mime-version:subject:from:in-reply-to:date :cc:content-transfer-encoding:message-id:references:to:x-mailer :x-gm-message-state; bh=Gi4mUzBYSWZdtawt5inC+6rbYV5PqqHyGjU9JXqNkv4=; b=F8vn7N9jtb/KXRwzcb1IjSlL1bXQ97DPaKoZFksAUbfdQ6+37uD18i8C2T1+opyINn OZ5Zkzsb9BniUo4mLcQnrLZ0IYC3i2z/mJhz1cYexlKqJX3x+DO7iHgAlnQwXDU5khnE my9ydFZNgTIljle481ptHv7e6BNCmjs17H1IPHcl5+N6c+rMv3fP5IpJb3xSUdT88pjX 1kcXl59gNbdX7omnWjPbQVjPiZMiye1oDffFVBEQEeUVT5Fsd1ZV7F1u3PfJZCKpgPfC Qt8+eJ629gbi+d9lPPmlQUy4eJf9xlU0ZNwtHl41gDrk7dokNvsWe7iNZDUsP3UjP8tf U9+w== X-Received: by 10.180.108.3 with SMTP id hg3mr15672211wib.17.1368632944865; Wed, 15 May 2013 08:49:04 -0700 (PDT) Received: from dfleuriot.paris.hi-media-techno.com ([83.167.62.196]) by mx.google.com with ESMTPSA id ay7sm21450167wib.9.2013.05.15.08.49.03 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Wed, 15 May 2013 08:49:03 -0700 (PDT) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 6.3 \(1503\)) Subject: Re: FreeBSD 9-RELEASE zpool Out of File Descriptors ?? From: Fleuriot Damien In-Reply-To: Date: Wed, 15 May 2013 17:49:03 +0200 Content-Transfer-Encoding: quoted-printable Message-Id: <1CE05B33-7639-4F7F-9FD6-78B27D31E186@my.gd> References: To: Outback Dingo X-Mailer: Apple Mail (2.1503) X-Gm-Message-State: ALoCoQleseUUzG1zOr/qUFyJFBZ0wmvXQe4gi4Z10c0ZyxPha8WzKvL7lW1nrbmQDlGIyqq8AHNS Cc: freebsd-fs@freebsd.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 15 May 2013 15:49:19 -0000 On May 15, 2013, at 2:46 PM, Outback Dingo = wrote: > On Wed, May 15, 2013 at 8:34 AM, Ronald Klop = wrote: >=20 >> On Wed, 15 May 2013 14:27:10 +0200, Outback Dingo = >> wrote: >>=20 >> On Wed, May 15, 2013 at 8:22 AM, Ronald Klop = >>> **wrote: >>>=20 >>> On Wed, 15 May 2013 14:13:32 +0200, Outback Dingo < >>>> outbackdingo@gmail.com> >>>> wrote: >>>>=20 >>>> So it seems a new deployment we just built with zfsonroot mirror = and a >>>>=20 >>>>> 48TB >>>>> master pool is already out of File Descriptors??? >>>>>=20 >>>>> pool: master >>>>> state: ONLINE >>>>> status: The pool is formatted using a legacy on-disk format. The = pool >>>>> can >>>>> still be used, but some features are unavailable. >>>>> action: Upgrade the pool using 'zpool upgrade'. Once this is = done, the >>>>> pool will no longer be accessible on software that does not >>>>> support >>>>> feature >>>>> flags. >>>>> scan: none requested >>>>> config: >>>>>=20 >>>>> NAME STATE READ WRITE CKSUM >>>>> master ONLINE 0 0 0 >>>>> raidz3-0 ONLINE 0 0 0 >>>>> multipath/SATA_LUN01 ONLINE 0 0 0 >>>>> multipath/SATA_LUN03 ONLINE 0 0 0 >>>>> multipath/SATA_LUN04 ONLINE 0 0 0 >>>>> multipath/SATA_LUN05 ONLINE 0 0 0 >>>>> multipath/SATA_LUN07 ONLINE 0 0 0 >>>>> multipath/SATA_LUN08 ONLINE 0 0 0 >>>>> multipath/SATA_LUN09 ONLINE 0 0 0 >>>>> multipath/SATA_LUN10 ONLINE 0 0 0 >>>>> raidz3-1 ONLINE 0 0 0 >>>>> multipath/SATA_LUN11 ONLINE 0 0 0 >>>>> multipath/SATA_LUN12 ONLINE 0 0 0 >>>>> multipath/SATA_LUN13 ONLINE 0 0 0 >>>>> multipath/SATA_LUN14 ONLINE 0 0 0 >>>>> multipath/SATA_LUN15 ONLINE 0 0 0 >>>>> multipath/SATA_LUN16 ONLINE 0 0 0 >>>>> multipath/SATA_LUN17 ONLINE 0 0 0 >>>>> multipath/SATA_LUN18 ONLINE 0 0 0 >>>>> raidz3-2 ONLINE 0 0 0 >>>>> multipath/SATA_LUN19 ONLINE 0 0 0 >>>>> multipath/SATA_LUN20 ONLINE 0 0 0 >>>>> multipath/SATA_LUN21 ONLINE 0 0 0 >>>>> multipath/SATA_LUN22 ONLINE 0 0 0 >>>>> multipath/SATA_LUN23 ONLINE 0 0 0 >>>>> multipath/SATA_LUN24 ONLINE 0 0 0 >>>>> multipath/SATA_LUN26 ONLINE 0 0 0 >>>>> multipath/SATA_LUN27 ONLINE 0 0 0 >>>>> raidz3-3 ONLINE 0 0 0 >>>>> multipath/SATA_LUN29 ONLINE 0 0 0 >>>>> multipath/SATA_LUN30 ONLINE 0 0 0 >>>>> multipath/SATA_LUN32 ONLINE 0 0 0 >>>>> multipath/SATA_LUN33 ONLINE 0 0 0 >>>>> multipath/SATA_LUN35 ONLINE 0 0 0 >>>>> multipath/SATA_LUN36 ONLINE 0 0 0 >>>>> multipath/SATA_LUN37 ONLINE 0 0 0 >>>>> multipath/SATA_LUN38 ONLINE 0 0 0 >>>>> logs >>>>> multipath/SATA_LUN06 ONLINE 0 0 0 >>>>> cache >>>>> multipath/SATA_LUN02 ONLINE 0 0 0 >>>>>=20 >>>>> errors: No known data errors >>>>> pool: tank >>>>> state: ONLINE >>>>> status: The pool is formatted using a legacy on-disk format. The = pool >>>>> can >>>>> still be used, but some features are unavailable. >>>>> action: Upgrade the pool using 'zpool upgrade'. Once this is = done, the >>>>> pool will no longer be accessible on software that does not >>>>> support >>>>> feature >>>>> flags. >>>>> scan: none requested >>>>> config: >>>>>=20 >>>>> NAME STATE READ WRITE CKSUM >>>>> tank ONLINE 0 0 0 >>>>> mirror-0 ONLINE 0 0 0 >>>>> da34p3 ONLINE 0 0 0 >>>>> da35p3 ONLINE 0 0 0 >>>>>=20 >>>>> errors: No known data errors >>>>>=20 >>>>> while compiling a kernel, buildworld worked and installed ok >>>>>=20 >>>>> lex -t >>>>> /usr/src/sys/modules/aic7xxx/****aicasm/../../../dev/aic7xxx/**** >>>>> aicasm/aicasm_scan.l >>>>>=20 >>>>> aicasm_scan.c >>>>>>=20 >>>>>> lex -t -Pmm >>>>> /usr/src/sys/modules/aic7xxx/****aicasm/../../../dev/aic7xxx/**** >>>>>=20 >>>>> aicasm/aicasm_macro_scan.l >>>>>=20 >>>>> aicasm_macro_scan.c >>>>>>=20 >>>>>> rm -f .depend_aicasm >>>>> mkdep -f .depend_aicasm -a -I. >>>>> -I/usr/src/sys/modules/****aic7xxx/aicasm/../../../dev/**** >>>>> aic7xxx/aicasm >>>>> -std=3Dgnu99 >>>>> /usr/src/sys/modules/aic7xxx/****aicasm/../../../dev/aic7xxx/**** >>>>> aicasm/aicasm.c >>>>> /usr/src/sys/modules/aic7xxx/****aicasm/../../../dev/aic7xxx/**** >>>>>=20 >>>>> aicasm/aicasm_symbol.c >>>>> aicasm_gram.c aicasm_macro_gram.c aicasm_scan.c = aicasm_macro_scan.c >>>>> Out of file descriptors >>>>> *** [.depend_aicasm] Error code 2 >>>>>=20 >>>>> Stop in /usr/src/sys/modules/aic7xxx/****aicasm. >>>>>=20 >>>>> *** [buildkernel] Error code 1 >>>>>=20 >>>>> Stop in /usr/src. >>>>> *** [buildkernel] Error code 1 >>>>>=20 >>>>>=20 >>>>=20 >>>> Do you know what a file descriptor is? >>>>=20 >>>> There is something about it here: = http://www.freebsd.org/doc/en/**** >>>> books/handbook/configtuning-****kernel-limits.html>>> = freebsd.org/doc/en/books/**handbook/configtuning-kernel-**limits.html>= (search >>>> for 'file descriptor') >>>> Or more info in here: = https://www.google.nl/search?**** >>>> q=3Dfreebsd+Out+of+file+****descriptors>>> = google.nl/search?q=3Dfreebsd+**Out+of+file+descriptors >>>>>=20 >>>>=20 >>>>=20 >>>> It mainly says that you have more files open than your system is >>>> configured to allow. This is often produced by a bug in a program = which >>>> does not close some files properly. >>>>=20 >>>>=20 >>>> Yes I do.... but this is a brand new zfsonroot with barely any data = on >>> it >>> and >>>=20 >>> sysctl -a | grep kern.openfiles >>> kern.openfiles: 68 >>> root@:/master/builder # sysctl -a | grep kern.maxfiles >>> kern.maxfiles: 24600 >>> kern.maxfilesperproc: 11095 >>>=20 >>> the openfiles and maxfiles seems to be plenty, but im out of = descriptors, >>> i >>> used to see this back in the 4.x days when you could format >>> ufs with larger inodes, but zfs ??? really? >>>=20 >>=20 >>=20 >> Is this kern.openfiles when idle or during your buildkernel? >> Did you check 'ulimit -a'? >> You say installworld worked ok while you are just now doing = buildkernel. >> Are your kernel and world out of sync? >>=20 >>=20 > well, i was upgrading FreeBSD 9-RELEASE to CURRENT, did a make = buildworld, > then a make installworld, when i went to do a make buildkernel > thats the error i got, so id say, yes it is "out of sync" Wow wow wow, hold on a sec here. You're installing your new world before the kernel ? I've had no trouble upgrading several boxes from 8-STABLE to = 10.0-CURRENT, following the regular procedure as described at: http://www.freebsd.org/doc/handbook/makeworld.html Note that the correct order is: - buildworld (or kernel-toolchain if you only want to test the kernel = and not do the whole world) - buildkernel - installkernel - reboot - mergemaster -p (preferably in single user, I've always run it = multiuser w/o problems) - installworld - mergemaster - reboot You don't want to run the new world on the old kernel.