From owner-freebsd-fs@FreeBSD.ORG Mon Nov 5 11:06:56 2007 Return-Path: Delivered-To: freebsd-fs@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 7404416A50B for ; Mon, 5 Nov 2007 11:06:56 +0000 (UTC) (envelope-from owner-bugmaster@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 6467313C4A3 for ; Mon, 5 Nov 2007 11:06:56 +0000 (UTC) (envelope-from owner-bugmaster@FreeBSD.org) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.1/8.14.1) with ESMTP id lA5B6uTH026227 for ; Mon, 5 Nov 2007 11:06:56 GMT (envelope-from owner-bugmaster@FreeBSD.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.1/8.14.1/Submit) id lA5B6tJ0026223 for freebsd-fs@FreeBSD.org; Mon, 5 Nov 2007 11:06:55 GMT (envelope-from owner-bugmaster@FreeBSD.org) Date: Mon, 5 Nov 2007 11:06:55 GMT Message-Id: <200711051106.lA5B6tJ0026223@freefall.freebsd.org> X-Authentication-Warning: freefall.freebsd.org: gnats set sender to owner-bugmaster@FreeBSD.org using -f From: FreeBSD bugmaster To: freebsd-fs@FreeBSD.org Cc: Subject: Current problem reports assigned to freebsd-fs@FreeBSD.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 05 Nov 2007 11:06:56 -0000 Current FreeBSD problem reports Critical problems Serious problems S Tracker Resp. Description -------------------------------------------------------------------------------- o kern/112658 fs [smbfs] [patch] smbfs and caching problems (resolves b o kern/114676 fs [ufs] snapshot creation panics: snapacct_ufs2: bad blo o kern/114856 fs [ntfs] [patch] Bug in NTFS allows bogus file modes. o kern/116170 fs Kernel panic when mounting /tmp 4 problems total. Non-critical problems S Tracker Resp. Description -------------------------------------------------------------------------------- o kern/114847 fs [ntfs] [patch] dirmask support for NTFS ala MSDOSFS 1 problem total. From owner-freebsd-fs@FreeBSD.ORG Mon Nov 5 22:02:56 2007 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 8BB3816A421; Mon, 5 Nov 2007 22:02:56 +0000 (UTC) (envelope-from thompsa@FreeBSD.org) Received: from heff.fud.org.nz (203-109-251-39.static.bliink.ihug.co.nz [203.109.251.39]) by mx1.freebsd.org (Postfix) with ESMTP id 1F17613C48E; Mon, 5 Nov 2007 22:02:56 +0000 (UTC) (envelope-from thompsa@FreeBSD.org) Received: by heff.fud.org.nz (Postfix, from userid 1001) id B928958AB; Tue, 6 Nov 2007 10:50:35 +1300 (NZDT) Date: Tue, 6 Nov 2007 10:50:35 +1300 From: Andrew Thompson To: Pawel Jakub Dawidek Message-ID: <20071105215035.GC26730@heff.fud.org.nz> References: <20071005000046.GC92272@garage.freebsd.pl> <20071008121523.GM2327@garage.freebsd.pl> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20071008121523.GM2327@garage.freebsd.pl> User-Agent: Mutt/1.5.16 (2007-06-09) Cc: freebsd-fs@FreeBSD.org, freebsd-current@FreeBSD.org Subject: Re: ZFS kmem_map too small. X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 05 Nov 2007 22:02:56 -0000 On Mon, Oct 08, 2007 at 02:15:23PM +0200, Pawel Jakub Dawidek wrote: > Here are some updates: > > I was able to reproduce the panic by rsyncing big files and trying > bonnie++ test suggested in this thread. > > Can you guys retry with this patch: > > http://people.freebsd.org/~pjd/patches/vm_kern.c.2.patch > > It's a hack, yes, but allows to mitigate the problem quite well. I'm > looking for a solution that can be used for 7.0 before we find a better > fix. Do you still plan to commit this for 7.0 or is a better solution in the works? I would assume something would need to be in before RC1. > BTW. To use ZFS you _must_ increase vm.kmem_size/vm.kmem_size_max. > If you have the problem discussed here and you're using standard values, > please retry with vm.kmem_size/vm.kmem_size_max set to at least 600MB in > /boot/loader.conf. > > I'm not sure if it's not too late to ask re@ about increasing the > default kmem size at least on amd64. ~300MB we have there is silly > small. > > -- > Pawel Jakub Dawidek http://www.wheel.pl > pjd@FreeBSD.org http://www.FreeBSD.org > FreeBSD committer Am I Evil? Yes, I Am! From owner-freebsd-fs@FreeBSD.ORG Mon Nov 5 23:59:23 2007 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id BFEE216A41B for ; Mon, 5 Nov 2007 23:59:23 +0000 (UTC) (envelope-from ndenev@gmail.com) Received: from rv-out-0910.google.com (rv-out-0910.google.com [209.85.198.189]) by mx1.freebsd.org (Postfix) with ESMTP id 8962A13C4A8 for ; Mon, 5 Nov 2007 23:59:23 +0000 (UTC) (envelope-from ndenev@gmail.com) Received: by rv-out-0910.google.com with SMTP id l15so1473910rvb for ; Mon, 05 Nov 2007 15:59:21 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=beta; h=domainkey-signature:received:received:message-id:date:from:sender:to:subject:cc:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references:x-google-sender-auth; bh=P7R3wxLm8r2Xfdg1pqJF2YGvRIwxhTx7k5+BHP1wfcc=; b=pB1nEiCmt1qbOiKeFpa4Lg7L5k736DyfgjKuOUhw26Cxqj3XENTsL39J/tT4ST96cb+85I4DJ5LOYYFRct44yWapbtWCADDWmJVvmYge5tGMLAODI3TMCvQH1zOyuYEd6t8uEFeksaRe2XaGrYa8D+GFsDiBcw/BAU0wz7vtU5o= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=beta; h=received:message-id:date:from:sender:to:subject:cc:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references:x-google-sender-auth; b=O6PIXY4QSlJg2W926n/KEYuijiwxoHCsb2xOYqP+mkV2LDiSr1MlXIugan6/uf7PoG0w73iKSbMEPo5gD8Dw2NjqRn1AQ5vhwEmz9hQ8Al9Gyn4WAeVt7XesWba9POeXjeu1WW+kEfQJMq2hLp+ghl83TIcV++IX+PqG26/4xEQ= Received: by 10.141.52.5 with SMTP id e5mr2697010rvk.1194305502073; Mon, 05 Nov 2007 15:31:42 -0800 (PST) Received: by 10.140.131.18 with HTTP; Mon, 5 Nov 2007 15:31:42 -0800 (PST) Message-ID: <2e77fc10711051531k41e7224dq6aaedb35cad8d9f2@mail.gmail.com> Date: Tue, 6 Nov 2007 01:31:42 +0200 From: "Niki Denev" Sender: ndenev@gmail.com To: freebsd-current@freebsd.org In-Reply-To: <20071105215035.GC26730@heff.fud.org.nz> MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline References: <20071005000046.GC92272@garage.freebsd.pl> <20071008121523.GM2327@garage.freebsd.pl> <20071105215035.GC26730@heff.fud.org.nz> X-Google-Sender-Auth: 448ea43501c3f746 Cc: freebsd-fs@freebsd.org Subject: Re: ZFS kmem_map too small. X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 05 Nov 2007 23:59:23 -0000 On Nov 5, 2007 11:50 PM, Andrew Thompson wrote: > On Mon, Oct 08, 2007 at 02:15:23PM +0200, Pawel Jakub Dawidek wrote: > > Here are some updates: > > > > I was able to reproduce the panic by rsyncing big files and trying > > bonnie++ test suggested in this thread. > > > > Can you guys retry with this patch: > > > > http://people.freebsd.org/~pjd/patches/vm_kern.c.2.patch > > > > It's a hack, yes, but allows to mitigate the problem quite well. I'm > > looking for a solution that can be used for 7.0 before we find a better > > fix. > > Do you still plan to commit this for 7.0 or is a better solution in the > works? I would assume something would need to be in before RC1. > > > BTW. To use ZFS you _must_ increase vm.kmem_size/vm.kmem_size_max. > > If you have the problem discussed here and you're using standard values, > > please retry with vm.kmem_size/vm.kmem_size_max set to at least 600MB in > > /boot/loader.conf. > > > > I'm not sure if it's not too late to ask re@ about increasing the > > default kmem size at least on amd64. ~300MB we have there is silly > > small. > > I have fairly easy reproductable (bonnie++) kmem_map too small panics on AMD64 even with kmem_size and kmem_size_max to 1G on 8Gig machine. I've tried setting them to 2G but the kernel panics on boot. I have compiled in debugger support and can provide backtraces and testing if needed (the machine is still not in production). From owner-freebsd-fs@FreeBSD.ORG Tue Nov 6 09:30:00 2007 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 40A4516A41B; Tue, 6 Nov 2007 09:30:00 +0000 (UTC) (envelope-from BORJAMAR@SARENET.ES) Received: from proxypop1.sarenet.es (proxypop1.sarenet.es [194.30.0.99]) by mx1.freebsd.org (Postfix) with ESMTP id EDFAF13C4BD; Tue, 6 Nov 2007 09:29:59 +0000 (UTC) (envelope-from BORJAMAR@SARENET.ES) Received: from localhost (matahari.sarenet.es [192.148.167.18]) by proxypop1.sarenet.es (Postfix) with ESMTP id 89A195CA5; Tue, 6 Nov 2007 10:04:55 +0100 (CET) Message-Id: <6214AB9C-9F9B-4B9D-8B05-0B3DF5F6C16D@SARENET.ES> From: Borja Marcos To: Niki Denev In-Reply-To: <2e77fc10711051531k41e7224dq6aaedb35cad8d9f2@mail.gmail.com> Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes Content-Transfer-Encoding: 7bit Mime-Version: 1.0 (Apple Message framework v912) Date: Tue, 6 Nov 2007 10:04:43 +0100 References: <20071005000046.GC92272@garage.freebsd.pl> <20071008121523.GM2327@garage.freebsd.pl> <20071105215035.GC26730@heff.fud.org.nz> <2e77fc10711051531k41e7224dq6aaedb35cad8d9f2@mail.gmail.com> X-Mailer: Apple Mail (2.912) Cc: freebsd-fs@freebsd.org, freebsd-current@freebsd.org Subject: Re: ZFS kmem_map too small. X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 06 Nov 2007 09:30:00 -0000 On Nov 6, 2007, at 12:31 AM, Niki Denev wrote: > I have fairly easy reproductable (bonnie++) kmem_map too small panics > on AMD64 even with kmem_size and kmem_size_max to 1G on 8Gig machine. > I've tried setting them to 2G but the kernel panics on boot. > I have compiled in debugger support and can provide backtraces and > testing if needed (the machine is still not in production). Same here. I can provoke an instant panic running (for example) a make buildworld and bonnie++. I have all the filesystems on ZFS, booting from a USB pendrive. I tried to set the kernel memory limits to 1.5 GB but it still crashes with the kmem_too_small panic. I also tried the patch supplied by Pawel. The machine has 6 hard disks configured as a raidz2 pool. Please let me know of any test anyone might need. Borja. ---------------- "The thing he realised about the windows was this: because they had been converted into openable windows after they had first been designed to be impregnable, they were, in fact, much less secure than if they had been designed as openable windows in the first place." Douglas Adams, "Mostly Harmless" From owner-freebsd-fs@FreeBSD.ORG Tue Nov 6 10:00:35 2007 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id A285216A46D; Tue, 6 Nov 2007 10:00:35 +0000 (UTC) (envelope-from pjd@garage.freebsd.pl) Received: from mail.garage.freebsd.pl (arm132.internetdsl.tpnet.pl [83.17.198.132]) by mx1.freebsd.org (Postfix) with ESMTP id 34A8813C480; Tue, 6 Nov 2007 10:00:34 +0000 (UTC) (envelope-from pjd@garage.freebsd.pl) Received: by mail.garage.freebsd.pl (Postfix, from userid 65534) id 9CAF545EA7; Tue, 6 Nov 2007 11:00:27 +0100 (CET) Received: from localhost (pjd.wheel.pl [10.0.1.1]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mail.garage.freebsd.pl (Postfix) with ESMTP id 141C9456AB; Tue, 6 Nov 2007 11:00:22 +0100 (CET) Date: Tue, 6 Nov 2007 11:00:15 +0100 From: Pawel Jakub Dawidek To: Borja Marcos Message-ID: <20071106100015.GB5268@garage.freebsd.pl> References: <20071005000046.GC92272@garage.freebsd.pl> <20071008121523.GM2327@garage.freebsd.pl> <20071105215035.GC26730@heff.fud.org.nz> <2e77fc10711051531k41e7224dq6aaedb35cad8d9f2@mail.gmail.com> <6214AB9C-9F9B-4B9D-8B05-0B3DF5F6C16D@SARENET.ES> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="DKU6Jbt7q3WqK7+M" Content-Disposition: inline In-Reply-To: <6214AB9C-9F9B-4B9D-8B05-0B3DF5F6C16D@SARENET.ES> User-Agent: Mutt/1.4.2.3i X-PGP-Key-URL: http://people.freebsd.org/~pjd/pjd.asc X-OS: FreeBSD 7.0-CURRENT i386 X-Spam-Checker-Version: SpamAssassin 3.0.4 (2005-06-05) on mail.garage.freebsd.pl X-Spam-Level: X-Spam-Status: No, score=-5.9 required=3.0 tests=ALL_TRUSTED,BAYES_00 autolearn=ham version=3.0.4 Cc: freebsd-fs@freebsd.org, Niki Denev , freebsd-current@freebsd.org Subject: Re: ZFS kmem_map too small. X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 06 Nov 2007 10:00:35 -0000 --DKU6Jbt7q3WqK7+M Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Tue, Nov 06, 2007 at 10:04:43AM +0100, Borja Marcos wrote: >=20 > On Nov 6, 2007, at 12:31 AM, Niki Denev wrote: > >I have fairly easy reproductable (bonnie++) kmem_map too small panics > >on AMD64 even with kmem_size and kmem_size_max to 1G on 8Gig machine. > >I've tried setting them to 2G but the kernel panics on boot. > >I have compiled in debugger support and can provide backtraces and > >testing if needed (the machine is still not in production). >=20 > Same here. I can provoke an instant panic running (for example) a make = =20 > buildworld and > bonnie++. I have all the filesystems on ZFS, booting from a USB =20 > pendrive. I tried to set > the kernel memory limits to 1.5 GB but it still crashes with the =20 > kmem_too_small panic. I also > tried the patch supplied by Pawel. >=20 > The machine has 6 hard disks configured as a raidz2 pool. >=20 > Please let me know of any test anyone might need. If you use vm_kern.c.2.patch, can you show loader.conf and exact command that can provke the panic? --=20 Pawel Jakub Dawidek http://www.wheel.pl pjd@FreeBSD.org http://www.FreeBSD.org FreeBSD committer Am I Evil? Yes, I Am! --DKU6Jbt7q3WqK7+M Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.4 (FreeBSD) iD8DBQFHMDsvForvXbEpPzQRApMqAJ0cen6EItnSGqa4TvbiKjrfxnD72gCcDeOb YxjrQgs7kLvzQs6FwWLueOw= =YFR9 -----END PGP SIGNATURE----- --DKU6Jbt7q3WqK7+M-- From owner-freebsd-fs@FreeBSD.ORG Tue Nov 6 10:21:04 2007 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id DAFC916A417; Tue, 6 Nov 2007 10:21:04 +0000 (UTC) (envelope-from BORJAMAR@SARENET.ES) Received: from proxypop2.sarenet.es (proxypop2.sarenet.es [194.30.0.95]) by mx1.freebsd.org (Postfix) with ESMTP id 8C2B513C4A8; Tue, 6 Nov 2007 10:21:04 +0000 (UTC) (envelope-from BORJAMAR@SARENET.ES) Received: from localhost (matahari.sarenet.es [192.148.167.18]) by proxypop2.sarenet.es (Postfix) with ESMTP id DCDA47326D; Tue, 6 Nov 2007 11:20:47 +0100 (CET) Message-Id: From: Borja Marcos To: Pawel Jakub Dawidek In-Reply-To: <20071106100015.GB5268@garage.freebsd.pl> Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes Content-Transfer-Encoding: 7bit Mime-Version: 1.0 (Apple Message framework v912) Date: Tue, 6 Nov 2007 11:20:34 +0100 References: <20071005000046.GC92272@garage.freebsd.pl> <20071008121523.GM2327@garage.freebsd.pl> <20071105215035.GC26730@heff.fud.org.nz> <2e77fc10711051531k41e7224dq6aaedb35cad8d9f2@mail.gmail.com> <6214AB9C-9F9B-4B9D-8B05-0B3DF5F6C16D@SARENET.ES> <20071106100015.GB5268@garage.freebsd.pl> X-Mailer: Apple Mail (2.912) Cc: freebsd-fs@freebsd.org, Niki Denev , freebsd-current@freebsd.org Subject: Re: ZFS kmem_map too small. X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 06 Nov 2007 10:21:05 -0000 On Nov 6, 2007, at 11:00 AM, Pawel Jakub Dawidek wrote: >> The machine has 6 hard disks configured as a raidz2 pool. >> >> Please let me know of any test anyone might need. > > If you use vm_kern.c.2.patch, can you show loader.conf and exact > command > that can provke the panic? loader.conf: zfs_load="YES" vfs.root.mountfrom="zfs:pool/root" vm.kmem_size_max="1610612736" vm.kmem_size="1610612736" Y bumped up the vnodes as well, as suggested at the ZFS tuning Wiki. (/etc/sysctl.conf) kern.maxvnodes: 400000 Anything else? (I just don't want to spam a big message with kernel config, etc) I don't have a fancy config as far as I know. I'm running the latest FreeBSD 7-BETA, amd64, and the machine has 4 GB of memory. It's a Dell Poweredge 2950, I'm using the mfi passthough devices as disks (I don't want to use the disk array functionality it provides), When I did the tests I did not have a swap partition configured, but I guess it shouldn't be related (4 GB is plenty of memory). Anyway I'm going to try again with a swap partition today. (_Not_ on ZFS) so that I can get a kernel dump as well. Thank you very much, Borja. > ---------------- "The thing he realised about the windows was this: because they had been converted into openable windows after they had first been designed to be impregnable, they were, in fact, much less secure than if they had been designed as openable windows in the first place." Douglas Adams, "Mostly Harmless" From owner-freebsd-fs@FreeBSD.ORG Tue Nov 6 10:39:35 2007 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 8692116A41A; Tue, 6 Nov 2007 10:39:35 +0000 (UTC) (envelope-from BORJAMAR@SARENET.ES) Received: from proxypop1.sarenet.es (proxypop1.sarenet.es [194.30.0.99]) by mx1.freebsd.org (Postfix) with ESMTP id 3DEDF13C4A5; Tue, 6 Nov 2007 10:39:35 +0000 (UTC) (envelope-from BORJAMAR@SARENET.ES) Received: from localhost (matahari.sarenet.es [192.148.167.18]) by proxypop1.sarenet.es (Postfix) with ESMTP id 3600C5CA0; Tue, 6 Nov 2007 11:39:21 +0100 (CET) Message-Id: <9A147CEC-D5E6-4117-8C69-16E40DB45B22@SARENET.ES> From: Borja Marcos To: Pawel Jakub Dawidek In-Reply-To: <20071106100015.GB5268@garage.freebsd.pl> Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes Content-Transfer-Encoding: 7bit Mime-Version: 1.0 (Apple Message framework v912) Date: Tue, 6 Nov 2007 11:38:55 +0100 References: <20071005000046.GC92272@garage.freebsd.pl> <20071008121523.GM2327@garage.freebsd.pl> <20071105215035.GC26730@heff.fud.org.nz> <2e77fc10711051531k41e7224dq6aaedb35cad8d9f2@mail.gmail.com> <6214AB9C-9F9B-4B9D-8B05-0B3DF5F6C16D@SARENET.ES> <20071106100015.GB5268@garage.freebsd.pl> X-Mailer: Apple Mail (2.912) Cc: freebsd-fs@freebsd.org, freebsd-current@freebsd.org Subject: Re: ZFS kmem_map too small. X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 06 Nov 2007 10:39:35 -0000 On Nov 6, 2007, at 11:00 AM, Pawel Jakub Dawidek wrote: > If you use vm_kern.c.2.patch, can you show loader.conf and exact > command > that can provke the panic? Sorry, forgot about the command. I just created a ZFS volume (scratch) and used: bonnie++ -s 12g -n 24 The machine dies consistently during the single file tests (sequential write, rewrite, etc). Borja. ---------------- "The thing he realised about the windows was this: because they had been converted into openable windows after they had first been designed to be impregnable, they were, in fact, much less secure than if they had been designed as openable windows in the first place." Douglas Adams, "Mostly Harmless" From owner-freebsd-fs@FreeBSD.ORG Tue Nov 6 12:05:59 2007 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id A96E316A41B for ; Tue, 6 Nov 2007 12:05:59 +0000 (UTC) (envelope-from freebsd-fs@m.gmane.org) Received: from ciao.gmane.org (main.gmane.org [80.91.229.2]) by mx1.freebsd.org (Postfix) with ESMTP id 57A7713C480 for ; Tue, 6 Nov 2007 12:05:59 +0000 (UTC) (envelope-from freebsd-fs@m.gmane.org) Received: from list by ciao.gmane.org with local (Exim 4.43) id 1IpMIz-0006jE-OG for freebsd-fs@freebsd.org; Tue, 06 Nov 2007 11:09:13 +0000 Received: from lara.cc.fer.hr ([161.53.72.113]) by main.gmane.org with esmtp (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Tue, 06 Nov 2007 11:09:13 +0000 Received: from ivoras by lara.cc.fer.hr with local (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Tue, 06 Nov 2007 11:09:13 +0000 X-Injected-Via-Gmane: http://gmane.org/ To: freebsd-fs@freebsd.org From: Ivan Voras Date: Tue, 06 Nov 2007 12:11:53 +0100 Lines: 10 Message-ID: References: <20071005000046.GC92272@garage.freebsd.pl> <20071008121523.GM2327@garage.freebsd.pl> <20071105215035.GC26730@heff.fud.org.nz> <2e77fc10711051531k41e7224dq6aaedb35cad8d9f2@mail.gmail.com> <6214AB9C-9F9B-4B9D-8B05-0B3DF5F6C16D@SARENET.ES> <20071106100015.GB5268@garage.freebsd.pl> Mime-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-Complaints-To: usenet@ger.gmane.org X-Gmane-NNTP-Posting-Host: lara.cc.fer.hr User-Agent: Thunderbird 2.0.0.6 (X11/20070801) In-Reply-To: Sender: news Cc: freebsd-current@freebsd.org Subject: Re: ZFS kmem_map too small. X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 06 Nov 2007 12:05:59 -0000 Borja Marcos wrote: > > Y bumped up the vnodes as well, as suggested at the ZFS tuning Wiki. > > (/etc/sysctl.conf) > kern.maxvnodes: 400000 This particular setting was suggested to increase performance, not reliability - I'll clarify that on the page. From owner-freebsd-fs@FreeBSD.ORG Tue Nov 6 12:55:38 2007 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id BC77116A41A; Tue, 6 Nov 2007 12:55:38 +0000 (UTC) (envelope-from pjd@garage.freebsd.pl) Received: from mail.garage.freebsd.pl (arm132.internetdsl.tpnet.pl [83.17.198.132]) by mx1.freebsd.org (Postfix) with ESMTP id 29DD913C48A; Tue, 6 Nov 2007 12:55:36 +0000 (UTC) (envelope-from pjd@garage.freebsd.pl) Received: by mail.garage.freebsd.pl (Postfix, from userid 65534) id 65B4B45F5C; Tue, 6 Nov 2007 13:55:19 +0100 (CET) Received: from localhost (pjd.wheel.pl [10.0.1.1]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mail.garage.freebsd.pl (Postfix) with ESMTP id 63FC945EE5; Tue, 6 Nov 2007 13:55:10 +0100 (CET) Date: Tue, 6 Nov 2007 13:55:03 +0100 From: Pawel Jakub Dawidek To: Borja Marcos Message-ID: <20071106125503.GN5268@garage.freebsd.pl> References: <20071005000046.GC92272@garage.freebsd.pl> <20071008121523.GM2327@garage.freebsd.pl> <20071105215035.GC26730@heff.fud.org.nz> <2e77fc10711051531k41e7224dq6aaedb35cad8d9f2@mail.gmail.com> <6214AB9C-9F9B-4B9D-8B05-0B3DF5F6C16D@SARENET.ES> <20071106100015.GB5268@garage.freebsd.pl> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="j/HO4hzKTNbM1mOX" Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.4.2.3i X-PGP-Key-URL: http://people.freebsd.org/~pjd/pjd.asc X-OS: FreeBSD 7.0-CURRENT i386 X-Spam-Checker-Version: SpamAssassin 3.0.4 (2005-06-05) on mail.garage.freebsd.pl X-Spam-Level: X-Spam-Status: No, score=-5.9 required=3.0 tests=ALL_TRUSTED,BAYES_00 autolearn=ham version=3.0.4 Cc: freebsd-fs@freebsd.org, Niki Denev , freebsd-current@freebsd.org Subject: Re: ZFS kmem_map too small. X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 06 Nov 2007 12:55:38 -0000 --j/HO4hzKTNbM1mOX Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Tue, Nov 06, 2007 at 11:20:34AM +0100, Borja Marcos wrote: >=20 > On Nov 6, 2007, at 11:00 AM, Pawel Jakub Dawidek wrote: >=20 > >>The machine has 6 hard disks configured as a raidz2 pool. > >> > >>Please let me know of any test anyone might need. > > > >If you use vm_kern.c.2.patch, can you show loader.conf and exact =20 > >command > >that can provke the panic? >=20 > loader.conf: > zfs_load=3D"YES" > vfs.root.mountfrom=3D"zfs:pool/root" > vm.kmem_size_max=3D"1610612736" > vm.kmem_size=3D"1610612736" >=20 > Y bumped up the vnodes as well, as suggested at the ZFS tuning Wiki. >=20 > (/etc/sysctl.conf) > kern.maxvnodes: 400000 The Wiki should be changed. Allow ZFS to autotune it, don't tune it by hand. > Anything else? (I just don't want to spam a big message with kernel =20 > config, etc) I asked about exact command to trigger the panic. > When I did the tests I did not have a swap partition configured, but I = =20 > guess it shouldn't be > related (4 GB is plenty of memory). Anyway I'm going to try again with = =20 > a swap partition today. > (_Not_ on ZFS) so that I can get a kernel dump as well. Yes, do not use swap on ZFS. --=20 Pawel Jakub Dawidek http://www.wheel.pl pjd@FreeBSD.org http://www.FreeBSD.org FreeBSD committer Am I Evil? Yes, I Am! --j/HO4hzKTNbM1mOX Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.4 (FreeBSD) iD8DBQFHMGQnForvXbEpPzQRAsZCAJ9ePHQT/PqSjkfe3wB1lCXNXUkYowCghloH y4FVicWAH0M7aqSnuoNy3Is= =CpAD -----END PGP SIGNATURE----- --j/HO4hzKTNbM1mOX-- From owner-freebsd-fs@FreeBSD.ORG Tue Nov 6 12:56:21 2007 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 0A70516A419; Tue, 6 Nov 2007 12:56:21 +0000 (UTC) (envelope-from pjd@garage.freebsd.pl) Received: from mail.garage.freebsd.pl (arm132.internetdsl.tpnet.pl [83.17.198.132]) by mx1.freebsd.org (Postfix) with ESMTP id 50F3E13C4A6; Tue, 6 Nov 2007 12:56:20 +0000 (UTC) (envelope-from pjd@garage.freebsd.pl) Received: by mail.garage.freebsd.pl (Postfix, from userid 65534) id A382145F59; Tue, 6 Nov 2007 13:56:05 +0100 (CET) Received: from localhost (pjd.wheel.pl [10.0.1.1]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mail.garage.freebsd.pl (Postfix) with ESMTP id 9423245EEB; Tue, 6 Nov 2007 13:55:58 +0100 (CET) Date: Tue, 6 Nov 2007 13:55:52 +0100 From: Pawel Jakub Dawidek To: Borja Marcos Message-ID: <20071106125552.GO5268@garage.freebsd.pl> References: <20071005000046.GC92272@garage.freebsd.pl> <20071008121523.GM2327@garage.freebsd.pl> <20071105215035.GC26730@heff.fud.org.nz> <2e77fc10711051531k41e7224dq6aaedb35cad8d9f2@mail.gmail.com> <6214AB9C-9F9B-4B9D-8B05-0B3DF5F6C16D@SARENET.ES> <20071106100015.GB5268@garage.freebsd.pl> <9A147CEC-D5E6-4117-8C69-16E40DB45B22@SARENET.ES> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="Qgd2S+2VS1hsWwXW" Content-Disposition: inline In-Reply-To: <9A147CEC-D5E6-4117-8C69-16E40DB45B22@SARENET.ES> User-Agent: Mutt/1.4.2.3i X-PGP-Key-URL: http://people.freebsd.org/~pjd/pjd.asc X-OS: FreeBSD 7.0-CURRENT i386 X-Spam-Checker-Version: SpamAssassin 3.0.4 (2005-06-05) on mail.garage.freebsd.pl X-Spam-Level: X-Spam-Status: No, score=-5.9 required=3.0 tests=ALL_TRUSTED,BAYES_00 autolearn=ham version=3.0.4 Cc: freebsd-fs@freebsd.org, freebsd-current@freebsd.org Subject: Re: ZFS kmem_map too small. X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 06 Nov 2007 12:56:21 -0000 --Qgd2S+2VS1hsWwXW Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Tue, Nov 06, 2007 at 11:38:55AM +0100, Borja Marcos wrote: >=20 > On Nov 6, 2007, at 11:00 AM, Pawel Jakub Dawidek wrote: >=20 > >If you use vm_kern.c.2.patch, can you show loader.conf and exact =20 > >command > >that can provke the panic? >=20 > Sorry, forgot about the command. >=20 > I just created a ZFS volume (scratch) and used: ZFS volume? You mean ZVOL? Can you paste 'zfs list; zpool status'? > bonnie++ -s 12g -n 24 >=20 > The machine dies consistently during the single file tests (sequential = =20 > write, rewrite, etc). --=20 Pawel Jakub Dawidek http://www.wheel.pl pjd@FreeBSD.org http://www.FreeBSD.org FreeBSD committer Am I Evil? Yes, I Am! --Qgd2S+2VS1hsWwXW Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.4 (FreeBSD) iD8DBQFHMGRYForvXbEpPzQRAopjAJ9bxAvXsWEZvd7QPuqwGRcfkycm4QCgl757 U3IVYTSYbdm5KmJzZQ8jsow= =xEIm -----END PGP SIGNATURE----- --Qgd2S+2VS1hsWwXW-- From owner-freebsd-fs@FreeBSD.ORG Tue Nov 6 13:18:22 2007 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 180BA16A417; Tue, 6 Nov 2007 13:18:22 +0000 (UTC) (envelope-from BORJAMAR@SARENET.ES) Received: from proxypop2.sarenet.es (proxypop2.sarenet.es [194.30.0.95]) by mx1.freebsd.org (Postfix) with ESMTP id C122013C48E; Tue, 6 Nov 2007 13:18:21 +0000 (UTC) (envelope-from BORJAMAR@SARENET.ES) Received: from localhost (unknown [212.81.200.214]) by proxypop2.sarenet.es (Postfix) with ESMTP id EBBAE731E2; Tue, 6 Nov 2007 14:18:12 +0100 (CET) Message-Id: From: Borja Marcos To: Pawel Jakub Dawidek In-Reply-To: <20071106100015.GB5268@garage.freebsd.pl> Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes Content-Transfer-Encoding: 7bit Mime-Version: 1.0 (Apple Message framework v912) Date: Tue, 6 Nov 2007 14:18:04 +0100 References: <20071005000046.GC92272@garage.freebsd.pl> <20071008121523.GM2327@garage.freebsd.pl> <20071105215035.GC26730@heff.fud.org.nz> <2e77fc10711051531k41e7224dq6aaedb35cad8d9f2@mail.gmail.com> <6214AB9C-9F9B-4B9D-8B05-0B3DF5F6C16D@SARENET.ES> <20071106100015.GB5268@garage.freebsd.pl> X-Mailer: Apple Mail (2.912) Cc: freebsd-fs@freebsd.org, freebsd-current@freebsd.org Subject: Re: ZFS kmem_map too small. X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 06 Nov 2007 13:18:22 -0000 On Nov 6, 2007, at 11:00 AM, Pawel Jakub Dawidek wrote: > If you use vm_kern.c.2.patch, can you show loader.conf and exact > command > that can provke the panic? I can elaborate a bit. I repeated the test while I kept some connections open, watching zpool iostat, the Bonnie++ command output, the make buildworld, and pinging the machine continuously. Wired memory reached 1696 MB according to top, and the programs seemed to be stopped. I didn't see more activity from top, zpool iostat, or make buildworld. But the kernel was still answering to pings, although with a very long delay, an average of 300 - 500 ms. I'm connected to the same local network, and the typical delay I get with the machine is around 0.3 ms. It was until things started to go wrong. The machine kept answering pings (very late, but with no packet loss), until I pressed ctrl-C at the terminal session where I was running Bonnie++. That seemed to trigger the panic, kmem_map to small. And I've found something. I was using SCHED_ULE instead of SCHED_4BSD. I've switched back to SCHED_4BSD (I apologise I didn't mention it, I thought it shouldn't make a difference) and the kmem usage climbs much slower, sustaning a 30 MB/s write throughput during the Bonnie's "writing intelligently" test. Using ULE, wired memory climbs and tops 1.6 GB in less than a minute, while using 4BSD it's climbing much slower. I've seen it reach 1.2 GB, go back to 800 MB... and suddenly go down to 256 MB, while it keeps sustaining a write throughput of about 25 - 30 MB/s (according to zpool iostat at 10 second intervals). I've compiled the kernel with debugger support and this is what the backtrace reads: panic kmem_malloc(131072): kmem_map too small: 1608417280 total allocated cpuid=1 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a panic() at panic+0x17a kmem_malloc at kmem_malloc+0x65e uma_large_malloc() at uma_large_malloc+0x4a malloc() at malloc+0x7d vdev_queue_io_to_issue() at vdev_queue_io_to_issue+0x147 vdev_queue_io_done() at vdev_queue_io_done+0x9c vdev_geom_io_done() at vdev_geom_io_done+0x11 taskq_thread() at taskq_thread+0x17b fork_exit() at fork_exit+0x11f fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xfffffffffee34d30, rbp = 0 --- KDB: enter: panic Hope it makes sense. Anything else I can provide? Borja ---------------- "The thing he realised about the windows was this: because they had been converted into openable windows after they had first been designed to be impregnable, they were, in fact, much less secure than if they had been designed as openable windows in the first place." Douglas Adams, "Mostly Harmless" From owner-freebsd-fs@FreeBSD.ORG Tue Nov 6 13:25:00 2007 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 0887916A46E; Tue, 6 Nov 2007 13:25:00 +0000 (UTC) (envelope-from BORJAMAR@SARENET.ES) Received: from proxypop1.sarenet.es (proxypop1.sarenet.es [194.30.0.99]) by mx1.freebsd.org (Postfix) with ESMTP id B22CC13C4AA; Tue, 6 Nov 2007 13:24:59 +0000 (UTC) (envelope-from BORJAMAR@SARENET.ES) Received: from localhost (unknown [212.81.200.214]) by proxypop1.sarenet.es (Postfix) with ESMTP id 2740D5CAC; Tue, 6 Nov 2007 14:24:27 +0100 (CET) Message-Id: <5C6AEFD0-FAC0-4E02-A969-D3B2E01DB099@SARENET.ES> From: Borja Marcos To: Pawel Jakub Dawidek In-Reply-To: <20071106125552.GO5268@garage.freebsd.pl> Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes Content-Transfer-Encoding: 7bit Mime-Version: 1.0 (Apple Message framework v912) Date: Tue, 6 Nov 2007 14:24:25 +0100 References: <20071005000046.GC92272@garage.freebsd.pl> <20071008121523.GM2327@garage.freebsd.pl> <20071105215035.GC26730@heff.fud.org.nz> <2e77fc10711051531k41e7224dq6aaedb35cad8d9f2@mail.gmail.com> <6214AB9C-9F9B-4B9D-8B05-0B3DF5F6C16D@SARENET.ES> <20071106100015.GB5268@garage.freebsd.pl> <9A147CEC-D5E6-4117-8C69-16E40DB45B22@SARENET.ES> <20071106125552.GO5268@garage.freebsd.pl> X-Mailer: Apple Mail (2.912) Cc: freebsd-fs@freebsd.org, freebsd-current@freebsd.org Subject: Re: ZFS kmem_map too small. X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 06 Nov 2007 13:25:00 -0000 On Nov 6, 2007, at 1:55 PM, Pawel Jakub Dawidek wrote: > ZFS volume? You mean ZVOL? Can you paste 'zfs list; zpool status'? Sorry, not a ZVOL, that's been an unfortunate usage of "volume". I created a single pool with all the disks, and I've created ZFS filesystems for /, /var, / usr... Of course I can :) prueba# zpool status pool: pool state: ONLINE scrub: none requested config: NAME STATE READ WRITE CKSUM pool ONLINE 0 0 0 raidz2 ONLINE 0 0 0 da1s2c ONLINE 0 0 0 da2s2c ONLINE 0 0 0 da3s2c ONLINE 0 0 0 da4s2c ONLINE 0 0 0 da5s2c ONLINE 0 0 0 da6s2c ONLINE 0 0 0 errors: No known data errors prueba# zfs list NAME USED AVAIL REFER MOUNTPOINT pool 30.8G 231G 46.0K /pool pool/expert 36.0K 231G 36.0K /expert pool/home 461K 231G 461K /home pool/root 385M 231G 385M legacy pool/scratch 16.8G 231G 16.8G /pool/scratch pool/swap 32.0K 243G 32.0K - pool/usr 1.63G 231G 1.63G /usr pool/var 52.2M 231G 52.2M /var prueba# If you want me to do some tests, I'm available. Best regards, Borja. ---------------- "The thing he realised about the windows was this: because they had been converted into openable windows after they had first been designed to be impregnable, they were, in fact, much less secure than if they had been designed as openable windows in the first place." Douglas Adams, "Mostly Harmless" From owner-freebsd-fs@FreeBSD.ORG Wed Nov 7 08:09:47 2007 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 1B46116A417 for ; Wed, 7 Nov 2007 08:09:47 +0000 (UTC) (envelope-from martijn@plak.net) Received: from smtp-vbr15.xs4all.nl (smtp-vbr15.xs4all.nl [194.109.24.35]) by mx1.freebsd.org (Postfix) with ESMTP id AD1B713C4A5 for ; Wed, 7 Nov 2007 08:09:45 +0000 (UTC) (envelope-from martijn@plak.net) Received: from [172.16.253.25] (83-108.bbned.dsl.internl.net [85.223.108.83]) (authenticated bits=0) by smtp-vbr15.xs4all.nl (8.13.8/8.13.8) with ESMTP id lA77kb10027993 (version=TLSv1/SSLv3 cipher=AES128-SHA bits=128 verify=NO) for ; Wed, 7 Nov 2007 08:46:38 +0100 (CET) (envelope-from martijn@plak.net) Mime-Version: 1.0 (Apple Message framework v752.3) In-Reply-To: <20071106125552.GO5268@garage.freebsd.pl> References: <20071005000046.GC92272@garage.freebsd.pl> <20071008121523.GM2327@garage.freebsd.pl> <20071105215035.GC26730@heff.fud.org.nz> <2e77fc10711051531k41e7224dq6aaedb35cad8d9f2@mail.gmail.com> <6214AB9C-9F9B-4B9D-8B05-0B3DF5F6C16D@SARENET.ES> <20071106100015.GB5268@garage.freebsd.pl> <9A147CEC-D5E6-4117-8C69-16E40DB45B22@SARENET.ES> <20071106125552.GO5268@garage.freebsd.pl> Content-Type: text/plain; charset=US-ASCII; delsp=yes; format=flowed Message-Id: <5ED261E6-A85A-4D7D-A01A-4688802FECB9@plak.net> Content-Transfer-Encoding: 7bit From: Martijn Plak Date: Wed, 7 Nov 2007 08:46:35 +0100 To: freebsd-fs@freebsd.org X-Mailer: Apple Mail (2.752.3) X-Virus-Scanned: by XS4ALL Virus Scanner Subject: Re: ZFS kmem_map too small. X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 07 Nov 2007 08:09:47 -0000 My 7.0-BETA2 (+ updates) system is crashing regularly on a kmem_map too small panic too. Yesterday I applied the vm_kern.c.2.patch patch and turned off all of my sysctl tweaks. At first, the system was stable for a day. During that day, I ran make -j4 buildworld several times without trouble. Then started a network transfer. This was a usenet download, on 4 TCP channels, for a total of 10Mbps, from a server at a 'ping distance' of about 20ms, using hellanzb.py. My guess was that this sort of transfer puts some memory pressure on the kernel space. Running that in parallel with the buildworld crashed the system again. I hope this helps the investigation. Let me know if I can run other tests. - Martijn === sudo cat /var/crash/info.2 Dump header from device /dev/da0 Architecture: i386 Architecture Version: 2 Dump Length: 370655232B (353 MB) Blocksize: 512 Dumptime: Tue Nov 6 22:00:28 2007 Hostname: dupont.plak.net Magic: FreeBSD Kernel Dump Version String: FreeBSD 7.0-BETA2 #6: Mon Nov 5 14:25:33 UTC 2007 root@dupont.plak.net:/usr/obj/usr/src/sys/DUPONT Panic String: kmem_malloc(90112): kmem_map too small: 295931904 total allocated Dump Parity: 2925288312 Bounds: 2 Dump Status: good Some hardware info: Intel P4 3GHz hyperthreading processor, 1GB RAM, intel i915 chipset, 4 SATA disks on i915 (ICH6), 1 compact flash on IDE to boot the system. Here is some information from the system, after reboot. === uname -a FreeBSD dupont.plak.net 7.0-BETA2 FreeBSD 7.0-BETA2 #6: Mon Nov 5 14:25:33 UTC 2007 root@dupont.plak.net:/usr/obj/usr/src/sys/ DUPONT i386 === diff /usr/src/sys/i386/conf/GENERIC /usr/src/sys/i386/conf/DUPONT 21,22d20 < cpu I486_CPU < cpu I586_CPU 24c22 < ident GENERIC --- > ident DUPONT === zfs list NAME USED AVAIL REFER MOUNTPOINT raid 552G 325G 1.50K none raid/data 547G 325G 471G /data raid/data/store1 76.5G 325G 76.5G /data/store1 raid/data/store2 84.8M 325G 84.8M /data/store2 raid/sys 4.65G 325G 26.9K none raid/sys/home 131M 325G 131M legacy raid/sys/root 135M 325G 135M legacy raid/sys/tmp 56.9K 325G 56.9K legacy raid/sys/usr 2.92G 325G 2.92G legacy raid/sys/var 1.47G 325G 1.47G legacy raid/sys2 300M 325G 26.9K none raid/sys2/root 178M 325G 178M legacy raid/sys2/tmp 35.2K 325G 35.2K legacy raid/sys2/usr 121M 325G 121M legacy raid/sys2/var 291K 325G 291K legacy === zpool status pool: raid state: ONLINE scrub: none requested config: NAME STATE READ WRITE CKSUM raid ONLINE 0 0 0 raidz1 ONLINE 0 0 0 ad4 ONLINE 0 0 0 ad5 ONLINE 0 0 0 ad6 ONLINE 0 0 0 ad7 ONLINE 0 0 0 errors: No known data errors === cat /boot/loader.conf # Load ZFS and load root system from the RAID-Z array. zfs_load="YES" vfs.root.mountfrom="zfs:raid/sys/root" # Tune ZFS and VM parameters. #vfs.zfs.arc_max="64M" #kern.maxvnodes="50000" #vm.kmem_size_max="512M" #vm.kmem_size="512M" === sysctl vm | grep kmem vm.kmem_size_scale: 3 vm.kmem_size_max: 335544320 vm.kmem_size_min: 0 vm.kmem_size: 335544320 === sysctl vfs.zfs vfs.zfs.arc_min: 16777216 vfs.zfs.arc_max: 251658240 vfs.zfs.mdcomp_disable: 0 vfs.zfs.prefetch_disable: 0 vfs.zfs.zio.taskq_threads: 0 vfs.zfs.recover: 0 vfs.zfs.vdev.cache.size: 10485760 vfs.zfs.vdev.cache.max: 16384 vfs.zfs.cache_flush_disable: 0 vfs.zfs.zil_disable: 0 vfs.zfs.debug: 0 === sysctl kstat.zfs kstat.zfs.misc.arcstats.hits: 602394 kstat.zfs.misc.arcstats.misses: 29556 kstat.zfs.misc.arcstats.demand_data_hits: 475248 kstat.zfs.misc.arcstats.demand_data_misses: 7156 kstat.zfs.misc.arcstats.demand_metadata_hits: 105827 kstat.zfs.misc.arcstats.demand_metadata_misses: 10413 kstat.zfs.misc.arcstats.prefetch_data_hits: 140 kstat.zfs.misc.arcstats.prefetch_data_misses: 4620 kstat.zfs.misc.arcstats.prefetch_metadata_hits: 21179 kstat.zfs.misc.arcstats.prefetch_metadata_misses: 7367 kstat.zfs.misc.arcstats.mru_hits: 242176 kstat.zfs.misc.arcstats.mru_ghost_hits: 3548 kstat.zfs.misc.arcstats.mfu_hits: 338953 kstat.zfs.misc.arcstats.mfu_ghost_hits: 2849 kstat.zfs.misc.arcstats.deleted: 40513 kstat.zfs.misc.arcstats.recycle_miss: 61805 kstat.zfs.misc.arcstats.mutex_miss: 66 kstat.zfs.misc.arcstats.evict_skip: 49115 kstat.zfs.misc.arcstats.hash_elements: 3528 kstat.zfs.misc.arcstats.hash_elements_max: 7423 kstat.zfs.misc.arcstats.hash_collisions: 15472 kstat.zfs.misc.arcstats.hash_chains: 290 kstat.zfs.misc.arcstats.hash_chain_max: 4 kstat.zfs.misc.arcstats.p: 124527736 kstat.zfs.misc.arcstats.c: 126306424 kstat.zfs.misc.arcstats.c_min: 16777216 kstat.zfs.misc.arcstats.c_max: 251658240 kstat.zfs.misc.arcstats.size: 126307328 === sysctl -a | grep vnode kern.maxvnodes: 52280 kern.minvnodes: 17426 vm.stats.vm.v_vnodepgsout: 60 vm.stats.vm.v_vnodepgsin: 3719 vm.stats.vm.v_vnodeout: 43 vm.stats.vm.v_vnodein: 3719 vfs.freevnodes: 17385 vfs.wantfreevnodes: 17426 vfs.numvnodes: 18986 debug.sizeof.vnode: 272 From owner-freebsd-fs@FreeBSD.ORG Wed Nov 7 11:57:36 2007 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id CAE5416A41B for ; Wed, 7 Nov 2007 11:57:36 +0000 (UTC) (envelope-from pjd@garage.freebsd.pl) Received: from mail.garage.freebsd.pl (arm132.internetdsl.tpnet.pl [83.17.198.132]) by mx1.freebsd.org (Postfix) with ESMTP id D571713C4B7 for ; Wed, 7 Nov 2007 11:57:34 +0000 (UTC) (envelope-from pjd@garage.freebsd.pl) Received: by mail.garage.freebsd.pl (Postfix, from userid 65534) id E237D45F59; Wed, 7 Nov 2007 12:57:22 +0100 (CET) Received: from localhost (pjd.wheel.pl [10.0.1.1]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mail.garage.freebsd.pl (Postfix) with ESMTP id 05FBF45684; Wed, 7 Nov 2007 12:57:18 +0100 (CET) Date: Wed, 7 Nov 2007 12:57:11 +0100 From: Pawel Jakub Dawidek To: Peter Schuller Message-ID: <20071107115711.GM15618@garage.freebsd.pl> References: <20071022153521.GB27594@hyperion.scode.org> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="jozmn01XJZjDjM3N" Content-Disposition: inline In-Reply-To: <20071022153521.GB27594@hyperion.scode.org> User-Agent: Mutt/1.4.2.3i X-PGP-Key-URL: http://people.freebsd.org/~pjd/pjd.asc X-OS: FreeBSD 7.0-CURRENT i386 X-Spam-Checker-Version: SpamAssassin 3.0.4 (2005-06-05) on mail.garage.freebsd.pl X-Spam-Level: X-Spam-Status: No, score=-5.9 required=3.0 tests=ALL_TRUSTED,BAYES_00 autolearn=ham version=3.0.4 Cc: freebsd-fs@freebsd.org Subject: Re: ZFS: reproducable inability to accesss a pool (process hangs; other pools fine) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 07 Nov 2007 11:57:36 -0000 --jozmn01XJZjDjM3N Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Mon, Oct 22, 2007 at 05:35:21PM +0200, Peter Schuller wrote: > Hello, >=20 > On the same system I recently posted about on -stable, with RELENG_7 > from a few days ago, I am now running a SiL 3114 on a raidz2 in > degraded mode with one disk missing (it is degraded by design because > I wanted to create a 5 disk array but only had 4). >=20 > For the purpose of discovering any stability issues with the 3114 > controller I did some stress tests that have yet to reveil controller > problems, but has triggered what appears to be a ZFS problem. >=20 > Test case: >=20 > /promraid - root of the pool in question > /promraid/ports - copy of /usr/ports tree from my machine > /promraid/1 - empty directory > /promraid/2 - empty directory >=20 > I now run concurrently in two shells: >=20 > while [ 1 ] ; do rsync -a /promraid/ports /promraid/1/pp ; rm -rf /promra= id/1/pp ; done >=20 > and: >=20 > while [ 1 ] ; do rsync -a /promraid/ports /promraid/2/pp ; rm -rf /promra= id/2/pp ; done >=20 > This runs fine for some hours, but eventually I end up with hung > rsyncs in "zfs" state according to op. Attempting to e.g. ls /promraid > hangs as well. Yet ZFS continues working (another pool is entirely > fine), and there are no errors in dmesg. >=20 > iostat -x does NOT indicate that it is perpetually waiting on I/O from > a disk or something likethat (0% utilization). The processes are > unkillable, even by SIGKILL. >=20 > I should have this environment for a few more days, so can hopefully > reproduce this again. It has happened at least twice already (the > first time I was in X and X hung; I thought I had a panic so re-ran > the tests in the console; these two times I didn't get a panic but I > am unsure whether the failure case is different). >=20 > Does anyone have suggestions for what to do to produce the best > information possible? Given that there are no errors, no panic, etc. >=20 > One obvious bit is to ktrace them I realize, if that gives me anything > (the size of the trace if I were to trace it from the beginning would, > I suspect, be prohibitive). Will do that next time. I've found a deadlock recently. Can you enter DDB, find spa_zio_intr_X threads, run 'tr ' on theirs PIDs and send me the output? --=20 Pawel Jakub Dawidek http://www.wheel.pl pjd@FreeBSD.org http://www.FreeBSD.org FreeBSD committer Am I Evil? Yes, I Am! --jozmn01XJZjDjM3N Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.4 (FreeBSD) iD8DBQFHMagXForvXbEpPzQRAl1kAJ9qLT7H8mmJdsrgwKDV3HmCZ3CEbwCgprXQ tAha09rlzRo4K9UtUjyxeYI= =+8sB -----END PGP SIGNATURE----- --jozmn01XJZjDjM3N-- From owner-freebsd-fs@FreeBSD.ORG Wed Nov 7 15:08:52 2007 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 69FE216A41A for ; Wed, 7 Nov 2007 15:08:52 +0000 (UTC) (envelope-from amdmi3@amdmi3.ru) Received: from cp65.agava.net (cp65.agava.net [89.108.66.215]) by mx1.freebsd.org (Postfix) with ESMTP id 2493513C4BD for ; Wed, 7 Nov 2007 15:08:51 +0000 (UTC) (envelope-from amdmi3@amdmi3.ru) Received: from [213.148.20.85] (helo=nexii.panopticon) by cp65.agava.net with esmtpsa (TLSv1:AES256-SHA:256) (Exim 4.44 (FreeBSD)) id 1IpmWI-0000oc-Al for freebsd-fs@freebsd.org; Wed, 07 Nov 2007 18:08:42 +0300 Received: from hades.panopticon (hades.panopticon [192.168.0.2]) by nexii.panopticon (Postfix) with ESMTP id 0207E1703E for ; Wed, 7 Nov 2007 18:10:08 +0300 (MSK) Received: by hades.panopticon (Postfix, from userid 1000) id 2DAE1419E; Wed, 7 Nov 2007 18:10:17 +0300 (MSK) Date: Wed, 7 Nov 2007 18:10:16 +0300 From: Dmitry Marakasov To: freebsd-fs@freebsd.org Message-ID: <20071107151015.GA99845@hades.panopticon> Mail-Followup-To: freebsd-fs@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=koi8-r Content-Disposition: inline User-Agent: Mutt/1.5.16 (2007-06-09) X-AntiAbuse: This header was added to track abuse, please include it with any abuse report X-AntiAbuse: Primary Hostname - cp65.agava.net X-AntiAbuse: Original Domain - freebsd.org X-AntiAbuse: Originator/Caller UID/GID - [0 0] / [26 6] X-AntiAbuse: Sender Address Domain - amdmi3.ru X-Source: X-Source-Args: X-Source-Dir: Subject: NFSv4: where to start? X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 07 Nov 2007 15:08:52 -0000 Hi! I would like to deploy NFSv4 with Kerberos auth with couple of 7.0 systems. I've seen it work very well on Linux, but FreeBSD seem to be a bit behind here, so I need a place to start. AFAIK, there is only NFSv4 client in FreeBSD 7.0, and the server is available somewhere as set of patches. Are there any guides on NFSv4@FreeBSD? -- Dmitry A. Marakasov | jabber: amdmi3@jabber.ru amdmi3@amdmi3.ru | http://www.amdmi3.ru From owner-freebsd-fs@FreeBSD.ORG Wed Nov 7 15:48:43 2007 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 789AD16A418 for ; Wed, 7 Nov 2007 15:48:43 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from mailhub.cs.uoguelph.ca (mailhub.cs.uoguelph.ca [131.104.94.205]) by mx1.freebsd.org (Postfix) with ESMTP id 254E813C4B9 for ; Wed, 7 Nov 2007 15:48:42 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from muncher.cs.uoguelph.ca (muncher.cs.uoguelph.ca [131.104.91.102]) by mailhub.cs.uoguelph.ca (8.13.1/8.13.1) with ESMTP id lA7FJVkE003555; Wed, 7 Nov 2007 10:19:31 -0500 Received: from localhost (rmacklem@localhost) by muncher.cs.uoguelph.ca (8.11.7p3+Sun/8.11.6) with ESMTP id lA7FJo815376; Wed, 7 Nov 2007 10:19:50 -0500 (EST) X-Authentication-Warning: muncher.cs.uoguelph.ca: rmacklem owned process doing -bs Date: Wed, 7 Nov 2007 10:19:50 -0500 (EST) From: Rick Macklem X-X-Sender: rmacklem@muncher.cs.uoguelph.ca To: Dmitry Marakasov In-Reply-To: <20071107151015.GA99845@hades.panopticon> Message-ID: References: <20071107151015.GA99845@hades.panopticon> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed X-Scanned-By: MIMEDefang 2.63 on 131.104.94.205 Cc: freebsd-fs@freebsd.org Subject: Re: NFSv4: where to start? X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 07 Nov 2007 15:48:43 -0000 On Wed, 7 Nov 2007, Dmitry Marakasov wrote: > Hi! > > I would like to deploy NFSv4 with Kerberos auth with couple of 7.0 > systems. I've seen it work very well on Linux, but FreeBSD seem to > be a bit behind here, so I need a place to start. AFAIK, there is > only NFSv4 client in FreeBSD 7.0, and the server is available > somewhere as set of patches. Are there any guides on NFSv4@FreeBSD? > My server patches are available anonymous ftp from: ftp.cis.uoguelph.ca/pub/nfsv4/FreeBSD-CURRENT - This patch was tested on a May 2007 snapshot, so I'm not sure if it will go in easily on the current kernel. If you try it, please let me know how it goes. I don't believe the client is usable at this point, but you can give it a try. rick From owner-freebsd-fs@FreeBSD.ORG Wed Nov 7 17:02:13 2007 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 9F08316A41A for ; Wed, 7 Nov 2007 17:02:13 +0000 (UTC) (envelope-from pjd@garage.freebsd.pl) Received: from mail.garage.freebsd.pl (arm132.internetdsl.tpnet.pl [83.17.198.132]) by mx1.freebsd.org (Postfix) with ESMTP id 2E47713C4B2 for ; Wed, 7 Nov 2007 17:02:12 +0000 (UTC) (envelope-from pjd@garage.freebsd.pl) Received: by mail.garage.freebsd.pl (Postfix, from userid 65534) id 0395D45E90; Wed, 7 Nov 2007 18:02:01 +0100 (CET) Received: from localhost (pjd.wheel.pl [10.0.1.1]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mail.garage.freebsd.pl (Postfix) with ESMTP id D3FD245CD9; Wed, 7 Nov 2007 18:01:55 +0100 (CET) Date: Wed, 7 Nov 2007 18:01:48 +0100 From: Pawel Jakub Dawidek To: Martijn Plak Message-ID: <20071107170147.GN15618@garage.freebsd.pl> References: <20071005000046.GC92272@garage.freebsd.pl> <20071008121523.GM2327@garage.freebsd.pl> <20071105215035.GC26730@heff.fud.org.nz> <2e77fc10711051531k41e7224dq6aaedb35cad8d9f2@mail.gmail.com> <6214AB9C-9F9B-4B9D-8B05-0B3DF5F6C16D@SARENET.ES> <20071106100015.GB5268@garage.freebsd.pl> <9A147CEC-D5E6-4117-8C69-16E40DB45B22@SARENET.ES> <20071106125552.GO5268@garage.freebsd.pl> <5ED261E6-A85A-4D7D-A01A-4688802FECB9@plak.net> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="SwC/mAzP11p89u4p" Content-Disposition: inline In-Reply-To: <5ED261E6-A85A-4D7D-A01A-4688802FECB9@plak.net> User-Agent: Mutt/1.4.2.3i X-PGP-Key-URL: http://people.freebsd.org/~pjd/pjd.asc X-OS: FreeBSD 7.0-CURRENT i386 X-Spam-Checker-Version: SpamAssassin 3.0.4 (2005-06-05) on mail.garage.freebsd.pl X-Spam-Level: X-Spam-Status: No, score=-5.9 required=3.0 tests=ALL_TRUSTED,BAYES_00 autolearn=ham version=3.0.4 Cc: freebsd-fs@freebsd.org Subject: Re: ZFS kmem_map too small. X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 07 Nov 2007 17:02:13 -0000 --SwC/mAzP11p89u4p Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Wed, Nov 07, 2007 at 08:46:35AM +0100, Martijn Plak wrote: > My 7.0-BETA2 (+ updates) system is crashing regularly on a kmem_map =20 > too small panic too. > Yesterday I applied the vm_kern.c.2.patch patch and turned off all of =20 > my sysctl tweaks. >=20 > At first, the system was stable for a day. During that day, I ran =20 > make -j4 buildworld several times without trouble. >=20 > Then started a network transfer. This was a usenet download, on 4 TCP =20 > channels, for a total of 10Mbps, from a server at a 'ping distance' =20 > of about 20ms, using hellanzb.py. My guess was that this sort of =20 > transfer puts some memory pressure on the kernel space. Running that =20 > in parallel with the buildworld crashed the system again. >=20 > I hope this helps the investigation. Let me know if I can run other =20 > tests. [...] > =3D=3D=3D cat /boot/loader.conf > # Load ZFS and load root system from the RAID-Z array. > zfs_load=3D"YES" > vfs.root.mountfrom=3D"zfs:raid/sys/root" >=20 > # Tune ZFS and VM parameters. > #vfs.zfs.arc_max=3D"64M" > #kern.maxvnodes=3D"50000" > #vm.kmem_size_max=3D"512M" > #vm.kmem_size=3D"512M" > =3D=3D=3D sysctl vm | grep kmem > vm.kmem_size_scale: 3 > vm.kmem_size_max: 335544320 > vm.kmem_size_min: 0 > vm.kmem_size: 335544320 The only tuning that is needed (and ZFS will warn now about this) is vm.kmem_size/vm.kmem_size_max. Can you set those back to 512M and retry? --=20 Pawel Jakub Dawidek http://www.wheel.pl pjd@FreeBSD.org http://www.FreeBSD.org FreeBSD committer Am I Evil? Yes, I Am! --SwC/mAzP11p89u4p Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.4 (FreeBSD) iD8DBQFHMe97ForvXbEpPzQRArDuAJ45U0Vm7EkW7Pt3jMyOzkuBdPNMHQCg9HuB zQ9URLta+KPLekOREBLBX6U= =OB5C -----END PGP SIGNATURE----- --SwC/mAzP11p89u4p-- From owner-freebsd-fs@FreeBSD.ORG Thu Nov 8 06:36:15 2007 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 6955E16A474; Thu, 8 Nov 2007 06:36:15 +0000 (UTC) (envelope-from martijn@plak.net) Received: from smtp-vbr15.xs4all.nl (smtp-vbr15.xs4all.nl [194.109.24.35]) by mx1.freebsd.org (Postfix) with ESMTP id E651213C4B8; Thu, 8 Nov 2007 06:36:14 +0000 (UTC) (envelope-from martijn@plak.net) Received: from [172.16.253.25] (83-108.bbned.dsl.internl.net [85.223.108.83]) (authenticated bits=0) by smtp-vbr15.xs4all.nl (8.13.8/8.13.8) with ESMTP id lA86a2Go069155 (version=TLSv1/SSLv3 cipher=AES128-SHA bits=128 verify=NO); Thu, 8 Nov 2007 07:36:03 +0100 (CET) (envelope-from martijn@plak.net) In-Reply-To: <20071107170147.GN15618@garage.freebsd.pl> References: <20071005000046.GC92272@garage.freebsd.pl> <20071008121523.GM2327@garage.freebsd.pl> <20071105215035.GC26730@heff.fud.org.nz> <2e77fc10711051531k41e7224dq6aaedb35cad8d9f2@mail.gmail.com> <6214AB9C-9F9B-4B9D-8B05-0B3DF5F6C16D@SARENET.ES> <20071106100015.GB5268@garage.freebsd.pl> <9A147CEC-D5E6-4117-8C69-16E40DB45B22@SARENET.ES> <20071106125552.GO5268@garage.freebsd.pl> <5ED261E6-A85A-4D7D-A01A-4688802FECB9@plak.net> <20071107170147.GN15618@garage.freebsd.pl> Mime-Version: 1.0 (Apple Message framework v752.3) Content-Type: text/plain; charset=US-ASCII; delsp=yes; format=flowed Message-Id: Content-Transfer-Encoding: 7bit From: Martijn Plak Date: Thu, 8 Nov 2007 07:36:01 +0100 To: Pawel Jakub Dawidek X-Mailer: Apple Mail (2.752.3) X-Virus-Scanned: by XS4ALL Virus Scanner Cc: freebsd-fs@freebsd.org Subject: Re: ZFS kmem_map too small. X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 08 Nov 2007 06:36:15 -0000 >> === cat /boot/loader.conf >> # Load ZFS and load root system from the RAID-Z array. >> zfs_load="YES" >> vfs.root.mountfrom="zfs:raid/sys/root" >> >> # Tune ZFS and VM parameters. >> #vfs.zfs.arc_max="64M" >> #kern.maxvnodes="50000" >> #vm.kmem_size_max="512M" >> #vm.kmem_size="512M" >> === sysctl vm | grep kmem >> vm.kmem_size_scale: 3 >> vm.kmem_size_max: 335544320 >> vm.kmem_size_min: 0 >> vm.kmem_size: 335544320 > > The only tuning that is needed (and ZFS will warn now about this) is > vm.kmem_size/vm.kmem_size_max. Can you set those back to 512M and > retry? I set the vm.kmem_size and vm.kmem_size_max values back to 512M and rebooted. After about 7 hours, during the second "make buildworld" in parallel with additional network and disk activity, the machine experienced a kmem_map too small panic again. This time, the machine hung, no crash dump was available. Do you want me to update the kernel sources? I'm using Nov 5 sources at the moment. From owner-freebsd-fs@FreeBSD.ORG Sat Nov 10 01:06:13 2007 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 3A2D916A419 for ; Sat, 10 Nov 2007 01:06:13 +0000 (UTC) (envelope-from scrappy@freebsd.org) Received: from hub.org (hub.org [200.46.204.220]) by mx1.freebsd.org (Postfix) with ESMTP id 0C00213C4BC for ; Sat, 10 Nov 2007 01:06:12 +0000 (UTC) (envelope-from scrappy@freebsd.org) Received: from localhost (unknown [200.46.204.183]) by hub.org (Postfix) with ESMTP id 6921EB4768A for ; Fri, 9 Nov 2007 20:46:43 -0400 (AST) Received: from hub.org ([200.46.204.220]) by localhost (mx1.hub.org [200.46.204.183]) (amavisd-maia, port 10024) with ESMTP id 06286-02 for ; Fri, 9 Nov 2007 20:46:41 -0400 (AST) Received: from fserv.hub.org (blk-137-93-67.eastlink.ca [24.137.93.67]) by hub.org (Postfix) with ESMTP id AB4EAB47689 for ; Fri, 9 Nov 2007 20:46:42 -0400 (AST) Received: from [192.168.1.2] (unknown [192.168.1.2]) by fserv.hub.org (Postfix) with ESMTP id 93C4C4E98B for ; Fri, 9 Nov 2007 20:46:45 -0400 (AST) Date: Fri, 09 Nov 2007 20:46:24 -0400 From: "Marc G. Fournier" To: freebsd-fs@freebsd.org Message-ID: <1E865507A930162B67FB75DE@ganymede.hub.org> X-Mailer: Mulberry/4.0.8 (Linux/x86) MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Content-Disposition: inline Subject: gclusterfs ... X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 10 Nov 2007 01:06:13 -0000 -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Anyone play with this? It says it runs under FreeBSD, but am curious if anyone is using it 'in production' and how well it works ... - ---- Marc G. Fournier Hub.Org Networking Services (http://www.hub.org) Email . scrappy@hub.org MSN . scrappy@hub.org Yahoo . yscrappy Skype: hub.org ICQ . 7615664 -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.4 (FreeBSD) iD8DBQFHNP9g4QvfyHIvDvMRAhZZAJ9zIhejwK4wSl4vzvH4c4YWBVhMOACdHeZf nn0OGXhWqOCjUpMWsQGrXJU= =jBtn -----END PGP SIGNATURE----- From owner-freebsd-fs@FreeBSD.ORG Sat Nov 10 11:27:31 2007 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 37D3C16A418; Sat, 10 Nov 2007 11:27:31 +0000 (UTC) (envelope-from kris@FreeBSD.org) Received: from weak.local (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 601F113C494; Sat, 10 Nov 2007 11:27:30 +0000 (UTC) (envelope-from kris@FreeBSD.org) Message-ID: <473595A1.5020904@FreeBSD.org> Date: Sat, 10 Nov 2007 12:27:29 +0100 From: Kris Kennaway User-Agent: Thunderbird 2.0.0.6 (Macintosh/20070728) MIME-Version: 1.0 To: "Marc G. Fournier" References: <1E865507A930162B67FB75DE@ganymede.hub.org> In-Reply-To: <1E865507A930162B67FB75DE@ganymede.hub.org> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-fs@freebsd.org Subject: Re: gclusterfs ... X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 10 Nov 2007 11:27:31 -0000 Marc G. Fournier wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > > Anyone play with this? No, have you? Try it and tell us about it. > It says it runs under FreeBSD, but am curious if anyone is using it 'in > production' and how well it works ... I dont see that statement made. kris From owner-freebsd-fs@FreeBSD.ORG Sat Nov 10 11:55:24 2007 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 7A7FB16A419 for ; Sat, 10 Nov 2007 11:55:24 +0000 (UTC) (envelope-from ndenev@gmail.com) Received: from rv-out-0910.google.com (rv-out-0910.google.com [209.85.198.187]) by mx1.freebsd.org (Postfix) with ESMTP id 4DBD813C4AA for ; Sat, 10 Nov 2007 11:55:24 +0000 (UTC) (envelope-from ndenev@gmail.com) Received: by rv-out-0910.google.com with SMTP id l15so649967rvb for ; Sat, 10 Nov 2007 03:55:08 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=beta; h=domainkey-signature:received:received:message-id:date:from:to:subject:cc:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references; bh=THgu3PDGO/xnMAQMYb4kqJ1LtjuwoE5q2ahKjxETfVo=; b=mEoToZaCglYnCq20tsAun5DatKg1SfOGoA0LLer5qpK7Sc96vtpEDk0N7yFYSzPaqsM04iw7C6DV0E/1HqNBdHrBQdqOoNpBrnMUpKAtclTneOaxwSlqde+uc6q2bzBPkhNBgdHdKXgrdHXtwrGopfbbDomtcksEMysqzAvDpcw= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=beta; h=received:message-id:date:from:to:subject:cc:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references; b=o6fF9q91bzM2BKj3rc1/HxnVSWHzZcXLVpEfVxnbKa/n8NBdVrL07CDzgs82lvMCJ7s56bszFoDw25yxIZET1DR+zEkJAuJdlL98t6HUiqApmN4h/mxLLMGvyBjIPLdBXuiJ1dLuwmD1z0kuqoGjhaZH+c75NSrbH+2I6XKDyhQ= Received: by 10.141.45.16 with SMTP id x16mr1471461rvj.1194694047710; Sat, 10 Nov 2007 03:27:27 -0800 (PST) Received: by 10.140.131.18 with HTTP; Sat, 10 Nov 2007 03:27:27 -0800 (PST) Message-ID: <2e77fc10711100327j3d563c21o74d216dd7c888da@mail.gmail.com> Date: Sat, 10 Nov 2007 13:27:27 +0200 From: "Niki Denev" To: "Marc G. Fournier" In-Reply-To: <1E865507A930162B67FB75DE@ganymede.hub.org> MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline References: <1E865507A930162B67FB75DE@ganymede.hub.org> Cc: freebsd-fs@freebsd.org Subject: Re: gclusterfs ... X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 10 Nov 2007 11:55:24 -0000 On Nov 10, 2007 2:46 AM, Marc G. Fournier wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > > Anyone play with this? > > It says it runs under FreeBSD, but am curious if anyone is using it 'in > production' and how well it works ... > > - ---- > Marc G. Fournier Hub.Org Networking Services (http://www.hub.org) > Email . scrappy@hub.org MSN . scrappy@hub.org > Yahoo . yscrappy Skype: hub.org ICQ . 7615664 > -----BEGIN PGP SIGNATURE----- > Version: GnuPG v2.0.4 (FreeBSD) > > iD8DBQFHNP9g4QvfyHIvDvMRAhZZAJ9zIhejwK4wSl4vzvH4c4YWBVhMOACdHeZf > nn0OGXhWqOCjUpMWsQGrXJU= > =jBtn > -----END PGP SIGNATURE----- > I've tried compiling the server part only (server part doesn't need fuse). It needed a little bit of doctoring (mainly properly including types.h in various places) to compile properly. Then the daemon started but didn't work very well.. for some reason the clients would stop seeing the server after some time, while the server continued to run with no errors in the logs. Nevertheless it looks very interesting, but would probably need some portability fixes to compile and work properly on anything not linux. (sigh...) From owner-freebsd-fs@FreeBSD.ORG Sat Nov 10 17:41:13 2007 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 467EA16A417; Sat, 10 Nov 2007 17:41:13 +0000 (UTC) (envelope-from scottl@samsco.org) Received: from pooker.samsco.org (pooker.samsco.org [168.103.85.57]) by mx1.freebsd.org (Postfix) with ESMTP id E96A113C48E; Sat, 10 Nov 2007 17:41:12 +0000 (UTC) (envelope-from scottl@samsco.org) Received: from phobos.samsco.home (phobos.samsco.home [192.168.254.11]) (authenticated bits=0) by pooker.samsco.org (8.13.8/8.13.8) with ESMTP id lAAHewMR089941; Sat, 10 Nov 2007 10:40:58 -0700 (MST) (envelope-from scottl@samsco.org) Message-ID: <4735ED29.7090301@samsco.org> Date: Sat, 10 Nov 2007 10:40:57 -0700 From: Scott Long User-Agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X; en-US; rv:1.8.1.6) Gecko/20070802 SeaMonkey/1.1.4 MIME-Version: 1.0 To: Kris Kennaway References: <1E865507A930162B67FB75DE@ganymede.hub.org> <473595A1.5020904@FreeBSD.org> In-Reply-To: <473595A1.5020904@FreeBSD.org> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Greylist: Sender succeeded SMTP AUTH authentication, not delayed by milter-greylist-2.0.2 (pooker.samsco.org [168.103.85.57]); Sat, 10 Nov 2007 10:40:58 -0700 (MST) X-Spam-Status: No, score=-1.4 required=5.5 tests=ALL_TRUSTED autolearn=failed version=3.1.8 X-Spam-Checker-Version: SpamAssassin 3.1.8 (2007-02-13) on pooker.samsco.org Cc: freebsd-fs@freebsd.org, "Marc G. Fournier" Subject: Re: gclusterfs ... X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 10 Nov 2007 17:41:13 -0000 Kris Kennaway wrote: > Marc G. Fournier wrote: >> -----BEGIN PGP SIGNED MESSAGE----- >> Hash: SHA1 >> >> >> Anyone play with this? > > No, have you? Try it and tell us about it. Are questions no longer allowed on this this? Do all emails need to be accompanied by a research paper and graphs? > >> It says it runs under FreeBSD, but am curious if anyone is using it >> 'in production' and how well it works ... > > I dont see that statement made. The statement is made in the FAQ. Took me approximately 2 clicks to find it. Scott From owner-freebsd-fs@FreeBSD.ORG Sat Nov 10 17:58:47 2007 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 130FC16A469 for ; Sat, 10 Nov 2007 17:58:47 +0000 (UTC) (envelope-from scrappy@freebsd.org) Received: from hub.org (hub.org [200.46.204.220]) by mx1.freebsd.org (Postfix) with ESMTP id B008013C491 for ; Sat, 10 Nov 2007 17:58:46 +0000 (UTC) (envelope-from scrappy@freebsd.org) Received: from localhost (unknown [200.46.204.184]) by hub.org (Postfix) with ESMTP id 1D684B46FFC; Sat, 10 Nov 2007 13:58:35 -0400 (AST) Received: from hub.org ([200.46.204.220]) by localhost (mx1.hub.org [200.46.204.184]) (amavisd-maia, port 10024) with ESMTP id 05129-02; Sat, 10 Nov 2007 13:58:33 -0400 (AST) Received: from fserv.hub.org (blk-137-93-67.eastlink.ca [24.137.93.67]) by hub.org (Postfix) with ESMTP id 43396B46FEC; Sat, 10 Nov 2007 13:58:34 -0400 (AST) Received: from [192.168.1.2] (unknown [192.168.1.2]) by fserv.hub.org (Postfix) with ESMTP id C5CA83E44C; Sat, 10 Nov 2007 13:58:33 -0400 (AST) Date: Sat, 10 Nov 2007 13:58:06 -0400 From: "Marc G. Fournier" To: Kris Kennaway Message-ID: In-Reply-To: <473595A1.5020904@FreeBSD.org> References: <1E865507A930162B67FB75DE@ganymede.hub.org> <473595A1.5020904@FreeBSD.org> X-Mailer: Mulberry/4.0.8 (Linux/x86) MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Content-Disposition: inline Cc: freebsd-fs@freebsd.org Subject: Re: gclusterfs ... X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 10 Nov 2007 17:58:47 -0000 -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 - --On Saturday, November 10, 2007 12:27:29 +0100 Kris Kennaway wrote: > Marc G. Fournier wrote: >> -----BEGIN PGP SIGNED MESSAGE----- >> Hash: SHA1 >> >> >> Anyone play with this? > > No, have you? Try it and tell us about it. > >> It says it runs under FreeBSD, but am curious if anyone is using it 'in >> production' and how well it works ... > > I dont see that statement made. In the FAQ: "As of now Linux, FreeBSD, OpenSolaris (work in progress) and Mac OS X kernels are known to support." But, I suspect that that is in reference to the fact that we support FUSE, not so much that gclusterfs works on us ... - ---- Marc G. Fournier Hub.Org Networking Services (http://www.hub.org) Email . scrappy@hub.org MSN . scrappy@hub.org Yahoo . yscrappy Skype: hub.org ICQ . 7615664 -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.4 (FreeBSD) iD8DBQFHNfE14QvfyHIvDvMRArqjAKDNS0KR+gLMdhHp9J+Q6TcZxEEA9QCghMc+ FgVv7QQ10Zf4n1R4YndhMOI= =KALy -----END PGP SIGNATURE----- From owner-freebsd-fs@FreeBSD.ORG Sat Nov 10 18:05:53 2007 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id A002C16A468 for ; Sat, 10 Nov 2007 18:05:53 +0000 (UTC) (envelope-from scrappy@freebsd.org) Received: from hub.org (hub.org [200.46.204.220]) by mx1.freebsd.org (Postfix) with ESMTP id 6B54B13C4C4 for ; Sat, 10 Nov 2007 18:05:52 +0000 (UTC) (envelope-from scrappy@freebsd.org) Received: from localhost (unknown [200.46.204.191]) by hub.org (Postfix) with ESMTP id CF384B47040; Sat, 10 Nov 2007 14:05:35 -0400 (AST) Received: from hub.org ([200.46.204.220]) by localhost (mx1.hub.org [200.46.204.191]) (amavisd-maia, port 10024) with ESMTP id 34672-08; Sat, 10 Nov 2007 14:05:35 -0400 (AST) Received: from fserv.hub.org (blk-137-93-67.eastlink.ca [24.137.93.67]) by hub.org (Postfix) with ESMTP id CFFBCB46FEC; Sat, 10 Nov 2007 14:05:34 -0400 (AST) Received: from [192.168.1.2] (unknown [192.168.1.2]) by fserv.hub.org (Postfix) with ESMTP id CF1363E44C; Sat, 10 Nov 2007 14:05:34 -0400 (AST) Date: Sat, 10 Nov 2007 14:05:14 -0400 From: "Marc G. Fournier" To: Scott Long , Kris Kennaway Message-ID: In-Reply-To: <4735ED29.7090301@samsco.org> References: <1E865507A930162B67FB75DE@ganymede.hub.org> <473595A1.5020904@FreeBSD.org> <4735ED29.7090301@samsco.org> X-Mailer: Mulberry/4.0.8 (Linux/x86) MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Content-Disposition: inline Cc: freebsd-fs@freebsd.org Subject: Re: gclusterfs ... X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 10 Nov 2007 18:05:53 -0000 -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 - --On Saturday, November 10, 2007 10:40:57 -0700 Scott Long wrote: > The statement is made in the FAQ. Took me approximately 2 clicks to > find it. After re-reading it, I suspect that I extrapolated was is supposed to mean 'FreeBSD supports FUSE' to 'gClusterFS works on FreeBSD' :( Based on Niki's attempt, I suspect the former ... but, as he said, looks promising, now just need to get some more machines ... - ---- Marc G. Fournier Hub.Org Networking Services (http://www.hub.org) Email . scrappy@hub.org MSN . scrappy@hub.org Yahoo . yscrappy Skype: hub.org ICQ . 7615664 -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.4 (FreeBSD) iD8DBQFHNfLa4QvfyHIvDvMRAh8YAJ9+vjy2QN0aTI6cFpKPaLYngHulagCdHrkN E7V2AqF2imhvtzJEUHRVXsw= =PUkj -----END PGP SIGNATURE----- From owner-freebsd-fs@FreeBSD.ORG Sat Nov 10 19:58:09 2007 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 1D1B616A41A; Sat, 10 Nov 2007 19:58:09 +0000 (UTC) (envelope-from kris@FreeBSD.org) Received: from weak.local (pointyhat.freebsd.org [IPv6:2001:4f8:fff6::2b]) by mx1.freebsd.org (Postfix) with ESMTP id 0513513C48A; Sat, 10 Nov 2007 19:58:03 +0000 (UTC) (envelope-from kris@FreeBSD.org) Message-ID: <47360D49.4060405@FreeBSD.org> Date: Sat, 10 Nov 2007 20:58:01 +0100 From: Kris Kennaway User-Agent: Thunderbird 2.0.0.6 (Macintosh/20070728) MIME-Version: 1.0 To: Scott Long References: <1E865507A930162B67FB75DE@ganymede.hub.org> <473595A1.5020904@FreeBSD.org> <4735ED29.7090301@samsco.org> In-Reply-To: <4735ED29.7090301@samsco.org> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-fs@freebsd.org, "Marc G. Fournier" Subject: Re: gclusterfs ... X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 10 Nov 2007 19:58:09 -0000 Scott Long wrote: >>> It says it runs under FreeBSD, but am curious if anyone is using it >>> 'in production' and how well it works ... >> >> I dont see that statement made. > > The statement is made in the FAQ. Took me approximately 2 clicks to > find it. I see no reference to FreeBSD in http://gluster.org/docs/index.php/Frequently_Asked_Questions, and the following statement: "Currently Gluster has been tested on * Debian * Slackware * Redhat Enterprise (and clones such as ScientificLinux and CentOS) * Fedora Core (1-4) * Ubuntu * Gentoo * SLES (Suse Linux Enterprise Server) 10 * OpenSUSE " and only references to being portable across Linux distributions. But maybe your search skills are more awesome than mine :D Kris From owner-freebsd-fs@FreeBSD.ORG Sat Nov 10 20:34:03 2007 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 1D7F816A420 for ; Sat, 10 Nov 2007 20:34:03 +0000 (UTC) (envelope-from freebsd-fs@m.gmane.org) Received: from ciao.gmane.org (main.gmane.org [80.91.229.2]) by mx1.freebsd.org (Postfix) with ESMTP id C6B9213C4C8 for ; Sat, 10 Nov 2007 20:34:02 +0000 (UTC) (envelope-from freebsd-fs@m.gmane.org) Received: from list by ciao.gmane.org with local (Exim 4.43) id 1Iqx14-0005ea-TP for freebsd-fs@freebsd.org; Sat, 10 Nov 2007 20:33:18 +0000 Received: from 89-172-61-156.adsl.net.t-com.hr ([89.172.61.156]) by main.gmane.org with esmtp (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Sat, 10 Nov 2007 20:33:18 +0000 Received: from ivoras by 89-172-61-156.adsl.net.t-com.hr with local (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Sat, 10 Nov 2007 20:33:18 +0000 X-Injected-Via-Gmane: http://gmane.org/ To: freebsd-fs@freebsd.org From: Ivan Voras Date: Sat, 10 Nov 2007 21:32:36 +0100 Lines: 34 Message-ID: References: <1E865507A930162B67FB75DE@ganymede.hub.org> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="------------enig409323C715EBD2D3DFFF833E" X-Complaints-To: usenet@ger.gmane.org X-Gmane-NNTP-Posting-Host: 89-172-61-156.adsl.net.t-com.hr User-Agent: Thunderbird 2.0.0.6 (Windows/20070728) In-Reply-To: <1E865507A930162B67FB75DE@ganymede.hub.org> X-Enigmail-Version: 0.95.5 Sender: news Subject: Re: gclusterfs ... X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 10 Nov 2007 20:34:03 -0000 This is an OpenPGP/MIME signed message (RFC 2440 and 3156) --------------enig409323C715EBD2D3DFFF833E Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Marc G. Fournier wrote: >=20 > Anyone play with this? >=20 > It says it runs under FreeBSD, but am curious if anyone is using it 'in= =20 > production' and how well it works ... As far as I can see it uses FUSE for the file system driver, and when I last looked at it (at least our version of...) FUSE doesn't support file locking (flock(2), etc., not kernel locking), so at best it would be of limited use (e.g. no databases, mbox mailboxes, etc. on it). --------------enig409323C715EBD2D3DFFF833E Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.5 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFHNhVqldnAQVacBcgRAuUZAKCm+hhIC1/OHG8hKxOXwcPPnlk43gCePgxR MKf7n8f6gqB1sCou1ip5mco= =WXMF -----END PGP SIGNATURE----- --------------enig409323C715EBD2D3DFFF833E-- From owner-freebsd-fs@FreeBSD.ORG Sat Nov 10 21:11:17 2007 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id ACE4D16A419; Sat, 10 Nov 2007 21:11:17 +0000 (UTC) (envelope-from freebsd@hub.org) Received: from hub.org (hub.org [200.46.204.220]) by mx1.freebsd.org (Postfix) with ESMTP id 6CC9A13C4B7; Sat, 10 Nov 2007 21:11:17 +0000 (UTC) (envelope-from freebsd@hub.org) Received: from localhost (unknown [200.46.204.187]) by hub.org (Postfix) with ESMTP id D4210B4721A; Sat, 10 Nov 2007 16:54:20 -0400 (AST) Received: from hub.org ([200.46.204.220]) by localhost (mx1.hub.org [200.46.204.187]) (amavisd-maia, port 10024) with ESMTP id 74135-01; Sat, 10 Nov 2007 16:54:17 -0400 (AST) Received: from fserv.hub.org (blk-137-93-67.eastlink.ca [24.137.93.67]) by hub.org (Postfix) with ESMTP id 71807B471D9; Sat, 10 Nov 2007 16:54:18 -0400 (AST) Received: from [192.168.1.2] (unknown [192.168.1.2]) by fserv.hub.org (Postfix) with ESMTP id 445023D0AE; Sat, 10 Nov 2007 16:54:19 -0400 (AST) Date: Sat, 10 Nov 2007 16:53:56 -0400 From: "Marc G. Fournier" To: Ivan Voras , freebsd-fs@freebsd.org Message-ID: <5803BE560EBD58C6EF0E8FC7@ganymede.hub.org> In-Reply-To: References: <1E865507A930162B67FB75DE@ganymede.hub.org> X-Mailer: Mulberry/4.0.8 (Linux/x86) MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Content-Disposition: inline Cc: Subject: Re: gclusterfs ... X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 10 Nov 2007 21:11:17 -0000 -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 - --On Saturday, November 10, 2007 21:32:36 +0100 Ivan Voras wrote: > Marc G. Fournier wrote: >> >> Anyone play with this? >> >> It says it runs under FreeBSD, but am curious if anyone is using it 'in >> production' and how well it works ... > > As far as I can see it uses FUSE for the file system driver, and when I > last looked at it (at least our version of...) FUSE doesn't support file > locking (flock(2), etc., not kernel locking), so at best it would be of > limited use (e.g. no databases, mbox mailboxes, etc. on it). Actually, FAQ 'How is locking handled?' NOTE: GlusterFS only supports fcntl() locks, as FUSE itself doesn't support flock() calls. > - ---- Marc G. Fournier Hub.Org Networking Services (http://www.hub.org) Email . scrappy@hub.org MSN . scrappy@hub.org Yahoo . yscrappy Skype: hub.org ICQ . 7615664 -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.4 (FreeBSD) iD8DBQFHNhpk4QvfyHIvDvMRAi6VAKDmt0vtcjC22zZDGwhhX/guO2DL2QCfdTQx K/4RyWPlTHioPsM2cyLbITQ= =bP6G -----END PGP SIGNATURE----- From owner-freebsd-fs@FreeBSD.ORG Sat Nov 10 23:11:19 2007 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id CCD9A16A418 for ; Sat, 10 Nov 2007 23:11:19 +0000 (UTC) (envelope-from ndenev@gmail.com) Received: from rv-out-0910.google.com (rv-out-0910.google.com [209.85.198.184]) by mx1.freebsd.org (Postfix) with ESMTP id 9E0D513C491 for ; Sat, 10 Nov 2007 23:11:19 +0000 (UTC) (envelope-from ndenev@gmail.com) Received: by rv-out-0910.google.com with SMTP id l15so743608rvb for ; Sat, 10 Nov 2007 15:11:10 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=beta; h=domainkey-signature:received:received:message-id:date:from:to:subject:cc:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references; bh=WIxzPAMRgNM0rCd4opYwkJ4r3viAHjtRl785zPWPzic=; b=OJyA91trOcn3ZcAJdCNeQWCiOyv2EO0vPuaLDyWrAwcHCYtOCYKEvjYmh/5PyGZa7ES3Trmz38mooOvTzigddw9DG7+w1J2CzzqP7kanho3CAOXjbg1y3z+Fl3G66+YSB5kS/tMnrIz2eZuaQmJAomHpD/ZfJeDPT/ffTf3VXlA= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=beta; h=received:message-id:date:from:to:subject:cc:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references; b=ijVyCc+c9ni2H8ZdQa9r08iC4XaQ2LjcdquN9mJdjlMlJBD9vkAD6kdK5hcUOUirZ9Csx9Zx3AuTFN04a2Bx/FgtX8mhGzkclfclWgTD3zQyK1shy3VNuiPfLZl1TlJ1EvyooGghZOgl8GGWhuodQ4asb2cp65S2nB6CgM3E/k8= Received: by 10.140.161.11 with SMTP id j11mr1691294rve.1194736270858; Sat, 10 Nov 2007 15:11:10 -0800 (PST) Received: by 10.140.131.18 with HTTP; Sat, 10 Nov 2007 15:11:10 -0800 (PST) Message-ID: <2e77fc10711101511r70dce5cbqf00cee53459458ad@mail.gmail.com> Date: Sun, 11 Nov 2007 01:11:10 +0200 From: "Niki Denev" To: "Ivan Voras" In-Reply-To: MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline References: <1E865507A930162B67FB75DE@ganymede.hub.org> Cc: freebsd-fs@freebsd.org Subject: Re: gclusterfs ... X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 10 Nov 2007 23:11:19 -0000 On Nov 10, 2007 10:32 PM, Ivan Voras wrote: > Marc G. Fournier wrote: > > > > Anyone play with this? > > > > It says it runs under FreeBSD, but am curious if anyone is using it 'in > > production' and how well it works ... > > As far as I can see it uses FUSE for the file system driver, and when I > last looked at it (at least our version of...) FUSE doesn't support file > locking (flock(2), etc., not kernel locking), so at best it would be of > limited use (e.g. no databases, mbox mailboxes, etc. on it). > > It uses FUSE only for the client. The server is just a normal user space daemon. (although obviously it is not written with portability in mind)