From owner-freebsd-current@FreeBSD.ORG Sun Jan 6 23:48:29 2008 Return-Path: Delivered-To: freebsd-current@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 6F09E16A417 for ; Sun, 6 Jan 2008 23:48:29 +0000 (UTC) (envelope-from mcdouga9@egr.msu.edu) Received: from mx.egr.msu.edu (surfnturf.egr.msu.edu [35.9.37.164]) by mx1.freebsd.org (Postfix) with ESMTP id 38F7C13C457 for ; Sun, 6 Jan 2008 23:48:28 +0000 (UTC) (envelope-from mcdouga9@egr.msu.edu) Received: from localhost (localhost.egr.msu.edu [127.0.0.1]) by mx.egr.msu.edu (Postfix) with ESMTP id 3ED9A2EB8C5 for ; Sun, 6 Jan 2008 18:32:55 -0500 (EST) X-Virus-Scanned: amavisd-new at egr.msu.edu Received: from mx.egr.msu.edu ([127.0.0.1]) by localhost (surfnturf.egr.msu.edu [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id uCaxuHDcVMQz for ; Sun, 6 Jan 2008 18:32:55 -0500 (EST) Received: from localhost (daemon.egr.msu.edu [35.9.44.65]) by mx.egr.msu.edu (Postfix) with ESMTP id 162822EB8C4 for ; Sun, 6 Jan 2008 18:32:55 -0500 (EST) Received: by localhost (Postfix, from userid 21281) id 12A3333C3D; Sun, 6 Jan 2008 18:32:55 -0500 (EST) Date: Sun, 6 Jan 2008 18:32:55 -0500 From: Adam McDougall To: freebsd-current@FreeBSD.org Message-ID: <20080106233254.GE1138@egr.msu.edu> References: <9bbcef730801060458k4bc9f2d6uc3f097d70e087b68@mail.gmail.com> <4780D289.7020509@FreeBSD.org> <4780E546.9050303@FreeBSD.org> <9bbcef730801060651y489f1f9bw269d0968407dd8fb@mail.gmail.com> <4780EF09.4090908@FreeBSD.org> <47810BE3.4080601@FreeBSD.org> <4781227B.5020800@rcn.com> <47814EAB.70405@FreeBSD.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <47814EAB.70405@FreeBSD.org> User-Agent: Mutt/1.5.17 (2007-11-01) Cc: Subject: Re: Should we simply disallow ZFS on FreeBSD/i386? X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 06 Jan 2008 23:48:29 -0000 On Sun, Jan 06, 2008 at 01:56:59PM -0800, Maxim Sobolev wrote: Gary Corcoran wrote: >> Perhaps the 7.0 release notes should include a note to the effect that >> ZFS is *strongly* NOT RECOMMENDED on 32-bit systems at this time, due By watching this and other threads on ZFS and reading Sun's own design papers I am getting strong impression that this should be even more strong than strong NOT RECOMMENDED. Perhaps ZFS should BE DISALLOWED to run on i386 at all (unless one does some manual source code tweaking or something like this, and hence can ask no official support from the project). I feel like stating my opinion on this, noting that I am usually stubborn enough to squeeze alot of value out of any rough product, while avoiding complaining about continuing problems if I am not prepared to put in appropriate effort to solve them. A summary of my opinion on this matter is that some i386 FreeBSD servers do have a place running zfs in a useful role, but some dedication and patience from the administrator is usually required, and the effort to tune at least kmem is nearly required on ALL hardware platforms, not just i386. I think kmem shortages from zfs are simply more touchy on i386, and with enough ram and slightly more tuning than amd64 the kmem can most likely be tuned away, but this does not do anything for other zfs problems such as zil deadlocks and other deadlocks. I think doing something to prevent FreeBSD/i386 users from using zfs will just rule out a portion of the people having problems, and admins who take a little time to tune zfs AND use it more than just lightly may continue to have problems, and will just come back to the lists. I have zfs on at least 4 systems presently, each one tuned to where I no longer receive kmem panics at least based on their expected system load. 2 of them are i386 and I would be quite dismayed to upgrade RELENG_7 to find ZFS has been disabled for me (although since I read the mailing lists I would expect it and deal appropriately). It would be a tradeoff between breaking a limited amount of existing setups versus somehow limiting the influx of new zfs users who _may_ encounter zfs problems (of course you will only hear from the people who complain). The amount of kmem required for a particular workload on any one machine can vary alot. Believe it or not, it is one of my AMD64 systems that I had to increase kmem to 1.6G to prevent kmem panics (it does some heavy nightly rsyncs); versus just having kmem set to 1G on a i386 system that constantly serves out files to the internet with various rsyncs running through the day. I don't think its exactly fair to punish all i386 users by disabling functionality, but I'm not the one that has to support all the users so I can understand the caution. I'm certainly in favor of more dire warnings where appropriate. Perhaps even a simple runtime warning when creating or importing a zpool? I still think problems with running zfs on i386 will be the tip of that iceberg. Certainly light use of zfs on an i386 might never see a crash. But I think this might be a new frontier for a FreeBSD release to include functionality that is blatently labeled experimental and considered unstable yet will have wide appeal and interest; almost anyone can use a filesystem on FreeBSD, its not like it is just an unstable network driver affecting a small portion of users. I believe that 95% of hardware today that realistically is capable of running ZFS is also capable of running 64bit code, so that potential ZFS users are far better off switching to FreeBSD/amd64 and help testing/improving that architecture than fighting architectural limitations of already dying i386. And we are as a project are better off too, by spending out limited resources on something that has future. One of my busiest zfs servers is a dual Xeon 2.0ghz (i386 only) with 2 gigs of ram (and I plan to add more 'just cuz'). I don't have any spare amd64 systems I could replace it with at this time, yet I feel the hardware is up to the job if configured as best I can. I have plenty of spares of this server type since it is older yet not useless. It pushes out data constantly to the internet, boots off of zfs, and I believe has never crashed from a kmem panic thanks to the tuning I have set below, gathered from wiki, mailing lists, and experience: vfs.zfs.prefetch_disable="1" vfs.zfs.arc_max="104857600" vm.kmem_size_max="1G" vfs.zfs.zil_disable="1" I will not pretend it is perfectly stable (it has hung every couple of weeks) but it is not due to a kmem panic. zil was disabled because it caused deadlocks. I think what I face now is some other kind of deadlock, which I will try harder to debug the next time it happens, although I have to balance time spent with benefit. It has only happened twice since I disabled zil. This server is doing something useful with zfs yet I can put up with occasional downtime without much complaint from users. I am using it as a semi-production testbed to kick the tires on zfs and use the experience to boost my zfs skills and hopefully help out the whole FreeBSD community with my results when possible. From my own experience FreeBSD/amd64 is quite mature for running most if not all of the server tasks today and ZFS is first and foremost a server FS. The only place where FreeBSD/i386 beats FreeBSD/amd64 is desktop, due to binary drivers and such, but ZFS is almost useless there. So that by simply officially disallowing ZFS on FreeBSD/i386 we could win by a great margin. Just my CAD0.02. -Maxim _______________________________________________ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscribe@freebsd.org"