From owner-freebsd-fs@FreeBSD.ORG Mon Feb 12 15:52:51 2007 Return-Path: X-Original-To: freebsd-fs@FreeBSD.ORG Delivered-To: freebsd-fs@FreeBSD.ORG Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id B38EA16A401 for ; Mon, 12 Feb 2007 15:52:51 +0000 (UTC) (envelope-from olli@lurza.secnetix.de) Received: from lurza.secnetix.de (lurza.secnetix.de [83.120.8.8]) by mx1.freebsd.org (Postfix) with ESMTP id 3215C13C4B5 for ; Mon, 12 Feb 2007 15:52:50 +0000 (UTC) (envelope-from olli@lurza.secnetix.de) Received: from lurza.secnetix.de (ezklyv@localhost [127.0.0.1]) by lurza.secnetix.de (8.13.4/8.13.4) with ESMTP id l1CFqiq3046651; Mon, 12 Feb 2007 16:52:49 +0100 (CET) (envelope-from oliver.fromme@secnetix.de) Received: (from olli@localhost) by lurza.secnetix.de (8.13.4/8.13.1/Submit) id l1CFqi6q046650; Mon, 12 Feb 2007 16:52:44 +0100 (CET) (envelope-from olli) Date: Mon, 12 Feb 2007 16:52:44 +0100 (CET) Message-Id: <200702121552.l1CFqi6q046650@lurza.secnetix.de> From: Oliver Fromme To: freebsd-fs@FreeBSD.ORG, aronesimi@yahoo.com In-Reply-To: <506226.11053.qm@web58610.mail.re3.yahoo.com> X-Newsgroups: list.freebsd-fs User-Agent: tin/1.8.2-20060425 ("Shillay") (UNIX) (FreeBSD/4.11-STABLE (i386)) X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-2.1.2 (lurza.secnetix.de [127.0.0.1]); Mon, 12 Feb 2007 16:52:50 +0100 (CET) Cc: Subject: Re: comments on newfs raw disk ? Safe ? (7 terabyte array) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: freebsd-fs@FreeBSD.ORG, aronesimi@yahoo.com List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 12 Feb 2007 15:52:51 -0000 Arone Silimantia wrote: > Thank you for your detailed response - my own response is inline below: You should indent the quoted text, so it's easier to tell the quoted text from your own text. Most MUAs offer a function to indent the quoted text automatically. I tried to fix it below. > Oliver Fromme wrote: > > Arone Silimantia wrote: > > > Big 3ware sata raid with 16 disks. First two disks are a mirror to > > > boot off of. I installed the system with sysinstall and created all the > > > partitions on the boot mirror, etc., and just didn't even touch the > > > 14-disk array that was also created. > > > [...] > > > newfs -m 0 /dev/da1 > > > > You didn't mention the size of the FS, but I guess it's at > > least 4 TB, probably more. > > Well, in the subject line I mentioned 7 TB, but I have > since rearranged some things and it will be 5.5 TB. Sorry, I didn't look at the subject line too closely. I expected that all important information was included in the mail body. > > You'll probably want to reduce the inode density (i.e. > > increase the bytes-per-inode ratio). With the default > > value, an fsck will be a royal pain, no matter whether you > > use background fsck (with snapshots) or not. It might even > > not work at all if you don't have a huge amount of RAM. > > Well, I have 4 GB of physical RAM, and 4 GB of swap - so does that > total of 8 GB satisfy the "1 GB per TB" requirement, or do I really > need >5.5 GB of actual swap space (in addition to the physical) ? That "1 GB per TB" requirement is just a rule of thumb. I don't know hoe accurate it is. Also note that it is desirable to avoid having fsck use swap, because it will be even slower then. A lot slower. > > If you increase the ratio to 64 K, it will lower the fsck > > time and RAM requirement by an order of magnitude, while > > there are still about 15 million inodes available per TB. > > If possible, increase the ratio (-i option) further. It > > depends on the expected average file size and the maximum > > number of files that you intend to store on the FS, of > > course. > > Ok, I will look into this. My data population uses a little less > than 5 million inodes per TB, so this may be workable to tune. So I > see the default is '4' - so I could run newfs with: The default is 4096 (one inode per 4 KB). > newfs -i 8 > > to do what you are suggesting ? I think you mean 8192. That will cut the number of inodes in half, but I think you can go even further. Try: # newfs -i 65536 That will leave room for about 15 million inodes per TB, which is plenty for your needs. By the way, reducing the inode density like that will also give your more space for actual file data. In UFS2, every inode takes 256 bytes. Reducing the bytes-per-inode ratio from 4 KB to 64 KB will give you additional 60 GB of space. _And_ it will reduce the memory and time requirements of fsck. > > Oh by the way, I also agree with Eric that you should have > > a look at gjournal. It pratically removes the fsck issues. > > At the moment it's only in -current, but I think Pawel > > provided a port for 6.x. > > Well, I don't mind a 24 hour fsck, and I would like to remove > complexity and not be so on the bleeding edge with things. Since I > am only using 5mil inodes per TB anyway, that ends up being 25-30 > million inodes in the 5 TB drive which I think could fsck in a day or > so. I suggest you test it before putting it into production, i.e. populate the file system with the expected number of files, then run fsck. > I just need to know if my 4+4 GB of memory is enough, and if this > option in loader.conf: > > kern.maxdsiz="2048000000" That will limit the process size to 2 GB. You might need to set it higher if fsck needs more than that. (I assume you're running FreeBSD/amd64, or otherwise you'll run into process size limitations anyway.) Best regards Oliver -- Oliver Fromme, secnetix GmbH & Co. KG, Marktplatz 29, 85567 Grafing b. M. Handelsregister: Registergericht Muenchen, HRA 74606, USt-Id: DE204219783 Any opinions expressed in this message are personal to the author and may not necessarily reflect the opinions of secnetix GmbH & Co KG in any way. FreeBSD-Dienstleistungen, -Produkte und mehr: http://www.secnetix.de/bsd Perl is worse than Python because people wanted it worse. -- Larry Wall From owner-freebsd-fs@FreeBSD.ORG Mon Feb 12 16:16:19 2007 Return-Path: X-Original-To: freebsd-fs@FreeBSD.ORG Delivered-To: freebsd-fs@FreeBSD.ORG Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 475C416A41F for ; Mon, 12 Feb 2007 16:16:19 +0000 (UTC) (envelope-from olli@lurza.secnetix.de) Received: from lurza.secnetix.de (lurza.secnetix.de [83.120.8.8]) by mx1.freebsd.org (Postfix) with ESMTP id C93AC13C47E for ; Mon, 12 Feb 2007 16:16:18 +0000 (UTC) (envelope-from olli@lurza.secnetix.de) Received: from lurza.secnetix.de (cxinax@localhost [127.0.0.1]) by lurza.secnetix.de (8.13.4/8.13.4) with ESMTP id l1CGGCjO048294; Mon, 12 Feb 2007 17:16:17 +0100 (CET) (envelope-from oliver.fromme@secnetix.de) Received: (from olli@localhost) by lurza.secnetix.de (8.13.4/8.13.1/Submit) id l1CGGBX3048293; Mon, 12 Feb 2007 17:16:11 +0100 (CET) (envelope-from olli) Date: Mon, 12 Feb 2007 17:16:11 +0100 (CET) Message-Id: <200702121616.l1CGGBX3048293@lurza.secnetix.de> From: Oliver Fromme To: freebsd-fs@FreeBSD.ORG, aronesimi@yahoo.com In-Reply-To: <200702121552.l1CFqi6q046650@lurza.secnetix.de> X-Newsgroups: list.freebsd-fs User-Agent: tin/1.8.2-20060425 ("Shillay") (UNIX) (FreeBSD/4.11-STABLE (i386)) X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-2.1.2 (lurza.secnetix.de [127.0.0.1]); Mon, 12 Feb 2007 17:16:17 +0100 (CET) Cc: Subject: Re: comments on newfs raw disk ? Safe ? (7 terabyte array) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: freebsd-fs@FreeBSD.ORG, aronesimi@yahoo.com List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 12 Feb 2007 16:16:19 -0000 I'm sorry I made a small mistake ... Oliver Fromme wrote: > Arone Silimantia wrote: > > Ok, I will look into this. My data population uses a little less > > than 5 million inodes per TB, so this may be workable to tune. So I > > see the default is '4' - so I could run newfs with: > > The default is 4096 (one inode per 4 KB). The default is 4 * fragsize, and the default fragsize is 2 KB, so the default bytes-per-inode ratio is 8 KB, not 4. (Historically the default UFS fragsize was 1 KB with a blocksize of 8 KB, so the default ratio was indeed 4 KB per inode. But that was changed quite some years ago.) Best regards Oliver -- Oliver Fromme, secnetix GmbH & Co. KG, Marktplatz 29, 85567 Grafing b. M. Handelsregister: Registergericht Muenchen, HRA 74606, USt-Id: DE204219783 Any opinions expressed in this message are personal to the author and may not necessarily reflect the opinions of secnetix GmbH & Co KG in any way. FreeBSD-Dienstleistungen, -Produkte und mehr: http://www.secnetix.de/bsd With Perl you can manipulate text, interact with programs, talk over networks, drive Web pages, perform arbitrary precision arithmetic, and write programs that look like Snoopy swearing. From owner-freebsd-fs@FreeBSD.ORG Mon Feb 12 16:39:21 2007 Return-Path: X-Original-To: freebsd-fs@FreeBSD.ORG Delivered-To: freebsd-fs@FreeBSD.ORG Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id DEFF816A401 for ; Mon, 12 Feb 2007 16:39:21 +0000 (UTC) (envelope-from randy@psg.com) Received: from rip.psg.com (rip.psg.com [147.28.0.39]) by mx1.freebsd.org (Postfix) with ESMTP id C8FC613C4AA for ; Mon, 12 Feb 2007 16:39:21 +0000 (UTC) (envelope-from randy@psg.com) Received: from cust16202.lava.net ([64.65.95.74] helo=[192.168.0.100]) by rip.psg.com with esmtpsa (TLSv1:AES256-SHA:256) (Exim 4.66 (FreeBSD)) (envelope-from ) id 1HGe18-0007sH-1a; Mon, 12 Feb 2007 16:27:02 +0000 Message-ID: <45D09549.8090807@psg.com> Date: Mon, 12 Feb 2007 06:26:49 -1000 From: Randy Bush User-Agent: Thunderbird 1.5.0.9 (Windows/20061207) MIME-Version: 1.0 To: Oliver Fromme References: <200702121616.l1CGGBX3048293@lurza.secnetix.de> In-Reply-To: <200702121616.l1CGGBX3048293@lurza.secnetix.de> X-Enigmail-Version: 0.94.2.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Cc: freebsd-fs@FreeBSD.ORG Subject: Re: comments on newfs raw disk ? Safe ? (7 terabyte array) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 12 Feb 2007 16:39:22 -0000 this thread has been great. but i suspect it would be greatly appreciated if the handbook had a page "How to format and use multi-terabyte drives, facts, trade-offs, and recipies." i am about to do this (5TB), have been gathering info, and feel a appropriately confused. randy From owner-freebsd-fs@FreeBSD.ORG Mon Feb 12 16:53:28 2007 Return-Path: X-Original-To: freebsd-fs@FreeBSD.ORG Delivered-To: freebsd-fs@FreeBSD.ORG Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 5FB1D16A406 for ; Mon, 12 Feb 2007 16:53:28 +0000 (UTC) (envelope-from olli@lurza.secnetix.de) Received: from lurza.secnetix.de (lurza.secnetix.de [83.120.8.8]) by mx1.freebsd.org (Postfix) with ESMTP id E128013C46B for ; Mon, 12 Feb 2007 16:53:27 +0000 (UTC) (envelope-from olli@lurza.secnetix.de) Received: from lurza.secnetix.de (tgbshk@localhost [127.0.0.1]) by lurza.secnetix.de (8.13.4/8.13.4) with ESMTP id l1CGrLS5055966; Mon, 12 Feb 2007 17:53:26 +0100 (CET) (envelope-from oliver.fromme@secnetix.de) Received: (from olli@localhost) by lurza.secnetix.de (8.13.4/8.13.1/Submit) id l1CGrLmJ055965; Mon, 12 Feb 2007 17:53:21 +0100 (CET) (envelope-from olli) From: Oliver Fromme Message-Id: <200702121653.l1CGrLmJ055965@lurza.secnetix.de> To: randy@psg.com (Randy Bush) Date: Mon, 12 Feb 2007 17:53:21 +0100 (CET) In-Reply-To: <45D09549.8090807@psg.com> X-Mailer: ELM [version 2.5 PL8] MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-2.1.2 (lurza.secnetix.de [127.0.0.1]); Mon, 12 Feb 2007 17:53:26 +0100 (CET) Cc: freebsd-fs@FreeBSD.ORG Subject: Re: comments on newfs raw disk ? Safe ? (7 terabyte array) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 12 Feb 2007 16:53:28 -0000 Randy Bush wrote: > this thread has been great. but i suspect it would be greatly > appreciated if the handbook had a page "How to format and use > multi-terabyte drives, facts, trade-offs, and recipies." That would indeed be great. Unfortunately I have other battle fields right now (fighting against sysinstall and loader code, among other things), so I have zero time to write a Handbook chapter from scratch. I'm also probably not that authoritative; there are people who are more knowledgable about UFS/UFS2. However, that whole issue -- formatting multi-TB FS -- is very much a moving target, especially right now that the new gjournal code is entering the arena. It will change quite a lot of things, and the recommandations will be different. You'll probably have to rewrite half of the chapter. With that in mind, maybe it makes more sense to wait a little bit until gjournal has matured some more an has officially hit the RELENG_6 branch. I don't think it will take long; the code has already proven quite stable. Best regards Oliver -- Oliver Fromme, secnetix GmbH & Co. KG, Marktplatz 29, 85567 Grafing b. M. Handelsregister: Registergericht Muenchen, HRA 74606, USt-Id: DE204219783 Any opinions expressed in this message are personal to the author and may not necessarily reflect the opinions of secnetix GmbH & Co KG in any way. FreeBSD-Dienstleistungen, -Produkte und mehr: http://www.secnetix.de/bsd "In My Egoistical Opinion, most people's C programs should be indented six feet downward and covered with dirt." -- Blair P. Houghton From owner-freebsd-fs@FreeBSD.ORG Mon Feb 12 21:04:32 2007 Return-Path: X-Original-To: freebsd-fs@freebsd.org Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 283E216A402 for ; Mon, 12 Feb 2007 21:04:32 +0000 (UTC) (envelope-from freebsd-fs@m.gmane.org) Received: from ciao.gmane.org (main.gmane.org [80.91.229.2]) by mx1.freebsd.org (Postfix) with ESMTP id D83B313C4AC for ; Mon, 12 Feb 2007 21:04:31 +0000 (UTC) (envelope-from freebsd-fs@m.gmane.org) Received: from list by ciao.gmane.org with local (Exim 4.43) id 1HGiLT-0000mL-8z for freebsd-fs@freebsd.org; Mon, 12 Feb 2007 22:04:19 +0100 Received: from 89-172-59-177.adsl.net.t-com.hr ([89.172.59.177]) by main.gmane.org with esmtp (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Mon, 12 Feb 2007 22:04:14 +0100 Received: from ivoras by 89-172-59-177.adsl.net.t-com.hr with local (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Mon, 12 Feb 2007 22:04:14 +0100 X-Injected-Via-Gmane: http://gmane.org/ To: freebsd-fs@freebsd.org From: Ivan Voras Date: Mon, 12 Feb 2007 22:03:47 +0100 Lines: 34 Message-ID: References: <200702121616.l1CGGBX3048293@lurza.secnetix.de> <45D09549.8090807@psg.com> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="------------enig292E1184065E6CF66B41D1D7" X-Complaints-To: usenet@sea.gmane.org X-Gmane-NNTP-Posting-Host: 89-172-59-177.adsl.net.t-com.hr User-Agent: Thunderbird 1.5.0.9 (Windows/20061207) In-Reply-To: <45D09549.8090807@psg.com> X-Enigmail-Version: 0.94.1.2 Sender: news Subject: Re: comments on newfs raw disk ? Safe ? (7 terabyte array) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 12 Feb 2007 21:04:32 -0000 This is an OpenPGP/MIME signed message (RFC 2440 and 3156) --------------enig292E1184065E6CF66B41D1D7 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Randy Bush wrote: > this thread has been great. but i suspect it would be greatly > appreciated if the handbook had a page "How to format and use > multi-terabyte drives, facts, trade-offs, and recipies." i am about to= > do this (5TB), have been gathering info, and feel a appropriately confu= sed. It probably doesn't have to be a handbook chapter - I think an article would also be good, especially because it would contain "rule of thumb" information. I'll probably start making one during the next week or two..= =2E --------------enig292E1184065E6CF66B41D1D7 Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.4 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFF0NY4ldnAQVacBcgRAgDRAKCXMVNO7afFRoeigrOncLCa5aPPHwCg700l eLy7m36xMY5eD1LDLloRlNM= =cU2B -----END PGP SIGNATURE----- --------------enig292E1184065E6CF66B41D1D7-- From owner-freebsd-fs@FreeBSD.ORG Mon Feb 12 21:13:03 2007 Return-Path: X-Original-To: freebsd-fs@freebsd.org Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 9CEC316A481 for ; Mon, 12 Feb 2007 21:13:03 +0000 (UTC) (envelope-from indigo@voda.cz) Received: from smtp.voda.cz (gw.voda.cz [212.24.154.90]) by mx1.freebsd.org (Postfix) with ESMTP id 5A94313C4BE for ; Mon, 12 Feb 2007 21:13:03 +0000 (UTC) (envelope-from indigo@voda.cz) Received: from localhost (localhost [127.0.0.1]) by smtp.voda.cz (Postfix) with ESMTP id D1F5D43FFE; Mon, 12 Feb 2007 22:13:01 +0100 (CET) Received: from smtp.voda.cz ([127.0.0.1]) by localhost (mail.voda.cz [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 06412-03; Mon, 12 Feb 2007 22:12:58 +0100 (CET) Received: from spyro.eiecon.net (unknown [213.151.77.190]) by smtp.voda.cz (Postfix) with ESMTP id 883BB43FF4; Mon, 12 Feb 2007 22:12:58 +0100 (CET) Date: Mon, 12 Feb 2007 22:12:58 +0100 To: "Ivan Voras" , freebsd-fs@freebsd.org From: Indigo Content-Type: text/plain; format=flowed; delsp=yes; charset=iso-8859-2 MIME-Version: 1.0 References: <200702121616.l1CGGBX3048293@lurza.secnetix.de> <45D09549.8090807@psg.com> Content-Transfer-Encoding: 7bit Message-ID: In-Reply-To: User-Agent: Opera Mail/9.10 (Win32) X-Virus-Scanned: by amavisd-new at voda.cz Cc: Subject: Re: comments on newfs raw disk ? Safe ? (7 terabyte array) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 12 Feb 2007 21:13:03 -0000 On Mon, 12 Feb 2007 22:03:47 +0100, Ivan Voras wrote: > Randy Bush wrote: >> this thread has been great. but i suspect it would be greatly >> appreciated if the handbook had a page "How to format and use >> multi-terabyte drives, facts, trade-offs, and recipies." i am about to >> do this (5TB), have been gathering info, and feel a appropriately >> confused. > > It probably doesn't have to be a handbook chapter - I think an article > would also be good, especially because it would contain "rule of thumb" > information. I'll probably start making one during the next week or > two... > It would be nice to have more information about partitioning and filesystem tuning in the handbook. Im not going to write it -but If anyone ever decides to write it Id love to help in any way I can. Vasek From owner-freebsd-fs@FreeBSD.ORG Mon Feb 12 23:48:52 2007 Return-Path: X-Original-To: freebsd-fs@freebsd.org Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 69AD816A400; Mon, 12 Feb 2007 23:48:52 +0000 (UTC) (envelope-from des@des.no) Received: from tim.des.no (tim.des.no [194.63.250.121]) by mx1.freebsd.org (Postfix) with ESMTP id 28C9113C48E; Mon, 12 Feb 2007 23:48:52 +0000 (UTC) (envelope-from des@des.no) Received: from tim.des.no (localhost [127.0.0.1]) by spam.des.no (Postfix) with ESMTP id 17D5D2089; Tue, 13 Feb 2007 00:27:17 +0100 (CET) X-Spam-Tests: AWL X-Spam-Learn: disabled X-Spam-Score: 0.0/3.0 X-Spam-Checker-Version: SpamAssassin 3.1.7 (2006-10-05) on tim.des.no Received: from dwp.des.no (des.no [80.203.243.180]) by tim.des.no (Postfix) with ESMTP id 0913B2087; Tue, 13 Feb 2007 00:27:17 +0100 (CET) Received: by dwp.des.no (Postfix, from userid 1001) id D3AF6B88C; Tue, 13 Feb 2007 00:27:16 +0100 (CET) From: des@des.no (Dag-Erling =?iso-8859-1?Q?Sm=F8rgrav?=) To: Eric Anderson References: <6FC9F9894A9F8C49A722CF9F2132FC2204C9DAAE@ms05.mailstreet2003.net> <45CD6AA6.1000003@freebsd.org> <6FC9F9894A9F8C49A722CF9F2132FC2204C9DAB0@ms05.mailstreet2003.net> <45CD6FF5.8070007@freebsd.org> Date: Tue, 13 Feb 2007 00:27:16 +0100 In-Reply-To: <45CD6FF5.8070007@freebsd.org> (Eric Anderson's message of "Sat, 10 Feb 2007 01:10:45 -0600") Message-ID: <86ire7gdnv.fsf@dwp.des.no> User-Agent: Gnus/5.110006 (No Gnus v0.6) Emacs/21.3 (berkeley-unix) MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: quoted-printable Cc: freebsd-fs@freebsd.org, Chris Haulmark Subject: Re: UFS2 with SAN X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 12 Feb 2007 23:48:52 -0000 Eric Anderson writes: > Well, I'm not sure what issues they had, but have had fantastic > success with NFS and FreeBSD. FreeBSD with the right hardware and > tweaks can make some NetApp boxes look weak. *cough* WAFL *cough* I'd be very surprised, considering that NetApp filers run FreeBSD. DES --=20 Dag-Erling Sm=F8rgrav - des@des.no From owner-freebsd-fs@FreeBSD.ORG Tue Feb 13 00:00:59 2007 Return-Path: X-Original-To: freebsd-fs@freebsd.org Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 5651016A406 for ; Tue, 13 Feb 2007 00:00:59 +0000 (UTC) (envelope-from speedtoys.racing@gmail.com) Received: from ug-out-1314.google.com (ug-out-1314.google.com [66.249.92.173]) by mx1.freebsd.org (Postfix) with ESMTP id D545F13C474 for ; Tue, 13 Feb 2007 00:00:58 +0000 (UTC) (envelope-from speedtoys.racing@gmail.com) Received: by ug-out-1314.google.com with SMTP id 71so1021ugh for ; Mon, 12 Feb 2007 16:00:54 -0800 (PST) DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=beta; h=received:message-id:date:from:to:subject:cc:in-reply-to:mime-version:content-type:references; b=WEuZHDklL9mui8dEPqaQmaRR1XQUnTQCAzN8TLcE2BGucOyl1JRWC1A5CqlrbPJMKX4s4QzTNfH7IlxctavllDS9aL1QFlRi0bWBQe3wi26RXvzJvftY3jJ/cmZEBJOkyxmNjmAVqeRGZDUp5ZuUAD2QbLFYuCvI+iWUo6Ulgds= Received: by 10.78.138.6 with SMTP id l6mr65940hud.1171324854139; Mon, 12 Feb 2007 16:00:54 -0800 (PST) Received: by 10.78.17.18 with HTTP; Mon, 12 Feb 2007 16:00:54 -0800 (PST) Message-ID: Date: Mon, 12 Feb 2007 16:00:54 -0800 From: "Jeff Mohler" To: "=?UTF-8?Q?Dag-Erling_Sm=C3=B8rgrav?=" In-Reply-To: <86ire7gdnv.fsf@dwp.des.no> MIME-Version: 1.0 References: <6FC9F9894A9F8C49A722CF9F2132FC2204C9DAAE@ms05.mailstreet2003.net> <45CD6AA6.1000003@freebsd.org> <6FC9F9894A9F8C49A722CF9F2132FC2204C9DAB0@ms05.mailstreet2003.net> <45CD6FF5.8070007@freebsd.org> <86ire7gdnv.fsf@dwp.des.no> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: base64 Content-Disposition: inline X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: freebsd-fs@freebsd.org, Chris Haulmark , Eric Anderson Subject: Re: UFS2 with SAN X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 13 Feb 2007 00:00:59 -0000 SW5jb3JyZWN0LgoKV0FGTCBhbmQgdGhlIGFzc29jaWF0ZWQgcmFpZCBsYXllcihzKSwgZXRjLCBh cmUgLW5vdC0gRnJlZWJzZC4KCgpHWC4uaXMgZGlmZmVyZW50LCBidXQgaXMgc3RpbGwgTk9UIGEg cHVyZSBGYnNkNiBwbGF5Z3JvdW5kLgoKaHR0cDovL2VuLndpa2lwZWRpYS5vcmcvd2lraS9GcmVl QlNEI0Rlcml2YXRpdmVzCgpPbiAyLzEyLzA3LCBEYWctRXJsaW5nIFNtw7hyZ3JhdiA8ZGVzQGRl cy5ubz4gd3JvdGU6Cj4KPiBFcmljIEFuZGVyc29uIDxhbmRlcnNvbkBmcmVlYnNkLm9yZz4gd3Jp dGVzOgo+ID4gV2VsbCwgSSdtIG5vdCBzdXJlIHdoYXQgaXNzdWVzIHRoZXkgaGFkLCBidXQgaGF2 ZSBoYWQgZmFudGFzdGljCj4gPiBzdWNjZXNzIHdpdGggTkZTIGFuZCBGcmVlQlNELiAgRnJlZUJT RCB3aXRoIHRoZSByaWdodCBoYXJkd2FyZSBhbmQKPiA+IHR3ZWFrcyBjYW4gbWFrZSBzb21lIE5l dEFwcCBib3hlcyBsb29rIHdlYWsuICpjb3VnaCogV0FGTCAqY291Z2gqCj4KPiBJJ2QgYmUgdmVy eSBzdXJwcmlzZWQsIGNvbnNpZGVyaW5nIHRoYXQgTmV0QXBwIGZpbGVycyBydW4gRnJlZUJTRC4K Pgo+IERFUwo+IC0tCj4gRGFnLUVybGluZyBTbcO4cmdyYXYgLSBkZXNAZGVzLm5vCj4gX19fX19f X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX18KPiBmcmVlYnNkLWZzQGZy ZWVic2Qub3JnIG1haWxpbmcgbGlzdAo+IGh0dHA6Ly9saXN0cy5mcmVlYnNkLm9yZy9tYWlsbWFu L2xpc3RpbmZvL2ZyZWVic2QtZnMKPiBUbyB1bnN1YnNjcmliZSwgc2VuZCBhbnkgbWFpbCB0byAi ZnJlZWJzZC1mcy11bnN1YnNjcmliZUBmcmVlYnNkLm9yZyIKPgo= From owner-freebsd-fs@FreeBSD.ORG Tue Feb 13 00:10:57 2007 Return-Path: X-Original-To: freebsd-fs@freebsd.org Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 8D87416A481 for ; Tue, 13 Feb 2007 00:10:57 +0000 (UTC) (envelope-from julian@elischer.org) Received: from outI.internet-mail-service.net (outI.internet-mail-service.net [216.240.47.232]) by mx1.freebsd.org (Postfix) with ESMTP id 6FB8D13C4B5 for ; Tue, 13 Feb 2007 00:10:57 +0000 (UTC) (envelope-from julian@elischer.org) Received: from mx0.idiom.com (HELO idiom.com) (216.240.32.160) by out.internet-mail-service.net (qpsmtpd/0.32) with ESMTP; Mon, 12 Feb 2007 15:32:57 -0800 Received: from [10.251.22.38] (nat.ironport.com [63.251.108.100]) by idiom.com (Postfix) with ESMTP id E1527125AD7; Mon, 12 Feb 2007 15:56:05 -0800 (PST) Message-ID: <45D0FE95.30600@elischer.org> Date: Mon, 12 Feb 2007 15:56:05 -0800 From: Julian Elischer User-Agent: Thunderbird 1.5.0.9 (Macintosh/20061207) MIME-Version: 1.0 To: =?ISO-8859-1?Q?Dag-Erling_Sm=F8rgrav?= , freebsd-fs@freebsd.org, chris@sigd.net References: <6FC9F9894A9F8C49A722CF9F2132FC2204C9DAAE@ms05.mailstreet2003.net> <45CD6AA6.1000003@freebsd.org> <6FC9F9894A9F8C49A722CF9F2132FC2204C9DAB0@ms05.mailstreet2003.net> <45CD6FF5.8070007@freebsd.org> <86ire7gdnv.fsf@dwp.des.no> In-Reply-To: <86ire7gdnv.fsf@dwp.des.no> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 8bit Cc: Subject: Re: UFS2 with SAN X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 13 Feb 2007 00:10:57 -0000 Dag-Erling Smørgrav wrote: > Eric Anderson writes: >> Well, I'm not sure what issues they had, but have had fantastic >> success with NFS and FreeBSD. FreeBSD with the right hardware and >> tweaks can make some NetApp boxes look weak. *cough* WAFL *cough* > > I'd be very surprised, considering that NetApp filers run FreeBSD. not 100 % true.. Until 6 months ago they were based on a proprietary kernel with NetBSD networking code grafted into it. Their newest stuff is based on FreeBSD, but any older model stuff they sell is still the old OS. > > DES From owner-freebsd-fs@FreeBSD.ORG Tue Feb 13 05:58:10 2007 Return-Path: X-Original-To: freebsd-fs@freebsd.org Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id DA27216A401 for ; Tue, 13 Feb 2007 05:58:09 +0000 (UTC) (envelope-from aronesimi@yahoo.com) Received: from web58601.mail.re3.yahoo.com (web58601.mail.re3.yahoo.com [68.142.236.199]) by mx1.freebsd.org (Postfix) with SMTP id 70E2D13C4AA for ; Tue, 13 Feb 2007 05:58:07 +0000 (UTC) (envelope-from aronesimi@yahoo.com) Received: (qmail 55406 invoked by uid 60001); 13 Feb 2007 05:58:06 -0000 Message-ID: <20070213055806.55404.qmail@web58601.mail.re3.yahoo.com> DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=s1024; d=yahoo.com; h=X-YMail-OSG:Received:Date:From:Subject:To:Cc:MIME-Version:Content-Type:Content-Transfer-Encoding:Message-ID; b=PhKGbwkFY0Px/s8odWv2DYWHxh1ZXYOWPuUmBNHxgSjVes/CeQm9nNVboJFSLzrzwKTw53vMoWyCX/tI0TAelZLOgAq3ah6L3nLV4/9boW/nLZ3pzTcAUL/mRFlsuZAjFAGvisXHZQjM8f70eLkfitOUJO+BC6SnPaDhs4VctyA=; X-YMail-OSG: Vr6Z8v8VM1lbVo187usqdqvexbHqiG5V_aIC8TU4S7CqsX2mAIXEWa9MiSFK5UWFS3eHXDMwek7G.NMsngFs1FE6Yh.hdtj6zKaQR7il4nU228X.n2JTcA-- Received: from [71.136.255.127] by web58601.mail.re3.yahoo.com via HTTP; Mon, 12 Feb 2007 21:58:06 PST Date: Mon, 12 Feb 2007 21:58:06 -0800 (PST) From: Arone Silimantia To: freebsd-fs@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: 8bit X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: olli@lurza.secnetix.de Subject: Re: comments on newfs raw disk ? Safe ? (7 terabyte array) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 13 Feb 2007 05:58:10 -0000 Oliver, On Mon, 12 Feb 2007, Oliver Fromme wrote: > > > You'll probably want to reduce the inode density (i.e. > > > increase the bytes-per-inode ratio). With the default > > > value, an fsck will be a royal pain, no matter whether you > > > use background fsck (with snapshots) or not. It might even > > > not work at all if you don't have a huge amount of RAM. > > > > Well, I have 4 GB of physical RAM, and 4 GB of swap - so does that > > total of 8 GB satisfy the "1 GB per TB" requirement, or do I really > > need >5.5 GB of actual swap space (in addition to the physical) ? > > That "1 GB per TB" requirement is just a rule of thumb. > I don't know hoe accurate it is. Also note that it is > desirable to avoid having fsck use swap, because it will > be even slower then. A lot slower. Ok, understood. But regardless of performance, fsck will use BOTH physical and swap, so as far as fsck is concerned, I have 8 GB of memory ? > # newfs -i 65536 > > That will leave room for about 15 million inodes per TB, > which is plenty for your needs. > > By the way, reducing the inode density like that will also > give your more space for actual file data. In UFS2, every > inode takes 256 bytes. Reducing the bytes-per-inode ratio > from 4 KB to 64 KB will give you additional 60 GB of space. > _And_ it will reduce the memory and time requirements of > fsck. Thank you - this is great advice. > > Well, I don't mind a 24 hour fsck, and I would like to remove > > complexity and not be so on the bleeding edge with things. Since I > > am only using 5mil inodes per TB anyway, that ends up being 25-30 > > million inodes in the 5 TB drive which I think could fsck in a day > > so. > > I suggest you test it before putting it into production, > i.e. populate the file system with the expected number of > files, then run fsck. Well, here is what I am assuming, and I would like to get some confirmation on these two points: - The time it takes to fsck is not a function of how many inodes are initialized from newfs, but how many you are _actually using_. - But the amount of memory the fsck takes is a function of how many inodes exist, regardless of how many you are actually using. Are these two interpretations correct ? > > I just need to know if my 4+4 GB of memory is enough, and if this > > option in loader.conf: > > > > kern.maxdsiz="2048000000" > > That will limit the process size to 2 GB. You might need > to set it higher if fsck needs more than that. (I assume > you're running FreeBSD/amd64, or otherwise you'll run into > process size limitations anyway.) Well ... no, I am using normal x86 FreeBSD on an Intel based system. I have 4 GB of physical ram, and 4 GB of swap. So I am tempted to just make that number 4096000000 and be done with it ... if fsck doesn't need that much memory, there is no harm to the system in simply having an inflated limit like that, is there ? I guess if I want to be safe and guard against a rogue, runaway, memory eating process, I could ratchet it up to (physical_ram - 256 megs). Which brings me to my last question: I understand why it's not useful to try to compute fsck _times_ - there are so many factors from disk speed to array speed to stripe size to population, etc. - who knows how long it will take. BUT, why isn't it possible to compute fsck _memory needs_ ? If I have a filesystem of A size with X inodes init'd, and Y inodes used, shouldn't I be able to compute how much memory fsck will need ? Thanks again. --------------------------------- No need to miss a message. Get email on-the-go with Yahoo! Mail for Mobile. Get started. From owner-freebsd-fs@FreeBSD.ORG Tue Feb 13 08:09:14 2007 Return-Path: X-Original-To: freebsd-fs@freebsd.org Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 6D88516A401 for ; Tue, 13 Feb 2007 08:09:14 +0000 (UTC) (envelope-from drumslayer2@yahoo.com) Received: from web34502.mail.mud.yahoo.com (web34502.mail.mud.yahoo.com [66.163.178.168]) by mx1.freebsd.org (Postfix) with SMTP id 3E59B13C467 for ; Tue, 13 Feb 2007 08:09:14 +0000 (UTC) (envelope-from drumslayer2@yahoo.com) Received: (qmail 68241 invoked by uid 60001); 13 Feb 2007 08:09:13 -0000 DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=s1024; d=yahoo.com; h=Message-ID:X-YMail-OSG:Received:Date:From:Subject:To:Cc:In-Reply-To:MIME-Version:Content-Type:Content-Transfer-Encoding; b=Lq401ePJ7TMZ/bviQNJSoNpyVpsd5UC+bFlEQ4gn5x1y6FJcYXr8OYvqp0dkGtEIUrOsfynq+fZG3PyLiY9cKKnVIb/ufnrRMykZcd/YwNmijIX2ZDG1cugZ4QMJs7t+KFFyiO0yRppvrdDomfOnUHZjxWUGjTcZfi6/Incj1d8= ; Message-ID: <20070213080913.68239.qmail@web34502.mail.mud.yahoo.com> X-YMail-OSG: T1IPApUVM1mBSn4gbePHONdViQjbiz18StsePHmrmUgKI00haWEQyz29ZEq0c.Fd0.mnGXMyqIttCQDM7q6Tt5GpvZlJnuDa.UK6EZRF66xhZpzhhPIfaDpVAsMsVFL8pYJJImufqJPEFesiYe7k0rFMibbCz.IyNw-- Received: from [67.112.21.27] by web34502.mail.mud.yahoo.com via HTTP; Tue, 13 Feb 2007 00:09:13 PST Date: Tue, 13 Feb 2007 00:09:13 -0800 (PST) From: Nicole Harrington To: Arone Silimantia In-Reply-To: <20070213055806.55404.qmail@web58601.mail.re3.yahoo.com> MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: 8bit Cc: freebsd-fs@freebsd.org, olli@lurza.secnetix.de Subject: Re: comments on newfs raw disk ? Safe ? (7 terabyte array) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 13 Feb 2007 08:09:14 -0000 --- Arone Silimantia wrote: > Oliver, > On Mon, 12 Feb 2007, Oliver Fromme wrote: > > > > > You'll probably want to reduce the inode > density (i.e. > > > > increase the bytes-per-inode ratio). With > the default > > > > value, an fsck will be a royal pain, no > matter whether you > > > > use background fsck (with snapshots) or not. > It might even > > > > not work at all if you don't have a huge > amount of RAM. > > > > > > Well, I have 4 GB of physical RAM, and 4 GB of > swap - so does that > > > total of 8 GB satisfy the "1 GB per TB" > requirement, or do I really > > > need >5.5 GB of actual swap space (in addition > to the physical) ? > > > > That "1 GB per TB" requirement is just a rule of > thumb. > > I don't know hoe accurate it is. Also note that > it is > > desirable to avoid having fsck use swap, because > it will > > be even slower then. A lot slower. > > > Ok, understood. But regardless of performance, fsck > will use > BOTH physical and swap, so as far as fsck is > concerned, I have 8 GB of > memory ? > > > > # newfs -i 65536 > > > > That will leave room for about 15 million inodes > per TB, > > which is plenty for your needs. > > > > By the way, reducing the inode density like that > will also > > give your more space for actual file data. In > UFS2, every > > inode takes 256 bytes. Reducing the > bytes-per-inode ratio > > from 4 KB to 64 KB will give you additional 60 GB > of space. > > _And_ it will reduce the memory and time > requirements of > > fsck. > > > Thank you - this is great advice. > > > > > > Well, I don't mind a 24 hour fsck, and I would > like to remove > > > complexity and not be so on the bleeding edge > with things. Since I > > > am only using 5mil inodes per TB anyway, that > ends up being 25-30 > > > million inodes in the 5 TB drive which I think > could fsck in a day > > > so. > > > > I suggest you test it before putting it into > production, > > i.e. populate the file system with the expected > number of > > files, then run fsck. > > > Well, here is what I am assuming, and I would like > to get some > confirmation on these two points: > > - The time it takes to fsck is not a function of how > many inodes are > initialized from newfs, but how many you are > _actually using_. > > - But the amount of memory the fsck takes is a > function of how many inodes > exist, regardless of how many you are actually > using. > > Are these two interpretations correct ? > > > > > I just need to know if my 4+4 GB of memory is > enough, and if this > > > option in loader.conf: > > > > > > kern.maxdsiz="2048000000" > > > > That will limit the process size to 2 GB. You > might need > > to set it higher if fsck needs more than that. > (I assume > > you're running FreeBSD/amd64, or otherwise you'll > run into > > process size limitations anyway.) > > > Well ... no, I am using normal x86 FreeBSD on an > Intel based system. I > have 4 GB of physical ram, and 4 GB of swap. So I > am tempted to just make > that number 4096000000 and be done with it ... if > fsck doesn't need > that much memory, there is no harm to the system in > simply having an > inflated limit like that, is there ? > > I guess if I want to be safe and guard against a > rogue, runaway, memory > eating process, I could ratchet it up to > (physical_ram - 256 megs). > > Which brings me to my last question: > > I understand why it's not useful to try to compute > fsck _times_ - there > are so many factors from disk speed to array speed > to stripe size to > population, etc. - who knows how long it will take. > > BUT, why isn't it possible to compute fsck _memory > needs_ ? If I have a > filesystem of A size with X inodes init'd, and Y > inodes used, shouldn't I > be able to compute how much memory fsck will need ? > > Thanks again. > Or, try not to worry about FSCK via a journaling file system. I have been using and having great success with File Journaling patches from: http://people.freebsd.org/~pjd/patches/ See gjournal patches. I really hope these get put into 6.3. They rock and it is something FreeBSD needs to stay competitive for large file systems. They no longer apply cleanly, but if you have any questions, feel free to ask me. There is also some basic info to be found vis goole for gjournal and old mail list postings. Nicole From owner-freebsd-fs@FreeBSD.ORG Tue Feb 13 08:10:35 2007 Return-Path: X-Original-To: freebsd-fs@FreeBSD.ORG Delivered-To: freebsd-fs@FreeBSD.ORG Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id C60AA16A402 for ; Tue, 13 Feb 2007 08:10:35 +0000 (UTC) (envelope-from olli@lurza.secnetix.de) Received: from lurza.secnetix.de (lurza.secnetix.de [83.120.8.8]) by mx1.freebsd.org (Postfix) with ESMTP id 323A113C471 for ; Tue, 13 Feb 2007 08:10:32 +0000 (UTC) (envelope-from olli@lurza.secnetix.de) Received: from lurza.secnetix.de (dapatm@localhost [127.0.0.1]) by lurza.secnetix.de (8.13.4/8.13.4) with ESMTP id l1D8AJmt026136; Tue, 13 Feb 2007 09:10:24 +0100 (CET) (envelope-from oliver.fromme@secnetix.de) Received: (from olli@localhost) by lurza.secnetix.de (8.13.4/8.13.1/Submit) id l1D8AIlZ026135; Tue, 13 Feb 2007 09:10:18 +0100 (CET) (envelope-from olli) Date: Tue, 13 Feb 2007 09:10:18 +0100 (CET) Message-Id: <200702130810.l1D8AIlZ026135@lurza.secnetix.de> From: Oliver Fromme To: freebsd-fs@FreeBSD.ORG, indigo@voda.cz, ivoras@fer.hr In-Reply-To: X-Newsgroups: list.freebsd-fs User-Agent: tin/1.8.2-20060425 ("Shillay") (UNIX) (FreeBSD/4.11-STABLE (i386)) MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 8bit X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-2.1.2 (lurza.secnetix.de [127.0.0.1]); Tue, 13 Feb 2007 09:10:24 +0100 (CET) Cc: Subject: Re: comments on newfs raw disk ? Safe ? (7 terabyte array) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: freebsd-fs@FreeBSD.ORG, indigo@voda.cz, ivoras@fer.hr List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 13 Feb 2007 08:10:35 -0000 Indigo wrote: > Ivan Voras wrote: > > Randy Bush wrote: > > > this thread has been great. but i suspect it would be greatly > > > appreciated if the handbook had a page "How to format and use > > > multi-terabyte drives, facts, trade-offs, and recipies." i am about to > > > do this (5TB), have been gathering info, and feel a appropriately > > > confused. > > > > It probably doesn't have to be a handbook chapter - I think an article > > would also be good, especially because it would contain "rule of thumb" > > information. I'll probably start making one during the next week or > > two... > > It would be nice to have more information about partitioning and > filesystem tuning in the handbook. Im not going to write it -but If anyone > ever decides to write it Id love to help in any way I can. Note that there's already quite some information about that in the tuning(7) manual page. Have you had a look at it? Best regards Oliver -- Oliver Fromme, secnetix GmbH & Co. KG, Marktplatz 29, 85567 Grafing b. M. Handelsregister: Registergericht Muenchen, HRA 74606, Geschäftsfuehrung: secnetix Verwaltungsgesellsch. mbH, Handelsregister: Registergericht Mün- chen, HRB 125758, Geschäftsführer: Maik Bachmann, Olaf Erb, Ralf Gebhart Any opinions expressed in this message are personal to the author and may not necessarily reflect the opinions of secnetix GmbH & Co KG in any way. FreeBSD-Dienstleistungen, -Produkte und mehr: http://www.secnetix.de/bsd "Perl will consistently give you what you want, unless what you want is consistency." -- Larry Wall From owner-freebsd-fs@FreeBSD.ORG Tue Feb 13 08:23:09 2007 Return-Path: X-Original-To: freebsd-fs@freebsd.org Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 0793816A401 for ; Tue, 13 Feb 2007 08:23:09 +0000 (UTC) (envelope-from drumslayer2@yahoo.com) Received: from web34502.mail.mud.yahoo.com (web34502.mail.mud.yahoo.com [66.163.178.168]) by mx1.freebsd.org (Postfix) with SMTP id C7F1913C481 for ; Tue, 13 Feb 2007 08:23:08 +0000 (UTC) (envelope-from drumslayer2@yahoo.com) Received: (qmail 63128 invoked by uid 60001); 13 Feb 2007 07:56:27 -0000 DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=s1024; d=yahoo.com; h=Message-ID:X-YMail-OSG:Received:Date:From:Subject:To:Cc:In-Reply-To:MIME-Version:Content-Type:Content-Transfer-Encoding; b=yzNEsKsI1A1rxtaZpTt0c37cgMtMesft6JJDRQ5jCC92L6AgsexAIYy72W7W4J8WpAFovB1b+1SN/keXKRX7SyiM3W0XhZXWZgqbbX8odLr3P040gsnQ0YeIwNZ0j4AsztwVUL5L1ypGVv+t12VmoOtfxS4VS1OiZ1qI1E3qf+E= ; Message-ID: <20070213075627.63126.qmail@web34502.mail.mud.yahoo.com> X-YMail-OSG: vRZqnqQVM1nlQhNE4k9lpp6VkHNZSUDZJkoD8AxYry8vLcNyXMsKMZM5TA195wC7lw-- Received: from [67.112.21.27] by web34502.mail.mud.yahoo.com via HTTP; Mon, 12 Feb 2007 23:56:27 PST Date: Mon, 12 Feb 2007 23:56:27 -0800 (PST) From: Nicole Harrington To: Eric Anderson , Chris Haulmark In-Reply-To: <45CD6FF5.8070007@freebsd.org> MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: 8bit Cc: freebsd-fs@freebsd.org Subject: Re: UFS2 with SAN X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 13 Feb 2007 08:23:09 -0000 --- Eric Anderson wrote: > On 02/10/07 00:54, Chris Haulmark wrote: > > > >> -----Original Message----- > >> From: Eric Anderson [mailto:anderson@freebsd.org] > >> Sent: Saturday, February 10, 2007 12:48 AM > >> To: Chris Haulmark > >> Cc: freebsd-fs@freebsd.org > >> Subject: Re: UFS2 with SAN > >> > >> On 02/09/07 19:30, Chris Haulmark wrote: > >>> Hello, > >>> > >>> I am looking into setting up a SAN with several > web servers that > >>> will be clustered. It would be a FC network > using Qlogic cards > >>> in each of those FreeBSD web servers. It would > be about 5+ > >>> of those web servers. > >>> > >>> I want to have the capability to share the same > web data across > >>> those web servers. I have scorched the entire > mailing list and > >>> found that there were some work on GFS porting > over to FreeBSD. > >>> It seems like that it is just all talk and if I > am wrong, could > >>> you have my head turned over to where I can find > out how to enable > >>> GFS on those FreeBSD systems. > >> GFS on FreeBSD is indeed dead. Not enough people > stepped up to help > >> port it. > > > > I really feared to hear that! > > > >>> If GFS is out of question, which file system am > I recommendeded > >>> to attempt to use for this SAN setup? > >> NFS. > >> > >>> My first thought to use UFS2 and attempt is to > allow only one web > >>> server to have a write/read access while the > reminder would be > >>> read only access. That should prevent from > lockings that is similar > >>> on NFS/NAS. > >> This will result it the read/write system seeing > the data ok, and the > >> rest getting corrupt data without knowing it, and > probably crashing. > >> UFS2 is not cluster aware. You could mount all > the hosts read only, > >> and > >> then update the mount point on one to rw, makes > changes, then back to > >> ro, then unmount/remount on the other boxes. > > > > That's my original idea if I do not have anything > else better to go > > with. > > > >> That's all still a kludge to simulate what NFS > will do for you. Why > >> won't NFS work for you? > > > > I have a client who wants to go from NAS to a true > SAN solution with > > full > > fibre channel network. I would hate to lose the > opportunity for this > > client > > to continue using FreeBSD as the choice of OS for > his web servers. > > Currently, > > his set up is using NAS with NFS. He complains of > locking files that > > occurs > > too often. > > > > I had hoped to find more better solution and make > this client much more > > happier > > with all the FreeBSD support that can be provided. > > > Well, I'm not sure what issues they had, but have > had fantastic success > with NFS and FreeBSD. FreeBSD with the right > hardware and tweaks can > make some NetApp boxes look weak. *cough* WAFL > *cough* > > > >> I agree that it would be fantastic to have a > clustered file system for > >> FreeBSD, and I've done lot's of hunting and > nagging vendors to support > >> it - but it's just not there. > > > > We should get few bandwagons and get in circle. > It could be likely that > > I could > > provide access for the developers to test and get > whatever file system > > and other > > necessaries needed to be working. :) > > > The problem isn't the environment or hardware, it's > developers skilled > to do the work. They're all either in NDA's, off > writing something > else, or just too busy to provide any amount of > input. > > Eric I have a set of servers NFS mounted to a Netapp and after hurs of tuning with netapp's help. (after getting through the idiots adking what FreeBSd was) I got very low performance. I was of course then told by Netapp to switch to Linux for better NFS support. I would love for any help with tuning this further, but I cannot say that FreeBSD with Netapp NFS will be great. Of course, I have not been able to test if indeed Linux would be any better. I will say however that I have a large number of small files which tends to not do well with NFS. Nicole From owner-freebsd-fs@FreeBSD.ORG Tue Feb 13 08:35:46 2007 Return-Path: X-Original-To: freebsd-fs@FreeBSD.ORG Delivered-To: freebsd-fs@FreeBSD.ORG Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id C775016A402 for ; Tue, 13 Feb 2007 08:35:46 +0000 (UTC) (envelope-from olli@lurza.secnetix.de) Received: from lurza.secnetix.de (lurza.secnetix.de [83.120.8.8]) by mx1.freebsd.org (Postfix) with ESMTP id 113C713C4AC for ; Tue, 13 Feb 2007 08:35:43 +0000 (UTC) (envelope-from olli@lurza.secnetix.de) Received: from lurza.secnetix.de (gfelop@localhost [127.0.0.1]) by lurza.secnetix.de (8.13.4/8.13.4) with ESMTP id l1D8ZaBW027347; Tue, 13 Feb 2007 09:35:42 +0100 (CET) (envelope-from oliver.fromme@secnetix.de) Received: (from olli@localhost) by lurza.secnetix.de (8.13.4/8.13.1/Submit) id l1D8Zams027346; Tue, 13 Feb 2007 09:35:36 +0100 (CET) (envelope-from olli) Date: Tue, 13 Feb 2007 09:35:36 +0100 (CET) Message-Id: <200702130835.l1D8Zams027346@lurza.secnetix.de> From: Oliver Fromme To: freebsd-fs@FreeBSD.ORG, aronesimi@yahoo.com In-Reply-To: <20070213055806.55404.qmail@web58601.mail.re3.yahoo.com> X-Newsgroups: list.freebsd-fs User-Agent: tin/1.8.2-20060425 ("Shillay") (UNIX) (FreeBSD/4.11-STABLE (i386)) MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 8bit X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-2.1.2 (lurza.secnetix.de [127.0.0.1]); Tue, 13 Feb 2007 09:35:42 +0100 (CET) Cc: Subject: Re: comments on newfs raw disk ? Safe ? (7 terabyte array) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: freebsd-fs@FreeBSD.ORG, aronesimi@yahoo.com List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 13 Feb 2007 08:35:46 -0000 Arone Silimantia wrote: > Oliver Fromme wrote: > > That "1 GB per TB" requirement is just a rule of thumb. > > I don't know hoe accurate it is. Also note that it is > > desirable to avoid having fsck use swap, because it will > > be even slower then. A lot slower. > > Ok, understood. But regardless of performance, fsck will use > BOTH physical and swap, Basically yes. fsck runs as a normal userland process, so it can use memory (RAM + swap) like any other programm, but it is also subject of the usual limitations (e.g. resource limits, address space limitations etc.). > so as far as fsck is concerned, I have 8 GB of > memory ? Only if you run a 64bit operating system (FreeBSD/amd64, /ia64 or /sparc64). In 32bit operating systems the address space is limited to 4 GB. Also note that the kernel needs some room from that address space, so the available space will be even smaller, usually 3 GB or less, depending on how your kernel is tuned. > > I suggest you test it before putting it into production, > > i.e. populate the file system with the expected number of > > files, then run fsck. > > Well, here is what I am assuming, and I would like to get some > confirmation on these two points: > > - The time it takes to fsck is not a function of how many inodes are > initialized from newfs, but how many you are _actually using_. > > - But the amount of memory the fsck takes is a function of how many inodes > exist, regardless of how many you are actually using. > > Are these two interpretations correct ? The answer is yeas and no. :-) I have to admit that I'm not 100% sure here, so please someone correct me if I'm wrong ... However, fsck runs several passes which do different things on the file system. One of the passes involves reading all directory information -- this pass is obviously dependant on the number of directories and files that are actually allocated on the file system. In another pass fsck checks for lost inodes -- this pass involves visiting _all_ inodes, no matter if they're currently marked as allocated or not. So you have both parameters in the equation, and they affect both the memory requirements and the run time of fsck. The exact function of inodes vs. memory/runtime is probably not very simple. That's why I suggested you try it yourself under the expected conditions before putting the machine into production. > > > I just need to know if my 4+4 GB of memory is enough, and if this > > > option in loader.conf: > > > > > > kern.maxdsiz="2048000000" > > > > That will limit the process size to 2 GB. You might need > > to set it higher if fsck needs more than that. (I assume > > you're running FreeBSD/amd64, or otherwise you'll run into > > process size limitations anyway.) > > Well ... no, I am using normal x86 FreeBSD on an Intel based system. I > have 4 GB of physical ram, and 4 GB of swap. So I am tempted to just make > that number 4096000000 and be done with it ... See my comment about 32bit vs. 64bit above. If you're running a 32bit OS (such as FreeBSD/i386), you have a 4GB address space limit, and it is shared between kernel and userland processes. Of course, every process has its own (virtual) address space, but the kernel virtual memory (KVM) is always mapped into it. So, for example, if the kernel uses 1 GB of KVM, then a single userland process can only be as big as 3 GB. (By the way, the PAE option does _not_ change the limit of the address space. It's still only 4 GB even with PAE.) > if fsck doesn't need > that much memory, there is no harm to the system in simply having an > inflated limit like that, is there ? Well, the process limits are useful for protection against run-away processes that just keep growing (because of a bug, an attack or other circumstances). If there's no limit, a single process can take the whol system down by using up all of its resources. However, there's a soft and a hard limit. The maxdsize parameter specifies the maximum hardlimit, so you can still have a lower soft limit for certain processes or users. You can modify the limits via /etc/login.conf. (The soft limit can only be increased up to the hard limit, and the hard limit can never be increased.) > BUT, why isn't it possible to compute fsck _memory needs_ ? If I have a > filesystem of A size with X inodes init'd, and Y inodes used, shouldn't I > be able to compute how much memory fsck will need ? Yes, in theory that should be possible. Either by carefully reading the fsck source code, or by running fsck on various test file systems and trying to build a function from the observed process sizes. However, it isn't _that_ trivial, because it also depends on the malloc implementation and on the malloc flags in use (e.g. via /etc/malloc.conf). By the way, I think fsck also records and checks the path names of all files, so those must be taken into the equation, too. Short names will take less space. I just did a "find /usr/src | wc" for testing, and it showed about 50,000 files, and the path names are 2 MB total. If you have 25,000,000 files with the same average file name length, those names alone will take 1 GB to store. (I'm assuming here that fsck indeed stores all the paths names at the same time. I don't know if it really does that. I haven't examined the source code closely.) Best regards Oliver -- Oliver Fromme, secnetix GmbH & Co. KG, Marktplatz 29, 85567 Grafing b. M. Handelsregister: Registergericht Muenchen, HRA 74606, Geschäftsfuehrung: secnetix Verwaltungsgesellsch. mbH, Handelsregister: Registergericht Mün- chen, HRB 125758, Geschäftsführer: Maik Bachmann, Olaf Erb, Ralf Gebhart Any opinions expressed in this message are personal to the author and may not necessarily reflect the opinions of secnetix GmbH & Co KG in any way. FreeBSD-Dienstleistungen, -Produkte und mehr: http://www.secnetix.de/bsd "It combines all the worst aspects of C and Lisp: a billion different sublanguages in one monolithic executable. It combines the power of C with the readability of PostScript." -- Jamie Zawinski, when asked: "What's wrong with perl?" From owner-freebsd-fs@FreeBSD.ORG Tue Feb 13 09:11:54 2007 Return-Path: X-Original-To: freebsd-fs@FreeBSD.ORG Delivered-To: freebsd-fs@FreeBSD.ORG Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 8574F16A402 for ; Tue, 13 Feb 2007 09:11:54 +0000 (UTC) (envelope-from olli@lurza.secnetix.de) Received: from lurza.secnetix.de (lurza.secnetix.de [83.120.8.8]) by mx1.freebsd.org (Postfix) with ESMTP id E5B2013C48E for ; Tue, 13 Feb 2007 09:11:53 +0000 (UTC) (envelope-from olli@lurza.secnetix.de) Received: from lurza.secnetix.de (spslgh@localhost [127.0.0.1]) by lurza.secnetix.de (8.13.4/8.13.4) with ESMTP id l1D9BkKD033953; Tue, 13 Feb 2007 10:11:51 +0100 (CET) (envelope-from oliver.fromme@secnetix.de) Received: (from olli@localhost) by lurza.secnetix.de (8.13.4/8.13.1/Submit) id l1D9Bkxl033952; Tue, 13 Feb 2007 10:11:46 +0100 (CET) (envelope-from olli) Date: Tue, 13 Feb 2007 10:11:46 +0100 (CET) Message-Id: <200702130911.l1D9Bkxl033952@lurza.secnetix.de> From: Oliver Fromme To: freebsd-fs@FreeBSD.ORG, des@des.no In-Reply-To: <86ire7gdnv.fsf@dwp.des.no> X-Newsgroups: list.freebsd-fs User-Agent: tin/1.8.2-20060425 ("Shillay") (UNIX) (FreeBSD/4.11-STABLE (i386)) MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 8bit X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-2.1.2 (lurza.secnetix.de [127.0.0.1]); Tue, 13 Feb 2007 10:11:52 +0100 (CET) Cc: Subject: Re: UFS2 with SAN X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: freebsd-fs@FreeBSD.ORG, des@des.no List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 13 Feb 2007 09:11:54 -0000 Dag-Erling Smørgrav wrote: > Eric Anderson writes: > > Well, I'm not sure what issues they had, but have had fantastic > > success with NFS and FreeBSD. FreeBSD with the right hardware and > > tweaks can make some NetApp boxes look weak. *cough* WAFL *cough* > > I'd be very surprised, considering that NetApp filers run FreeBSD. AFAIK they run a heavily modified NetBSD, but at least it's some sort of BSD. ;-) In fact I'm very satisfied with NetApp filers and their WAFL system. We have NetApp Filer clusters in production at customers, they exhibit exceptional reliability, even if a disk, a disk shelf, or an entire filer head fails, the cluster keeps running. Not to mention that it submits a support mail automatically and you get the replacement without having to do anything yourself. I'm afraid it is not possible to implement such a redundant NFS cluster with FreeBSD. Not to mention snapshots ... Years before FreeBSD intro- duced snapshots, WAFL supported them, and they work a whole lot better. Creating a snapshot takes just a few seconds, and file system access never freezes during that period. It works so well that you can create hourly snapshots for simple "undelete" usage without affecting the usability and performance of the file system at all. Best regards Oliver -- Oliver Fromme, secnetix GmbH & Co. KG, Marktplatz 29, 85567 Grafing b. M. Handelsregister: Registergericht Muenchen, HRA 74606, Geschäftsfuehrung: secnetix Verwaltungsgesellsch. mbH, Handelsregister: Registergericht Mün- chen, HRB 125758, Geschäftsführer: Maik Bachmann, Olaf Erb, Ralf Gebhart Any opinions expressed in this message are personal to the author and may not necessarily reflect the opinions of secnetix GmbH & Co KG in any way. FreeBSD-Dienstleistungen, -Produkte und mehr: http://www.secnetix.de/bsd Python is executable pseudocode. Perl is executable line noise. From owner-freebsd-fs@FreeBSD.ORG Tue Feb 13 09:13:53 2007 Return-Path: X-Original-To: freebsd-fs@FreeBSD.ORG Delivered-To: freebsd-fs@FreeBSD.ORG Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id BC17F16A400 for ; Tue, 13 Feb 2007 09:13:53 +0000 (UTC) (envelope-from randy@psg.com) Received: from rip.psg.com (rip.psg.com [147.28.0.39]) by mx1.freebsd.org (Postfix) with ESMTP id A0DE813C491 for ; Tue, 13 Feb 2007 09:13:53 +0000 (UTC) (envelope-from randy@psg.com) Received: from localhost ([127.0.0.1] helo=roam.psg.com) by rip.psg.com with esmtp (Exim 4.66 (FreeBSD)) (envelope-from ) id 1HGtjT-0007Yf-TV; Tue, 13 Feb 2007 09:13:52 +0000 Received: from localhost ([127.0.0.1] helo=roam.psg.com) by roam.psg.com with esmtp (Exim 4.66 (FreeBSD)) (envelope-from ) id 1HGtin-0007D2-Ca; Tue, 13 Feb 2007 01:13:09 -0800 From: Randy Bush MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Message-ID: <17873.33060.746546.773210@roam.psg.com> Date: Mon, 12 Feb 2007 23:13:08 -1000 To: Oliver Fromme References: <506226.11053.qm@web58610.mail.re3.yahoo.com> <200702121552.l1CFqi6q046650@lurza.secnetix.de> Cc: freebsd-fs@FreeBSD.ORG, aronesimi@yahoo.com Subject: Re: comments on newfs raw disk ? Safe ? (7 terabyte array) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 13 Feb 2007 09:13:53 -0000 this has been a wonderfully well-timed thread as i am about to hack a 4tb array tomorrow afternoon. the normal spindle is separate and partitioned to death and newfsed using the defaults. with 2gb of ram, i figure 6gb swap just in case two userland hogs are running at once, e.g. some hog while background fsck is running. the 4tb will be used as a dump/restore target only. so i am thinking few files, relatively big ones, little i/o and more write than read. so my current plan is newfs -b 16384 -f 2048 -i 262144 i would crank up even further, but these are the largest numbers mentioned in tuning(7). i will leave -m alone for now. does this seem reasonable? thank you all for this thread. randy From owner-freebsd-fs@FreeBSD.ORG Tue Feb 13 09:27:18 2007 Return-Path: X-Original-To: freebsd-fs@FreeBSD.ORG Delivered-To: freebsd-fs@FreeBSD.ORG Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id AE65416A401 for ; Tue, 13 Feb 2007 09:27:18 +0000 (UTC) (envelope-from olli@lurza.secnetix.de) Received: from lurza.secnetix.de (lurza.secnetix.de [83.120.8.8]) by mx1.freebsd.org (Postfix) with ESMTP id 150A113C48E for ; Tue, 13 Feb 2007 09:27:17 +0000 (UTC) (envelope-from olli@lurza.secnetix.de) Received: from lurza.secnetix.de (bmtuhk@localhost [127.0.0.1]) by lurza.secnetix.de (8.13.4/8.13.4) with ESMTP id l1D9RBl5034762; Tue, 13 Feb 2007 10:27:16 +0100 (CET) (envelope-from oliver.fromme@secnetix.de) Received: (from olli@localhost) by lurza.secnetix.de (8.13.4/8.13.1/Submit) id l1D9RBn9034761; Tue, 13 Feb 2007 10:27:11 +0100 (CET) (envelope-from olli) Date: Tue, 13 Feb 2007 10:27:11 +0100 (CET) Message-Id: <200702130927.l1D9RBn9034761@lurza.secnetix.de> From: Oliver Fromme To: freebsd-fs@FreeBSD.ORG, randy@psg.com, nesimi@yahoo.com In-Reply-To: <17873.33060.746546.773210@roam.psg.com> X-Newsgroups: list.freebsd-fs User-Agent: tin/1.8.2-20060425 ("Shillay") (UNIX) (FreeBSD/4.11-STABLE (i386)) MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 8bit X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-2.1.2 (lurza.secnetix.de [127.0.0.1]); Tue, 13 Feb 2007 10:27:17 +0100 (CET) Cc: Subject: Re: comments on newfs raw disk ? Safe ? (7 terabyte array) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: freebsd-fs@FreeBSD.ORG, randy@psg.com, nesimi@yahoo.com List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 13 Feb 2007 09:27:18 -0000 Randy Bush wrote: > this has been a wonderfully well-timed thread as i am about > to hack a 4tb array tomorrow afternoon. the normal spindle > is separate and partitioned to death and newfsed using the > defaults. with 2gb of ram, i figure 6gb swap just in case > two userland hogs are running at once, e.g. some hog while > background fsck is running. A bit careful here ... Background fsck had some issues, especially when the machine crashed or is otherwise reset while the background fsck is still running. It resulted in corruption that could not be repaired by fsck anymore. I don't know if all of those issues have been resolved in RELENG_6, but personally I always disable background fsck on all of my machines, just to be safe. Also note that background fsck will take longer than regular fscp, and it puts more stress on the disk, depending on what kind of applications run at the same time. > the 4tb will be used as a dump/restore target only. so i > am thinking few files, relatively big ones, little i/o and > more write than read. > > so my current plan is > > newfs -b 16384 -f 2048 -i 262144 > > i would crank up even further, but these are the largest > numbers mentioned in tuning(7). > > i will leave -m alone for now. > > does this seem reasonable? Yes, I think so. I would probably use the very same numbers in that case. I've also heard of people using 1 MB per inode (-i 1048576), but I haven't tried that high a number myself. -i 262144 works fine. In a thread some time ago, Matt Dillon commented on very high -i numbers: Matt Dillon wrote: > [...] > :> newsfeed-inn# newfs -i 67108864 /dev/twed0d > :> [stuff deleted] > :> 1048576032, 1048641568, 1048707104, 1048772640, 1048838176, 1048903712, > :> 1048969248, 1049034784, 1049100320, 1049165856, 1049231392, 1049296928, > :> 1049362464, 1049428000, 1049493536, 1049559072, 1049624608, 1049690144, > :> 1049755680, 1049821216, 1049886752, 1049952288, 1050017824, 1050083360, > :> 1050148896, 1050214432, 1050279968, 1050345504, 1050411040, 1050476576, > :> 1050542112, 1050607648, 1050673184, 1050738720, 1050804256, 1050869792, > :> 1050935328 > :> fsinit: inode value out of range (2). > :> > :> Tried larger -i parameters, same thing. > :> > :> Can't newfs figure this out before it gets to this point that something > :> isn't going to work? > :> > :> I'll try some different block/frag sizes, see if it helps. > > Specifying a byte-to-inode ratio that large is kind of silly. I > usually use something like -i 262144 -b 16384 -f 2048 on my larger > filesystems. A couple of million would also probably work, but beyond > that you aren't really saving anything, not even fsck time. I'm not > surprised that 67 million doesn't work. > > -Matt Best regards Oliver -- Oliver Fromme, secnetix GmbH & Co. KG, Marktplatz 29, 85567 Grafing b. M. Handelsregister: Registergericht Muenchen, HRA 74606, Geschäftsfuehrung: secnetix Verwaltungsgesellsch. mbH, Handelsregister: Registergericht Mün- chen, HRB 125758, Geschäftsführer: Maik Bachmann, Olaf Erb, Ralf Gebhart Any opinions expressed in this message are personal to the author and may not necessarily reflect the opinions of secnetix GmbH & Co KG in any way. FreeBSD-Dienstleistungen, -Produkte und mehr: http://www.secnetix.de/bsd "[...] one observation we can make here is that Python makes an excellent pseudocoding language, with the wonderful attribute that it can actually be executed." -- Bruce Eckel From owner-freebsd-fs@FreeBSD.ORG Tue Feb 13 16:00:48 2007 Return-Path: X-Original-To: freebsd-fs@freebsd.org Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 8629116A402 for ; Tue, 13 Feb 2007 16:00:48 +0000 (UTC) (envelope-from speedtoys.racing@gmail.com) Received: from ug-out-1314.google.com (ug-out-1314.google.com [66.249.92.170]) by mx1.freebsd.org (Postfix) with ESMTP id 74CA013C471 for ; Tue, 13 Feb 2007 16:00:45 +0000 (UTC) (envelope-from speedtoys.racing@gmail.com) Received: by ug-out-1314.google.com with SMTP id 71so217053ugh for ; Tue, 13 Feb 2007 08:00:43 -0800 (PST) DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=beta; h=received:message-id:date:from:to:subject:cc:in-reply-to:mime-version:content-type:references; b=ZfwnJJZwXESWYNAyayjd9dsABvszERPSQXPthY0r0DJqBqVuFczdCwqe9uu15kX6R+z8K8iQGpTazXnvTKI8KJp3/x9j6NS7cPJbY8yYakaMuvDwcUEVSXzhtEG36EAm88jMebYalnDotqVOKHdlbHnzmd0Y43bK4lv7w76xUSs= Received: by 10.78.106.3 with SMTP id e3mr82018huc.1171382442793; Tue, 13 Feb 2007 08:00:42 -0800 (PST) Received: by 10.78.17.18 with HTTP; Tue, 13 Feb 2007 08:00:42 -0800 (PST) Message-ID: Date: Tue, 13 Feb 2007 08:00:42 -0800 From: "Jeff Mohler" To: "Nicole Harrington" In-Reply-To: <20070213075627.63126.qmail@web34502.mail.mud.yahoo.com> MIME-Version: 1.0 References: <45CD6FF5.8070007@freebsd.org> <20070213075627.63126.qmail@web34502.mail.mud.yahoo.com> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit Content-Disposition: inline X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: freebsd-fs@freebsd.org, Chris Haulmark , Eric Anderson Subject: Re: UFS2 with SAN X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 13 Feb 2007 16:00:48 -0000 For certain workloads, linux is far better tuned for NFS than Fbsd..netapp as a server or not. On 2/12/07, Nicole Harrington wrote: > > --- Eric Anderson wrote: > > > On 02/10/07 00:54, Chris Haulmark wrote: > > > > > >> -----Original Message----- > > >> From: Eric Anderson [mailto:anderson@freebsd.org] > > >> Sent: Saturday, February 10, 2007 12:48 AM > > >> To: Chris Haulmark > > >> Cc: freebsd-fs@freebsd.org > > >> Subject: Re: UFS2 with SAN > > >> > > >> On 02/09/07 19:30, Chris Haulmark wrote: > > >>> Hello, > > >>> > > >>> I am looking into setting up a SAN with several > > web servers that > > >>> will be clustered. It would be a FC network > > using Qlogic cards > > >>> in each of those FreeBSD web servers. It would > > be about 5+ > > >>> of those web servers. > > >>> > > >>> I want to have the capability to share the same > > web data across > > >>> those web servers. I have scorched the entire > > mailing list and > > >>> found that there were some work on GFS porting > > over to FreeBSD. > > >>> It seems like that it is just all talk and if I > > am wrong, could > > >>> you have my head turned over to where I can find > > out how to enable > > >>> GFS on those FreeBSD systems. > > >> GFS on FreeBSD is indeed dead. Not enough people > > stepped up to help > > >> port it. > > > > > > I really feared to hear that! > > > > > >>> If GFS is out of question, which file system am > > I recommendeded > > >>> to attempt to use for this SAN setup? > > >> NFS. > > >> > > >>> My first thought to use UFS2 and attempt is to > > allow only one web > > >>> server to have a write/read access while the > > reminder would be > > >>> read only access. That should prevent from > > lockings that is similar > > >>> on NFS/NAS. > > >> This will result it the read/write system seeing > > the data ok, and the > > >> rest getting corrupt data without knowing it, and > > probably crashing. > > >> UFS2 is not cluster aware. You could mount all > > the hosts read only, > > >> and > > >> then update the mount point on one to rw, makes > > changes, then back to > > >> ro, then unmount/remount on the other boxes. > > > > > > That's my original idea if I do not have anything > > else better to go > > > with. > > > > > >> That's all still a kludge to simulate what NFS > > will do for you. Why > > >> won't NFS work for you? > > > > > > I have a client who wants to go from NAS to a true > > SAN solution with > > > full > > > fibre channel network. I would hate to lose the > > opportunity for this > > > client > > > to continue using FreeBSD as the choice of OS for > > his web servers. > > > Currently, > > > his set up is using NAS with NFS. He complains of > > locking files that > > > occurs > > > too often. > > > > > > I had hoped to find more better solution and make > > this client much more > > > happier > > > with all the FreeBSD support that can be provided. > > > > > > Well, I'm not sure what issues they had, but have > > had fantastic success > > with NFS and FreeBSD. FreeBSD with the right > > hardware and tweaks can > > make some NetApp boxes look weak. *cough* WAFL > > *cough* > > > > > > >> I agree that it would be fantastic to have a > > clustered file system for > > >> FreeBSD, and I've done lot's of hunting and > > nagging vendors to support > > >> it - but it's just not there. > > > > > > We should get few bandwagons and get in circle. > > It could be likely that > > > I could > > > provide access for the developers to test and get > > whatever file system > > > and other > > > necessaries needed to be working. :) > > > > > > The problem isn't the environment or hardware, it's > > developers skilled > > to do the work. They're all either in NDA's, off > > writing something > > else, or just too busy to provide any amount of > > input. > > > > Eric > > > I have a set of servers NFS mounted to a Netapp and > after hurs of tuning with netapp's help. (after > getting through the idiots adking what FreeBSd was) > I got very low performance. I was of course then told > by Netapp to switch to Linux for better NFS support. > > I would love for any help with tuning this further, > but I cannot say that FreeBSD with Netapp NFS will be > great. Of course, I have not been able to test if > indeed Linux would be any better. > I will say however that I have a large number of > small files which tends to not do well with NFS. > > > Nicole > > > _______________________________________________ > freebsd-fs@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" > From owner-freebsd-fs@FreeBSD.ORG Tue Feb 13 16:30:50 2007 Return-Path: X-Original-To: freebsd-fs@freebsd.org Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id B9FDE16A401; Tue, 13 Feb 2007 16:30:50 +0000 (UTC) (envelope-from chris@sigd.net) Received: from ms05.mailstreet2003.net (ms05.mailstreet2003.net [69.25.50.235]) by mx1.freebsd.org (Postfix) with ESMTP id 8A54513C491; Tue, 13 Feb 2007 16:30:50 +0000 (UTC) (envelope-from chris@sigd.net) X-MimeOLE: Produced By Microsoft Exchange V6.5.7226.0 Content-class: urn:content-classes:message MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Date: Tue, 13 Feb 2007 11:30:49 -0500 Message-ID: <6FC9F9894A9F8C49A722CF9F2132FC2204C9DAB4@ms05.mailstreet2003.net> In-Reply-To: <20070213075627.63126.qmail@web34502.mail.mud.yahoo.com> X-MS-Has-Attach: X-MS-TNEF-Correlator: Thread-Topic: UFS2 with SAN Thread-Index: AcdPRH0nhuQ8BExxQVu84pHXJ42ALgARzuug References: <45CD6FF5.8070007@freebsd.org> <20070213075627.63126.qmail@web34502.mail.mud.yahoo.com> From: "Chris Haulmark" To: "Nicole Harrington" , "Eric Anderson" Cc: freebsd-fs@freebsd.org Subject: RE: UFS2 with SAN X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 13 Feb 2007 16:30:50 -0000 > -----Original Message----- > From: Nicole Harrington [mailto:drumslayer2@yahoo.com] > Sent: Tuesday, February 13, 2007 1:56 AM > To: Eric Anderson; Chris Haulmark > Cc: freebsd-fs@freebsd.org > Subject: Re: UFS2 with SAN >=20 > --- Eric Anderson wrote: >=20 > > On 02/10/07 00:54, Chris Haulmark wrote: > > > > > >> -----Original Message----- > > >> From: Eric Anderson [mailto:anderson@freebsd.org] > > >> Sent: Saturday, February 10, 2007 12:48 AM > > >> To: Chris Haulmark > > >> Cc: freebsd-fs@freebsd.org > > >> Subject: Re: UFS2 with SAN > > >> > > >> On 02/09/07 19:30, Chris Haulmark wrote: > > >>> Hello, > > >>> > > >>> I am looking into setting up a SAN with several > > web servers that > > >>> will be clustered. It would be a FC network > > using Qlogic cards > > >>> in each of those FreeBSD web servers. It would > > be about 5+ > > >>> of those web servers. > > >>> > > >>> I want to have the capability to share the same > > web data across > > >>> those web servers. I have scorched the entire > > mailing list and > > >>> found that there were some work on GFS porting > > over to FreeBSD. > > >>> It seems like that it is just all talk and if I > > am wrong, could > > >>> you have my head turned over to where I can find > > out how to enable > > >>> GFS on those FreeBSD systems. > > >> GFS on FreeBSD is indeed dead. Not enough people > > stepped up to help > > >> port it. > > > > > > I really feared to hear that! > > > > > >>> If GFS is out of question, which file system am > > I recommendeded > > >>> to attempt to use for this SAN setup? > > >> NFS. > > >> > > >>> My first thought to use UFS2 and attempt is to > > allow only one web > > >>> server to have a write/read access while the > > reminder would be > > >>> read only access. That should prevent from > > lockings that is similar > > >>> on NFS/NAS. > > >> This will result it the read/write system seeing > > the data ok, and the > > >> rest getting corrupt data without knowing it, and > > probably crashing. > > >> UFS2 is not cluster aware. You could mount all > > the hosts read only, > > >> and > > >> then update the mount point on one to rw, makes > > changes, then back to > > >> ro, then unmount/remount on the other boxes. > > > > > > That's my original idea if I do not have anything > > else better to go > > > with. > > > > > >> That's all still a kludge to simulate what NFS > > will do for you. Why > > >> won't NFS work for you? > > > > > > I have a client who wants to go from NAS to a true > > SAN solution with > > > full > > > fibre channel network. I would hate to lose the > > opportunity for this > > > client > > > to continue using FreeBSD as the choice of OS for > > his web servers. > > > Currently, > > > his set up is using NAS with NFS. He complains of > > locking files that > > > occurs > > > too often. > > > > > > I had hoped to find more better solution and make > > this client much more > > > happier > > > with all the FreeBSD support that can be provided. > > > > > > Well, I'm not sure what issues they had, but have > > had fantastic success > > with NFS and FreeBSD. FreeBSD with the right > > hardware and tweaks can > > make some NetApp boxes look weak. *cough* WAFL > > *cough* > > > > > > >> I agree that it would be fantastic to have a > > clustered file system for > > >> FreeBSD, and I've done lot's of hunting and > > nagging vendors to support > > >> it - but it's just not there. > > > > > > We should get few bandwagons and get in circle. > > It could be likely that > > > I could > > > provide access for the developers to test and get > > whatever file system > > > and other > > > necessaries needed to be working. :) > > > > > > The problem isn't the environment or hardware, it's > > developers skilled > > to do the work. They're all either in NDA's, off > > writing something > > else, or just too busy to provide any amount of > > input. > > > > Eric >=20 >=20 > I have a set of servers NFS mounted to a Netapp and > after hurs of tuning with netapp's help. (after > getting through the idiots adking what FreeBSd was) > I got very low performance. I was of course then told > by Netapp to switch to Linux for better NFS support. That is what I would like to avoid telling my client to do The same thing. "Stay with NFS and tolerate it." I had hoped a SAN solution would be possible for FreeBSD. So far, it appears that it is not possible to share the same file system across several web servers. Chris >=20 > I would love for any help with tuning this further, > but I cannot say that FreeBSD with Netapp NFS will be > great. Of course, I have not been able to test if > indeed Linux would be any better. > I will say however that I have a large number of > small files which tends to not do well with NFS. >=20 >=20 > Nicole >=20 From owner-freebsd-fs@FreeBSD.ORG Tue Feb 13 16:39:40 2007 Return-Path: X-Original-To: freebsd-fs@freebsd.org Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id BE6C416A407 for ; Tue, 13 Feb 2007 16:39:40 +0000 (UTC) (envelope-from speedtoys.racing@gmail.com) Received: from hu-out-0506.google.com (hu-out-0506.google.com [72.14.214.227]) by mx1.freebsd.org (Postfix) with ESMTP id 16D8213C4A7 for ; Tue, 13 Feb 2007 16:39:39 +0000 (UTC) (envelope-from speedtoys.racing@gmail.com) Received: by hu-out-0506.google.com with SMTP id 38so3303974huc for ; Tue, 13 Feb 2007 08:39:38 -0800 (PST) DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=beta; h=received:message-id:date:from:to:subject:cc:in-reply-to:mime-version:content-type:references; b=TYnvER8RvYwWoBiuRohhO20CMZzBcnZL6LYlIEd3FDy/vp0rkTe5EuL5H0+F7Hb0OTLZuNkjUnHFOlt6alBWzEaZ0Pm/KhbVDzdDLxOPVcFb2F8BECT337dEHrfGsev/ncYF4dQUysbOWeqIlcalM79qUtHVcYhOzGdkN2yVnag= Received: by 10.78.200.3 with SMTP id x3mr89374huf.1171384778598; Tue, 13 Feb 2007 08:39:38 -0800 (PST) Received: by 10.78.17.18 with HTTP; Tue, 13 Feb 2007 08:39:38 -0800 (PST) Message-ID: Date: Tue, 13 Feb 2007 08:39:38 -0800 From: "Jeff Mohler" To: "Chris Haulmark" In-Reply-To: <6FC9F9894A9F8C49A722CF9F2132FC2204C9DAB4@ms05.mailstreet2003.net> MIME-Version: 1.0 References: <45CD6FF5.8070007@freebsd.org> <20070213075627.63126.qmail@web34502.mail.mud.yahoo.com> <6FC9F9894A9F8C49A722CF9F2132FC2204C9DAB4@ms05.mailstreet2003.net> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit Content-Disposition: inline X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: freebsd-fs@freebsd.org, Eric Anderson Subject: Re: UFS2 with SAN X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 13 Feb 2007 16:39:40 -0000 Its pretty much that simple. You cannot share SAN data..well..you -can- make a LUN appear as a shared NFS or CIFS share on a Netapp, but I havent tried it in a while..but you -can-. But in the normal world..you cannot...you would have to make it available via NFS to other client. Thats the key difference between SAN and NAS. On 2/13/07, Chris Haulmark wrote: > > > > > -----Original Message----- > > From: Nicole Harrington [mailto:drumslayer2@yahoo.com] > > Sent: Tuesday, February 13, 2007 1:56 AM > > To: Eric Anderson; Chris Haulmark > > Cc: freebsd-fs@freebsd.org > > Subject: Re: UFS2 with SAN > > > > --- Eric Anderson wrote: > > > > > On 02/10/07 00:54, Chris Haulmark wrote: > > > > > > > >> -----Original Message----- > > > >> From: Eric Anderson [mailto:anderson@freebsd.org] > > > >> Sent: Saturday, February 10, 2007 12:48 AM > > > >> To: Chris Haulmark > > > >> Cc: freebsd-fs@freebsd.org > > > >> Subject: Re: UFS2 with SAN > > > >> > > > >> On 02/09/07 19:30, Chris Haulmark wrote: > > > >>> Hello, > > > >>> > > > >>> I am looking into setting up a SAN with several > > > web servers that > > > >>> will be clustered. It would be a FC network > > > using Qlogic cards > > > >>> in each of those FreeBSD web servers. It would > > > be about 5+ > > > >>> of those web servers. > > > >>> > > > >>> I want to have the capability to share the same > > > web data across > > > >>> those web servers. I have scorched the entire > > > mailing list and > > > >>> found that there were some work on GFS porting > > > over to FreeBSD. > > > >>> It seems like that it is just all talk and if I > > > am wrong, could > > > >>> you have my head turned over to where I can find > > > out how to enable > > > >>> GFS on those FreeBSD systems. > > > >> GFS on FreeBSD is indeed dead. Not enough people > > > stepped up to help > > > >> port it. > > > > > > > > I really feared to hear that! > > > > > > > >>> If GFS is out of question, which file system am > > > I recommendeded > > > >>> to attempt to use for this SAN setup? > > > >> NFS. > > > >> > > > >>> My first thought to use UFS2 and attempt is to > > > allow only one web > > > >>> server to have a write/read access while the > > > reminder would be > > > >>> read only access. That should prevent from > > > lockings that is similar > > > >>> on NFS/NAS. > > > >> This will result it the read/write system seeing > > > the data ok, and the > > > >> rest getting corrupt data without knowing it, and > > > probably crashing. > > > >> UFS2 is not cluster aware. You could mount all > > > the hosts read only, > > > >> and > > > >> then update the mount point on one to rw, makes > > > changes, then back to > > > >> ro, then unmount/remount on the other boxes. > > > > > > > > That's my original idea if I do not have anything > > > else better to go > > > > with. > > > > > > > >> That's all still a kludge to simulate what NFS > > > will do for you. Why > > > >> won't NFS work for you? > > > > > > > > I have a client who wants to go from NAS to a true > > > SAN solution with > > > > full > > > > fibre channel network. I would hate to lose the > > > opportunity for this > > > > client > > > > to continue using FreeBSD as the choice of OS for > > > his web servers. > > > > Currently, > > > > his set up is using NAS with NFS. He complains of > > > locking files that > > > > occurs > > > > too often. > > > > > > > > I had hoped to find more better solution and make > > > this client much more > > > > happier > > > > with all the FreeBSD support that can be provided. > > > > > > > > > Well, I'm not sure what issues they had, but have > > > had fantastic success > > > with NFS and FreeBSD. FreeBSD with the right > > > hardware and tweaks can > > > make some NetApp boxes look weak. *cough* WAFL > > > *cough* > > > > > > > > > >> I agree that it would be fantastic to have a > > > clustered file system for > > > >> FreeBSD, and I've done lot's of hunting and > > > nagging vendors to support > > > >> it - but it's just not there. > > > > > > > > We should get few bandwagons and get in circle. > > > It could be likely that > > > > I could > > > > provide access for the developers to test and get > > > whatever file system > > > > and other > > > > necessaries needed to be working. :) > > > > > > > > > The problem isn't the environment or hardware, it's > > > developers skilled > > > to do the work. They're all either in NDA's, off > > > writing something > > > else, or just too busy to provide any amount of > > > input. > > > > > > Eric > > > > > > I have a set of servers NFS mounted to a Netapp and > > after hurs of tuning with netapp's help. (after > > getting through the idiots adking what FreeBSd was) > > I got very low performance. I was of course then told > > by Netapp to switch to Linux for better NFS support. > > That is what I would like to avoid telling my client to do > The same thing. "Stay with NFS and tolerate it." > > I had hoped a SAN solution would be possible for > FreeBSD. So far, it appears that it is not possible to > share the same file system across several web servers. > > Chris > > > > I would love for any help with tuning this further, > > but I cannot say that FreeBSD with Netapp NFS will be > > great. Of course, I have not been able to test if > > indeed Linux would be any better. > > I will say however that I have a large number of > > small files which tends to not do well with NFS. > > > > > > Nicole > > > > _______________________________________________ > freebsd-fs@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" > From owner-freebsd-fs@FreeBSD.ORG Tue Feb 13 17:00:25 2007 Return-Path: X-Original-To: freebsd-fs@freebsd.org Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 1995016A401; Tue, 13 Feb 2007 17:00:25 +0000 (UTC) (envelope-from chris@sigd.net) Received: from ms05.mailstreet2003.net (ms05.mailstreet2003.net [69.25.50.235]) by mx1.freebsd.org (Postfix) with ESMTP id DA40D13C4A7; Tue, 13 Feb 2007 17:00:22 +0000 (UTC) (envelope-from chris@sigd.net) X-MimeOLE: Produced By Microsoft Exchange V6.5.7226.0 Content-class: urn:content-classes:message MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 Date: Tue, 13 Feb 2007 12:00:22 -0500 Message-ID: <6FC9F9894A9F8C49A722CF9F2132FC2204C9DAB6@ms05.mailstreet2003.net> In-Reply-To: <6FC9F9894A9F8C49A722CF9F2132FC2204C9DAB5@ms05.mailstreet2003.net> X-MS-Has-Attach: X-MS-TNEF-Correlator: Thread-Topic: UFS2 with SAN Thread-Index: AcdPjZH8o+oiJtW/T+uEWt6ASumE1wAAYbVQAAADd3A= References: <45CD6FF5.8070007@freebsd.org> <20070213075627.63126.qmail@web34502.mail.mud.yahoo.com> <6FC9F9894A9F8C49A722CF9F2132FC2204C9DAB4@ms05.mailstreet2003.net> <6FC9F9894A9F8C49A722CF9F2132FC2204C9DAB5@ms05.mailstreet2003.net> From: "Chris Haulmark" To: "Jeff Mohler" Cc: freebsd-fs@freebsd.org, Eric Anderson Subject: RE: UFS2 with SAN X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 13 Feb 2007 17:00:25 -0000 DQo+IEZyb206IEplZmYgTW9obGVyIFttYWlsdG86c3BlZWR0b3lzLnJhY2luZ0BnbWFpbC5jb21d DQo+IFNlbnQ6IFR1ZXNkYXksIEZlYnJ1YXJ5IDEzLCAyMDA3IDEwOjQwIEFNDQo+IFRvOiBDaHJp cyBIYXVsbWFyaw0KPiBDYzogTmljb2xlIEhhcnJpbmd0b247IEVyaWMgQW5kZXJzb247IGZyZWVi c2QtZnNAZnJlZWJzZC5vcmcNCj4gU3ViamVjdDogUmU6IFVGUzIgd2l0aCBTQU4NCj4gDQo+IEl0 cyBwcmV0dHkgbXVjaCB0aGF0IHNpbXBsZS4NCj4gDQo+IFlvdSBjYW5ub3Qgc2hhcmUgU0FOIGRh dGEuLndlbGwuLnlvdSAtY2FuLSBtYWtlIGEgTFVOIGFwcGVhciBhcyBhDQo+IHNoYXJlZCBORlMg b3IgQ0lGUyBzaGFyZSBvbiBhIE5ldGFwcCwgYnV0IEkgaGF2ZW50IHRyaWVkIGl0IGluIGENCj4g d2hpbGUuLmJ1dCB5b3UgLWNhbi0uDQoNCkkgc2V0IHVwIGEgTGludXggY2x1c3RlciB3aXRoIEdG UyBvbiBhIHNpbmdsZSBTQU4gZmlsZSBzeXN0ZW0uICBJdA0KaW5jbHVkZWQgOCBub2Rlcy4NCg0K PiANCj4gQnV0IGluIHRoZSBub3JtYWwgd29ybGQuLnlvdSBjYW5ub3QuLi55b3Ugd291bGQgaGF2 ZSB0byBtYWtlIGl0DQo+IGF2YWlsYWJsZSB2aWEgTkZTIHRvIG90aGVyIGNsaWVudC7CoMKgIFRo YXRzIHRoZSBrZXkgZGlmZmVyZW5jZSBiZXR3ZWVuDQo+IFNBTiBhbmQgTkFTLg0KDQpOby4gIFNB TiBhbmQgTkFTIGlzIGRpZmZlcmVudCBiYXNlZCBvbiB0aGVpciBhY2Nlc3MuICBTQU4gd2FzIGRl cGVuZGVudA0Kb24gZWl0aGVyIFNDU0kgb3IgRmlicmUgdW50aWwgaVNDU0kgKElQU0FOKSB3YXMg aW50cm9kdWNlZC4gIE5BUyB3aWxsDQphbHdheXMgcmVxdWlyZSBhbiBJUCBhZGRyZXNzLiAgSWYg eW91IHdlcmUgdXNpbmcgRmlicmUgYW5kIHdhbnRlZCB0bw0KdXNlIE5BUywgeW91IHdvdWxkIGhh dmUgdG8gdXNlIGlGQ1Agb3IgRkNJUCBwcm90b2NvbHMuICBJIGRvIG5vdCBuZWVkIHRvDQpleHBs YWluIHRoZSBsaW1pdGF0aW9uIG9mIHVzaW5nIFNDU0kgd2l0aCBOQVMuDQoNCklmIHlvdXIgcmVz cG9uc2VzIG9uIHRoaXMgU0FOIHRocmVhZCB3b3VsZCBub3QgYmUgcHJvZHVjdGl2ZSwgcGxlYXNl DQpzdGF5IG91dC4gIEkgYW0gbm90IGludGVyZXN0ZWQgdG8gaGVhciBsZWN0dXJlcyBhYm91dCB3 aGF0IGlzIGltcG9zc2libGUuDQoNCkkgYXNrZWQgYWJvdXQgaWYgYW55b25lIGhhcyB0cmllZCB0 byB1c2UgVUZTMiB3aXRoIG9ubHkgb25lIG5vZGUgdG8NCmhhdmUgd3JpdGUvcmVhZCBvbmx5IHdo aWxlIHRoZSByZXN0IHdvdWxkIGJlIHJlYWQgb25seS4NCg0KVGhhbmsgeW91LA0KDQpDaHJpcw0K DQo+IE9uIDIvMTMvMDcsIENocmlzIEhhdWxtYXJrIDxjaHJpc0BzaWdkLm5ldD4gd3JvdGU6DQo+ IA0KPiANCj4gPiAtLS0tLU9yaWdpbmFsIE1lc3NhZ2UtLS0tLQ0KPiA+IEZyb206IE5pY29sZSBI YXJyaW5ndG9uIFttYWlsdG86ZHJ1bXNsYXllcjJAeWFob28uY29tXQ0KPiA+IFNlbnQ6IFR1ZXNk YXksIEZlYnJ1YXJ5IDEzLCAyMDA3IDE6NTYgQU0NCj4gPiBUbzogRXJpYyBBbmRlcnNvbjsgQ2hy aXMgSGF1bG1hcmsNCj4gPiBDYzogZnJlZWJzZC1mc0BmcmVlYnNkLm9yZw0KPiA+IFN1YmplY3Q6 IFJlOiBVRlMyIHdpdGggU0FODQo+ID4NCj4gPiAtLS0gRXJpYyBBbmRlcnNvbiA8YW5kZXJzb25A ZnJlZWJzZC5vcmc+IHdyb3RlOg0KPiA+DQo+ID4gPiBPbiAwMi8xMC8wNyAwMDo1NCwgQ2hyaXMg SGF1bG1hcmsgd3JvdGU6DQo+ID4gPiA+DQo+ID4gPiA+PiAtLS0tLU9yaWdpbmFsIE1lc3NhZ2Ut LS0tLQ0KPiA+ID4gPj4gRnJvbTogRXJpYyBBbmRlcnNvbiBbbWFpbHRvOmFuZGVyc29uQGZyZWVi c2Qub3JnIF0NCj4gPiA+ID4+IFNlbnQ6IFNhdHVyZGF5LCBGZWJydWFyeSAxMCwgMjAwNyAxMjo0 OCBBTQ0KPiA+ID4gPj4gVG86IENocmlzIEhhdWxtYXJrDQo+ID4gPiA+PiBDYzogZnJlZWJzZC1m c0BmcmVlYnNkLm9yZw0KPiA+ID4gPj4gU3ViamVjdDogUmU6IFVGUzIgd2l0aCBTQU4NCj4gPiA+ ID4+DQo+ID4gPiA+PiBPbiAwMi8wOS8wNyAxOTozMCwgQ2hyaXMgSGF1bG1hcmsgd3JvdGU6DQo+ ID4gPiA+Pj4gSGVsbG8sDQo+ID4gPiA+Pj4NCj4gPiA+ID4+PiBJIGFtIGxvb2tpbmcgaW50byBz ZXR0aW5nIHVwIGEgU0FOIHdpdGggc2V2ZXJhbA0KPiA+ID4gd2ViIHNlcnZlcnMgdGhhdA0KPiA+ ID4gPj4+IHdpbGwgYmUgY2x1c3RlcmVkLsKgwqBJdCB3b3VsZCBiZSBhIEZDIG5ldHdvcmsNCj4g PiA+IHVzaW5nIFFsb2dpYyBjYXJkcw0KPiA+ID4gPj4+IGluIGVhY2ggb2YgdGhvc2UgRnJlZUJT RCB3ZWIgc2VydmVycy7CoMKgSXQgd291bGQNCj4gPiA+IGJlIGFib3V0IDUrDQo+ID4gPiA+Pj4g b2YgdGhvc2Ugd2ViIHNlcnZlcnMuDQo+ID4gPiA+Pj4NCj4gPiA+ID4+PiBJIHdhbnQgdG8gaGF2 ZSB0aGUgY2FwYWJpbGl0eSB0byBzaGFyZSB0aGUgc2FtZQ0KPiA+ID4gd2ViIGRhdGEgYWNyb3Nz DQo+ID4gPiA+Pj4gdGhvc2Ugd2ViIHNlcnZlcnMuwqDCoEkgaGF2ZSBzY29yY2hlZCB0aGUgZW50 aXJlDQo+ID4gPiBtYWlsaW5nIGxpc3QgYW5kDQo+ID4gPiA+Pj4gZm91bmQgdGhhdCB0aGVyZSB3 ZXJlIHNvbWUgd29yayBvbiBHRlMgcG9ydGluZw0KPiA+ID4gb3ZlciB0byBGcmVlQlNELg0KPiA+ ID4gPj4+IEl0IHNlZW1zIGxpa2UgdGhhdCBpdCBpcyBqdXN0IGFsbCB0YWxrIGFuZCBpZiBJDQo+ ID4gPiBhbSB3cm9uZywgY291bGQNCj4gPiA+ID4+PiB5b3UgaGF2ZSBteSBoZWFkIHR1cm5lZCBv dmVyIHRvIHdoZXJlIEkgY2FuIGZpbmQNCj4gPiA+IG91dCBob3cgdG8gZW5hYmxlDQo+ID4gPiA+ Pj4gR0ZTIG9uIHRob3NlIEZyZWVCU0Qgc3lzdGVtcy4NCj4gPiA+ID4+IEdGUyBvbiBGcmVlQlNE IGlzIGluZGVlZCBkZWFkLsKgwqBOb3QgZW5vdWdoIHBlb3BsZQ0KPiA+ID4gc3RlcHBlZCB1cCB0 byBoZWxwDQo+ID4gPiA+PiBwb3J0IGl0Lg0KPiA+ID4gPg0KPiA+ID4gPiBJIHJlYWxseSBmZWFy ZWQgdG8gaGVhciB0aGF0IQ0KPiA+ID4gPg0KPiA+ID4gPj4+IElmIEdGUyBpcyBvdXQgb2YgcXVl c3Rpb24sIHdoaWNoIGZpbGUgc3lzdGVtIGFtDQo+ID4gPiBJIHJlY29tbWVuZGVkZWQNCj4gPiA+ ID4+PiB0byBhdHRlbXB0IHRvIHVzZSBmb3IgdGhpcyBTQU4gc2V0dXA/DQo+ID4gPiA+PiBORlMu DQo+ID4gPiA+Pg0KPiA+ID4gPj4+IE15IGZpcnN0IHRob3VnaHQgdG8gdXNlIFVGUzIgYW5kIGF0 dGVtcHQgaXMgdG8NCj4gPiA+IGFsbG93IG9ubHkgb25lIHdlYg0KPiA+ID4gPj4+IHNlcnZlciB0 byBoYXZlIGEgd3JpdGUvcmVhZCBhY2Nlc3Mgd2hpbGUgdGhlDQo+ID4gPiByZW1pbmRlciB3b3Vs ZCBiZQ0KPiA+ID4gPj4+IHJlYWQgb25seSBhY2Nlc3MuIFRoYXQgc2hvdWxkIHByZXZlbnQgZnJv bQ0KPiA+ID4gbG9ja2luZ3MgdGhhdCBpcyBzaW1pbGFyDQo+ID4gPiA+Pj4gb24gTkZTL05BUy4N Cj4gPiA+ID4+IFRoaXMgd2lsbCByZXN1bHQgaXQgdGhlIHJlYWQvd3JpdGUgc3lzdGVtIHNlZWlu Zw0KPiA+ID4gdGhlIGRhdGEgb2ssIGFuZCB0aGUNCj4gPiA+ID4+IHJlc3QgZ2V0dGluZyBjb3Jy dXB0IGRhdGEgd2l0aG91dCBrbm93aW5nIGl0LCBhbmQNCj4gPiA+IHByb2JhYmx5IGNyYXNoaW5n Lg0KPiA+ID4gPj4gVUZTMiBpcyBub3QgY2x1c3RlciBhd2FyZS7CoMKgWW91IGNvdWxkIG1vdW50 IGFsbA0KPiA+ID4gdGhlIGhvc3RzIHJlYWQgb25seSwNCj4gPiA+ID4+IGFuZA0KPiA+ID4gPj4g dGhlbiB1cGRhdGUgdGhlIG1vdW50IHBvaW50IG9uIG9uZSB0byBydywgbWFrZXMNCj4gPiA+IGNo YW5nZXMsIHRoZW4gYmFjayB0bw0KPiA+ID4gPj4gcm8sIHRoZW4gdW5tb3VudC9yZW1vdW50IG9u IHRoZSBvdGhlciBib3hlcy4NCj4gPiA+ID4NCj4gPiA+ID4gVGhhdCdzIG15IG9yaWdpbmFsIGlk ZWEgaWYgSSBkbyBub3QgaGF2ZSBhbnl0aGluZw0KPiA+ID4gZWxzZSBiZXR0ZXIgdG8gZ28NCj4g PiA+ID4gd2l0aC4NCj4gPiA+ID4NCj4gPiA+ID4+IFRoYXQncyBhbGwgc3RpbGwgYSBrbHVkZ2Ug dG8gc2ltdWxhdGUgd2hhdCBORlMNCj4gPiA+IHdpbGwgZG8gZm9yIHlvdS7CoMKgV2h5DQo+ID4g PiA+PiB3b24ndCBORlMgd29yayBmb3IgeW91Pw0KPiA+ID4gPg0KPiA+ID4gPiBJIGhhdmUgYSBj bGllbnQgd2hvIHdhbnRzIHRvIGdvIGZyb20gTkFTIHRvIGEgdHJ1ZQ0KPiA+ID4gU0FOIHNvbHV0 aW9uIHdpdGgNCj4gPiA+ID4gZnVsbA0KPiA+ID4gPiBmaWJyZSBjaGFubmVsIG5ldHdvcmsuwqDC oEkgd291bGQgaGF0ZSB0byBsb3NlIHRoZQ0KPiA+ID4gb3Bwb3J0dW5pdHkgZm9yIHRoaXMNCj4g PiA+ID4gY2xpZW50DQo+ID4gPiA+IHRvIGNvbnRpbnVlIHVzaW5nIEZyZWVCU0QgYXMgdGhlIGNo b2ljZSBvZiBPUyBmb3INCj4gPiA+IGhpcyB3ZWIgc2VydmVycy4NCj4gPiA+ID4gQ3VycmVudGx5 LA0KPiA+ID4gPiBoaXMgc2V0IHVwIGlzIHVzaW5nIE5BUyB3aXRoIE5GUy7CoMKgSGUgY29tcGxh aW5zIG9mDQo+ID4gPiBsb2NraW5nIGZpbGVzIHRoYXQNCj4gPiA+ID4gb2NjdXJzDQo+ID4gPiA+ IHRvbyBvZnRlbi4NCj4gPiA+ID4NCj4gPiA+ID4gSSBoYWQgaG9wZWQgdG8gZmluZCBtb3JlIGJl dHRlciBzb2x1dGlvbiBhbmQgbWFrZQ0KPiA+ID4gdGhpcyBjbGllbnQgbXVjaCBtb3JlDQo+ID4g PiA+IGhhcHBpZXINCj4gPiA+ID4gd2l0aCBhbGwgdGhlIEZyZWVCU0Qgc3VwcG9ydCB0aGF0IGNh biBiZSBwcm92aWRlZC4NCj4gPiA+DQo+ID4gPg0KPiA+ID4gV2VsbCwgSSdtIG5vdCBzdXJlIHdo YXQgaXNzdWVzIHRoZXkgaGFkLCBidXQgaGF2ZQ0KPiA+ID4gaGFkIGZhbnRhc3RpYyBzdWNjZXNz DQo+ID4gPiB3aXRoIE5GUyBhbmQgRnJlZUJTRC7CoMKgRnJlZUJTRCB3aXRoIHRoZSByaWdodA0K PiA+ID4gaGFyZHdhcmUgYW5kIHR3ZWFrcyBjYW4NCj4gPiA+IG1ha2Ugc29tZSBOZXRBcHAgYm94 ZXMgbG9vayB3ZWFrLiAqY291Z2gqIFdBRkwNCj4gPiA+ICpjb3VnaCoNCj4gPiA+DQo+ID4gPg0K PiA+ID4gPj4gSSBhZ3JlZSB0aGF0IGl0IHdvdWxkIGJlIGZhbnRhc3RpYyB0byBoYXZlIGENCj4g PiA+IGNsdXN0ZXJlZCBmaWxlIHN5c3RlbSBmb3INCj4gPiA+ID4+IEZyZWVCU0QsIGFuZCBJJ3Zl IGRvbmUgbG90J3Mgb2YgaHVudGluZyBhbmQNCj4gPiA+IG5hZ2dpbmcgdmVuZG9ycyB0byBzdXBw b3J0DQo+ID4gPiA+PiBpdCAtIGJ1dCBpdCdzIGp1c3Qgbm90IHRoZXJlLg0KPiA+ID4gPg0KPiA+ ID4gPiBXZSBzaG91bGQgZ2V0IGZldyBiYW5kd2Fnb25zIGFuZCBnZXQgaW4gY2lyY2xlLg0KPiA+ ID4gSXQgY291bGQgYmUgbGlrZWx5IHRoYXQNCj4gPiA+ID4gSSBjb3VsZA0KPiA+ID4gPiBwcm92 aWRlIGFjY2VzcyBmb3IgdGhlIGRldmVsb3BlcnMgdG8gdGVzdCBhbmQgZ2V0DQo+ID4gPiB3aGF0 ZXZlciBmaWxlIHN5c3RlbQ0KPiA+ID4gPiBhbmQgb3RoZXINCj4gPiA+ID4gbmVjZXNzYXJpZXMg bmVlZGVkIHRvIGJlIHdvcmtpbmcuIDopDQo+ID4gPg0KPiA+ID4NCj4gPiA+IFRoZSBwcm9ibGVt IGlzbid0IHRoZSBlbnZpcm9ubWVudCBvciBoYXJkd2FyZSwgaXQncw0KPiA+ID4gZGV2ZWxvcGVy cyBza2lsbGVkDQo+ID4gPiB0byBkbyB0aGUgd29yay7CoMKgVGhleSdyZSBhbGwgZWl0aGVyIGlu IE5EQSdzLCBvZmYNCj4gPiA+IHdyaXRpbmcgc29tZXRoaW5nDQo+ID4gPiBlbHNlLCBvciBqdXN0 IHRvbyBidXN5IHRvIHByb3ZpZGUgYW55IGFtb3VudCBvZg0KPiA+ID4gaW5wdXQuDQo+ID4gPg0K PiA+ID4gRXJpYw0KPiA+DQo+ID4NCj4gPiBJIGhhdmUgYSBzZXQgb2Ygc2VydmVycyBORlMgbW91 bnRlZCB0byBhIE5ldGFwcCBhbmQNCj4gPiBhZnRlciBodXJzIG9mIHR1bmluZyB3aXRoIG5ldGFw cCdzIGhlbHAuIChhZnRlcg0KPiA+IGdldHRpbmcgdGhyb3VnaCB0aGUgaWRpb3RzIGFka2luZyB3 aGF0IEZyZWVCU2Qgd2FzKQ0KPiA+wqDCoEkgZ290IHZlcnkgbG93IHBlcmZvcm1hbmNlLiBJIHdh cyBvZiBjb3Vyc2UgdGhlbiB0b2xkDQo+ID7CoMKgYnkgTmV0YXBwIHRvIHN3aXRjaCB0byBMaW51 eCBmb3IgYmV0dGVyIE5GUyBzdXBwb3J0Lg0KPiANCj4gVGhhdCBpcyB3aGF0IEkgd291bGQgbGlr ZSB0byBhdm9pZCB0ZWxsaW5nIG15IGNsaWVudCB0byBkbw0KPiBUaGUgc2FtZSB0aGluZy4gIlN0 YXkgd2l0aCBORlMgYW5kIHRvbGVyYXRlIGl0LiINCj4gDQo+IEkgaGFkIGhvcGVkIGEgU0FOIHNv bHV0aW9uIHdvdWxkIGJlIHBvc3NpYmxlIGZvcg0KPiBGcmVlQlNELsKgwqBTbyBmYXIsIGl0IGFw cGVhcnMgdGhhdCBpdCBpcyBub3QgcG9zc2libGUgdG8NCj4gc2hhcmUgdGhlIHNhbWUgZmlsZSBz eXN0ZW0gYWNyb3NzIHNldmVyYWwgd2ViIHNlcnZlcnMuDQo+IA0KPiBDaHJpcw0KPiA+DQo+ID7C oMKgSSB3b3VsZCBsb3ZlIGZvciBhbnkgaGVscCB3aXRoIHR1bmluZyB0aGlzIGZ1cnRoZXIsDQo+ ID4gYnV0IEkgY2Fubm90IHNheSB0aGF0IEZyZWVCU0Qgd2l0aCBOZXRhcHAgTkZTIHdpbGwgYmUN Cj4gPiBncmVhdC4gT2YgY291cnNlLCBJIGhhdmUgbm90IGJlZW4gYWJsZSB0byB0ZXN0IGlmDQo+ ID4gaW5kZWVkIExpbnV4IHdvdWxkIGJlIGFueSBiZXR0ZXIuDQo+ID7CoMKgSSB3aWxsIHNheSBo b3dldmVyIHRoYXQgSSBoYXZlIGEgbGFyZ2UgbnVtYmVyIG9mDQo+ID4gc21hbGwgZmlsZXMgd2hp Y2ggdGVuZHMgdG8gbm90IGRvIHdlbGwgd2l0aCBORlMuDQo+ID4NCj4gPg0KPiA+wqDCoE5pY29s ZQ0KPiA+DQo+IA0KPiBfX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19f X19fXw0KPiBmcmVlYnNkLWZzQGZyZWVic2Qub3JnIG1haWxpbmcgbGlzdA0KPiBodHRwOi8vbGlz dHMuZnJlZWJzZC5vcmcvbWFpbG1hbi9saXN0aW5mby9mcmVlYnNkLWZzDQo+IFRvIHVuc3Vic2Ny aWJlLCBzZW5kIGFueSBtYWlsIHRvICJmcmVlYnNkLWZzLXVuc3Vic2NyaWJlQGZyZWVic2Qub3Jn Ig0KDQo= From owner-freebsd-fs@FreeBSD.ORG Tue Feb 13 17:16:40 2007 Return-Path: X-Original-To: freebsd-fs@FreeBSD.ORG Delivered-To: freebsd-fs@FreeBSD.ORG Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id D075C16A407 for ; Tue, 13 Feb 2007 17:16:39 +0000 (UTC) (envelope-from olli@lurza.secnetix.de) Received: from lurza.secnetix.de (lurza.secnetix.de [83.120.8.8]) by mx1.freebsd.org (Postfix) with ESMTP id 45F4C13C4BB for ; Tue, 13 Feb 2007 17:16:38 +0000 (UTC) (envelope-from olli@lurza.secnetix.de) Received: from lurza.secnetix.de (bmtuhk@localhost [127.0.0.1]) by lurza.secnetix.de (8.13.4/8.13.4) with ESMTP id l1DHGVxp000720; Tue, 13 Feb 2007 18:16:37 +0100 (CET) (envelope-from oliver.fromme@secnetix.de) Received: (from olli@localhost) by lurza.secnetix.de (8.13.4/8.13.1/Submit) id l1DHGVsN000719; Tue, 13 Feb 2007 18:16:31 +0100 (CET) (envelope-from olli) Date: Tue, 13 Feb 2007 18:16:31 +0100 (CET) Message-Id: <200702131716.l1DHGVsN000719@lurza.secnetix.de> From: Oliver Fromme To: freebsd-fs@FreeBSD.ORG, chris@sigd.net In-Reply-To: <6FC9F9894A9F8C49A722CF9F2132FC2204C9DAB6@ms05.mailstreet2003.net> X-Newsgroups: list.freebsd-fs User-Agent: tin/1.8.2-20060425 ("Shillay") (UNIX) (FreeBSD/4.11-STABLE (i386)) MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 8bit X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-2.1.2 (lurza.secnetix.de [127.0.0.1]); Tue, 13 Feb 2007 18:16:37 +0100 (CET) Cc: Subject: Re: UFS2 with SAN X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: freebsd-fs@FreeBSD.ORG, chris@sigd.net List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 13 Feb 2007 17:16:40 -0000 Chris Haulmark wrote: > If your responses on this SAN thread would not be productive, please > stay out. I am not interested to hear lectures about what is impossible. But maybe others are interested to hear that information. > I asked about if anyone has tried to use UFS2 with only one node to > have write/read only while the rest would be read only. And you already got the answer, see Eric's mail. He wrote: | This will result it the read/write system seeing the data ok, | and the rest getting corrupt data without knowing it, and | probably crashing. UFS2 is not cluster aware. Eric is right. For UFS to be cluster-aware, it would need to implement a cache-coherency protocol, so every node knows what data is up-to-date and what data is stale. UFS doesn't do that at all because it was designed as a local-only file system. The only way it would work is to mount _all_ of the nodes read-only. When you need to update data, you must remount the file system read-write on one node _and_ unmount it on all other nodes. I don't think that you want to do that, though. Best regards Oliver -- Oliver Fromme, secnetix GmbH & Co. KG, Marktplatz 29, 85567 Grafing b. M. Handelsregister: Registergericht Muenchen, HRA 74606, Geschäftsfuehrung: secnetix Verwaltungsgesellsch. mbH, Handelsregister: Registergericht Mün- chen, HRB 125758, Geschäftsführer: Maik Bachmann, Olaf Erb, Ralf Gebhart Any opinions expressed in this message are personal to the author and may not necessarily reflect the opinions of secnetix GmbH & Co KG in any way. FreeBSD-Dienstleistungen, -Produkte und mehr: http://www.secnetix.de/bsd "UNIX was not designed to stop you from doing stupid things, because that would also stop you from doing clever things." -- Doug Gwyn From owner-freebsd-fs@FreeBSD.ORG Tue Feb 13 17:19:09 2007 Return-Path: X-Original-To: freebsd-fs@freebsd.org Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 273F616A408 for ; Tue, 13 Feb 2007 17:19:09 +0000 (UTC) (envelope-from anderson@freebsd.org) Received: from mh1.centtech.com (moat3.centtech.com [64.129.166.50]) by mx1.freebsd.org (Postfix) with ESMTP id 0633813C47E for ; Tue, 13 Feb 2007 17:19:08 +0000 (UTC) (envelope-from anderson@freebsd.org) Received: from [10.177.171.220] (neutrino.centtech.com [10.177.171.220]) by mh1.centtech.com (8.13.8/8.13.8) with ESMTP id l1DHJ6fA007609; Tue, 13 Feb 2007 11:19:06 -0600 (CST) (envelope-from anderson@freebsd.org) Message-ID: <45D1F30A.6080403@freebsd.org> Date: Tue, 13 Feb 2007 11:19:06 -0600 From: Eric Anderson User-Agent: Thunderbird 1.5.0.9 (X11/20070204) MIME-Version: 1.0 To: Chris Haulmark References: <45CD6FF5.8070007@freebsd.org> <20070213075627.63126.qmail@web34502.mail.mud.yahoo.com> <6FC9F9894A9F8C49A722CF9F2132FC2204C9DAB4@ms05.mailstreet2003.net> <6FC9F9894A9F8C49A722CF9F2132FC2204C9DAB5@ms05.mailstreet2003.net> <6FC9F9894A9F8C49A722CF9F2132FC2204C9DAB6@ms05.mailstreet2003.net> In-Reply-To: <6FC9F9894A9F8C49A722CF9F2132FC2204C9DAB6@ms05.mailstreet2003.net> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 8bit X-Virus-Scanned: ClamAV 0.88.4/2562/Tue Feb 13 09:18:56 2007 on mh1.centtech.com X-Virus-Status: Clean X-Spam-Status: No, score=-2.6 required=8.0 tests=BAYES_00 autolearn=ham version=3.1.6 X-Spam-Checker-Version: SpamAssassin 3.1.6 (2006-10-03) on mh1.centtech.com Cc: freebsd-fs@freebsd.org Subject: Re: UFS2 with SAN X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 13 Feb 2007 17:19:09 -0000 On 02/13/07 11:00, Chris Haulmark wrote: >> From: Jeff Mohler [mailto:speedtoys.racing@gmail.com] >> Sent: Tuesday, February 13, 2007 10:40 AM >> To: Chris Haulmark >> Cc: Nicole Harrington; Eric Anderson; freebsd-fs@freebsd.org >> Subject: Re: UFS2 with SAN >> >> Its pretty much that simple. >> >> You cannot share SAN data..well..you -can- make a LUN appear as a >> shared NFS or CIFS share on a Netapp, but I havent tried it in a >> while..but you -can-. > > I set up a Linux cluster with GFS on a single SAN file system. It > included 8 nodes. Yep, and there's a handful of other linux supported cluster file systems out there, some good, some not so good. >> But in the normal world..you cannot...you would have to make it >> available via NFS to other client.  Thats the key difference between >> SAN and NAS. > > No. SAN and NAS is different based on their access. SAN was dependent > on either SCSI or Fibre until iSCSI (IPSAN) was introduced. NAS will > always require an IP address. If you were using Fibre and wanted to > use NAS, you would have to use iFCP or FCIP protocols. I do not need to > explain the limitation of using SCSI with NAS. > > If your responses on this SAN thread would not be productive, please > stay out. I am not interested to hear lectures about what is impossible. > > I asked about if anyone has tried to use UFS2 with only one node to > have write/read only while the rest would be read only. Right, and the answer comes down to - UFS is not cluster aware, and so it doesn't know to talk to other nodes to tell them to reload data from disk, nor does it know if two nodes are writing to the same file at the same time (think: appending to a log file from many nodes), locking, etc. Sharing the storage works (I have lots of FreeBSD boxen using the same fiber backend, with disks visible to all the nodes). You can mount the file system RO on many systems just fine too. *BUT* as soon as you mount an fs rw on another box, it's going to start making changes to the file system, and those changes will not be see on the other nodes. I have indeed tried it. You might look at tdfs (fuse module from Ivan Voras I believe). It's probably horribly beta (not knocking it Ivan - I'm sure it's an amazing start), so lots of testing would need to be done. It's also most likely not high performance. Do you need one node to be doing *lots* of writes to the file system, or just nightly updates, etc? Eric >> On 2/13/07, Chris Haulmark wrote: >> >> >>> -----Original Message----- >>> From: Nicole Harrington [mailto:drumslayer2@yahoo.com] >>> Sent: Tuesday, February 13, 2007 1:56 AM >>> To: Eric Anderson; Chris Haulmark >>> Cc: freebsd-fs@freebsd.org >>> Subject: Re: UFS2 with SAN >>> >>> --- Eric Anderson wrote: >>> >>>> On 02/10/07 00:54, Chris Haulmark wrote: >>>>>> -----Original Message----- >>>>>> From: Eric Anderson [mailto:anderson@freebsd.org ] >>>>>> Sent: Saturday, February 10, 2007 12:48 AM >>>>>> To: Chris Haulmark >>>>>> Cc: freebsd-fs@freebsd.org >>>>>> Subject: Re: UFS2 with SAN >>>>>> >>>>>> On 02/09/07 19:30, Chris Haulmark wrote: >>>>>>> Hello, >>>>>>> >>>>>>> I am looking into setting up a SAN with several >>>> web servers that >>>>>>> will be clustered.  It would be a FC network >>>> using Qlogic cards >>>>>>> in each of those FreeBSD web servers.  It would >>>> be about 5+ >>>>>>> of those web servers. >>>>>>> >>>>>>> I want to have the capability to share the same >>>> web data across >>>>>>> those web servers.  I have scorched the entire >>>> mailing list and >>>>>>> found that there were some work on GFS porting >>>> over to FreeBSD. >>>>>>> It seems like that it is just all talk and if I >>>> am wrong, could >>>>>>> you have my head turned over to where I can find >>>> out how to enable >>>>>>> GFS on those FreeBSD systems. >>>>>> GFS on FreeBSD is indeed dead.  Not enough people >>>> stepped up to help >>>>>> port it. >>>>> I really feared to hear that! >>>>> >>>>>>> If GFS is out of question, which file system am >>>> I recommendeded >>>>>>> to attempt to use for this SAN setup? >>>>>> NFS. >>>>>> >>>>>>> My first thought to use UFS2 and attempt is to >>>> allow only one web >>>>>>> server to have a write/read access while the >>>> reminder would be >>>>>>> read only access. That should prevent from >>>> lockings that is similar >>>>>>> on NFS/NAS. >>>>>> This will result it the read/write system seeing >>>> the data ok, and the >>>>>> rest getting corrupt data without knowing it, and >>>> probably crashing. >>>>>> UFS2 is not cluster aware.  You could mount all >>>> the hosts read only, >>>>>> and >>>>>> then update the mount point on one to rw, makes >>>> changes, then back to >>>>>> ro, then unmount/remount on the other boxes. >>>>> That's my original idea if I do not have anything >>>> else better to go >>>>> with. >>>>> >>>>>> That's all still a kludge to simulate what NFS >>>> will do for you.  Why >>>>>> won't NFS work for you? >>>>> I have a client who wants to go from NAS to a true >>>> SAN solution with >>>>> full >>>>> fibre channel network.  I would hate to lose the >>>> opportunity for this >>>>> client >>>>> to continue using FreeBSD as the choice of OS for >>>> his web servers. >>>>> Currently, >>>>> his set up is using NAS with NFS.  He complains of >>>> locking files that >>>>> occurs >>>>> too often. >>>>> >>>>> I had hoped to find more better solution and make >>>> this client much more >>>>> happier >>>>> with all the FreeBSD support that can be provided. >>>> >>>> Well, I'm not sure what issues they had, but have >>>> had fantastic success >>>> with NFS and FreeBSD.  FreeBSD with the right >>>> hardware and tweaks can >>>> make some NetApp boxes look weak. *cough* WAFL >>>> *cough* >>>> >>>> >>>>>> I agree that it would be fantastic to have a >>>> clustered file system for >>>>>> FreeBSD, and I've done lot's of hunting and >>>> nagging vendors to support >>>>>> it - but it's just not there. >>>>> We should get few bandwagons and get in circle. >>>> It could be likely that >>>>> I could >>>>> provide access for the developers to test and get >>>> whatever file system >>>>> and other >>>>> necessaries needed to be working. :) >>>> >>>> The problem isn't the environment or hardware, it's >>>> developers skilled >>>> to do the work.  They're all either in NDA's, off >>>> writing something >>>> else, or just too busy to provide any amount of >>>> input. >>>> >>>> Eric >>> >>> I have a set of servers NFS mounted to a Netapp and >>> after hurs of tuning with netapp's help. (after >>> getting through the idiots adking what FreeBSd was) >>>   I got very low performance. I was of course then told >>>   by Netapp to switch to Linux for better NFS support. >> That is what I would like to avoid telling my client to do >> The same thing. "Stay with NFS and tolerate it." >> >> I had hoped a SAN solution would be possible for >> FreeBSD.  So far, it appears that it is not possible to >> share the same file system across several web servers. >> >> Chris >>>   I would love for any help with tuning this further, >>> but I cannot say that FreeBSD with Netapp NFS will be >>> great. Of course, I have not been able to test if >>> indeed Linux would be any better. >>>   I will say however that I have a large number of >>> small files which tends to not do well with NFS. >>> >>> >>>   Nicole >>> >> _______________________________________________ >> freebsd-fs@freebsd.org mailing list >> http://lists.freebsd.org/mailman/listinfo/freebsd-fs >> To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" > From owner-freebsd-fs@FreeBSD.ORG Tue Feb 13 17:28:29 2007 Return-Path: X-Original-To: freebsd-fs@freebsd.org Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id A885516A402; Tue, 13 Feb 2007 17:28:29 +0000 (UTC) (envelope-from chris@sigd.net) Received: from ms05.mailstreet2003.net (ms05.mailstreet2003.net [69.25.50.235]) by mx1.freebsd.org (Postfix) with ESMTP id 6073913C4A6; Tue, 13 Feb 2007 17:28:28 +0000 (UTC) (envelope-from chris@sigd.net) X-MimeOLE: Produced By Microsoft Exchange V6.5.7226.0 Content-class: urn:content-classes:message MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Date: Tue, 13 Feb 2007 12:28:27 -0500 Message-ID: <6FC9F9894A9F8C49A722CF9F2132FC2204C9DAB7@ms05.mailstreet2003.net> In-Reply-To: <45D1F30A.6080403@freebsd.org> X-MS-Has-Attach: X-MS-TNEF-Correlator: Thread-Topic: UFS2 with SAN Thread-Index: AcdPkxawNHXk2RCtSnSIXeJRpl2FGAAAHXnQ References: <45CD6FF5.8070007@freebsd.org> <20070213075627.63126.qmail@web34502.mail.mud.yahoo.com> <6FC9F9894A9F8C49A722CF9F2132FC2204C9DAB4@ms05.mailstreet2003.net> <6FC9F9894A9F8C49A722CF9F2132FC2204C9DAB5@ms05.mailstreet2003.net> <6FC9F9894A9F8C49A722CF9F2132FC2204C9DAB6@ms05.mailstreet2003.net> <45D1F30A.6080403@freebsd.org> From: "Chris Haulmark" To: "Eric Anderson" Cc: freebsd-fs@freebsd.org Subject: RE: UFS2 with SAN X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 13 Feb 2007 17:28:29 -0000 > -----Original Message----- > From: Eric Anderson [mailto:anderson@freebsd.org] > Sent: Tuesday, February 13, 2007 11:19 AM > To: Chris Haulmark > Cc: freebsd-fs@freebsd.org > Subject: Re: UFS2 with SAN >=20 > On 02/13/07 11:00, Chris Haulmark wrote: > >> From: Jeff Mohler [mailto:speedtoys.racing@gmail.com] > >> Sent: Tuesday, February 13, 2007 10:40 AM > >> To: Chris Haulmark > >> Cc: Nicole Harrington; Eric Anderson; freebsd-fs@freebsd.org > >> Subject: Re: UFS2 with SAN > >> > >> Its pretty much that simple. > >> > >> You cannot share SAN data..well..you -can- make a LUN appear as a > >> shared NFS or CIFS share on a Netapp, but I havent tried it in a > >> while..but you -can-. > > > > I set up a Linux cluster with GFS on a single SAN file system. It > > included 8 nodes. >=20 >=20 > Yep, and there's a handful of other linux supported cluster file > systems > out there, some good, some not so good. >=20 >=20 > >> But in the normal world..you cannot...you would have to make it > >> available via NFS to other client.=C3' =C3' Thats the key = difference > between > >> SAN and NAS. > > > > No. SAN and NAS is different based on their access. SAN was > dependent > > on either SCSI or Fibre until iSCSI (IPSAN) was introduced. NAS = will > > always require an IP address. If you were using Fibre and wanted to > > use NAS, you would have to use iFCP or FCIP protocols. I do not = need > to > > explain the limitation of using SCSI with NAS. > > > > If your responses on this SAN thread would not be productive, please > > stay out. I am not interested to hear lectures about what is > impossible. > > > > I asked about if anyone has tried to use UFS2 with only one node to > > have write/read only while the rest would be read only. >=20 >=20 > Right, and the answer comes down to - UFS is not cluster aware, and so > it doesn't know to talk to other nodes to tell them to reload data = from > disk, nor does it know if two nodes are writing to the same file at = the > same time (think: appending to a log file from many nodes), locking, > etc. >=20 I figured that if UFS was my only solution, I would write up a script that would handle the talking to the other nodes to umount/mount the file system on demand. > Sharing the storage works (I have lots of FreeBSD boxen using the same > fiber backend, with disks visible to all the nodes). You can mount = the > file system RO on many systems just fine too. *BUT* as soon as you > mount an fs rw on another box, it's going to start making changes to > the > file system, and those changes will not be see on the other nodes. I > have indeed tried it. >=20 > You might look at tdfs (fuse module from Ivan Voras I believe). It's > probably horribly beta (not knocking it Ivan - I'm sure it's an = amazing > start), so lots of testing would need to be done. It's also most > likely > not high performance. Thanks for the tip! >=20 > Do you need one node to be doing *lots* of writes to the file system, > or > just nightly updates, etc? Just updates from the staging production. I doubt it would happen as often as the nightly updates. The client mentioned having a staging environment so I am sure that the site would be tested before doing a final update. It feels more likely to be weekly updates. Oliver, thanks too. Chris >=20 >=20 > Eric >=20 >=20 > >> On 2/13/07, Chris Haulmark wrote: > >> > >> > >>> -----Original Message----- > >>> From: Nicole Harrington [mailto:drumslayer2@yahoo.com] > >>> Sent: Tuesday, February 13, 2007 1:56 AM > >>> To: Eric Anderson; Chris Haulmark > >>> Cc: freebsd-fs@freebsd.org > >>> Subject: Re: UFS2 with SAN > >>> > >>> --- Eric Anderson wrote: > >>> > >>>> On 02/10/07 00:54, Chris Haulmark wrote: > >>>>>> -----Original Message----- > >>>>>> From: Eric Anderson [mailto:anderson@freebsd.org ] > >>>>>> Sent: Saturday, February 10, 2007 12:48 AM > >>>>>> To: Chris Haulmark > >>>>>> Cc: freebsd-fs@freebsd.org > >>>>>> Subject: Re: UFS2 with SAN > >>>>>> > >>>>>> On 02/09/07 19:30, Chris Haulmark wrote: > >>>>>>> Hello, > >>>>>>> > >>>>>>> I am looking into setting up a SAN with several > >>>> web servers that > >>>>>>> will be clustered.=C3' =C3' It would be a FC network > >>>> using Qlogic cards > >>>>>>> in each of those FreeBSD web servers.=C3' =C3' It would > >>>> be about 5+ > >>>>>>> of those web servers. > >>>>>>> > >>>>>>> I want to have the capability to share the same > >>>> web data across > >>>>>>> those web servers.=C3' =C3' I have scorched the entire > >>>> mailing list and > >>>>>>> found that there were some work on GFS porting > >>>> over to FreeBSD. > >>>>>>> It seems like that it is just all talk and if I > >>>> am wrong, could > >>>>>>> you have my head turned over to where I can find > >>>> out how to enable > >>>>>>> GFS on those FreeBSD systems. > >>>>>> GFS on FreeBSD is indeed dead.=C3' =C3' Not enough people > >>>> stepped up to help > >>>>>> port it. > >>>>> I really feared to hear that! > >>>>> > >>>>>>> If GFS is out of question, which file system am > >>>> I recommendeded > >>>>>>> to attempt to use for this SAN setup? > >>>>>> NFS. > >>>>>> > >>>>>>> My first thought to use UFS2 and attempt is to > >>>> allow only one web > >>>>>>> server to have a write/read access while the > >>>> reminder would be > >>>>>>> read only access. That should prevent from > >>>> lockings that is similar > >>>>>>> on NFS/NAS. > >>>>>> This will result it the read/write system seeing > >>>> the data ok, and the > >>>>>> rest getting corrupt data without knowing it, and > >>>> probably crashing. > >>>>>> UFS2 is not cluster aware.=C3' =C3' You could mount all > >>>> the hosts read only, > >>>>>> and > >>>>>> then update the mount point on one to rw, makes > >>>> changes, then back to > >>>>>> ro, then unmount/remount on the other boxes. > >>>>> That's my original idea if I do not have anything > >>>> else better to go > >>>>> with. > >>>>> > >>>>>> That's all still a kludge to simulate what NFS > >>>> will do for you.=C3' =C3' Why > >>>>>> won't NFS work for you? > >>>>> I have a client who wants to go from NAS to a true > >>>> SAN solution with > >>>>> full > >>>>> fibre channel network.=C3' =C3' I would hate to lose the > >>>> opportunity for this > >>>>> client > >>>>> to continue using FreeBSD as the choice of OS for > >>>> his web servers. > >>>>> Currently, > >>>>> his set up is using NAS with NFS.=C3' =C3' He complains of > >>>> locking files that > >>>>> occurs > >>>>> too often. > >>>>> > >>>>> I had hoped to find more better solution and make > >>>> this client much more > >>>>> happier > >>>>> with all the FreeBSD support that can be provided. > >>>> > >>>> Well, I'm not sure what issues they had, but have > >>>> had fantastic success > >>>> with NFS and FreeBSD.=C3' =C3' FreeBSD with the right > >>>> hardware and tweaks can > >>>> make some NetApp boxes look weak. *cough* WAFL > >>>> *cough* > >>>> > >>>> > >>>>>> I agree that it would be fantastic to have a > >>>> clustered file system for > >>>>>> FreeBSD, and I've done lot's of hunting and > >>>> nagging vendors to support > >>>>>> it - but it's just not there. > >>>>> We should get few bandwagons and get in circle. > >>>> It could be likely that > >>>>> I could > >>>>> provide access for the developers to test and get > >>>> whatever file system > >>>>> and other > >>>>> necessaries needed to be working. :) > >>>> > >>>> The problem isn't the environment or hardware, it's > >>>> developers skilled > >>>> to do the work.=C3' =C3' They're all either in NDA's, off > >>>> writing something > >>>> else, or just too busy to provide any amount of > >>>> input. > >>>> > >>>> Eric > >>> > >>> I have a set of servers NFS mounted to a Netapp and > >>> after hurs of tuning with netapp's help. (after > >>> getting through the idiots adking what FreeBSd was) > >>> =C3' =C3' I got very low performance. I was of course then told > >>> =C3' =C3' by Netapp to switch to Linux for better NFS support. > >> That is what I would like to avoid telling my client to do > >> The same thing. "Stay with NFS and tolerate it." > >> > >> I had hoped a SAN solution would be possible for > >> FreeBSD.=C3' =C3' So far, it appears that it is not possible to > >> share the same file system across several web servers. > >> > >> Chris > >>> =C3' =C3' I would love for any help with tuning this further, > >>> but I cannot say that FreeBSD with Netapp NFS will be > >>> great. Of course, I have not been able to test if > >>> indeed Linux would be any better. > >>> =C3' =C3' I will say however that I have a large number of > >>> small files which tends to not do well with NFS. > >>> > >>> > >>> =C3' =C3' Nicole > >>> > >> _______________________________________________ > >> freebsd-fs@freebsd.org mailing list > >> http://lists.freebsd.org/mailman/listinfo/freebsd-fs > >> To unsubscribe, send any mail to "freebsd-fs- > unsubscribe@freebsd.org" > > From owner-freebsd-fs@FreeBSD.ORG Tue Feb 13 18:00:31 2007 Return-Path: X-Original-To: freebsd-fs@freebsd.org Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 52A8D16A407 for ; Tue, 13 Feb 2007 18:00:31 +0000 (UTC) (envelope-from john@kozubik.com) Received: from kozubik.com (kozubik.com [69.43.165.2]) by mx1.freebsd.org (Postfix) with ESMTP id 27FBA13C478 for ; Tue, 13 Feb 2007 18:00:31 +0000 (UTC) (envelope-from john@kozubik.com) Received: from kozubik.com (localhost [127.0.0.1]) by kozubik.com (8.12.11/8.12.11) with ESMTP id l1DHngJN023744 for ; Tue, 13 Feb 2007 09:49:42 -0800 (PST) (envelope-from john@kozubik.com) Received: from localhost (john@localhost) by kozubik.com (8.12.11/8.12.11/Submit) with ESMTP id l1DHng8G023741 for ; Tue, 13 Feb 2007 09:49:42 -0800 (PST) (envelope-from john@kozubik.com) Date: Tue, 13 Feb 2007 09:49:42 -0800 (PST) From: John Kozubik To: freebsd-fs@freebsd.org In-Reply-To: <200702130927.l1D9RBn9034761@lurza.secnetix.de> Message-ID: <20070213093220.C95571@kozubik.com> References: <200702130927.l1D9RBn9034761@lurza.secnetix.de> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Subject: Re: comments on newfs raw disk ? Safe ? (7 terabyte array) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 13 Feb 2007 18:00:31 -0000 Friends, On Tue, 13 Feb 2007, Oliver Fromme wrote: > Randy Bush wrote: > > this has been a wonderfully well-timed thread as i am about > > to hack a 4tb array tomorrow afternoon. the normal spindle > > is separate and partitioned to death and newfsed using the > > defaults. with 2gb of ram, i figure 6gb swap just in case > > two userland hogs are running at once, e.g. some hog while > > background fsck is running. > > A bit careful here ... Background fsck had some issues, > especially when the machine crashed or is otherwise reset > while the background fsck is still running. It resulted > in corruption that could not be repaired by fsck anymore. > I don't know if all of those issues have been resolved in > RELENG_6, but personally I always disable background fsck > on all of my machines, just to be safe. Also remember that filling a filesystem to capacity _while_ it is being snapshotted will lock your system up[1]. I suppose some interesting crash loops could arise from this bug on a near full filesystem that someone is unlucky enough to background fsck. I think that FreeBSD needs to address the default implementation of background fsck in general. UFS2 snapshots are dangerous and unstable, and have been since their introduction in 5.x [2]. Oliver and I and everyone else here knows the dangers of UFS2 snapshots and background fsck, and it's very telling that Oliver (like myself) refuses to use them. I won't touch either of them, despite overwhelming financial incentive to implement them [3]. But how many innocent sysadmins and less well informed unix engineers in the world are loading up FreeBSD because of a perceived history of safety and stability and putting very important data and services on systems, which _by default_ have a dangerous ticking time bomb on them ? Are these people supposed to fall out of the womb knowing that UFS2 snapshots are unstable and dangerous, and that _4 years later_ they still aren't safe ? Until well-informed members of this list feel safe and secure with snapshots and background fsck in general use, I think background fsck should be disabled by default. John Kozubik - john@kozubik.com - http://www.kozubik.com [1] http://lists.freebsd.org/pipermail/freebsd-bugs/2006-January/016703.html [2] http://lists.freebsd.org/pipermail/freebsd-bugs/2004-July/007574.html [3] http://www.rsync.net From owner-freebsd-fs@FreeBSD.ORG Tue Feb 13 18:09:53 2007 Return-Path: X-Original-To: freebsd-fs@freebsd.org Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id E49E916A407 for ; Tue, 13 Feb 2007 18:09:53 +0000 (UTC) (envelope-from freebsd-fs@m.gmane.org) Received: from ciao.gmane.org (main.gmane.org [80.91.229.2]) by mx1.freebsd.org (Postfix) with ESMTP id 78A0613C4A3 for ; Tue, 13 Feb 2007 18:09:53 +0000 (UTC) (envelope-from freebsd-fs@m.gmane.org) Received: from list by ciao.gmane.org with local (Exim 4.43) id 1HH267-0005BS-04 for freebsd-fs@freebsd.org; Tue, 13 Feb 2007 19:09:47 +0100 Received: from 89-172-245-141.adsl.net.t-com.hr ([89.172.245.141]) by main.gmane.org with esmtp (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Tue, 13 Feb 2007 19:09:46 +0100 Received: from ivoras by 89-172-245-141.adsl.net.t-com.hr with local (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Tue, 13 Feb 2007 19:09:46 +0100 X-Injected-Via-Gmane: http://gmane.org/ To: freebsd-fs@freebsd.org From: Ivan Voras Date: Tue, 13 Feb 2007 19:09:32 +0100 Lines: 43 Message-ID: References: <45CD6FF5.8070007@freebsd.org> <20070213075627.63126.qmail@web34502.mail.mud.yahoo.com> <6FC9F9894A9F8C49A722CF9F2132FC2204C9DAB4@ms05.mailstreet2003.net> <6FC9F9894A9F8C49A722CF9F2132FC2204C9DAB5@ms05.mailstreet2003.net> <6FC9F9894A9F8C49A722CF9F2132FC2204C9DAB6@ms05.mailstreet2003.net> <45D1F30A.6080403@freebsd.org> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="------------enig34D9E7B78EA9224567D4C440" X-Complaints-To: usenet@sea.gmane.org X-Gmane-NNTP-Posting-Host: 89-172-245-141.adsl.net.t-com.hr User-Agent: Thunderbird 1.5.0.9 (Windows/20061207) In-Reply-To: <45D1F30A.6080403@freebsd.org> X-Enigmail-Version: 0.94.1.2 Sender: news Subject: Re: UFS2 with SAN X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 13 Feb 2007 18:09:54 -0000 This is an OpenPGP/MIME signed message (RFC 2440 and 3156) --------------enig34D9E7B78EA9224567D4C440 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Eric Anderson wrote: > You might look at tdfs (fuse module from Ivan Voras I believe). It's > probably horribly beta (not knocking it Ivan - I'm sure it's an amazing= > start), so lots of testing would need to be done. It's also most likel= y > not high performance. Yes, I'm realistic about it - it is "horribly beta" :) Two reasons: one: I couldn't amass enough interested parties to help test it (and possibly work on it), and second: while it's a nice problem to tackle, I don't [yet] need it that badly to start working on it outside of free time). In any case, TDFS doesn't have locking because FUSE doesn't have it (or didn't when I started doing it), so it stays single-writer-multiple-readers until that changes. I'm thinking of re-creating it in kernel mode (and in this form it could also have locking) but not any time soon. Maybe during the summer (SoC? probably.). --------------enig34D9E7B78EA9224567D4C440 Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.4 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFF0f7ildnAQVacBcgRAgLmAKDLJ6+65xAyR1C8Drue2g5nZFH6rwCgmG9l BguPYBTfFQr59E1yYbiaGtY= =Pb5x -----END PGP SIGNATURE----- --------------enig34D9E7B78EA9224567D4C440-- From owner-freebsd-fs@FreeBSD.ORG Tue Feb 13 18:15:28 2007 Return-Path: X-Original-To: freebsd-fs@freebsd.org Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id D00E516A400 for ; Tue, 13 Feb 2007 18:15:28 +0000 (UTC) (envelope-from anderson@freebsd.org) Received: from mh1.centtech.com (moat3.centtech.com [64.129.166.50]) by mx1.freebsd.org (Postfix) with ESMTP id A6B9913C4A3 for ; Tue, 13 Feb 2007 18:15:28 +0000 (UTC) (envelope-from anderson@freebsd.org) Received: from [10.177.171.220] (neutrino.centtech.com [10.177.171.220]) by mh1.centtech.com (8.13.8/8.13.8) with ESMTP id l1DIFIoD019230; Tue, 13 Feb 2007 12:15:19 -0600 (CST) (envelope-from anderson@freebsd.org) Message-ID: <45D20037.1040102@freebsd.org> Date: Tue, 13 Feb 2007 12:15:19 -0600 From: Eric Anderson User-Agent: Thunderbird 1.5.0.9 (X11/20070204) MIME-Version: 1.0 To: Ivan Voras References: <45CD6FF5.8070007@freebsd.org> <20070213075627.63126.qmail@web34502.mail.mud.yahoo.com> <6FC9F9894A9F8C49A722CF9F2132FC2204C9DAB4@ms05.mailstreet2003.net> <6FC9F9894A9F8C49A722CF9F2132FC2204C9DAB5@ms05.mailstreet2003.net> <6FC9F9894A9F8C49A722CF9F2132FC2204C9DAB6@ms05.mailstreet2003.net> <45D1F30A.6080403@freebsd.org> In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Virus-Scanned: ClamAV 0.88.4/2562/Tue Feb 13 09:18:56 2007 on mh1.centtech.com X-Virus-Status: Clean X-Spam-Status: No, score=-2.6 required=8.0 tests=AWL,BAYES_00 autolearn=ham version=3.1.6 X-Spam-Checker-Version: SpamAssassin 3.1.6 (2006-10-03) on mh1.centtech.com Cc: freebsd-fs@freebsd.org Subject: Re: UFS2 with SAN X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 13 Feb 2007 18:15:28 -0000 On 02/13/07 12:09, Ivan Voras wrote: > Eric Anderson wrote: > >> You might look at tdfs (fuse module from Ivan Voras I believe). It's >> probably horribly beta (not knocking it Ivan - I'm sure it's an amazing >> start), so lots of testing would need to be done. It's also most likely >> not high performance. > > Yes, I'm realistic about it - it is "horribly beta" :) > > Two reasons: one: I couldn't amass enough interested parties to help > test it (and possibly work on it), and second: while it's a nice problem > to tackle, I don't [yet] need it that badly to start working on it > outside of free time). In any case, TDFS doesn't have locking because > FUSE doesn't have it (or didn't when I started doing it), so it stays > single-writer-multiple-readers until that changes. > > I'm thinking of re-creating it in kernel mode (and in this form it could > also have locking) but not any time soon. Maybe during the summer (SoC? > probably.). > I've thought about this too - and even thought about making it a file system layer, like nullfs, with a network-kernel piece to notify the readers of changes, etc. I'm not certain if it would work or not.. Eric From owner-freebsd-fs@FreeBSD.ORG Tue Feb 13 18:21:06 2007 Return-Path: X-Original-To: freebsd-fs@FreeBSD.ORG Delivered-To: freebsd-fs@FreeBSD.ORG Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id ADD0D16A400 for ; Tue, 13 Feb 2007 18:21:06 +0000 (UTC) (envelope-from olli@lurza.secnetix.de) Received: from lurza.secnetix.de (lurza.secnetix.de [83.120.8.8]) by mx1.freebsd.org (Postfix) with ESMTP id 22DC013C474 for ; Tue, 13 Feb 2007 18:21:03 +0000 (UTC) (envelope-from olli@lurza.secnetix.de) Received: from lurza.secnetix.de (xeryjg@localhost [127.0.0.1]) by lurza.secnetix.de (8.13.4/8.13.4) with ESMTP id l1DIKvAr009659; Tue, 13 Feb 2007 19:21:02 +0100 (CET) (envelope-from oliver.fromme@secnetix.de) Received: (from olli@localhost) by lurza.secnetix.de (8.13.4/8.13.1/Submit) id l1DIKumD009658; Tue, 13 Feb 2007 19:20:56 +0100 (CET) (envelope-from olli) Date: Tue, 13 Feb 2007 19:20:56 +0100 (CET) Message-Id: <200702131820.l1DIKumD009658@lurza.secnetix.de> From: Oliver Fromme To: freebsd-fs@FreeBSD.ORG, john@kozubik.com In-Reply-To: <20070213093220.C95571@kozubik.com> X-Newsgroups: list.freebsd-fs User-Agent: tin/1.8.2-20060425 ("Shillay") (UNIX) (FreeBSD/4.11-STABLE (i386)) MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 8bit X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-2.1.2 (lurza.secnetix.de [127.0.0.1]); Tue, 13 Feb 2007 19:21:03 +0100 (CET) Cc: Subject: Re: comments on newfs raw disk ? Safe ? (7 terabyte array) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: freebsd-fs@FreeBSD.ORG, john@kozubik.com List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 13 Feb 2007 18:21:06 -0000 John Kozubik wrote: > Oliver Fromme wrote: > > Randy Bush wrote: > > > this has been a wonderfully well-timed thread as i am about > > > to hack a 4tb array tomorrow afternoon. the normal spindle > > > is separate and partitioned to death and newfsed using the > > > defaults. with 2gb of ram, i figure 6gb swap just in case > > > two userland hogs are running at once, e.g. some hog while > > > background fsck is running. > > > > A bit careful here ... Background fsck had some issues, > > especially when the machine crashed or is otherwise reset > > while the background fsck is still running. It resulted > > in corruption that could not be repaired by fsck anymore. > > I don't know if all of those issues have been resolved in > > RELENG_6, but personally I always disable background fsck > > on all of my machines, just to be safe. > > [...] > UFS2 snapshots are dangerous and unstable, > and have been since their introduction in 5.x [2]. That's not what I wrote. I wrote that they _had_ issues, and that I do not know if they have been fixed. I don't recall any reports of problems recently (i.e. in the past few months), and there are no open PRs that seem to relate to the current code, so those issues may very well have been fixed. It's just my personal paranoia that lets me disable bg fsck on my machines (and I don't really need bg fsck anyway). You have to be very careful with what you claim, or people might accuse you of spreading FUD. Best regards Oliver -- Oliver Fromme, secnetix GmbH & Co. KG, Marktplatz 29, 85567 Grafing b. M. Handelsregister: Registergericht Muenchen, HRA 74606, Geschäftsfuehrung: secnetix Verwaltungsgesellsch. mbH, Handelsregister: Registergericht Mün- chen, HRB 125758, Geschäftsführer: Maik Bachmann, Olaf Erb, Ralf Gebhart Any opinions expressed in this message are personal to the author and may not necessarily reflect the opinions of secnetix GmbH & Co KG in any way. FreeBSD-Dienstleistungen, -Produkte und mehr: http://www.secnetix.de/bsd "UNIX was not designed to stop you from doing stupid things, because that would also stop you from doing clever things." -- Doug Gwyn From owner-freebsd-fs@FreeBSD.ORG Tue Feb 13 18:23:28 2007 Return-Path: X-Original-To: freebsd-fs@freebsd.org Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 1A5AD16A400; Tue, 13 Feb 2007 18:23:28 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from galileo.cs.uoguelph.ca (galileo.cs.uoguelph.ca [131.104.94.215]) by mx1.freebsd.org (Postfix) with ESMTP id 79C7D13C478; Tue, 13 Feb 2007 18:23:27 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from muncher.cs.uoguelph.ca (muncher.cs.uoguelph.ca [131.104.96.170]) by galileo.cs.uoguelph.ca (8.13.1/8.13.1) with ESMTP id l1DHugFq007111; Tue, 13 Feb 2007 12:56:42 -0500 Received: from localhost (rmacklem@localhost) by muncher.cs.uoguelph.ca (8.11.7p1+Sun/8.11.6) with ESMTP id l1DI46R13667; Tue, 13 Feb 2007 13:04:06 -0500 (EST) X-Authentication-Warning: muncher.cs.uoguelph.ca: rmacklem owned process doing -bs Date: Tue, 13 Feb 2007 13:04:06 -0500 (EST) From: Rick Macklem X-X-Sender: rmacklem@muncher To: drumslayer2@yahoo.com In-Reply-To: Message-ID: References: <45CD6FF5.8070007@freebsd.org> <20070213075627.63126.qmail@web34502.mail.mud.yahoo.com> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed X-Scanned-By: MIMEDefang 2.57 on 131.104.94.215 Cc: freebsd-fs@freebsd.org, Chris Haulmark , Eric Anderson Subject: Re: UFS2 with SAN X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 13 Feb 2007 18:23:28 -0000 > On Tue, 13 Feb 2007, Jeff Mohler wrote: > For certain workloads, linux is far better tuned for NFS than Fbsd..netapp > as a server or not. > On 2/12/07, Nicole Harrington wrote: >> >> >> I have a set of servers NFS mounted to a Netapp and >> after hurs of tuning with netapp's help. (after >> getting through the idiots adking what FreeBSd was) >> I got very low performance. I was of course then told >> by Netapp to switch to Linux for better NFS support. >> >> I would love for any help with tuning this further, >> but I cannot say that FreeBSD with Netapp NFS will be >> great. Of course, I have not been able to test if >> indeed Linux would be any better. >> I will say however that I have a large number of >> small files which tends to not do well with NFS. >> >> >> Nicole I don't know if any of this will help, but... My experience is that abismal NFS perf is usually caused by one of: - Transport/Network problems: If the network interconnect introduces delays or drops packets, you've got to fix it. This includes network device drivers that aren't up to it. If you use NFS over UDP, one thing to check for is the # of "fragments dropped due to timeout" in the IP section of "netstat -s". If there are any of those happenning, you need to fix your network, or at least switch to TCP or reduce your read/write data size. (Make rsize, wsize as large as you can, given what your network can handle.) - Byte range locking (the NLM protocol, not really NFS). It's a dog in mho. All you can do is avoid apps that do byte range locking. (NFSv4 has good byte range locking support, but the client in FreeBSD isn't useful for production work yet, from what I've seem.) Beyond that, I'm not sure there is much you can do, rick From owner-freebsd-fs@FreeBSD.ORG Tue Feb 13 18:37:25 2007 Return-Path: X-Original-To: freebsd-fs@freebsd.org Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 076B616A408 for ; Tue, 13 Feb 2007 18:37:25 +0000 (UTC) (envelope-from randy@psg.com) Received: from rip.psg.com (rip.psg.com [147.28.0.39]) by mx1.freebsd.org (Postfix) with ESMTP id EAAE913C474 for ; Tue, 13 Feb 2007 18:37:22 +0000 (UTC) (envelope-from randy@psg.com) Received: from localhost ([127.0.0.1] helo=roam.psg.com) by rip.psg.com with esmtp (Exim 4.66 (FreeBSD)) (envelope-from ) id 1HH2Wn-000MVN-Br; Tue, 13 Feb 2007 18:37:21 +0000 Received: from localhost ([127.0.0.1] helo=roam.psg.com) by roam.psg.com with esmtp (Exim 4.66 (FreeBSD)) (envelope-from ) id 1HH2Tx-00081d-Od; Tue, 13 Feb 2007 10:34:25 -0800 From: Randy Bush MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Message-ID: <17874.1201.233347.849783@roam.psg.com> Date: Tue, 13 Feb 2007 08:34:25 -1000 To: John Kozubik References: <200702130927.l1D9RBn9034761@lurza.secnetix.de> <20070213093220.C95571@kozubik.com> Cc: freebsd-fs@freebsd.org Subject: Re: comments on newfs raw disk ? Safe ? (7 terabyte array) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 13 Feb 2007 18:37:25 -0000 > Also remember that filling a filesystem to capacity _while_ it is > being snapshotted will lock your system up[1]. I suppose some > interesting crash loops could arise from this bug on a near full > filesystem that someone is unlucky enough to background fsck. > > I think that FreeBSD needs to address the default implementation > of background fsck in general. UFS2 snapshots are dangerous and > unstable, and have been since their introduction in 5.x [2]. first, this smells of fud. second, this is not a problem for me. as the 4tb drive is the target of everyone else's dump/restore, i do not intend to snapshot it and dump it live. rasndh From owner-freebsd-fs@FreeBSD.ORG Tue Feb 13 18:51:57 2007 Return-Path: X-Original-To: freebsd-fs@freebsd.org Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id E712816A401 for ; Tue, 13 Feb 2007 18:51:57 +0000 (UTC) (envelope-from freebsd-fs@m.gmane.org) Received: from ciao.gmane.org (main.gmane.org [80.91.229.2]) by mx1.freebsd.org (Postfix) with ESMTP id A774B13C428 for ; Tue, 13 Feb 2007 18:51:57 +0000 (UTC) (envelope-from freebsd-fs@m.gmane.org) Received: from list by ciao.gmane.org with local (Exim 4.43) id 1HH2kq-0003dX-Ic for freebsd-fs@freebsd.org; Tue, 13 Feb 2007 19:51:52 +0100 Received: from 89-172-32-143.adsl.net.t-com.hr ([89.172.32.143]) by main.gmane.org with esmtp (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Tue, 13 Feb 2007 19:51:52 +0100 Received: from ivoras by 89-172-32-143.adsl.net.t-com.hr with local (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Tue, 13 Feb 2007 19:51:52 +0100 X-Injected-Via-Gmane: http://gmane.org/ To: freebsd-fs@freebsd.org From: Ivan Voras Date: Tue, 13 Feb 2007 19:51:38 +0100 Lines: 30 Message-ID: References: <45CD6FF5.8070007@freebsd.org> <20070213075627.63126.qmail@web34502.mail.mud.yahoo.com> <6FC9F9894A9F8C49A722CF9F2132FC2204C9DAB4@ms05.mailstreet2003.net> <6FC9F9894A9F8C49A722CF9F2132FC2204C9DAB5@ms05.mailstreet2003.net> <6FC9F9894A9F8C49A722CF9F2132FC2204C9DAB6@ms05.mailstreet2003.net> <45D1F30A.6080403@freebsd.org> <45D20037.1040102@freebsd.org> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="------------enig74B0243BCD0836FBEEBFFF16" X-Complaints-To: usenet@sea.gmane.org X-Gmane-NNTP-Posting-Host: 89-172-32-143.adsl.net.t-com.hr User-Agent: Thunderbird 1.5.0.9 (Windows/20061207) In-Reply-To: <45D20037.1040102@freebsd.org> X-Enigmail-Version: 0.94.1.2 Sender: news Subject: Re: UFS2 with SAN X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 13 Feb 2007 18:51:58 -0000 This is an OpenPGP/MIME signed message (RFC 2440 and 3156) --------------enig74B0243BCD0836FBEEBFFF16 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Eric Anderson wrote: > I've thought about this too - and even thought about making it a file > system layer, like nullfs, with a network-kernel piece to notify the > readers of changes, etc. I'm not certain if it would work or not.. Yes, I was thinking the same. Barring "unusual" problems during implementation, it would probably work. --------------enig74B0243BCD0836FBEEBFFF16 Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.4 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFF0gjBldnAQVacBcgRAgwlAKDFEqMvnXMHSL991AlJiIW0fEipuACfaFTV haZlbcFDonEsTFOLwp7GmOw= =vdaS -----END PGP SIGNATURE----- --------------enig74B0243BCD0836FBEEBFFF16-- From owner-freebsd-fs@FreeBSD.ORG Tue Feb 13 18:54:30 2007 Return-Path: X-Original-To: freebsd-fs@freebsd.org Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 5E64016A402 for ; Tue, 13 Feb 2007 18:54:30 +0000 (UTC) (envelope-from chris@arnold.se) Received: from av9-2-sn2.hy.skanova.net (av9-2-sn2.hy.skanova.net [81.228.8.180]) by mx1.freebsd.org (Postfix) with ESMTP id 1F16F13C48E for ; Tue, 13 Feb 2007 18:54:30 +0000 (UTC) (envelope-from chris@arnold.se) Received: by av9-2-sn2.hy.skanova.net (Postfix, from userid 502) id 9BA0638186; Tue, 13 Feb 2007 19:32:56 +0100 (CET) Received: from smtp4-2-sn2.hy.skanova.net (smtp4-2-sn2.hy.skanova.net [81.228.8.93]) by av9-2-sn2.hy.skanova.net (Postfix) with ESMTP id 4700338070; Tue, 13 Feb 2007 19:32:56 +0100 (CET) Received: from chrishome.localnet (81-224-156-16-o1033.telia.com [81.224.156.16]) by smtp4-2-sn2.hy.skanova.net (Postfix) with ESMTP id 05EB237E44; Tue, 13 Feb 2007 19:32:55 +0100 (CET) Received: from localhost (localhost [127.0.0.1]) by chrishome.localnet (8.13.6/8.13.6) with ESMTP id l1DIWs5T004767; Tue, 13 Feb 2007 19:32:54 +0100 (CET) (envelope-from chris@arnold.se) Date: Tue, 13 Feb 2007 19:32:54 +0100 (CET) From: Christopher Arnold X-X-Sender: chris@chrishome.localnet To: Ivan Voras In-Reply-To: Message-ID: <20070213192906.U726@chrishome.localnet> References: <45CD6FF5.8070007@freebsd.org> <20070213075627.63126.qmail@web34502.mail.mud.yahoo.com> <6FC9F9894A9F8C49A722CF9F2132FC2204C9DAB4@ms05.mailstreet2003.net> <6FC9F9894A9F8C49A722CF9F2132FC2204C9DAB5@ms05.mailstreet2003.net> <6FC9F9894A9F8C49A722CF9F2132FC2204C9DAB6@ms05.mailstreet2003.net> <45D1F30A.6080403@freebsd.org> X-message-flag: =?ISO-8859-1?Q?Outlook_isn=B4t_compliant_with_current_standards?= =?ISO-8859-1?Q?_please_install_another_mail_client!?= MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: freebsd-fs@freebsd.org Subject: Re: UFS2 with SAN X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 13 Feb 2007 18:54:30 -0000 On Tue, 13 Feb 2007, Ivan Voras wrote: > Eric Anderson wrote: > >> You might look at tdfs (fuse module from Ivan Voras I believe). It's >> probably horribly beta (not knocking it Ivan - I'm sure it's an amazing >> start), so lots of testing would need to be done. It's also most likely >> not high performance. > > Yes, I'm realistic about it - it is "horribly beta" :) > Is there anyone who has looked into using AFS as a distributed filestore on FreeBSD. Whouldn't that be ideal for a farm of webservers doing mostly read? Yes im aware of issues running AFS on 6.x but i belive there exists an existing implementation under 5.x. And if enough people need it i guess we could make shure it runs ok under 6.x to. /Chris -- www.strexa.com From owner-freebsd-fs@FreeBSD.ORG Tue Feb 13 19:01:26 2007 Return-Path: X-Original-To: freebsd-fs@FreeBSD.ORG Delivered-To: freebsd-fs@FreeBSD.ORG Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 07E3116A468 for ; Tue, 13 Feb 2007 19:01:26 +0000 (UTC) (envelope-from john@kozubik.com) Received: from kozubik.com (kozubik.com [69.43.165.2]) by mx1.freebsd.org (Postfix) with ESMTP id EA2C613C4B6 for ; Tue, 13 Feb 2007 19:01:25 +0000 (UTC) (envelope-from john@kozubik.com) Received: from kozubik.com (localhost [127.0.0.1]) by kozubik.com (8.12.11/8.12.11) with ESMTP id l1DJ1RDl025456 for ; Tue, 13 Feb 2007 11:01:27 -0800 (PST) (envelope-from john@kozubik.com) Received: from localhost (john@localhost) by kozubik.com (8.12.11/8.12.11/Submit) with ESMTP id l1DJ1RLg025450 for ; Tue, 13 Feb 2007 11:01:27 -0800 (PST) (envelope-from john@kozubik.com) Date: Tue, 13 Feb 2007 11:01:27 -0800 (PST) From: John Kozubik To: freebsd-fs@FreeBSD.ORG In-Reply-To: <200702131820.l1DIKumD009658@lurza.secnetix.de> Message-ID: <20070213104959.K95571@kozubik.com> References: <200702131820.l1DIKumD009658@lurza.secnetix.de> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Cc: Subject: Re: comments on newfs raw disk ? Safe ? (7 terabyte array) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 13 Feb 2007 19:01:26 -0000 On Tue, 13 Feb 2007, Oliver Fromme wrote: > > > A bit careful here ... Background fsck had some issues, > > > especially when the machine crashed or is otherwise reset > > > while the background fsck is still running. It resulted > > > in corruption that could not be repaired by fsck anymore. > > > I don't know if all of those issues have been resolved in > > > RELENG_6, but personally I always disable background fsck > > > on all of my machines, just to be safe. > > > > [...] > > UFS2 snapshots are dangerous and unstable, > > and have been since their introduction in 5.x [2]. > > That's not what I wrote. I wrote that they _had_ issues, > and that I do not know if they have been fixed. I don't > recall any reports of problems recently (i.e. in the past > few months), and there are no open PRs that seem to relate > to the current code, so those issues may very well have > been fixed. It's just my personal paranoia that lets me > disable bg fsck on my machines (and I don't really need > bg fsck anyway). Fair enough. For your information, they are still dangerous and unstable[1][2][3]. Your initial assessment is still valid today, unfortunately. FWIW, [1] is open and relates to the current code. It (bg_fsck and UFS2 snapshots) has gotten better over time - but it is still not something that I feel is fair to enable by default, as if it were rock solid, and force it onto unsuspecting end users who are not as well informed as you and I are. John Kozubik - john@kozubik.com - http://www.kozubik.com [1] http://lists.freebsd.org/pipermail/freebsd-bugs/2006-January/016703.html [2] http://lists.freebsd.org/pipermail/freebsd-bugs/2004-July/007574.html [3] [2, above] has been fixed, but large quantity inode movements keep coming back to haunt snapshots every other release or so... From owner-freebsd-fs@FreeBSD.ORG Tue Feb 13 19:12:34 2007 Return-Path: X-Original-To: freebsd-fs@freebsd.org Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id CBB7F16A4A1 for ; Tue, 13 Feb 2007 19:12:34 +0000 (UTC) (envelope-from julian@elischer.org) Received: from outV.internet-mail-service.net (outV.internet-mail-service.net [216.240.47.245]) by mx1.freebsd.org (Postfix) with ESMTP id ADB7E13C48D for ; Tue, 13 Feb 2007 19:12:34 +0000 (UTC) (envelope-from julian@elischer.org) Received: from mx0.idiom.com (HELO idiom.com) (216.240.32.160) by out.internet-mail-service.net (qpsmtpd/0.32) with ESMTP; Tue, 13 Feb 2007 10:49:18 -0800 Received: from [192.168.2.5] (home.elischer.org [216.240.48.38]) by idiom.com (Postfix) with ESMTP id 2636E125B03; Tue, 13 Feb 2007 11:12:33 -0800 (PST) Message-ID: <45D20DA0.4050600@elischer.org> Date: Tue, 13 Feb 2007 11:12:32 -0800 From: Julian Elischer User-Agent: Thunderbird 1.5.0.9 (Macintosh/20061207) MIME-Version: 1.0 To: Eric Anderson References: <45CD6FF5.8070007@freebsd.org> <20070213075627.63126.qmail@web34502.mail.mud.yahoo.com> <6FC9F9894A9F8C49A722CF9F2132FC2204C9DAB4@ms05.mailstreet2003.net> <6FC9F9894A9F8C49A722CF9F2132FC2204C9DAB5@ms05.mailstreet2003.net> <6FC9F9894A9F8C49A722CF9F2132FC2204C9DAB6@ms05.mailstreet2003.net> <45D1F30A.6080403@freebsd.org> <45D20037.1040102@freebsd.org> In-Reply-To: <45D20037.1040102@freebsd.org> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-fs@freebsd.org, Ivan Voras Subject: Re: UFS2 with SAN X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 13 Feb 2007 19:12:34 -0000 Eric Anderson wrote: > On 02/13/07 12:09, Ivan Voras wrote: >> Eric Anderson wrote: >> >>> You might look at tdfs (fuse module from Ivan Voras I believe). It's >>> probably horribly beta (not knocking it Ivan - I'm sure it's an amazing >>> start), so lots of testing would need to be done. It's also most likely >>> not high performance. >> >> Yes, I'm realistic about it - it is "horribly beta" :) >> >> Two reasons: one: I couldn't amass enough interested parties to help >> test it (and possibly work on it), and second: while it's a nice problem >> to tackle, I don't [yet] need it that badly to start working on it >> outside of free time). In any case, TDFS doesn't have locking because >> FUSE doesn't have it (or didn't when I started doing it), so it stays >> single-writer-multiple-readers until that changes. >> >> I'm thinking of re-creating it in kernel mode (and in this form it could >> also have locking) but not any time soon. Maybe during the summer (SoC? >> probably.). >> > > > I've thought about this too - and even thought about making it a file > system layer, like nullfs, with a network-kernel piece to notify the > readers of changes, etc. I'm not certain if it would work or not.. it will be very interesting to see the outcome from Matt's cluster filesystem on Dragonfly. > > Eric > > _______________________________________________ > freebsd-fs@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" From owner-freebsd-fs@FreeBSD.ORG Tue Feb 13 19:37:51 2007 Return-Path: X-Original-To: freebsd-fs@freebsd.org Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 5097816A400 for ; Tue, 13 Feb 2007 19:37:51 +0000 (UTC) (envelope-from rees@citi.umich.edu) Received: from citi.umich.edu (citi.umich.edu [141.211.133.111]) by mx1.freebsd.org (Postfix) with ESMTP id 36D1D13C461 for ; Tue, 13 Feb 2007 19:37:51 +0000 (UTC) (envelope-from rees@citi.umich.edu) Received: from citi.umich.edu (d-107.209.eecs.umich.edu [141.212.107.209]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by citi.umich.edu (Postfix) with ESMTP id 07AB439605; Tue, 13 Feb 2007 14:20:23 -0500 (EST) Date: Tue, 13 Feb 2007 14:20:22 -0500 From: Jim Rees To: Christopher Arnold Message-ID: <20070213192022.GA15500@citi.umich.edu> References: <45CD6FF5.8070007@freebsd.org> <20070213075627.63126.qmail@web34502.mail.mud.yahoo.com> <6FC9F9894A9F8C49A722CF9F2132FC2204C9DAB4@ms05.mailstreet2003.net> <6FC9F9894A9F8C49A722CF9F2132FC2204C9DAB5@ms05.mailstreet2003.net> <6FC9F9894A9F8C49A722CF9F2132FC2204C9DAB6@ms05.mailstreet2003.net> <45D1F30A.6080403@freebsd.org> <20070213192906.U726@chrishome.localnet> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20070213192906.U726@chrishome.localnet> Cc: freebsd-fs@freebsd.org, Ivan Voras Subject: Re: UFS2 with SAN X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 13 Feb 2007 19:37:51 -0000 Afs does make a very nice web back end, but there is enough administrative overhead that you might not want it just for web service. It works particularly well if your working set fits in the cache (I suppose that's true of most things). It wouldn't take much to get OpenAFS in pretty good shape on FreeBSD. It mostly has some locking and vnode ref count issues that shouldn't be too hard to fix for someone who knows what they're doing. I'd be happy to point anyone in the right direction. From owner-freebsd-fs@FreeBSD.ORG Tue Feb 13 19:47:39 2007 Return-Path: X-Original-To: freebsd-fs@freebsd.org Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 6C02616A402 for ; Tue, 13 Feb 2007 19:47:39 +0000 (UTC) (envelope-from anderson@freebsd.org) Received: from mh1.centtech.com (moat3.centtech.com [64.129.166.50]) by mx1.freebsd.org (Postfix) with ESMTP id 4914213C442 for ; Tue, 13 Feb 2007 19:47:39 +0000 (UTC) (envelope-from anderson@freebsd.org) Received: from [10.177.171.220] (neutrino.centtech.com [10.177.171.220]) by mh1.centtech.com (8.13.8/8.13.8) with ESMTP id l1DJlcex038757; Tue, 13 Feb 2007 13:47:38 -0600 (CST) (envelope-from anderson@freebsd.org) Message-ID: <45D215DA.2000306@freebsd.org> Date: Tue, 13 Feb 2007 13:47:38 -0600 From: Eric Anderson User-Agent: Thunderbird 1.5.0.9 (X11/20070204) MIME-Version: 1.0 To: John Kozubik References: <200702131820.l1DIKumD009658@lurza.secnetix.de> <20070213104959.K95571@kozubik.com> In-Reply-To: <20070213104959.K95571@kozubik.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Virus-Scanned: ClamAV 0.88.4/2562/Tue Feb 13 09:18:56 2007 on mh1.centtech.com X-Virus-Status: Clean X-Spam-Status: No, score=-2.6 required=8.0 tests=AWL,BAYES_00 autolearn=ham version=3.1.6 X-Spam-Checker-Version: SpamAssassin 3.1.6 (2006-10-03) on mh1.centtech.com Cc: freebsd-fs@freebsd.org Subject: Re: comments on newfs raw disk ? Safe ? (7 terabyte array) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 13 Feb 2007 19:47:39 -0000 On 02/13/07 13:01, John Kozubik wrote: > > On Tue, 13 Feb 2007, Oliver Fromme wrote: > >> > > A bit careful here ... Background fsck had some issues, >> > > especially when the machine crashed or is otherwise reset >> > > while the background fsck is still running. It resulted >> > > in corruption that could not be repaired by fsck anymore. >> > > I don't know if all of those issues have been resolved in >> > > RELENG_6, but personally I always disable background fsck >> > > on all of my machines, just to be safe. >> > >> > [...] >> > UFS2 snapshots are dangerous and unstable, >> > and have been since their introduction in 5.x [2]. >> >> That's not what I wrote. I wrote that they _had_ issues, >> and that I do not know if they have been fixed. I don't >> recall any reports of problems recently (i.e. in the past >> few months), and there are no open PRs that seem to relate >> to the current code, so those issues may very well have >> been fixed. It's just my personal paranoia that lets me >> disable bg fsck on my machines (and I don't really need >> bg fsck anyway). > > > Fair enough. For your information, they are still dangerous and > unstable[1][2][3]. Your initial assessment is still valid today, > unfortunately. FWIW, [1] is open and relates to the current code. > > It (bg_fsck and UFS2 snapshots) has gotten better over time - but it is > still not something that I feel is fair to enable by default, as if it > were rock solid, and force it onto unsuspecting end users who are not as > well informed as you and I are. Uhh, aren't those threads below at *least* a year old, or am I misreading it? If so - then I think you in fact need to become more informed, since massive UFS updates have been done in the past 6 months. If you have pointers to more recent issues, please post them.. Eric > [1] http://lists.freebsd.org/pipermail/freebsd-bugs/2006-January/016703.html > [2] http://lists.freebsd.org/pipermail/freebsd-bugs/2004-July/007574.html > [3] [2, above] has been fixed, but large quantity inode movements keep > coming back to haunt snapshots every other release or so... > _______________________________________________ > freebsd-fs@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" From owner-freebsd-fs@FreeBSD.ORG Tue Feb 13 20:03:53 2007 Return-Path: X-Original-To: freebsd-fs@freebsd.org Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 9E48616A402 for ; Tue, 13 Feb 2007 20:03:53 +0000 (UTC) (envelope-from john@kozubik.com) Received: from kozubik.com (kozubik.com [69.43.165.2]) by mx1.freebsd.org (Postfix) with ESMTP id 8211E13C48E for ; Tue, 13 Feb 2007 20:03:53 +0000 (UTC) (envelope-from john@kozubik.com) Received: from kozubik.com (localhost [127.0.0.1]) by kozubik.com (8.12.11/8.12.11) with ESMTP id l1DK3sRC027013; Tue, 13 Feb 2007 12:03:54 -0800 (PST) (envelope-from john@kozubik.com) Received: from localhost (john@localhost) by kozubik.com (8.12.11/8.12.11/Submit) with ESMTP id l1DK3sTE027010; Tue, 13 Feb 2007 12:03:54 -0800 (PST) (envelope-from john@kozubik.com) Date: Tue, 13 Feb 2007 12:03:54 -0800 (PST) From: John Kozubik To: Eric Anderson In-Reply-To: <45D215DA.2000306@freebsd.org> Message-ID: <20070213115237.A95571@kozubik.com> References: <200702131820.l1DIKumD009658@lurza.secnetix.de> <20070213104959.K95571@kozubik.com> <45D215DA.2000306@freebsd.org> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Cc: freebsd-fs@freebsd.org Subject: Re: comments on newfs raw disk ? Safe ? (7 terabyte array) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 13 Feb 2007 20:03:53 -0000 On Tue, 13 Feb 2007, Eric Anderson wrote: > > Fair enough. For your information, they are still dangerous and > > unstable[1][2][3]. Your initial assessment is still valid today, > > unfortunately. FWIW, [1] is open and relates to the current code. > > > > It (bg_fsck and UFS2 snapshots) has gotten better over time - but it is > > still not something that I feel is fair to enable by default, as if it > > were rock solid, and force it onto unsuspecting end users who are not as > > well informed as you and I are. > > Uhh, aren't those threads below at *least* a year old, or am I > misreading it? If so - then I think you in fact need to become more > informed, since massive UFS updates have been done in the past 6 months. > If you have pointers to more recent issues, please post them.. [1] Is from January 2006, and is currently acknowledged as an existing problem that _is not_ fixed in 6.2. Apparently there is some pretty heavy lifting that needs to be done to fix this "fill disk while snapshotting" problem. It is an open, current problem. [2], as I state below, has been fixed, but I keep re-demonstrating it every other release or so. It has been my observation that high volume inode movement on snapshotted UFS2 filesystems keeps popping up as a problem. I think you misunderstand my point in all of this. None of it affects me at all - I keep abreast of freebsd-fs, I test things, and I, like many others, simply don't use these features. The end. My point is not to complain about the current state of snapshots and bg_fsck. My point is that the average user is not active on these lists and should not be subject to an _enabled by default_ feature set that is dangerous. If they want to use snapshots and bg_fsck, then by all means - but have them turn it on themselves with some warning as to the ramifications of doing so. > > [1] http://lists.freebsd.org/pipermail/freebsd-bugs/2006-January/016703.html > > [2] http://lists.freebsd.org/pipermail/freebsd-bugs/2004-July/007574.html > > [3] [2, above] has been fixed, but large quantity inode movements keep > > coming back to haunt snapshots every other release or so... John Kozubik - john@kozubik.com - http://www.kozubik.com From owner-freebsd-fs@FreeBSD.ORG Tue Feb 13 20:07:37 2007 Return-Path: X-Original-To: freebsd-fs@freebsd.org Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id DAD6416A47E for ; Tue, 13 Feb 2007 20:07:37 +0000 (UTC) (envelope-from anderson@freebsd.org) Received: from mh1.centtech.com (moat3.centtech.com [64.129.166.50]) by mx1.freebsd.org (Postfix) with ESMTP id B3E7713C4B3 for ; Tue, 13 Feb 2007 20:07:37 +0000 (UTC) (envelope-from anderson@freebsd.org) Received: from [10.177.171.220] (neutrino.centtech.com [10.177.171.220]) by mh1.centtech.com (8.13.8/8.13.8) with ESMTP id l1DK7Udr042858; Tue, 13 Feb 2007 14:07:30 -0600 (CST) (envelope-from anderson@freebsd.org) Message-ID: <45D21A82.7000900@freebsd.org> Date: Tue, 13 Feb 2007 14:07:30 -0600 From: Eric Anderson User-Agent: Thunderbird 1.5.0.9 (X11/20070204) MIME-Version: 1.0 To: Julian Elischer References: <45CD6FF5.8070007@freebsd.org> <20070213075627.63126.qmail@web34502.mail.mud.yahoo.com> <6FC9F9894A9F8C49A722CF9F2132FC2204C9DAB4@ms05.mailstreet2003.net> <6FC9F9894A9F8C49A722CF9F2132FC2204C9DAB5@ms05.mailstreet2003.net> <6FC9F9894A9F8C49A722CF9F2132FC2204C9DAB6@ms05.mailstreet2003.net> <45D1F30A.6080403@freebsd.org> <45D20037.1040102@freebsd.org> <45D20DA0.4050600@elischer.org> In-Reply-To: <45D20DA0.4050600@elischer.org> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Virus-Scanned: ClamAV 0.88.4/2562/Tue Feb 13 09:18:56 2007 on mh1.centtech.com X-Virus-Status: Clean X-Spam-Status: No, score=-2.6 required=8.0 tests=AWL,BAYES_00 autolearn=ham version=3.1.6 X-Spam-Checker-Version: SpamAssassin 3.1.6 (2006-10-03) on mh1.centtech.com Cc: freebsd-fs@freebsd.org, Ivan Voras Subject: Re: UFS2 with SAN X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 13 Feb 2007 20:07:38 -0000 On 02/13/07 13:12, Julian Elischer wrote: > Eric Anderson wrote: >> On 02/13/07 12:09, Ivan Voras wrote: >>> Eric Anderson wrote: >>> >>>> You might look at tdfs (fuse module from Ivan Voras I believe). It's >>>> probably horribly beta (not knocking it Ivan - I'm sure it's an amazing >>>> start), so lots of testing would need to be done. It's also most likely >>>> not high performance. >>> Yes, I'm realistic about it - it is "horribly beta" :) >>> >>> Two reasons: one: I couldn't amass enough interested parties to help >>> test it (and possibly work on it), and second: while it's a nice problem >>> to tackle, I don't [yet] need it that badly to start working on it >>> outside of free time). In any case, TDFS doesn't have locking because >>> FUSE doesn't have it (or didn't when I started doing it), so it stays >>> single-writer-multiple-readers until that changes. >>> >>> I'm thinking of re-creating it in kernel mode (and in this form it could >>> also have locking) but not any time soon. Maybe during the summer (SoC? >>> probably.). >>> >> >> I've thought about this too - and even thought about making it a file >> system layer, like nullfs, with a network-kernel piece to notify the >> readers of changes, etc. I'm not certain if it would work or not.. > > it will be very interesting to see the outcome from Matt's cluster > filesystem on Dragonfly. Is it actually being developed? Last I checked, there was talk, but no code.. :( Eric From owner-freebsd-fs@FreeBSD.ORG Tue Feb 13 20:20:45 2007 Return-Path: X-Original-To: freebsd-fs@freebsd.org Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id E144A16A46B for ; Tue, 13 Feb 2007 20:20:45 +0000 (UTC) (envelope-from julian@elischer.org) Received: from outD.internet-mail-service.net (outD.internet-mail-service.net [216.240.47.227]) by mx1.freebsd.org (Postfix) with ESMTP id C9B7A13C4AA for ; Tue, 13 Feb 2007 20:20:45 +0000 (UTC) (envelope-from julian@elischer.org) Received: from mx0.idiom.com (HELO idiom.com) (216.240.32.160) by out.internet-mail-service.net (qpsmtpd/0.32) with ESMTP; Tue, 13 Feb 2007 11:57:25 -0800 Received: from [192.168.2.5] (home.elischer.org [216.240.48.38]) by idiom.com (Postfix) with ESMTP id 555AC125B31; Tue, 13 Feb 2007 12:19:53 -0800 (PST) Message-ID: <45D21D68.5030503@elischer.org> Date: Tue, 13 Feb 2007 12:19:52 -0800 From: Julian Elischer User-Agent: Thunderbird 1.5.0.9 (Macintosh/20061207) MIME-Version: 1.0 To: Eric Anderson References: <45CD6FF5.8070007@freebsd.org> <20070213075627.63126.qmail@web34502.mail.mud.yahoo.com> <6FC9F9894A9F8C49A722CF9F2132FC2204C9DAB4@ms05.mailstreet2003.net> <6FC9F9894A9F8C49A722CF9F2132FC2204C9DAB5@ms05.mailstreet2003.net> <6FC9F9894A9F8C49A722CF9F2132FC2204C9DAB6@ms05.mailstreet2003.net> <45D1F30A.6080403@freebsd.org> <45D20037.1040102@freebsd.org> <45D20DA0.4050600@elischer.org> <45D21A82.7000900@freebsd.org> In-Reply-To: <45D21A82.7000900@freebsd.org> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-fs@freebsd.org, Ivan Voras Subject: Re: UFS2 with SAN X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 13 Feb 2007 20:20:46 -0000 Eric Anderson wrote: > On 02/13/07 13:12, Julian Elischer wrote: >> Eric Anderson wrote: >>> On 02/13/07 12:09, Ivan Voras wrote: >>>> Eric Anderson wrote: >>>> >>>>> You might look at tdfs (fuse module from Ivan Voras I believe). It's >>>>> probably horribly beta (not knocking it Ivan - I'm sure it's an >>>>> amazing >>>>> start), so lots of testing would need to be done. It's also most >>>>> likely >>>>> not high performance. >>>> Yes, I'm realistic about it - it is "horribly beta" :) >>>> >>>> Two reasons: one: I couldn't amass enough interested parties to help >>>> test it (and possibly work on it), and second: while it's a nice >>>> problem >>>> to tackle, I don't [yet] need it that badly to start working on it >>>> outside of free time). In any case, TDFS doesn't have locking because >>>> FUSE doesn't have it (or didn't when I started doing it), so it stays >>>> single-writer-multiple-readers until that changes. >>>> >>>> I'm thinking of re-creating it in kernel mode (and in this form it >>>> could >>>> also have locking) but not any time soon. Maybe during the summer (SoC? >>>> probably.). >>>> >>> >>> I've thought about this too - and even thought about making it a file >>> system layer, like nullfs, with a network-kernel piece to notify the >>> readers of changes, etc. I'm not certain if it would work or not.. >> >> it will be very interesting to see the outcome from Matt's cluster >> filesystem on Dragonfly. > > > Is it actually being developed? Last I checked, there was talk, but no > code.. :( He's doing all the underlying stuff. for example he has done a lot in the VFS layer to get ready for it.. (e.g. cache coherency support). > > > Eric From owner-freebsd-fs@FreeBSD.ORG Tue Feb 13 20:24:02 2007 Return-Path: X-Original-To: freebsd-fs@FreeBSD.ORG Delivered-To: freebsd-fs@FreeBSD.ORG Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 3520D16A407 for ; Tue, 13 Feb 2007 20:24:02 +0000 (UTC) (envelope-from olli@lurza.secnetix.de) Received: from lurza.secnetix.de (lurza.secnetix.de [83.120.8.8]) by mx1.freebsd.org (Postfix) with ESMTP id 9D2B013C46B for ; Tue, 13 Feb 2007 20:24:01 +0000 (UTC) (envelope-from olli@lurza.secnetix.de) Received: from lurza.secnetix.de (uhktgb@localhost [127.0.0.1]) by lurza.secnetix.de (8.13.4/8.13.4) with ESMTP id l1DKNsS2022199; Tue, 13 Feb 2007 21:24:00 +0100 (CET) (envelope-from oliver.fromme@secnetix.de) Received: (from olli@localhost) by lurza.secnetix.de (8.13.4/8.13.1/Submit) id l1DKNsjM022198; Tue, 13 Feb 2007 21:23:54 +0100 (CET) (envelope-from olli) Date: Tue, 13 Feb 2007 21:23:54 +0100 (CET) Message-Id: <200702132023.l1DKNsjM022198@lurza.secnetix.de> From: Oliver Fromme To: freebsd-fs@FreeBSD.ORG, john@kozubik.com In-Reply-To: <20070213104959.K95571@kozubik.com> X-Newsgroups: list.freebsd-fs User-Agent: tin/1.8.2-20060425 ("Shillay") (UNIX) (FreeBSD/4.11-STABLE (i386)) MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 8bit X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-2.1.2 (lurza.secnetix.de [127.0.0.1]); Tue, 13 Feb 2007 21:24:00 +0100 (CET) Cc: Subject: Re: comments on newfs raw disk ? Safe ? (7 terabyte array) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: freebsd-fs@FreeBSD.ORG, john@kozubik.com List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 13 Feb 2007 20:24:02 -0000 John Kozubik wrote: > Oliver Fromme wrote: > > That's not what I wrote. I wrote that they _had_ issues, > > and that I do not know if they have been fixed. I don't > > recall any reports of problems recently (i.e. in the past > > few months), and there are no open PRs that seem to relate > > to the current code, so those issues may very well have > > been fixed. It's just my personal paranoia that lets me > > disable bg fsck on my machines (and I don't really need > > bg fsck anyway). > > Fair enough. For your information, they are still dangerous and > unstable[1][2][3]. Well, lets see ... [1] is from January 2006, applies to 6.0-RELEASE and has zero follow-ups. [2] is from July 2004 and was closed in May 2005 because it wasn't reproducible with the new compiler and no more feedback from the submitter was provided. [3] doesn't link to any PR or other information. > Your initial assessment is still valid today, > unfortunately. FWIW, [1] is open and relates to the current code. It says 6.0-RELEASE. FreeBSD 6 might have been "-current" at that time, but it's hardly current today. :-) > [1] http://lists.freebsd.org/pipermail/freebsd-bugs/2006-January/016703.html > [2] http://lists.freebsd.org/pipermail/freebsd-bugs/2004-July/007574.html > [3] [2, above] has been fixed, but large quantity inode movements keep > coming back to haunt snapshots every other release or so... In that case a new PR should be opened, I think. Best regards Oliver -- Oliver Fromme, secnetix GmbH & Co. KG, Marktplatz 29, 85567 Grafing b. M. Handelsregister: Registergericht Muenchen, HRA 74606, Geschäftsfuehrung: secnetix Verwaltungsgesellsch. mbH, Handelsregister: Registergericht Mün- chen, HRB 125758, Geschäftsführer: Maik Bachmann, Olaf Erb, Ralf Gebhart Any opinions expressed in this message are personal to the author and may not necessarily reflect the opinions of secnetix GmbH & Co KG in any way. FreeBSD-Dienstleistungen, -Produkte und mehr: http://www.secnetix.de/bsd "I learned Java 3 years before Python. It was my language of choice. It took me two weekends with Python before I was more productive with it than with Java." -- Anthony Roberts From owner-freebsd-fs@FreeBSD.ORG Wed Feb 14 06:49:44 2007 Return-Path: X-Original-To: freebsd-fs@freebsd.org Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 890C216A406 for ; Wed, 14 Feb 2007 06:49:44 +0000 (UTC) (envelope-from randy@psg.com) Received: from rip.psg.com (rip.psg.com [147.28.0.39]) by mx1.freebsd.org (Postfix) with ESMTP id 70D1E13C4AA for ; Wed, 14 Feb 2007 06:49:44 +0000 (UTC) (envelope-from randy@psg.com) Received: from localhost ([127.0.0.1] helo=roam.psg.com) by rip.psg.com with esmtp (Exim 4.66 (FreeBSD)) (envelope-from ) id 1HHDxW-000FEw-T4; Wed, 14 Feb 2007 06:49:43 +0000 Received: from localhost ([127.0.0.1] helo=roam.psg.com) by roam.psg.com with esmtp (Exim 4.66 (FreeBSD)) (envelope-from ) id 1HHDwb-0000Yj-TC; Tue, 13 Feb 2007 20:48:46 -1000 From: Randy Bush MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Message-ID: <17874.45261.367361.188024@roam.psg.com> Date: Tue, 13 Feb 2007 20:48:45 -1000 To: Oliver Fromme References: <17873.33060.746546.773210@roam.psg.com> <200702130927.l1D9RBn9034761@lurza.secnetix.de> Cc: freebsd-fs@freebsd.org Subject: Re: comments on newfs raw disk ? Safe ? (7 terabyte array) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 14 Feb 2007 06:49:44 -0000 4TB newfs -b 16384 -f 2048 -i 262144 # time fsck -y /dev/da1 ** /dev/da1 ** Last Mounted on /mnt ** Phase 1 - Check Blocks and Sizes ** Phase 2 - Check Pathnames ** Phase 3 - Check Connectivity ** Phase 4 - Check Reference Counts ** Phase 5 - Check Cyl groups 2 files, 2 used, 2173822273 free (17 frags, 271727782 blocks, 0.0% fragmentation) ***** FILE SYSTEM MARKED CLEAN ***** real 8m58.571s user 0m31.769s sys 0m1.246s randy From owner-freebsd-fs@FreeBSD.ORG Wed Feb 14 12:55:15 2007 Return-Path: X-Original-To: fs@freebsd.org Delivered-To: freebsd-fs@FreeBSD.ORG Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id A9B6D16A401; Wed, 14 Feb 2007 12:55:15 +0000 (UTC) (envelope-from joe@tao.org.uk) Received: from mailhost.tao.org.uk (transwarp.tao.org.uk [87.74.4.34]) by mx1.freebsd.org (Postfix) with ESMTP id 5E05A13C428; Wed, 14 Feb 2007 12:55:15 +0000 (UTC) (envelope-from joe@tao.org.uk) Received: from genius.tao.org.uk (wireless58.dhcp.tao.org.uk [87.74.4.58]) by mailhost.tao.org.uk (Postfix) with ESMTP id 8507865CE; Wed, 14 Feb 2007 12:55:14 +0000 (GMT) Received: by genius.tao.org.uk (Postfix, from userid 1000) id 8155F40D4; Wed, 14 Feb 2007 12:55:10 +0000 (GMT) Date: Wed, 14 Feb 2007 12:55:10 +0000 From: Josef Karthauser To: Mark Saad , Joe Koberg , stable@freebsd.org, fs@freebsd.org Message-ID: <20070214125510.GB14160@genius.tao.org.uk> Mail-Followup-To: Josef Karthauser , Mark Saad , Joe Koberg , stable@freebsd.org, fs@freebsd.org References: <20070115112106.GA2304@genius.tao.org.uk> <20070115115650.GB2304@genius.tao.org.uk> <45AB9BE4.1030606@osoft.us> <20070127224749.GA8203@genius.tao.org.uk> <20070130114459.GI2616@genius.tao.org.uk> <45BF7797.6000708@datapipe.com> <20070130174414.GE6751@genius.tao.org.uk> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="6sX45UoQRIJXqkqR" Content-Disposition: inline In-Reply-To: <20070130174414.GE6751@genius.tao.org.uk> User-Agent: Mutt/1.5.11 Cc: Subject: Re: mpt problems. (Re: Dell hardware raid 0 (sas5ir) or gmirror?) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 14 Feb 2007 12:55:15 -0000 --6sX45UoQRIJXqkqR Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Tue, Jan 30, 2007 at 05:44:14PM +0000, Josef Karthauser wrote: > On Tue, Jan 30, 2007 at 11:51:35AM -0500, Mark Saad wrote: > > Hello > > So the HP DL320G4 servers come with a similar LSI=20 > > Logic SAS controller. In 6.2-RELEASE was seeing the same=20 > > issue; > > The fix I found was a firmware update from HP HP Firmware=20 > > Update 7.6.0 . After applied the mpt messages went away . > > I beelive this is a bug in the LSI firmware more then the=20 > > FreeBSD OS, as this controller has been very odd=20 > > regardless of what os I am running on it. > >=20 >=20 > I can believe that. I wonder how I should put the issue to Dell so that > they can understand what I'm on about and fix it! The controller > appears to be a custom version, at least the card probes as Dell part > not an LSIlogic part. >=20 I spoke to Dell, but they don't have a newer firmware yet. I've reported the issue, and I guess that I'll have to see whether they fix it at some point. I the mean time I've nailed the tags in rc.local, so that they get fixed at boot time. % cat /etc/rc.local =20 camcontrol tags da0 -N 119 camcontrol tags da1 -N 119 Joe --6sX45UoQRIJXqkqR Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.4 (FreeBSD) iEYEARECAAYFAkXTBq4ACgkQXVIcjOaxUBbMvgCgtMVDY/WSWPZ536MdVjWWOAm4 UAcAoNnajaw6tAzRC7toG7JrQSlMN3V3 =z4S/ -----END PGP SIGNATURE----- --6sX45UoQRIJXqkqR-- From owner-freebsd-fs@FreeBSD.ORG Wed Feb 14 12:59:52 2007 Return-Path: X-Original-To: freebsd-fs@freebsd.org Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 5E9ED16A400 for ; Wed, 14 Feb 2007 12:59:52 +0000 (UTC) (envelope-from anderson@freebsd.org) Received: from mh1.centtech.com (moat3.centtech.com [64.129.166.50]) by mx1.freebsd.org (Postfix) with ESMTP id 3179613C491 for ; Wed, 14 Feb 2007 12:59:52 +0000 (UTC) (envelope-from anderson@freebsd.org) Received: from [10.177.171.220] (neutrino.centtech.com [10.177.171.220]) by mh1.centtech.com (8.13.8/8.13.8) with ESMTP id l1ECxopo036565; Wed, 14 Feb 2007 06:59:51 -0600 (CST) (envelope-from anderson@freebsd.org) Message-ID: <45D307C7.6020409@freebsd.org> Date: Wed, 14 Feb 2007 06:59:51 -0600 From: Eric Anderson User-Agent: Thunderbird 1.5.0.9 (X11/20070204) MIME-Version: 1.0 To: Randy Bush References: <17873.33060.746546.773210@roam.psg.com> <200702130927.l1D9RBn9034761@lurza.secnetix.de> <17874.45261.367361.188024@roam.psg.com> In-Reply-To: <17874.45261.367361.188024@roam.psg.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Virus-Scanned: ClamAV 0.88.4/2566/Wed Feb 14 05:44:20 2007 on mh1.centtech.com X-Virus-Status: Clean X-Spam-Status: No, score=-2.6 required=8.0 tests=AWL,BAYES_00 autolearn=ham version=3.1.6 X-Spam-Checker-Version: SpamAssassin 3.1.6 (2006-10-03) on mh1.centtech.com Cc: freebsd-fs@freebsd.org, Oliver Fromme Subject: Re: comments on newfs raw disk ? Safe ? (7 terabyte array) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 14 Feb 2007 12:59:52 -0000 On 02/14/07 00:48, Randy Bush wrote: > 4TB > newfs -b 16384 -f 2048 -i 262144 > > # time fsck -y /dev/da1 > ** /dev/da1 > ** Last Mounted on /mnt > ** Phase 1 - Check Blocks and Sizes > ** Phase 2 - Check Pathnames > ** Phase 3 - Check Connectivity > ** Phase 4 - Check Reference Counts > ** Phase 5 - Check Cyl groups > 2 files, 2 used, 2173822273 free (17 frags, 271727782 blocks, 0.0% fragmentation) > > ***** FILE SYSTEM MARKED CLEAN ***** > > real 8m58.571s > user 0m31.769s > sys 0m1.246s > > randy Perfect.. Now, if you could only leave it empty, your fscks would take no time at all!! :) Eric From owner-freebsd-fs@FreeBSD.ORG Wed Feb 14 14:51:38 2007 Return-Path: X-Original-To: freebsd-fs@freebsd.org Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 8A61D16A406 for ; Wed, 14 Feb 2007 14:51:38 +0000 (UTC) (envelope-from anderson@centtech.com) Received: from mh1.centtech.com (moat3.centtech.com [64.129.166.50]) by mx1.freebsd.org (Postfix) with ESMTP id 61BC913C467 for ; Wed, 14 Feb 2007 14:51:38 +0000 (UTC) (envelope-from anderson@centtech.com) Received: from [10.177.171.220] (neutrino.centtech.com [10.177.171.220]) by mh1.centtech.com (8.13.8/8.13.8) with ESMTP id l1EEpbOE059016 for ; Wed, 14 Feb 2007 08:51:37 -0600 (CST) (envelope-from anderson@centtech.com) Message-ID: <45D321F8.2000808@centtech.com> Date: Wed, 14 Feb 2007 08:51:36 -0600 From: Eric Anderson User-Agent: Thunderbird 1.5.0.9 (X11/20070204) MIME-Version: 1.0 To: "freebsd-fs@freebsd.org" Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Virus-Scanned: ClamAV 0.88.4/2566/Wed Feb 14 05:44:20 2007 on mh1.centtech.com X-Virus-Status: Clean X-Spam-Status: No, score=-2.5 required=8.0 tests=AWL,BAYES_00 autolearn=ham version=3.1.6 X-Spam-Checker-Version: SpamAssassin 3.1.6 (2006-10-03) on mh1.centtech.com Subject: fsck times/memory sizes/etc X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 14 Feb 2007 14:51:38 -0000 I just did some quick playing around, doing newfs and fsck's on fresh file systems. I did one set on a 1Gb malloc'ed md, and another on a 65GB real disk device. The disk was only being used for this test, nothing else. I don't claim these numbers are perfect, but I did run each test 3-4 times to make sure they were consistent. I found it interesting that the fsck times didn't reduce once all the files/directories were deleted. ################################################ # mdconfig -a -t malloc -s 1024m # time fsck_ufs -f /dev/md0 ** /dev/md0 ** Last Mounted on ** Phase 1 - Check Blocks and Sizes ** Phase 2 - Check Pathnames ** Phase 3 - Check Connectivity ** Phase 4 - Check Reference Counts ** Phase 5 - Check Cyl groups 2 files, 2 used, 506485 free (21 frags, 63308 blocks, 0.0% fragmentation) real 0m0.014s user 0m0.009s sys 0m0.002s # mount it # fill it full of directories and empty files # df -i /mnt Filesystem 1K-blocks Used Avail Capacity iused ifree %iused Mounted on /dev/md0 1012974 2340 929598 0% 141310 0 100% /mnt # umount it # time fsck_ufs -f /dev/md0 ** /dev/md0 ** Last Mounted on /mnt ** Phase 1 - Check Blocks and Sizes ** Phase 2 - Check Pathnames ** Phase 3 - Check Connectivity ** Phase 4 - Check Reference Counts ** Phase 5 - Check Cyl groups 141310 files, 1170 used, 505317 free (21 frags, 63162 blocks, 0.0% fragmentation) real 0m0.266s user 0m0.103s sys 0m0.015s # mount it, rm all the files # umount it # time fsck_ufs -f /dev/md0 ** /dev/md0 ** Last Mounted on /mnt ** Phase 1 - Check Blocks and Sizes ** Phase 2 - Check Pathnames ** Phase 3 - Check Connectivity ** Phase 4 - Check Reference Counts ** Phase 5 - Check Cyl groups 2 files, 2 used, 506485 free (21 frags, 63308 blocks, 0.0% fragmentation) real 0m0.151s user 0m0.045s sys 0m0.024s ################################################## # newfs -U /dev/ad2s2a /dev/ad2s2a: 65397.4MB (133933888 sectors) block size 16384, fragment size 2048 using 356 cylinder groups of 183.77MB, 11761 blks, 23552 inodes. with soft updates # time fsck -f /dev/ad2s2a ** /dev/ad2s2a ** Last Mounted on ** Phase 1 - Check Blocks and Sizes ** Phase 2 - Check Pathnames ** Phase 3 - Check Connectivity ** Phase 4 - Check Reference Counts ** Phase 5 - Check Cyl groups 2 files, 2 used, 32429667 free (19 frags, 4053706 blocks, 0.0% fragmentation) real 0m9.673s user 0m0.449s sys 0m0.000s [root@neutrino ~]# mount /dev/ad2s2a /mnt [root@neutrino ~]# df -i /mnt Filesystem 1K-blocks Used Avail Capacity iused ifree %iused Mounted on /dev/ad2s2a 64859338 4 59670588 0% 2 8384508 0% /mnt # fill it full of directories and empty files # df -i /mnt Filesystem 1K-blocks Used Avail Capacity iused ifree %iused Mounted on /dev/ad2s2a 64859338 138346 59532246 0% 8384510 0 100% /mnt # umount it # time fsck -f /dev/ad2s2a ** /dev/ad2s2a ** Last Mounted on /mnt ** Phase 1 - Check Blocks and Sizes ** Phase 2 - Check Pathnames ** Phase 3 - Check Connectivity ** Phase 4 - Check Reference Counts ** Phase 5 - Check Cyl groups 8384510 files, 69173 used, 32360496 free (24 frags, 4045059 blocks, 0.0% fragmentation) real 1m57.315s user 0m6.046s sys 0m0.811s # memory usage for fsck maxed at 56448kb (~55MB) # mount, rm -rf all created files/dirs # df -i /mnt Filesystem 1K-blocks Used Avail Capacity iused ifree %iused Mounted on /dev/ad2s2a 64859338 6 59670586 0% 2 8384508 0% /mnt # umount it # time fsck -f /dev/ad2s2a ** /dev/ad2s2a ** Last Mounted on /mnt ** Phase 1 - Check Blocks and Sizes ** Phase 2 - Check Pathnames ** Phase 3 - Check Connectivity ** Phase 4 - Check Reference Counts ** Phase 5 - Check Cyl groups 2 files, 3 used, 32429666 free (18 frags, 4053706 blocks, 0.0% fragmentation) real 1m37.347s user 0m3.054s sys 0m0.693s # memory usage for fsck (on this empty fs) maxed at 9344kb (about 9MB) ################################################## Eric -- ------------------------------------------------------------------------ Eric Anderson Sr. Systems Administrator Centaur Technology An undefined problem has an infinite number of solutions. ------------------------------------------------------------------------ From owner-freebsd-fs@FreeBSD.ORG Wed Feb 14 15:07:31 2007 Return-Path: X-Original-To: freebsd-fs@freebsd.org Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 5D54116A400 for ; Wed, 14 Feb 2007 15:07:31 +0000 (UTC) (envelope-from anderson@freebsd.org) Received: from mh1.centtech.com (moat3.centtech.com [64.129.166.50]) by mx1.freebsd.org (Postfix) with ESMTP id 3506E13C478 for ; Wed, 14 Feb 2007 15:07:31 +0000 (UTC) (envelope-from anderson@freebsd.org) Received: from [10.177.171.220] (neutrino.centtech.com [10.177.171.220]) by mh1.centtech.com (8.13.8/8.13.8) with ESMTP id l1EF7UI5062259 for ; Wed, 14 Feb 2007 09:07:30 -0600 (CST) (envelope-from anderson@freebsd.org) Message-ID: <45D325B1.7030607@freebsd.org> Date: Wed, 14 Feb 2007 09:07:29 -0600 From: Eric Anderson User-Agent: Thunderbird 1.5.0.9 (X11/20070204) MIME-Version: 1.0 To: "freebsd-fs@freebsd.org" References: <45D321F8.2000808@centtech.com> In-Reply-To: <45D321F8.2000808@centtech.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Virus-Scanned: ClamAV 0.88.4/2566/Wed Feb 14 05:44:20 2007 on mh1.centtech.com X-Virus-Status: Clean X-Spam-Status: No, score=-2.6 required=8.0 tests=AWL,BAYES_00 autolearn=ham version=3.1.6 X-Spam-Checker-Version: SpamAssassin 3.1.6 (2006-10-03) on mh1.centtech.com Subject: Re: fsck times/memory sizes/etc X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 14 Feb 2007 15:07:31 -0000 One more test appended to this mail: On 02/14/07 08:51, Eric Anderson wrote: > I just did some quick playing around, doing newfs and fsck's on fresh > file systems. I did one set on a 1Gb malloc'ed md, and another on a > 65GB real disk device. The disk was only being used for this test, > nothing else. > > I don't claim these numbers are perfect, but I did run each test 3-4 > times to make sure they were consistent. > > I found it interesting that the fsck times didn't reduce once all the > files/directories were deleted. > > ################################################ > # mdconfig -a -t malloc -s 1024m > > # time fsck_ufs -f /dev/md0 > ** /dev/md0 > ** Last Mounted on > ** Phase 1 - Check Blocks and Sizes > ** Phase 2 - Check Pathnames > ** Phase 3 - Check Connectivity > ** Phase 4 - Check Reference Counts > ** Phase 5 - Check Cyl groups > 2 files, 2 used, 506485 free (21 frags, 63308 blocks, 0.0% fragmentation) > > real 0m0.014s > user 0m0.009s > sys 0m0.002s > > # mount it > # fill it full of directories and empty files > # df -i /mnt > Filesystem 1K-blocks Used Avail Capacity iused ifree %iused Mounted on > /dev/md0 1012974 2340 929598 0% 141310 0 100% /mnt > > # umount it > # time fsck_ufs -f /dev/md0 > ** /dev/md0 > ** Last Mounted on /mnt > ** Phase 1 - Check Blocks and Sizes > ** Phase 2 - Check Pathnames > ** Phase 3 - Check Connectivity > ** Phase 4 - Check Reference Counts > ** Phase 5 - Check Cyl groups > 141310 files, 1170 used, 505317 free (21 frags, 63162 blocks, 0.0% > fragmentation) > > real 0m0.266s > user 0m0.103s > sys 0m0.015s > > # mount it, rm all the files > # umount it > # time fsck_ufs -f /dev/md0 > ** /dev/md0 > ** Last Mounted on /mnt > ** Phase 1 - Check Blocks and Sizes > ** Phase 2 - Check Pathnames > ** Phase 3 - Check Connectivity > ** Phase 4 - Check Reference Counts > ** Phase 5 - Check Cyl groups > 2 files, 2 used, 506485 free (21 frags, 63308 blocks, 0.0% fragmentation) > > real 0m0.151s > user 0m0.045s > sys 0m0.024s > > > ################################################## > > # newfs -U /dev/ad2s2a > /dev/ad2s2a: 65397.4MB (133933888 sectors) block size 16384, fragment > size 2048 > using 356 cylinder groups of 183.77MB, 11761 blks, 23552 inodes. > with soft updates > > # time fsck -f /dev/ad2s2a > ** /dev/ad2s2a > ** Last Mounted on > ** Phase 1 - Check Blocks and Sizes > ** Phase 2 - Check Pathnames > ** Phase 3 - Check Connectivity > ** Phase 4 - Check Reference Counts > ** Phase 5 - Check Cyl groups > 2 files, 2 used, 32429667 free (19 frags, 4053706 blocks, 0.0% > fragmentation) > > real 0m9.673s > user 0m0.449s > sys 0m0.000s > > [root@neutrino ~]# mount /dev/ad2s2a /mnt > [root@neutrino ~]# df -i /mnt > Filesystem 1K-blocks Used Avail Capacity iused ifree %iused > Mounted on > /dev/ad2s2a 64859338 4 59670588 0% 2 8384508 0% /mnt > > # fill it full of directories and empty files > # df -i /mnt > Filesystem 1K-blocks Used Avail Capacity iused ifree %iused > Mounted on > /dev/ad2s2a 64859338 138346 59532246 0% 8384510 0 100% /mnt > > # umount it > # time fsck -f /dev/ad2s2a > ** /dev/ad2s2a > ** Last Mounted on /mnt > ** Phase 1 - Check Blocks and Sizes > ** Phase 2 - Check Pathnames > ** Phase 3 - Check Connectivity > ** Phase 4 - Check Reference Counts > ** Phase 5 - Check Cyl groups > 8384510 files, 69173 used, 32360496 free (24 frags, 4045059 blocks, 0.0% > fragmentation) > > real 1m57.315s > user 0m6.046s > sys 0m0.811s > > # memory usage for fsck maxed at 56448kb (~55MB) > > # mount, rm -rf all created files/dirs > # df -i /mnt > Filesystem 1K-blocks Used Avail Capacity iused ifree %iused > Mounted on > /dev/ad2s2a 64859338 6 59670586 0% 2 8384508 0% /mnt > > # umount it > > # time fsck -f /dev/ad2s2a > ** /dev/ad2s2a > ** Last Mounted on /mnt > ** Phase 1 - Check Blocks and Sizes > ** Phase 2 - Check Pathnames > ** Phase 3 - Check Connectivity > ** Phase 4 - Check Reference Counts > ** Phase 5 - Check Cyl groups > 2 files, 3 used, 32429666 free (18 frags, 4053706 blocks, 0.0% > fragmentation) > > real 1m37.347s > user 0m3.054s > sys 0m0.693s > > # memory usage for fsck (on this empty fs) maxed at 9344kb (about 9MB) > ################################################## > > > Eric > > > > > ################################################## # newfs -b 16384 -f 2048 -i 262144 /dev/ad2s2a /dev/ad2s2a: 65397.4MB (133933888 sectors) block size 16384, fragment size 2048 using 294 cylinder groups of 223.11MB, 14279 blks, 896 inodes. # time fsck -f /dev/ad2s2a ** /dev/ad2s2a ** Last Mounted on ** Phase 1 - Check Blocks and Sizes ** Phase 2 - Check Pathnames ** Phase 3 - Check Connectivity ** Phase 4 - Check Reference Counts ** Phase 5 - Check Cyl groups 2 files, 2 used, 33445795 free (19 frags, 4180722 blocks, 0.0% fragmentation) real 0m9.703s user 0m0.410s sys 0m0.049s # memory usage reached max of 9344kb (9MB) # fill it up # df -i /mnt Filesystem 1K-blocks Used Avail Capacity iused ifree %iused Mounted on /dev/ad2s2a 66891594 4342 61535926 0% 263422 0 100% /mnt # umount it # time fsck -f /dev/ad2s2a ** /dev/ad2s2a ** Last Mounted on /mnt ** Phase 1 - Check Blocks and Sizes ** Phase 2 - Check Pathnames ** Phase 3 - Check Connectivity ** Phase 4 - Check Reference Counts ** Phase 5 - Check Cyl groups 263422 files, 2171 used, 33443626 free (26 frags, 4180450 blocks, 0.0% fragmentation) real 0m10.491s user 0m0.610s sys 0m0.020s # memory usage reached max of 9344kb (9MB) # mount, rm -rf all created files/dirs # df -i /mnt Filesystem 1K-blocks Used Avail Capacity iused ifree %iused Mounted on /dev/ad2s2a 66891594 4 61540264 0% 2 263420 0% /mnt # time fsck -f /dev/ad2s2a ** /dev/ad2s2a ** Last Mounted on /mnt ** Phase 1 - Check Blocks and Sizes ** Phase 2 - Check Pathnames ** Phase 3 - Check Connectivity ** Phase 4 - Check Reference Counts ** Phase 5 - Check Cyl groups 2 files, 2 used, 33445795 free (19 frags, 4180722 blocks, 0.0% fragmentation) real 0m10.427s user 0m0.570s sys 0m0.012s # memory usage for fsck (on this empty fs) maxed at 9344kb (about 9MB) ################################################## From owner-freebsd-fs@FreeBSD.ORG Wed Feb 14 16:56:21 2007 Return-Path: X-Original-To: freebsd-fs@freebsd.org Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 2ACDB16A409 for ; Wed, 14 Feb 2007 16:56:21 +0000 (UTC) (envelope-from rick@kiwi-computer.com) Received: from kiwi-computer.com (keira.kiwi-computer.com [63.224.10.3]) by mx1.freebsd.org (Postfix) with SMTP id 9B5C713C474 for ; Wed, 14 Feb 2007 16:56:20 +0000 (UTC) (envelope-from rick@kiwi-computer.com) Received: (qmail 97236 invoked by uid 2001); 14 Feb 2007 16:29:38 -0000 Date: Wed, 14 Feb 2007 10:29:38 -0600 From: "Rick C. Petty" To: Christopher Arnold Message-ID: <20070214162938.GA96725@keira.kiwi-computer.com> References: <45CD6FF5.8070007@freebsd.org> <20070213075627.63126.qmail@web34502.mail.mud.yahoo.com> <6FC9F9894A9F8C49A722CF9F2132FC2204C9DAB4@ms05.mailstreet2003.net> <6FC9F9894A9F8C49A722CF9F2132FC2204C9DAB5@ms05.mailstreet2003.net> <6FC9F9894A9F8C49A722CF9F2132FC2204C9DAB6@ms05.mailstreet2003.net> <45D1F30A.6080403@freebsd.org> <20070213192906.U726@chrishome.localnet> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20070213192906.U726@chrishome.localnet> User-Agent: Mutt/1.4.2.1i Cc: freebsd-fs@freebsd.org Subject: Re: UFS2 with SAN X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: rick-freebsd@kiwi-computer.com List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 14 Feb 2007 16:56:21 -0000 On Tue, Feb 13, 2007 at 07:32:54PM +0100, Christopher Arnold wrote: > > Is there anyone who has looked into using AFS as a distributed filestore > on FreeBSD. Yes, plenty of people. A few of us have even tried to get arla working in 6.x (stacken.kth.se), but I never had time to complete the project. Too many VFS changes broke this. > Yes im aware of issues running AFS on 6.x but i belive there exists an > existing implementation under 5.x. And if enough people need it i guess we > could make shure it runs ok under 6.x to. Arla works okay under 5.x, but there are a number of bugs. -- Rick C. Petty From owner-freebsd-fs@FreeBSD.ORG Wed Feb 14 16:58:51 2007 Return-Path: X-Original-To: freebsd-fs@freebsd.org Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 92A7616A402 for ; Wed, 14 Feb 2007 16:58:51 +0000 (UTC) (envelope-from rick@kiwi-computer.com) Received: from kiwi-computer.com (keira.kiwi-computer.com [63.224.10.3]) by mx1.freebsd.org (Postfix) with SMTP id 17A4813C441 for ; Wed, 14 Feb 2007 16:58:50 +0000 (UTC) (envelope-from rick@kiwi-computer.com) Received: (qmail 97268 invoked by uid 2001); 14 Feb 2007 16:32:09 -0000 Date: Wed, 14 Feb 2007 10:32:09 -0600 From: "Rick C. Petty" To: Jim Rees Message-ID: <20070214163209.GB96725@keira.kiwi-computer.com> References: <45CD6FF5.8070007@freebsd.org> <20070213075627.63126.qmail@web34502.mail.mud.yahoo.com> <6FC9F9894A9F8C49A722CF9F2132FC2204C9DAB4@ms05.mailstreet2003.net> <6FC9F9894A9F8C49A722CF9F2132FC2204C9DAB5@ms05.mailstreet2003.net> <6FC9F9894A9F8C49A722CF9F2132FC2204C9DAB6@ms05.mailstreet2003.net> <45D1F30A.6080403@freebsd.org> <20070213192906.U726@chrishome.localnet> <20070213192022.GA15500@citi.umich.edu> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20070213192022.GA15500@citi.umich.edu> User-Agent: Mutt/1.4.2.1i Cc: freebsd-fs@freebsd.org Subject: Re: UFS2 with SAN X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: rick-freebsd@kiwi-computer.com List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 14 Feb 2007 16:58:51 -0000 On Tue, Feb 13, 2007 at 02:20:22PM -0500, Jim Rees wrote: > > It wouldn't take much to get OpenAFS in pretty good shape on FreeBSD. It I completely disagree. It would take a lot more work to port OpenAFS than it would to finish porting Arla to 6.x (or even 7.x). If anyone has the time and VFS knowledge, there are a number of FreeBSD users who would appreciate your efforts. Their project page: http://www.stacken.kth.se/project/arla/ -- Rick C. Petty From owner-freebsd-fs@FreeBSD.ORG Wed Feb 14 17:03:36 2007 Return-Path: X-Original-To: freebsd-fs@freebsd.org Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 4107316A401 for ; Wed, 14 Feb 2007 17:03:36 +0000 (UTC) (envelope-from chris@arnold.se) Received: from av8-1-sn3.vrr.skanova.net (av8-1-sn3.vrr.skanova.net [81.228.9.183]) by mx1.freebsd.org (Postfix) with ESMTP id A83AF13C4A6 for ; Wed, 14 Feb 2007 17:03:35 +0000 (UTC) (envelope-from chris@arnold.se) Received: by av8-1-sn3.vrr.skanova.net (Postfix, from userid 502) id C07D238281; Wed, 14 Feb 2007 17:42:14 +0100 (CET) Received: from smtp3-2-sn3.vrr.skanova.net (smtp3-2-sn3.vrr.skanova.net [81.228.9.102]) by av8-1-sn3.vrr.skanova.net (Postfix) with ESMTP id A4E1638271; Wed, 14 Feb 2007 17:42:14 +0100 (CET) Received: from chrishome.localnet (81-224-156-16-o1033.telia.com [81.224.156.16]) by smtp3-2-sn3.vrr.skanova.net (Postfix) with ESMTP id 1A34F37E47; Wed, 14 Feb 2007 17:42:13 +0100 (CET) Received: from localhost (localhost [127.0.0.1]) by chrishome.localnet (8.13.6/8.13.6) with ESMTP id l1EGgCZP001076; Wed, 14 Feb 2007 17:42:13 +0100 (CET) (envelope-from chris@arnold.se) Date: Wed, 14 Feb 2007 17:42:12 +0100 (CET) From: Christopher Arnold X-X-Sender: chris@chrishome.localnet To: "Rick C. Petty" In-Reply-To: <20070214162938.GA96725@keira.kiwi-computer.com> Message-ID: <20070214173211.L1054@chrishome.localnet> References: <45CD6FF5.8070007@freebsd.org> <20070213075627.63126.qmail@web34502.mail.mud.yahoo.com> <6FC9F9894A9F8C49A722CF9F2132FC2204C9DAB4@ms05.mailstreet2003.net> <6FC9F9894A9F8C49A722CF9F2132FC2204C9DAB5@ms05.mailstreet2003.net> <6FC9F9894A9F8C49A722CF9F2132FC2204C9DAB6@ms05.mailstreet2003.net> <45D1F30A.6080403@freebsd.org> <20070213192906.U726@chrishome.localnet> <20070214162938.GA96725@keira.kiwi-computer.com> X-message-flag: =?ISO-8859-1?Q?Outlook_isn=B4t_compliant_with_current_standards?= =?ISO-8859-1?Q?_please_install_another_mail_client!?= MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: freebsd-fs@freebsd.org, arla-drinkers@stacken.kth.se Subject: Re: UFS2 with SAN X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 14 Feb 2007 17:03:36 -0000 On Wed, 14 Feb 2007, Rick C. Petty wrote: > On Tue, Feb 13, 2007 at 07:32:54PM +0100, Christopher Arnold wrote: >> >> Is there anyone who has looked into using AFS as a distributed filestore >> on FreeBSD. > > Yes, plenty of people. A few of us have even tried to get arla working in > 6.x (stacken.kth.se), but I never had time to complete the project. Too > many VFS changes broke this. > Yes i know, im from .se after all... Im really interested in getting arla and/or openafs working on the FreeBSD's after the VFS overhaul. Do you have an stimate on how much effort is needed? Even thou i don't have a huge budget i'm villing to put some dollars into making that happen so i could make a proper evaluation in using AFS as a backend for web and ftp servers. Ill start with 1000 USD, anyone else who would like to contribute? >> Yes im aware of issues running AFS on 6.x but i belive there exists an >> existing implementation under 5.x. And if enough people need it i guess we >> could make shure it runs ok under 6.x to. > > Arla works okay under 5.x, but there are a number of bugs. > Last time i tried related to load so i couldn't actually continue my efforts. Thats a shame. But i would love to overcome those hurdles and do some serious testing. /Chris -- www.infotropic.com From owner-freebsd-fs@FreeBSD.ORG Wed Feb 14 17:08:10 2007 Return-Path: X-Original-To: freebsd-fs@freebsd.org Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 1F48C16A41F for ; Wed, 14 Feb 2007 17:08:10 +0000 (UTC) (envelope-from rick@kiwi-computer.com) Received: from kiwi-computer.com (keira.kiwi-computer.com [63.224.10.3]) by mx1.freebsd.org (Postfix) with SMTP id B3B0F13C4A6 for ; Wed, 14 Feb 2007 17:08:09 +0000 (UTC) (envelope-from rick@kiwi-computer.com) Received: (qmail 98292 invoked by uid 2001); 14 Feb 2007 17:08:08 -0000 Date: Wed, 14 Feb 2007 11:08:08 -0600 From: "Rick C. Petty" To: Christopher Arnold Message-ID: <20070214170808.GC96725@keira.kiwi-computer.com> References: <20070213075627.63126.qmail@web34502.mail.mud.yahoo.com> <6FC9F9894A9F8C49A722CF9F2132FC2204C9DAB4@ms05.mailstreet2003.net> <6FC9F9894A9F8C49A722CF9F2132FC2204C9DAB5@ms05.mailstreet2003.net> <6FC9F9894A9F8C49A722CF9F2132FC2204C9DAB6@ms05.mailstreet2003.net> <45D1F30A.6080403@freebsd.org> <20070213192906.U726@chrishome.localnet> <20070214162938.GA96725@keira.kiwi-computer.com> <20070214173211.L1054@chrishome.localnet> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20070214173211.L1054@chrishome.localnet> User-Agent: Mutt/1.4.2.1i Cc: freebsd-fs@freebsd.org, arla-drinkers@stacken.kth.se Subject: Re: UFS2 with SAN X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: rick-freebsd@kiwi-computer.com List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 14 Feb 2007 17:08:10 -0000 On Wed, Feb 14, 2007 at 05:42:12PM +0100, Christopher Arnold wrote: > > Im really interested in getting arla and/or openafs working on the > FreeBSD's after the VFS overhaul. Do you have an stimate on how much > effort is needed? At one point I submitted patches which allowed arla to *compile* properly, but it kernel panicked when loading the module. I suspect it was related to changes in locking strategies between 5.x and 6.x. A few other people on the arla-drinkers mailing list were able to patch the module a bit further and it didn't panic at kldload (but did when trying to do any FS operation). Those patches never made it into arla and AFAIK no one has made any further progress. A coworker of mine was trying to apply those patches to 7.x, thinking fbsd folks would be more willing to accept it. I'm not sure how far he got. > Even thou i don't have a huge budget i'm villing to put some dollars into > making that happen so i could make a proper evaluation in using AFS as a > backend for web and ftp servers. Ill start with 1000 USD, anyone > else who would like to contribute? I'm hoping there will be some takers. We use AFS at work on 5.x for production and release engineering as well as backups. The OpenAFS server works just fine in FreeBSD (because it's all userland) and the Arla client seems to behave just fine on DragonFly, Mac OS X, and all other BSDs. In FreeBSD, it's a wash. -- Rick C. Petty From owner-freebsd-fs@FreeBSD.ORG Wed Feb 14 17:22:43 2007 Return-Path: X-Original-To: freebsd-fs@freebsd.org Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 3552516A402 for ; Wed, 14 Feb 2007 17:22:43 +0000 (UTC) (envelope-from randy@psg.com) Received: from rip.psg.com (rip.psg.com [147.28.0.39]) by mx1.freebsd.org (Postfix) with ESMTP id 1B67613C49D for ; Wed, 14 Feb 2007 17:22:43 +0000 (UTC) (envelope-from randy@psg.com) Received: from localhost ([127.0.0.1] helo=roam.psg.com) by rip.psg.com with esmtp (Exim 4.66 (FreeBSD)) (envelope-from ) id 1HHNq3-00063A-J8; Wed, 14 Feb 2007 17:22:40 +0000 Received: from localhost ([127.0.0.1] helo=roam.psg.com) by roam.psg.com with esmtp (Exim 4.66 (FreeBSD)) (envelope-from ) id 1HHNpq-0001Oe-U0; Wed, 14 Feb 2007 07:22:27 -1000 From: Randy Bush MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Message-ID: <17875.17746.301886.885155@roam.psg.com> Date: Wed, 14 Feb 2007 07:22:26 -1000 To: Eric Anderson References: <17873.33060.746546.773210@roam.psg.com> <200702130927.l1D9RBn9034761@lurza.secnetix.de> <17874.45261.367361.188024@roam.psg.com> <45D307C7.6020409@freebsd.org> Cc: freebsd-fs@freebsd.org, Oliver Fromme Subject: Re: comments on newfs raw disk ? Safe ? (7 terabyte array) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 14 Feb 2007 17:22:43 -0000 > On 02/14/07 00:48, Randy Bush wrote: >> 4TB >> newfs -b 16384 -f 2048 -i 262144 >> >> # time fsck -y /dev/da1 >> ** /dev/da1 >> ** Last Mounted on /mnt >> ** Phase 1 - Check Blocks and Sizes >> ** Phase 2 - Check Pathnames >> ** Phase 3 - Check Connectivity >> ** Phase 4 - Check Reference Counts >> ** Phase 5 - Check Cyl groups >> 2 files, 2 used, 2173822273 free (17 frags, 271727782 blocks, 0.0% fragmentation) >> >> ***** FILE SYSTEM MARKED CLEAN ***** >> >> real 8m58.571s >> user 0m31.769s >> sys 0m1.246s > > Perfect.. Now, if you could only leave it empty, your fscks would take > no time at all!! :) sorry. forgot to remention that this is a backup store for a dozen systems' rdump/restores. so it will have a few hundred large files. so we do not expect fsck time to go radically gigh. randy From owner-freebsd-fs@FreeBSD.ORG Wed Feb 14 19:35:44 2007 Return-Path: X-Original-To: freebsd-fs@freebsd.org Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 6BC4516A400 for ; Wed, 14 Feb 2007 19:35:44 +0000 (UTC) (envelope-from freebsd-fs@m.gmane.org) Received: from ciao.gmane.org (main.gmane.org [80.91.229.2]) by mx1.freebsd.org (Postfix) with ESMTP id 26BAE13C491 for ; Wed, 14 Feb 2007 19:35:43 +0000 (UTC) (envelope-from freebsd-fs@m.gmane.org) Received: from list by ciao.gmane.org with local (Exim 4.43) id 1HHPuc-0002kh-JU for freebsd-fs@freebsd.org; Wed, 14 Feb 2007 20:35:30 +0100 Received: from 89-172-61-156.adsl.net.t-com.hr ([89.172.61.156]) by main.gmane.org with esmtp (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Wed, 14 Feb 2007 20:35:30 +0100 Received: from ivoras by 89-172-61-156.adsl.net.t-com.hr with local (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Wed, 14 Feb 2007 20:35:30 +0100 X-Injected-Via-Gmane: http://gmane.org/ To: freebsd-fs@freebsd.org From: Ivan Voras Date: Wed, 14 Feb 2007 20:35:01 +0100 Lines: 31 Message-ID: References: <17873.33060.746546.773210@roam.psg.com> <200702130927.l1D9RBn9034761@lurza.secnetix.de> <17874.45261.367361.188024@roam.psg.com> <45D307C7.6020409@freebsd.org> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="------------enigB1FA506DAA693D39CABB698F" X-Complaints-To: usenet@sea.gmane.org X-Gmane-NNTP-Posting-Host: 89-172-61-156.adsl.net.t-com.hr User-Agent: Thunderbird 1.5.0.9 (Windows/20061207) In-Reply-To: <45D307C7.6020409@freebsd.org> X-Enigmail-Version: 0.94.1.2 Sender: news Subject: Re: comments on newfs raw disk ? Safe ? (7 terabyte array) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 14 Feb 2007 19:35:44 -0000 This is an OpenPGP/MIME signed message (RFC 2440 and 3156) --------------enigB1FA506DAA693D39CABB698F Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Eric Anderson wrote: >=20 > Perfect.. Now, if you could only leave it empty, your fscks would take > no time at all!! :) Yes, this leads us to the age old discussion about users, and how the world would be a better place without them :) Nice, clean terabytes of storage, sitting there quietly... --------------enigB1FA506DAA693D39CABB698F Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.4 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFF02RrldnAQVacBcgRAsRdAJ9KAbAqWJC24w88Up2lK284UI1hVQCfULRR T5tr6pMsNi4i/00JHhvZRjQ= =W4hz -----END PGP SIGNATURE----- --------------enigB1FA506DAA693D39CABB698F-- From owner-freebsd-fs@FreeBSD.ORG Wed Feb 14 21:17:49 2007 Return-Path: X-Original-To: freebsd-fs@freebsd.org Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id A615116A406 for ; Wed, 14 Feb 2007 21:17:49 +0000 (UTC) (envelope-from rick@snowhite.cis.uoguelph.ca) Received: from gigi.cs.uoguelph.ca (gigi.cs.uoguelph.ca [131.104.94.210]) by mx1.freebsd.org (Postfix) with ESMTP id 586F813C467 for ; Wed, 14 Feb 2007 21:17:49 +0000 (UTC) (envelope-from rick@snowhite.cis.uoguelph.ca) Received: from snowhite.cis.uoguelph.ca (snowhite.cis.uoguelph.ca [131.104.48.1]) by gigi.cs.uoguelph.ca (8.13.1/8.13.1) with ESMTP id l1EJr0Kg017924; Wed, 14 Feb 2007 14:53:00 -0500 Received: (from rick@localhost) by snowhite.cis.uoguelph.ca (8.9.3/8.9.3) id OAA84075; Wed, 14 Feb 2007 14:51:42 -0500 (EST) Date: Wed, 14 Feb 2007 14:51:42 -0500 (EST) From: rick@snowhite.cis.uoguelph.ca Message-Id: <200702141951.OAA84075@snowhite.cis.uoguelph.ca> To: drumslayer2@yahoo.com X-Scanned-By: MIMEDefang 2.57 on 131.104.94.210 Cc: freebsd-fs@freebsd.org Subject: large lookup cache for nfs X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 14 Feb 2007 21:17:49 -0000 If you wanted to create a large lookup name cache on FreeBSD6, from a quick glance, it looks like you have to set these very large, since it is sized based on desiredvnodes, which is sized based on these: kern.maxproc vm.kmem_size (Hopefully someone will correct me, if I got that wrong.) For FreeBSD4, you could just set kern.maxvnodes. That's all I can suggest if you are seeing lots of Lookup rpcs when checking via nfsstat. (If you are seeing lots of Getattr and/or Access, then it's a different story. If you look at nfs_lookup() in nfs_vnops.c, you'll see the checks it does after a cache hit, but if its lots'a'lookups, it isn't getting that far. Good luck with it, rick From owner-freebsd-fs@FreeBSD.ORG Wed Feb 14 22:57:32 2007 Return-Path: X-Original-To: freebsd-fs@freebsd.org Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 3664716A400 for ; Wed, 14 Feb 2007 22:57:32 +0000 (UTC) (envelope-from tol@pdc.kth.se) Received: from mx1.kth.se (mx1.kth.se [130.237.32.140]) by mx1.freebsd.org (Postfix) with ESMTP id E1C2113C428 for ; Wed, 14 Feb 2007 22:57:31 +0000 (UTC) (envelope-from tol@pdc.kth.se) Received: from localhost (localhost.localdomain [127.0.0.1]) by mx1.kth.se (Postfix) with ESMTP id 7F3B014075A; Wed, 14 Feb 2007 23:39:04 +0100 (CET) Received: from mx1.kth.se ([127.0.0.1]) by localhost (mx1.kth.se [127.0.0.1]) (amavisd-new, port 10024) with LMTP id 05433-03-57; Wed, 14 Feb 2007 23:39:02 +0100 (CET) Received: from kashyyyk.ite.kth.se (kashyyyk.ite.kth.se [130.237.31.35]) by mx1.kth.se (Postfix) with ESMTP id 68E971405CC; Wed, 14 Feb 2007 23:39:02 +0100 (CET) Received: by kashyyyk.ite.kth.se (Postfix, from userid 18404) id 5253089CA62; Wed, 14 Feb 2007 23:39:02 +0100 (CET) Sender: tol@kashyyyk.ite.kth.se From: Tomas Olsson To: "Rick C. Petty" References: <20070213075627.63126.qmail@web34502.mail.mud.yahoo.com> <6FC9F9894A9F8C49A722CF9F2132FC2204C9DAB4@ms05.mailstreet2003.net> <6FC9F9894A9F8C49A722CF9F2132FC2204C9DAB5@ms05.mailstreet2003.net> <6FC9F9894A9F8C49A722CF9F2132FC2204C9DAB6@ms05.mailstreet2003.net> <45D1F30A.6080403@freebsd.org> <20070213192906.U726@chrishome.localnet> <20070214162938.GA96725@keira.kiwi-computer.com> <20070214173211.L1054@chrishome.localnet> <20070214170808.GC96725@keira.kiwi-computer.com> Date: 14 Feb 2007 23:39:02 +0100 In-Reply-To: <20070214170808.GC96725@keira.kiwi-computer.com> Message-ID: Lines: 20 User-Agent: Gnus/5.09 (Gnus v5.9.0) Emacs/21.3 MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii X-Virus-Scanned: by amavisd-new at kth.se Cc: freebsd-fs@freebsd.org, arla-drinkers@stacken.kth.se Subject: Re: UFS2 with SAN X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 14 Feb 2007 22:57:32 -0000 "Rick C. Petty" writes: > the Arla client seems to behave just fine on DragonFly, Mac OS X, and all > other BSDs. In FreeBSD, it's a wash. > DragonFly support is most likely fading away, AFAIK nobody's testing it. arla-0.43 (and arla-0-44-branch) should work on FreeBSD 5, let us know if there are any issues with that. As for arla post the blockcache rewrite, I now have OpenBSD going through the basic tests (fails on heavier tests). If there's anyone with good knowledge of FreeBSD's vfs who's willing to help me out a bit I think we could have 6.2 support done in a week or so. If all is well, that is :) Last time I tried on 5.5 it kept crashing on me in the turnstile walk. I'm already funded and can work full time on this, but a FreeBSD hacker would help a lot. Any volunteers? thanks /t From owner-freebsd-fs@FreeBSD.ORG Wed Feb 14 23:39:55 2007 Return-Path: X-Original-To: freebsd-fs@freebsd.org Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id E412816A409 for ; Wed, 14 Feb 2007 23:39:55 +0000 (UTC) (envelope-from pjd@garage.freebsd.pl) Received: from mail.garage.freebsd.pl (arm132.internetdsl.tpnet.pl [83.17.198.132]) by mx1.freebsd.org (Postfix) with ESMTP id 82AEB13C4A6 for ; Wed, 14 Feb 2007 23:39:55 +0000 (UTC) (envelope-from pjd@garage.freebsd.pl) Received: by mail.garage.freebsd.pl (Postfix, from userid 65534) id 5D9DD45685; Thu, 15 Feb 2007 00:39:53 +0100 (CET) Received: from localhost (154.81.datacomsa.pl [195.34.81.154]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mail.garage.freebsd.pl (Postfix) with ESMTP id 4089A45681; Thu, 15 Feb 2007 00:39:48 +0100 (CET) Date: Thu, 15 Feb 2007 00:38:48 +0100 From: Pawel Jakub Dawidek To: Eric Anderson Message-ID: <20070214233848.GA13680@garage.freebsd.pl> References: <45D321F8.2000808@centtech.com> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="+QahgC5+KEYLbs62" Content-Disposition: inline In-Reply-To: <45D321F8.2000808@centtech.com> X-PGP-Key-URL: http://people.freebsd.org/~pjd/pjd.asc X-OS: FreeBSD 7.0-CURRENT i386 User-Agent: mutt-ng/devel-r804 (FreeBSD) X-Spam-Checker-Version: SpamAssassin 3.0.4 (2005-06-05) on mail.garage.freebsd.pl X-Spam-Level: X-Spam-Status: No, score=-2.6 required=3.0 tests=BAYES_00 autolearn=ham version=3.0.4 Cc: "freebsd-fs@freebsd.org" Subject: Re: fsck times/memory sizes/etc X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 14 Feb 2007 23:39:56 -0000 --+QahgC5+KEYLbs62 Content-Type: text/plain; charset=iso-8859-2 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Wed, Feb 14, 2007 at 08:51:36AM -0600, Eric Anderson wrote: > I just did some quick playing around, doing newfs and fsck's on fresh fil= e systems. I did one set on a 1Gb malloc'ed md, and another on a 65GB real= disk device. The disk=20 > was only being used for this test, nothing else. >=20 > I don't claim these numbers are perfect, but I did run each test 3-4 time= s to make sure they were consistent. >=20 > I found it interesting that the fsck times didn't reduce once all the fil= es/directories were deleted. This may be because of UFS2 has lazy inodes allocator - it doesn't initialize all inode blocks at newfs time, so fsck verifies only allocated inode blocks, thus runs faster on new file system. When you fill your file system, inode blocks are allocated, but are not reclaimed when you delete file/directories. You may try the same tests with UFS1, which allocates all inode blocks at newfs time. --=20 Pawel Jakub Dawidek http://www.wheel.pl pjd@FreeBSD.org http://www.FreeBSD.org FreeBSD committer Am I Evil? Yes, I Am! --+QahgC5+KEYLbs62 Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (FreeBSD) iD8DBQFF052IForvXbEpPzQRAnu8AKDPcMJel6A9qtndNIcDJCRWQdJLRwCePZt4 g8uPLFERBVPnTEvkfQU7/2w= =2sib -----END PGP SIGNATURE----- --+QahgC5+KEYLbs62-- From owner-freebsd-fs@FreeBSD.ORG Thu Feb 15 00:45:48 2007 Return-Path: X-Original-To: freebsd-fs@freebsd.org Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 1675616A401 for ; Thu, 15 Feb 2007 00:45:48 +0000 (UTC) (envelope-from rwatson@FreeBSD.org) Received: from cyrus.watson.org (cyrus.watson.org [209.31.154.42]) by mx1.freebsd.org (Postfix) with ESMTP id DC3F213C481 for ; Thu, 15 Feb 2007 00:45:47 +0000 (UTC) (envelope-from rwatson@FreeBSD.org) Received: from fledge.watson.org (fledge.watson.org [209.31.154.41]) by cyrus.watson.org (Postfix) with ESMTP id 088C04770F; Wed, 14 Feb 2007 19:26:09 -0500 (EST) Date: Thu, 15 Feb 2007 00:26:08 +0000 (GMT) From: Robert Watson X-X-Sender: robert@fledge.watson.org To: Tomas Olsson In-Reply-To: Message-ID: <20070215002206.G16383@fledge.watson.org> References: <20070213075627.63126.qmail@web34502.mail.mud.yahoo.com> <6FC9F9894A9F8C49A722CF9F2132FC2204C9DAB4@ms05.mailstreet2003.net> <6FC9F9894A9F8C49A722CF9F2132FC2204C9DAB5@ms05.mailstreet2003.net> <6FC9F9894A9F8C49A722CF9F2132FC2204C9DAB6@ms05.mailstreet2003.net> <45D1F30A.6080403@freebsd.org> <20070213192906.U726@chrishome.localnet> <20070214162938.GA96725@keira.kiwi-computer.com> <20070214173211.L1054@chrishome.localnet> <20070214170808.GC96725@keira.kiwi-computer.com> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: freebsd-fs@freebsd.org, "Rick C. Petty" , arla-drinkers@stacken.kth.se Subject: Re: UFS2 with SAN X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 15 Feb 2007 00:45:48 -0000 On Wed, 14 Feb 2007, Tomas Olsson wrote: > "Rick C. Petty" writes: >> the Arla client seems to behave just fine on DragonFly, Mac OS X, and all >> other BSDs. In FreeBSD, it's a wash. >> > DragonFly support is most likely fading away, AFAIK nobody's testing it. > arla-0.43 (and arla-0-44-branch) should work on FreeBSD 5, let us know if > there are any issues with that. > > As for arla post the blockcache rewrite, I now have OpenBSD going through > the basic tests (fails on heavier tests). If there's anyone with good > knowledge of FreeBSD's vfs who's willing to help me out a bit I think we > could have 6.2 support done in a week or so. If all is well, that is :) > > Last time I tried on 5.5 it kept crashing on me in the turnstile walk. > > I'm already funded and can work full time on this, but a FreeBSD hacker > would help a lot. Any volunteers? I'm not the right person to talk to you about FreeBSD VFS/buffering, but I'm very interested in seeing support for Arla improve for recent FreeBSD versions. In particular, I think it would be very helpful it it worked on FreeBSD 7-CURRENT, which would facilitate merging nnpfs into CVS. This might help solve the "forever playing catchup" problem that Arla seems constantly to experience with FreeBSD, in which nnpfs (previously xfs) gets left behind and works only on stale FreeBSD versions. In any recent FreeBSD kernel, nnpfs will need to run as an MPSAFE file system, however. Robert N M Watson Computer Laboratory University of Cambridge From owner-freebsd-fs@FreeBSD.ORG Thu Feb 15 00:52:55 2007 Return-Path: X-Original-To: freebsd-fs@freebsd.org Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 028D816A407 for ; Thu, 15 Feb 2007 00:52:55 +0000 (UTC) (envelope-from chris@arnold.se) Received: from av9-2-sn3.vrr.skanova.net (av9-2-sn3.vrr.skanova.net [81.228.9.186]) by mx1.freebsd.org (Postfix) with ESMTP id 8420913C494 for ; Thu, 15 Feb 2007 00:52:54 +0000 (UTC) (envelope-from chris@arnold.se) Received: by av9-2-sn3.vrr.skanova.net (Postfix, from userid 502) id 45A27382E1; Thu, 15 Feb 2007 01:52:53 +0100 (CET) Received: from smtp3-2-sn3.vrr.skanova.net (smtp3-2-sn3.vrr.skanova.net [81.228.9.102]) by av9-2-sn3.vrr.skanova.net (Postfix) with ESMTP id 2964E382D3; Thu, 15 Feb 2007 01:52:53 +0100 (CET) Received: from chrishome.localnet (81-224-156-16-o1033.telia.com [81.224.156.16]) by smtp3-2-sn3.vrr.skanova.net (Postfix) with ESMTP id BA10D37E45; Thu, 15 Feb 2007 01:52:52 +0100 (CET) Received: from localhost (localhost [127.0.0.1]) by chrishome.localnet (8.13.6/8.13.6) with ESMTP id l1F0qpO1006418; Thu, 15 Feb 2007 01:52:52 +0100 (CET) (envelope-from chris@arnold.se) Date: Thu, 15 Feb 2007 01:52:51 +0100 (CET) From: Christopher Arnold X-X-Sender: chris@chrishome.localnet To: Tomas Olsson In-Reply-To: Message-ID: <20070215011402.A6280@chrishome.localnet> References: <20070213075627.63126.qmail@web34502.mail.mud.yahoo.com> <6FC9F9894A9F8C49A722CF9F2132FC2204C9DAB4@ms05.mailstreet2003.net> <6FC9F9894A9F8C49A722CF9F2132FC2204C9DAB5@ms05.mailstreet2003.net> <6FC9F9894A9F8C49A722CF9F2132FC2204C9DAB6@ms05.mailstreet2003.net> <45D1F30A.6080403@freebsd.org> <20070213192906.U726@chrishome.localnet> <20070214162938.GA96725@keira.kiwi-computer.com> <20070214173211.L1054@chrishome.localnet> <20070214170808.GC96725@keira.kiwi-computer.com> X-message-flag: =?ISO-8859-1?Q?Outlook_isn=B4t_compliant_with_current_standards?= =?ISO-8859-1?Q?_please_install_another_mail_client!?= MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: freebsd-fs@freebsd.org, arla-drinkers@stacken.kth.se Subject: Re: UFS2 with SAN X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 15 Feb 2007 00:52:55 -0000 On Wed, 14 Feb 2007, Tomas Olsson wrote: > "Rick C. Petty" writes: >> the Arla client seems to behave just fine on DragonFly, Mac OS X, and all >> other BSDs. In FreeBSD, it's a wash. >> > DragonFly support is most likely fading away, AFAIK nobody's testing > it. arla-0.43 (and arla-0-44-branch) should work on FreeBSD 5, let us know > if there are any issues with that. > Hmm i managed to crash my system by doing an "ls -R" in /afs/something on FreeBSD 5.5. > As for arla post the blockcache rewrite, I now have OpenBSD going through > the basic tests (fails on heavier tests). If there's anyone with good > knowledge of FreeBSD's vfs who's willing to help me out a bit I think we > could have 6.2 support done in a week or so. If all is well, that is :) > Great news! In an earlier message on freebsd-fs@freebsd.org Jim Rees, rees@umich.edu. stated: "It wouldn't take much to get OpenAFS in pretty good shape on FreeBSD. It mostly has some locking and vnode ref count issues that shouldn't be too hard to fix for someone who knows what they're doing. I'd be happy to point anyone in the right direction." Maybe he could give you the push in the right direction? (Even thou i suspect your motivation is more connected to getting arla up and running under 6.2, Jim's input could perhaps help you in that direction too?) > Last time I tried on 5.5 it kept crashing on me in the turnstile walk. > My experience above with "ls -R" was also on 5.5 running arla-0.43. > I'm already funded and can work full time on this, but a FreeBSD hacker > would help a lot. Any volunteers? > Thats nice news. And i realise that my little bounty might not make any difference when people are "already funded" and have other motivations. So let me rephrase my bounty. 1000 USD to the person who needs it and makes a difference in implementing this. Otherwise given to the FreeBSD foundation or some other Open source project/fund etc. when a working afs server and client implementation exists for FreeBSD 6.2 or higher, preferably as a port or in the base system. Those involved in the implementation will have a say in which project or individual the money will go. And please if others are intrested in increasing the bounty, do so! /Chris -- www.infotropic.com From owner-freebsd-fs@FreeBSD.ORG Thu Feb 15 01:00:20 2007 Return-Path: X-Original-To: freebsd-fs@freebsd.org Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 3916F16A4A7 for ; Thu, 15 Feb 2007 01:00:20 +0000 (UTC) (envelope-from rick@kiwi-computer.com) Received: from kiwi-computer.com (keira.kiwi-computer.com [63.224.10.3]) by mx1.freebsd.org (Postfix) with SMTP id 6040513C4A6 for ; Thu, 15 Feb 2007 01:00:19 +0000 (UTC) (envelope-from rick@kiwi-computer.com) Received: (qmail 9349 invoked by uid 2001); 15 Feb 2007 01:00:17 -0000 Date: Wed, 14 Feb 2007 19:00:17 -0600 From: "Rick C. Petty" To: Christopher Arnold Message-ID: <20070215010017.GA9175@keira.kiwi-computer.com> References: <6FC9F9894A9F8C49A722CF9F2132FC2204C9DAB5@ms05.mailstreet2003.net> <6FC9F9894A9F8C49A722CF9F2132FC2204C9DAB6@ms05.mailstreet2003.net> <45D1F30A.6080403@freebsd.org> <20070213192906.U726@chrishome.localnet> <20070214162938.GA96725@keira.kiwi-computer.com> <20070214173211.L1054@chrishome.localnet> <20070214170808.GC96725@keira.kiwi-computer.com> <20070215011402.A6280@chrishome.localnet> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20070215011402.A6280@chrishome.localnet> User-Agent: Mutt/1.4.2.1i Cc: freebsd-fs@freebsd.org, arla-drinkers@stacken.kth.se Subject: Re: UFS2 with SAN X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: rick-arla@kiwi-computer.com List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 15 Feb 2007 01:00:20 -0000 On Thu, Feb 15, 2007 at 01:52:51AM +0100, Christopher Arnold wrote: > > > DragonFly support is most likely fading away, AFAIK nobody's testing > > it. arla-0.43 (and arla-0-44-branch) should work on FreeBSD 5, let us know > > if there are any issues with that. > > > Hmm i managed to crash my system by doing an "ls -R" in /afs/something on > FreeBSD 5.5. Hmm, works fine here. Are you on an SMP system? PAE enabled? nvidia driver? I'm not sure what else to ask. > > Last time I tried on 5.5 it kept crashing on me in the turnstile walk. > > > My experience above with "ls -R" was also on 5.5 running arla-0.43. Works fine, just tested it: arla-0.38 on 5.4-RELEASE-p22 arla-0.39 on 5.4-RELEASE-p8 arla-0.41 on 5.5-RELEASE-p8 arla-0.43 on 5.5-RELEASE-p8 Granted, I didn't wait until the ls completed, cuz that would take too long. Maybe your cache was full? I'm not the arla expert here (obviously). -- Rick C. Petty From owner-freebsd-fs@FreeBSD.ORG Thu Feb 15 01:09:53 2007 Return-Path: X-Original-To: freebsd-fs@freebsd.org Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 6536316A400 for ; Thu, 15 Feb 2007 01:09:53 +0000 (UTC) (envelope-from chris@arnold.se) Received: from av8-1-sn3.vrr.skanova.net (av8-1-sn3.vrr.skanova.net [81.228.9.183]) by mx1.freebsd.org (Postfix) with ESMTP id E776013C442 for ; Thu, 15 Feb 2007 01:09:52 +0000 (UTC) (envelope-from chris@arnold.se) Received: by av8-1-sn3.vrr.skanova.net (Postfix, from userid 502) id 1BAC238241; Thu, 15 Feb 2007 02:09:52 +0100 (CET) Received: from smtp3-2-sn3.vrr.skanova.net (smtp3-2-sn3.vrr.skanova.net [81.228.9.102]) by av8-1-sn3.vrr.skanova.net (Postfix) with ESMTP id F421237E61; Thu, 15 Feb 2007 02:09:51 +0100 (CET) Received: from chrishome.localnet (81-224-156-16-o1033.telia.com [81.224.156.16]) by smtp3-2-sn3.vrr.skanova.net (Postfix) with ESMTP id D5BDD37E42; Thu, 15 Feb 2007 02:09:51 +0100 (CET) Received: from localhost (localhost [127.0.0.1]) by chrishome.localnet (8.13.6/8.13.6) with ESMTP id l1F19nK4006472; Thu, 15 Feb 2007 02:09:50 +0100 (CET) (envelope-from chris@arnold.se) Date: Thu, 15 Feb 2007 02:09:49 +0100 (CET) From: Christopher Arnold X-X-Sender: chris@chrishome.localnet To: "Rick C. Petty" In-Reply-To: <20070215010017.GA9175@keira.kiwi-computer.com> Message-ID: <20070215020643.B6280@chrishome.localnet> References: <6FC9F9894A9F8C49A722CF9F2132FC2204C9DAB5@ms05.mailstreet2003.net> <6FC9F9894A9F8C49A722CF9F2132FC2204C9DAB6@ms05.mailstreet2003.net> <45D1F30A.6080403@freebsd.org> <20070213192906.U726@chrishome.localnet> <20070214162938.GA96725@keira.kiwi-computer.com> <20070214173211.L1054@chrishome.localnet> <20070214170808.GC96725@keira.kiwi-computer.com> <20070215011402.A6280@chrishome.localnet> <20070215010017.GA9175@keira.kiwi-computer.com> X-message-flag: =?ISO-8859-1?Q?Outlook_isn=B4t_compliant_with_current_standards?= =?ISO-8859-1?Q?_please_install_another_mail_client!?= MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: freebsd-fs@freebsd.org, arla-drinkers@stacken.kth.se Subject: Re: UFS2 with SAN X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 15 Feb 2007 01:09:53 -0000 On Wed, 14 Feb 2007, Rick C. Petty wrote: > On Thu, Feb 15, 2007 at 01:52:51AM +0100, Christopher Arnold wrote: >> >>> DragonFly support is most likely fading away, AFAIK nobody's testing >>> it. arla-0.43 (and arla-0-44-branch) should work on FreeBSD 5, let us know >>> if there are any issues with that. >>> >> Hmm i managed to crash my system by doing an "ls -R" in /afs/something on >> FreeBSD 5.5. > > Hmm, works fine here. Are you on an SMP system? PAE enabled? nvidia > driver? I'm not sure what else to ask. > Hmm, no PAE or nvidia. But SMP. An rather old machine just for testing: CPU: Pentium II/Pentium II Xeon/Celeron (332.80-MHz 686-class CPU) Origin = "GenuineIntel" Id = 0x651 Stepping = 1 Features=0x183fbff real memory = 671088640 (640 MB) avail memory = 647073792 (617 MB) MPTable: %uname -a FreeBSD kerb.infotropic.com 5.5-RELEASE FreeBSD 5.5-RELEASE #0: Sat Oct 28 01:12:00 CEST 2006 >> My experience above with "ls -R" was also on 5.5 running arla-0.43. > > Granted, I didn't wait until the ls completed, cuz that would take too > long. Maybe your cache was full? I'm not the arla expert here (obviously). > It took some time, and maybe the cache managed to fill up. But still thats a scenario that could happen in real life... /Chris From owner-freebsd-fs@FreeBSD.ORG Thu Feb 15 01:22:23 2007 Return-Path: X-Original-To: freebsd-fs@freebsd.org Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 260A316A400 for ; Thu, 15 Feb 2007 01:22:23 +0000 (UTC) (envelope-from rick@kiwi-computer.com) Received: from kiwi-computer.com (keira.kiwi-computer.com [63.224.10.3]) by mx1.freebsd.org (Postfix) with SMTP id 9559113C442 for ; Thu, 15 Feb 2007 01:22:22 +0000 (UTC) (envelope-from rick@kiwi-computer.com) Received: (qmail 9952 invoked by uid 2001); 15 Feb 2007 01:22:21 -0000 Date: Wed, 14 Feb 2007 19:22:21 -0600 From: "Rick C. Petty" To: Christopher Arnold Message-ID: <20070215012221.GA9607@keira.kiwi-computer.com> References: <45D1F30A.6080403@freebsd.org> <20070213192906.U726@chrishome.localnet> <20070214162938.GA96725@keira.kiwi-computer.com> <20070214173211.L1054@chrishome.localnet> <20070214170808.GC96725@keira.kiwi-computer.com> <20070215011402.A6280@chrishome.localnet> <20070215010017.GA9175@keira.kiwi-computer.com> <20070215020643.B6280@chrishome.localnet> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20070215020643.B6280@chrishome.localnet> User-Agent: Mutt/1.4.2.1i Cc: freebsd-fs@freebsd.org, arla-drinkers@stacken.kth.se Subject: Re: UFS2 with SAN X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: rick-arla@kiwi-computer.com List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 15 Feb 2007 01:22:23 -0000 On Thu, Feb 15, 2007 at 02:09:49AM +0100, Christopher Arnold wrote: > > > Hmm, no PAE or nvidia. But SMP. I don't think SMP would do it, although it's possible. I just noticed that even though the 5.5 machines I found were using an SMP kernel, hw.ncpu=1 for them all. > %uname -a > FreeBSD kerb.infotropic.com 5.5-RELEASE FreeBSD 5.5-RELEASE #0: Sat Oct 28 > 01:12:00 CEST 2006 More probable is that there were 5.5 bugs which were fixed by 5.5-p8. Maybe you want to give that a try? I will certainly ask around here if anyone had to do anything special to get things to work. > It took some time, and maybe the cache managed to fill up. But still thats > a scenario that could happen in real life... Sure, but maybe arla doesn't handle that case well. I know that arla had some various bugs in the past; without a more detailed message, I can't say for sure what happened. I certainly hope someone can futz around with getting arla to work with the post-nerfed VFS changes in 6.x. Maybe that same person would be willing to help me get HFS+ working too. I gave up on both these projects because the APIs kept shifting while I was trying to get them to work, and I just don't have the time to relearn everything and familiarize myself with the new locking mechanisms. -- Rick C. Petty From owner-freebsd-fs@FreeBSD.ORG Thu Feb 15 04:41:51 2007 Return-Path: X-Original-To: freebsd-fs@freebsd.org Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 025D216A408 for ; Thu, 15 Feb 2007 04:41:51 +0000 (UTC) (envelope-from anderson@freebsd.org) Received: from mh1.centtech.com (moat3.centtech.com [64.129.166.50]) by mx1.freebsd.org (Postfix) with ESMTP id C605313C494 for ; Thu, 15 Feb 2007 04:41:50 +0000 (UTC) (envelope-from anderson@freebsd.org) Received: from [192.168.42.21] (andersonbox1.centtech.com [192.168.42.21]) by mh1.centtech.com (8.13.8/8.13.8) with ESMTP id l1F4fdGp018920; Wed, 14 Feb 2007 22:41:40 -0600 (CST) (envelope-from anderson@freebsd.org) Message-ID: <45D3E483.2000206@freebsd.org> Date: Wed, 14 Feb 2007 22:41:39 -0600 From: Eric Anderson User-Agent: Thunderbird 1.5.0.9 (X11/20070204) MIME-Version: 1.0 To: rick-arla@kiwi-computer.com References: <45D1F30A.6080403@freebsd.org> <20070213192906.U726@chrishome.localnet> <20070214162938.GA96725@keira.kiwi-computer.com> <20070214173211.L1054@chrishome.localnet> <20070214170808.GC96725@keira.kiwi-computer.com> <20070215011402.A6280@chrishome.localnet> <20070215010017.GA9175@keira.kiwi-computer.com> <20070215020643.B6280@chrishome.localnet> <20070215012221.GA9607@keira.kiwi-computer.com> In-Reply-To: <20070215012221.GA9607@keira.kiwi-computer.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Virus-Scanned: ClamAV 0.88.4/2570/Wed Feb 14 21:47:13 2007 on mh1.centtech.com X-Virus-Status: Clean X-Spam-Status: No, score=-2.6 required=8.0 tests=BAYES_00 autolearn=ham version=3.1.6 X-Spam-Checker-Version: SpamAssassin 3.1.6 (2006-10-03) on mh1.centtech.com Cc: freebsd-fs@freebsd.org, arla-drinkers@stacken.kth.se Subject: Re: UFS2 with SAN X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 15 Feb 2007 04:41:51 -0000 On 02/14/07 19:22, Rick C. Petty wrote: > On Thu, Feb 15, 2007 at 02:09:49AM +0100, Christopher Arnold wrote: >> >> Hmm, no PAE or nvidia. But SMP. > > I don't think SMP would do it, although it's possible. I just noticed that > even though the 5.5 machines I found were using an SMP kernel, hw.ncpu=1 > for them all. > >> %uname -a >> FreeBSD kerb.infotropic.com 5.5-RELEASE FreeBSD 5.5-RELEASE #0: Sat Oct 28 >> 01:12:00 CEST 2006 > > More probable is that there were 5.5 bugs which were fixed by 5.5-p8. > Maybe you want to give that a try? I will certainly ask around here if > anyone had to do anything special to get things to work. > >> It took some time, and maybe the cache managed to fill up. But still thats >> a scenario that could happen in real life... > > Sure, but maybe arla doesn't handle that case well. I know that arla had > some various bugs in the past; without a more detailed message, I can't say > for sure what happened. > > I certainly hope someone can futz around with getting arla to work with the > post-nerfed VFS changes in 6.x. Maybe that same person would be willing to > help me get HFS+ working too. I gave up on both these projects because the > APIs kept shifting while I was trying to get them to work, and I just don't > have the time to relearn everything and familiarize myself with the new > locking mechanisms. How about gathering all the information into a single spot on the web somewhere, with the old patches, ports, etc, along with whatever someone non-arla aware would need to know (cheat sheets?) to get into the hard work as quickly as possible? I know next to nothing about arla, but if the cheat sheets were there so I didn't have to spend a ton of time on learning arla completely before looking at it, it would make tinkering a lot easier. Start a new thread on HFS+ if you need to. :) Eric From owner-freebsd-fs@FreeBSD.ORG Thu Feb 15 04:47:31 2007 Return-Path: X-Original-To: freebsd-fs@freebsd.org Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id AA1CA16A400 for ; Thu, 15 Feb 2007 04:47:31 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from relay01.kiev.sovam.com (relay01.kiev.sovam.com [62.64.120.200]) by mx1.freebsd.org (Postfix) with ESMTP id 3FD1A13C4A5 for ; Thu, 15 Feb 2007 04:47:31 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from [212.82.216.227] (helo=fw.zoral.com.ua) by relay01.kiev.sovam.com with esmtps (TLSv1:AES256-SHA:256) (Exim 4.60) (envelope-from ) id 1HHYWf-0009Tb-HO for freebsd-fs@freebsd.org; Thu, 15 Feb 2007 06:47:30 +0200 Received: from deviant.kiev.zoral.com.ua (root@deviant.kiev.zoral.com.ua [10.1.1.148]) by fw.zoral.com.ua (8.13.4/8.13.4) with ESMTP id l1F4l8cj089809 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Thu, 15 Feb 2007 06:47:08 +0200 (EET) (envelope-from kostikbel@gmail.com) Received: from deviant.kiev.zoral.com.ua (kostik@localhost [127.0.0.1]) by deviant.kiev.zoral.com.ua (8.13.8/8.13.8) with ESMTP id l1F4l7iR039348; Thu, 15 Feb 2007 06:47:08 +0200 (EET) (envelope-from kostikbel@gmail.com) Received: (from kostik@localhost) by deviant.kiev.zoral.com.ua (8.13.8/8.13.8/Submit) id l1F4l7VO039347; Thu, 15 Feb 2007 06:47:07 +0200 (EET) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: deviant.kiev.zoral.com.ua: kostik set sender to kostikbel@gmail.com using -f Date: Thu, 15 Feb 2007 06:47:07 +0200 From: Kostik Belousov To: Tomas Olsson Message-ID: <20070215044707.GA39168@deviant.kiev.zoral.com.ua> References: <6FC9F9894A9F8C49A722CF9F2132FC2204C9DAB5@ms05.mailstreet2003.net> <6FC9F9894A9F8C49A722CF9F2132FC2204C9DAB6@ms05.mailstreet2003.net> <45D1F30A.6080403@freebsd.org> <20070213192906.U726@chrishome.localnet> <20070214162938.GA96725@keira.kiwi-computer.com> <20070214173211.L1054@chrishome.localnet> <20070214170808.GC96725@keira.kiwi-computer.com> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="cWoXeonUoKmBZSoM" Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.4.2.2i X-Virus-Scanned: ClamAV version 0.88.7, clamav-milter version 0.88.7 on fw.zoral.com.ua X-Virus-Status: Clean X-Spam-Status: No, score=-0.1 required=5.0 tests=ALL_TRUSTED,SPF_NEUTRAL autolearn=failed version=3.1.7 X-Spam-Checker-Version: SpamAssassin 3.1.7 (2006-10-05) on fw.zoral.com.ua X-Scanner-Signature: b3971b7ac160fe45482707327a257f38 X-DrWeb-checked: yes X-SpamTest-Envelope-From: kostikbel@gmail.com X-SpamTest-Group-ID: 00000000 X-SpamTest-Info: Profiles 773 [Feb 14 2007] X-SpamTest-Info: helo_type=3 X-SpamTest-Info: {received from trusted relay: not dialup} X-SpamTest-Method: none X-SpamTest-Method: Local Lists X-SpamTest-Rate: 0 X-SpamTest-Status: Not detected X-SpamTest-Status-Extended: not_detected X-SpamTest-Version: SMTP-Filter Version 3.0.0 [0255], KAS30/Release Cc: freebsd-fs@freebsd.org, "Rick C. Petty" , arla-drinkers@stacken.kth.se Subject: Re: UFS2 with SAN X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 15 Feb 2007 04:47:31 -0000 --cWoXeonUoKmBZSoM Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Wed, Feb 14, 2007 at 11:39:02PM +0100, Tomas Olsson wrote: > "Rick C. Petty" writes: > > the Arla client seems to behave just fine on DragonFly, Mac OS X, and a= ll > > other BSDs. In FreeBSD, it's a wash. > > > DragonFly support is most likely fading away, AFAIK nobody's testing > it. arla-0.43 (and arla-0-44-branch) should work on FreeBSD 5, let us know > if there are any issues with that. >=20 > As for arla post the blockcache rewrite, I now have OpenBSD going through > the basic tests (fails on heavier tests). If there's anyone with good > knowledge of FreeBSD's vfs who's willing to help me out a bit I think we > could have 6.2 support done in a week or so. If all is well, that is :) >=20 > Last time I tried on 5.5 it kept crashing on me in the turnstile walk. >=20 > I'm already funded and can work full time on this, but a FreeBSD hacker > would help a lot. Any volunteers? Sorry for me pointing out obvious, why not continue to use fs@ as place where to ask ? This would be better then having private communication with some developer for at least two reasons: - fs@ has bigger audience - answers would be archived and available for further reference by other interested parties. I suggest to go this way. --cWoXeonUoKmBZSoM Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (FreeBSD) iD8DBQFF0+XKC3+MBN1Mb4gRArgsAJ9dwwwsmV0DDkDvNfNqGFFDblR7+QCg9/HL COtnFTRVCFWAjfSXDdMefYg= =vVEI -----END PGP SIGNATURE----- --cWoXeonUoKmBZSoM-- From owner-freebsd-fs@FreeBSD.ORG Thu Feb 15 10:16:06 2007 Return-Path: X-Original-To: freebsd-fs@freebsd.org Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id D089516A406 for ; Thu, 15 Feb 2007 10:16:06 +0000 (UTC) (envelope-from tol@pdc.kth.se) Received: from mx2.kth.se (mx2.kth.se [130.237.48.98]) by mx1.freebsd.org (Postfix) with ESMTP id 5EEE513C4B9 for ; Thu, 15 Feb 2007 10:16:03 +0000 (UTC) (envelope-from tol@pdc.kth.se) Received: from localhost (localhost.localdomain [127.0.0.1]) by mx2.kth.se (Postfix) with ESMTP id A24BD140554; Thu, 15 Feb 2007 11:16:02 +0100 (CET) Received: from mx2.kth.se ([127.0.0.1]) by localhost (mx2.kth.se [127.0.0.1]) (amavisd-new, port 10024) with LMTP id 16277-03-17; Thu, 15 Feb 2007 11:16:00 +0100 (CET) Received: from kashyyyk.ite.kth.se (kashyyyk.ite.kth.se [130.237.31.35]) by mx2.kth.se (Postfix) with ESMTP id 6807F140584; Thu, 15 Feb 2007 11:16:00 +0100 (CET) Received: by kashyyyk.ite.kth.se (Postfix, from userid 18404) id 58CF189CA62; Thu, 15 Feb 2007 11:16:00 +0100 (CET) Sender: tol@kashyyyk.ite.kth.se From: Tomas Olsson To: Kostik Belousov References: <6FC9F9894A9F8C49A722CF9F2132FC2204C9DAB5@ms05.mailstreet2003.net> <6FC9F9894A9F8C49A722CF9F2132FC2204C9DAB6@ms05.mailstreet2003.net> <45D1F30A.6080403@freebsd.org> <20070213192906.U726@chrishome.localnet> <20070214162938.GA96725@keira.kiwi-computer.com> <20070214173211.L1054@chrishome.localnet> <20070214170808.GC96725@keira.kiwi-computer.com> <20070215044707.GA39168@deviant.kiev.zoral.com.ua> Date: 15 Feb 2007 11:16:00 +0100 In-Reply-To: <20070215044707.GA39168@deviant.kiev.zoral.com.ua> Message-ID: Lines: 47 User-Agent: Gnus/5.09 (Gnus v5.9.0) Emacs/21.3 MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii X-Virus-Scanned: by amavisd-new at kth.se Cc: freebsd-fs@freebsd.org, "Rick C. Petty" , arla-drinkers@stacken.kth.se Subject: Arla on FreeBSD X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 15 Feb 2007 10:16:06 -0000 Kostik Belousov writes: > > I'm already funded and can work full time on this, but a FreeBSD hacker > > would help a lot. Any volunteers? > > Sorry for me pointing out obvious, why not continue to use fs@ as place > where to ask ? > You're very right, I'm just to shy to do it... Thanks. Anyway; Arla is built around a "small" caching fs driver (nnpfs) servicing user requests by asking the 'arlad' daemon for help or just operating on local files created/fetched by arlad. They communicate over a char device. A simple read would be handled as such: getnode/getdata rpc to arlad installnode/installdata + wakeup msgs from arlad VOP_READ() on newly fetched cache file Subsequent reads on the same data would skip the rpc part, unless arlad has invalidated the node. Previously, there was a 1:1 mapping between nnpfs vnode and cache file. The installdata message was then handled by fetching the cache file's vnode (in arlad's context), storing it in the nnpfs_node for future reference/access. Now we ended up with one cache file per "block" (large) of data, and decided that it would be better to open/access/close the cache "block" file on each access. The closest we could get to the olden ways was to open the directory where a node's cache blocks reside, in arlad's context. The interesting part is how we open and access the cache files, and from what context. arlad is in chroot() to avoid recursive lookups across /, and it seems like a good idea to avoid such lookups now too. So the main question is how to properly do VOP_{LOOKUP,CREATE,WRITE} etc on cache files in this dual context world, without mixing identities in bad ways or confusing the OS too much. The currently messed up code lives in http://cvsweb.stacken.kth.se/cvsweb.cgi/arla/nnpfs/bsd/ Most interesting is nnpfs_vnodeops-common.c (nnpfs_write_common()) and nnpfs_blocks.c (open_file()) thanks /t From owner-freebsd-fs@FreeBSD.ORG Thu Feb 15 10:45:59 2007 Return-Path: X-Original-To: freebsd-fs@freebsd.org Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 0CD7716A400 for ; Thu, 15 Feb 2007 10:45:59 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from relay02.kiev.sovam.com (relay02.kiev.sovam.com [62.64.120.197]) by mx1.freebsd.org (Postfix) with ESMTP id 99FF313C4B4 for ; Thu, 15 Feb 2007 10:45:58 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from [212.82.216.227] (helo=fw.zoral.com.ua) by relay02.kiev.sovam.com with esmtps (TLSv1:AES256-SHA:256) (Exim 4.60) (envelope-from ) id 1HHe7Z-000LfV-5X for freebsd-fs@freebsd.org; Thu, 15 Feb 2007 12:45:57 +0200 Received: from deviant.kiev.zoral.com.ua (root@deviant.kiev.zoral.com.ua [10.1.1.148]) by fw.zoral.com.ua (8.13.4/8.13.4) with ESMTP id l1FAjchC098938 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Thu, 15 Feb 2007 12:45:38 +0200 (EET) (envelope-from kostikbel@gmail.com) Received: from deviant.kiev.zoral.com.ua (kostik@localhost [127.0.0.1]) by deviant.kiev.zoral.com.ua (8.13.8/8.13.8) with ESMTP id l1FAjcPK084447; Thu, 15 Feb 2007 12:45:38 +0200 (EET) (envelope-from kostikbel@gmail.com) Received: (from kostik@localhost) by deviant.kiev.zoral.com.ua (8.13.8/8.13.8/Submit) id l1FAjbrR084446; Thu, 15 Feb 2007 12:45:38 +0200 (EET) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: deviant.kiev.zoral.com.ua: kostik set sender to kostikbel@gmail.com using -f Date: Thu, 15 Feb 2007 12:45:37 +0200 From: Kostik Belousov To: Tomas Olsson Message-ID: <20070215104537.GC39168@deviant.kiev.zoral.com.ua> References: <6FC9F9894A9F8C49A722CF9F2132FC2204C9DAB6@ms05.mailstreet2003.net> <45D1F30A.6080403@freebsd.org> <20070213192906.U726@chrishome.localnet> <20070214162938.GA96725@keira.kiwi-computer.com> <20070214173211.L1054@chrishome.localnet> <20070214170808.GC96725@keira.kiwi-computer.com> <20070215044707.GA39168@deviant.kiev.zoral.com.ua> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="p2kqVDKq5asng8Dg" Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.4.2.2i X-Virus-Scanned: ClamAV version 0.88.7, clamav-milter version 0.88.7 on fw.zoral.com.ua X-Virus-Status: Clean X-Spam-Status: No, score=-0.1 required=5.0 tests=ALL_TRUSTED,SPF_NEUTRAL autolearn=failed version=3.1.7 X-Spam-Checker-Version: SpamAssassin 3.1.7 (2006-10-05) on fw.zoral.com.ua X-Scanner-Signature: cdcac4e9833a162831dcdc7415f73b92 X-DrWeb-checked: yes X-SpamTest-Envelope-From: kostikbel@gmail.com X-SpamTest-Group-ID: 00000000 X-SpamTest-Info: Profiles 775 [Feb 15 2007] X-SpamTest-Info: helo_type=3 X-SpamTest-Info: {received from trusted relay: not dialup} X-SpamTest-Method: none X-SpamTest-Method: Local Lists X-SpamTest-Rate: 0 X-SpamTest-Status: Not detected X-SpamTest-Status-Extended: not_detected X-SpamTest-Version: SMTP-Filter Version 3.0.0 [0255], KAS30/Release Cc: freebsd-fs@freebsd.org, "Rick C. Petty" , arla-drinkers@stacken.kth.se Subject: Re: Arla on FreeBSD X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 15 Feb 2007 10:45:59 -0000 --p2kqVDKq5asng8Dg Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Thu, Feb 15, 2007 at 11:16:00AM +0100, Tomas Olsson wrote: > Kostik Belousov writes: > > > I'm already funded and can work full time on this, but a FreeBSD hack= er > > > would help a lot. Any volunteers? > >=20 > > Sorry for me pointing out obvious, why not continue to use fs@ as place > > where to ask ? > > > You're very right, I'm just to shy to do it... Thanks. >=20 > Anyway; >=20 > Arla is built around a "small" caching fs driver (nnpfs) servicing user > requests by asking the 'arlad' daemon for help or just operating on local > files created/fetched by arlad. They communicate over a char device. >=20 > A simple read would be handled as such: > getnode/getdata rpc to arlad > installnode/installdata + wakeup msgs from arlad > VOP_READ() on newly fetched cache file >=20 > Subsequent reads on the same data would skip the rpc part, unless arlad h= as > invalidated the node. >=20 > Previously, there was a 1:1 mapping between nnpfs vnode and cache file. T= he > installdata message was then handled by fetching the cache file's vnode (= in > arlad's context), storing it in the nnpfs_node for future reference/acces= s. > Now we ended up with one cache file per "block" (large) of data, and > decided that it would be better to open/access/close the cache "block" fi= le > on each access. The closest we could get to the olden ways was to open t= he > directory where a node's cache blocks reside, in arlad's context. >=20 > The interesting part is how we open and access the cache files, and from > what context. arlad is in chroot() to avoid recursive lookups across /, a= nd > it seems like a good idea to avoid such lookups now too. >=20 > So the main question is how to properly do VOP_{LOOKUP,CREATE,WRITE} etc = on > cache files in this dual context world, without mixing identities in bad > ways or confusing the OS too much. >=20 > The currently messed up code lives in > http://cvsweb.stacken.kth.se/cvsweb.cgi/arla/nnpfs/bsd/ >=20 > Most interesting is nnpfs_vnodeops-common.c (nnpfs_write_common()) and > nnpfs_blocks.c (open_file()) I made really quick look at the places you mentioned. I have some comment for open_file(). For FreeBSD >=3D 6.x, the right way to open vnode from the kernel code is to use vn_open() (and then vn_close()) API. Something along the lines (this is for already existing file): td =3D curthread; NDINIT(&nd, LOOKUP, FOLLOW | MPSAFE, UIO_USERSPACE, fname, td); flags =3D FREAD | FWRITE; error =3D vn_open(&nd, &flags, 0, -1); if (error) return (error); vfslocked =3D NDHASGIANT(&nd); NDFREE(&nd, NDF_ONLY_PNBUF); vp =3D nd.ni_vp; vp is now locked, shall be unlocked by VOP_UNLOCK() before returning to usermode. Giant is conditionally locked based on MP-safeness of the fs vp belongs to. When Giant-protected region shall be leaved, use VFS_UNLOCK_GIANT(vfslocked); To close the vnode, use vn_close(vp, FREAD|FWRITE, td->td_ucred, td); See, for instance, kern/kern_ktrace.c, ufs/ufs/ufs_quota.c or security/audit/audit_syscalls.c for real code that does this. --p2kqVDKq5asng8Dg Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (FreeBSD) iD8DBQFF1DnRC3+MBN1Mb4gRAiqAAKCSlzaxnV3rWWufc/oKuB54/U0/sgCg3gY0 jAtwlMRvtskEz0U0iWUVkuA= =jnpE -----END PGP SIGNATURE----- --p2kqVDKq5asng8Dg-- From owner-freebsd-fs@FreeBSD.ORG Thu Feb 15 10:56:07 2007 Return-Path: X-Original-To: freebsd-fs@freebsd.org Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 3AA9E16A402 for ; Thu, 15 Feb 2007 10:56:07 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from relay02.kiev.sovam.com (relay02.kiev.sovam.com [62.64.120.197]) by mx1.freebsd.org (Postfix) with ESMTP id CB54013C4B3 for ; Thu, 15 Feb 2007 10:56:06 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from [212.82.216.227] (helo=fw.zoral.com.ua) by relay02.kiev.sovam.com with esmtps (TLSv1:AES256-SHA:256) (Exim 4.60) (envelope-from ) id 1HHeHN-000P62-6l for freebsd-fs@freebsd.org; Thu, 15 Feb 2007 12:56:05 +0200 Received: from deviant.kiev.zoral.com.ua (root@deviant.kiev.zoral.com.ua [10.1.1.148]) by fw.zoral.com.ua (8.13.4/8.13.4) with ESMTP id l1FAtkcE099204 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Thu, 15 Feb 2007 12:55:46 +0200 (EET) (envelope-from kostikbel@gmail.com) Received: from deviant.kiev.zoral.com.ua (kostik@localhost [127.0.0.1]) by deviant.kiev.zoral.com.ua (8.13.8/8.13.8) with ESMTP id l1FAtkGn084694; Thu, 15 Feb 2007 12:55:46 +0200 (EET) (envelope-from kostikbel@gmail.com) Received: (from kostik@localhost) by deviant.kiev.zoral.com.ua (8.13.8/8.13.8/Submit) id l1FAtkGf084693; Thu, 15 Feb 2007 12:55:46 +0200 (EET) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: deviant.kiev.zoral.com.ua: kostik set sender to kostikbel@gmail.com using -f Date: Thu, 15 Feb 2007 12:55:46 +0200 From: Kostik Belousov To: Tomas Olsson Message-ID: <20070215105546.GD39168@deviant.kiev.zoral.com.ua> References: <45D1F30A.6080403@freebsd.org> <20070213192906.U726@chrishome.localnet> <20070214162938.GA96725@keira.kiwi-computer.com> <20070214173211.L1054@chrishome.localnet> <20070214170808.GC96725@keira.kiwi-computer.com> <20070215044707.GA39168@deviant.kiev.zoral.com.ua> <20070215104537.GC39168@deviant.kiev.zoral.com.ua> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="J5MfuwkIyy7RmF4Q" Content-Disposition: inline In-Reply-To: <20070215104537.GC39168@deviant.kiev.zoral.com.ua> User-Agent: Mutt/1.4.2.2i X-Virus-Scanned: ClamAV version 0.88.7, clamav-milter version 0.88.7 on fw.zoral.com.ua X-Virus-Status: Clean X-Spam-Status: No, score=-0.1 required=5.0 tests=ALL_TRUSTED,SPF_NEUTRAL, UPPERCASE_25_50 autolearn=failed version=3.1.7 X-Spam-Checker-Version: SpamAssassin 3.1.7 (2006-10-05) on fw.zoral.com.ua X-Scanner-Signature: 69470ef176344869aa12c1af2e75e870 X-DrWeb-checked: yes X-SpamTest-Envelope-From: kostikbel@gmail.com X-SpamTest-Group-ID: 00000000 X-SpamTest-Info: Profiles 775 [Feb 15 2007] X-SpamTest-Info: helo_type=3 X-SpamTest-Info: {received from trusted relay: not dialup} X-SpamTest-Method: none X-SpamTest-Method: Local Lists X-SpamTest-Rate: 0 X-SpamTest-Status: Not detected X-SpamTest-Status-Extended: not_detected X-SpamTest-Version: SMTP-Filter Version 3.0.0 [0255], KAS30/Release Cc: freebsd-fs@freebsd.org, "Rick C. Petty" , arla-drinkers@stacken.kth.se Subject: Re: Arla on FreeBSD X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 15 Feb 2007 10:56:07 -0000 --J5MfuwkIyy7RmF4Q Content-Type: text/plain; charset=us-ascii Content-Disposition: inline On Thu, Feb 15, 2007 at 12:45:37PM +0200, Kostik Belousov wrote: > td = curthread; > NDINIT(&nd, LOOKUP, FOLLOW | MPSAFE, UIO_USERSPACE, fname, td); > flags = FREAD | FWRITE; > error = vn_open(&nd, &flags, 0, -1); And, forgot to mention that there is very subtle difference in 6.x/CURRENT API that fd == -1 causes unconditional Giant aquision for 6.x. > if (error) > return (error); > vfslocked = NDHASGIANT(&nd); --J5MfuwkIyy7RmF4Q Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (FreeBSD) iD8DBQFF1DwxC3+MBN1Mb4gRAky7AKC7e/G7xJsHOXZhCZV6ntORqNHjTQCgtU2S HmpTCFOIMVqA04G5ghlDgrg= =hJHR -----END PGP SIGNATURE----- --J5MfuwkIyy7RmF4Q-- From owner-freebsd-fs@FreeBSD.ORG Thu Feb 15 11:59:09 2007 Return-Path: X-Original-To: freebsd-fs@freebsd.org Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 56E8A16A406 for ; Thu, 15 Feb 2007 11:59:09 +0000 (UTC) (envelope-from tol@pdc.kth.se) Received: from mx1.kth.se (mx1.kth.se [130.237.32.140]) by mx1.freebsd.org (Postfix) with ESMTP id DA4BF13C441 for ; Thu, 15 Feb 2007 11:59:08 +0000 (UTC) (envelope-from tol@pdc.kth.se) Received: from localhost (localhost.localdomain [127.0.0.1]) by mx1.kth.se (Postfix) with ESMTP id BF1581407A8; Thu, 15 Feb 2007 12:59:07 +0100 (CET) Received: from mx1.kth.se ([127.0.0.1]) by localhost (mx1.kth.se [127.0.0.1]) (amavisd-new, port 10024) with LMTP id 21775-01-44; Thu, 15 Feb 2007 12:59:04 +0100 (CET) Received: from kashyyyk.ite.kth.se (kashyyyk.ite.kth.se [130.237.31.35]) by mx1.kth.se (Postfix) with ESMTP id CA3CC1407EC; Thu, 15 Feb 2007 12:59:04 +0100 (CET) Received: by kashyyyk.ite.kth.se (Postfix, from userid 18404) id BD35E89CA62; Thu, 15 Feb 2007 12:59:04 +0100 (CET) Sender: tol@kashyyyk.ite.kth.se From: Tomas Olsson To: Kostik Belousov References: <6FC9F9894A9F8C49A722CF9F2132FC2204C9DAB6@ms05.mailstreet2003.net> <45D1F30A.6080403@freebsd.org> <20070213192906.U726@chrishome.localnet> <20070214162938.GA96725@keira.kiwi-computer.com> <20070214173211.L1054@chrishome.localnet> <20070214170808.GC96725@keira.kiwi-computer.com> <20070215044707.GA39168@deviant.kiev.zoral.com.ua> <20070215104537.GC39168@deviant.kiev.zoral.com.ua> Date: 15 Feb 2007 12:59:04 +0100 In-Reply-To: <20070215104537.GC39168@deviant.kiev.zoral.com.ua> Message-ID: Lines: 34 User-Agent: Gnus/5.09 (Gnus v5.9.0) Emacs/21.3 MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii X-Virus-Scanned: by amavisd-new at kth.se Cc: freebsd-fs@freebsd.org, arla-drinkers@stacken.kth.se Subject: Re: Arla on FreeBSD X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 15 Feb 2007 11:59:09 -0000 Kostik Belousov writes: > On Thu, Feb 15, 2007 at 11:16:00AM +0100, Tomas Olsson wrote: > > The interesting part is how we open and access the cache files, and from > > what context. arlad is in chroot() to avoid recursive lookups across /, and > > it seems like a good idea to avoid such lookups now too. > > > > So the main question is how to properly do VOP_{LOOKUP,CREATE,WRITE} etc on > > cache files in this dual context world, without mixing identities in bad > > ways or confusing the OS too much. > > > > The currently messed up code lives in > > http://cvsweb.stacken.kth.se/cvsweb.cgi/arla/nnpfs/bsd/ > > > > Most interesting is nnpfs_vnodeops-common.c (nnpfs_write_common()) and > > nnpfs_blocks.c (open_file()) > > I made really quick look at the places you mentioned. I have some > comment for open_file(). For FreeBSD >= 6.x, the right way to open vnode > from the kernel code is to use vn_open() (and then vn_close()) API. > Great! Sounds reasonable. We currently open the cache files from nnpfs' VOPs, are there any risks (deadlock?) involved if one passes an absolute path to vn_open() in such a context? I'd have liked to do use arlad's thread for this, but vput() explicitly uses curthread deep down in namei. Also, users are not normally allowed to access the cache files directly so some OSes complain on such a lookup with user creds; would that be a problem here? Of course, we wouldn't have to worry about such things if we just kept the vnode handy for each cache block file. Maybe it's a price worth paying. thanks /t From owner-freebsd-fs@FreeBSD.ORG Thu Feb 15 12:09:20 2007 Return-Path: X-Original-To: freebsd-fs@freebsd.org Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 2A20D16A476 for ; Thu, 15 Feb 2007 12:09:20 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from relay01.kiev.sovam.com (relay01.kiev.sovam.com [62.64.120.200]) by mx1.freebsd.org (Postfix) with ESMTP id B34D913C49D for ; Thu, 15 Feb 2007 12:09:19 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from [212.82.216.227] (helo=fw.zoral.com.ua) by relay01.kiev.sovam.com with esmtps (TLSv1:AES256-SHA:256) (Exim 4.60) (envelope-from ) id 1HHfQE-000CEm-4a for freebsd-fs@freebsd.org; Thu, 15 Feb 2007 14:09:18 +0200 Received: from deviant.kiev.zoral.com.ua (root@deviant.kiev.zoral.com.ua [10.1.1.148]) by fw.zoral.com.ua (8.13.4/8.13.4) with ESMTP id l1FC8u4G001261 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Thu, 15 Feb 2007 14:08:56 +0200 (EET) (envelope-from kostikbel@gmail.com) Received: from deviant.kiev.zoral.com.ua (kostik@localhost [127.0.0.1]) by deviant.kiev.zoral.com.ua (8.13.8/8.13.8) with ESMTP id l1FC8uJa086319; Thu, 15 Feb 2007 14:08:56 +0200 (EET) (envelope-from kostikbel@gmail.com) Received: (from kostik@localhost) by deviant.kiev.zoral.com.ua (8.13.8/8.13.8/Submit) id l1FC8tA3086318; Thu, 15 Feb 2007 14:08:55 +0200 (EET) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: deviant.kiev.zoral.com.ua: kostik set sender to kostikbel@gmail.com using -f Date: Thu, 15 Feb 2007 14:08:55 +0200 From: Kostik Belousov To: Tomas Olsson Message-ID: <20070215120855.GE39168@deviant.kiev.zoral.com.ua> References: <20070213192906.U726@chrishome.localnet> <20070214162938.GA96725@keira.kiwi-computer.com> <20070214173211.L1054@chrishome.localnet> <20070214170808.GC96725@keira.kiwi-computer.com> <20070215044707.GA39168@deviant.kiev.zoral.com.ua> <20070215104537.GC39168@deviant.kiev.zoral.com.ua> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="4Epv4kl9IRBfg3rk" Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.4.2.2i X-Virus-Scanned: ClamAV version 0.88.7, clamav-milter version 0.88.7 on fw.zoral.com.ua X-Virus-Status: Clean X-Spam-Status: No, score=-0.1 required=5.0 tests=ALL_TRUSTED,SPF_NEUTRAL autolearn=failed version=3.1.7 X-Spam-Checker-Version: SpamAssassin 3.1.7 (2006-10-05) on fw.zoral.com.ua X-Scanner-Signature: 9f090f397804869cf00fade203417d62 X-DrWeb-checked: yes X-SpamTest-Envelope-From: kostikbel@gmail.com X-SpamTest-Group-ID: 00000000 X-SpamTest-Info: Profiles 775 [Feb 15 2007] X-SpamTest-Info: helo_type=3 X-SpamTest-Info: {received from trusted relay: not dialup} X-SpamTest-Method: none X-SpamTest-Method: Local Lists X-SpamTest-Rate: 0 X-SpamTest-Status: Not detected X-SpamTest-Status-Extended: not_detected X-SpamTest-Version: SMTP-Filter Version 3.0.0 [0255], KAS30/Release Cc: freebsd-fs@freebsd.org, arla-drinkers@stacken.kth.se Subject: Re: Arla on FreeBSD X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 15 Feb 2007 12:09:20 -0000 --4Epv4kl9IRBfg3rk Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Thu, Feb 15, 2007 at 12:59:04PM +0100, Tomas Olsson wrote: > Kostik Belousov writes: > > On Thu, Feb 15, 2007 at 11:16:00AM +0100, Tomas Olsson wrote: > > > The interesting part is how we open and access the cache files, and f= rom > > > what context. arlad is in chroot() to avoid recursive lookups across = /, and > > > it seems like a good idea to avoid such lookups now too. > > >=20 > > > So the main question is how to properly do VOP_{LOOKUP,CREATE,WRITE} = etc on > > > cache files in this dual context world, without mixing identities in = bad > > > ways or confusing the OS too much. > > >=20 > > > The currently messed up code lives in > > > http://cvsweb.stacken.kth.se/cvsweb.cgi/arla/nnpfs/bsd/ > > >=20 > > > Most interesting is nnpfs_vnodeops-common.c (nnpfs_write_common()) and > > > nnpfs_blocks.c (open_file()) > >=20 > > I made really quick look at the places you mentioned. I have some > > comment for open_file(). For FreeBSD >=3D 6.x, the right way to open vn= ode > > from the kernel code is to use vn_open() (and then vn_close()) API. > > > Great! Sounds reasonable. >=20 > We currently open the cache files from nnpfs' VOPs, are there any risks > (deadlock?) involved if one passes an absolute path to vn_open() in such a > context? I'd have liked to do use arlad's thread for this, but vput() There, you already have nnpfs vnode locked. The right lock order for vnodes is from root down by the tree. As such, you may end up with reversals, that would result in deadlocks, IMHO. > explicitly uses curthread deep down in namei. Also, users are not normally > allowed to access the cache files directly so some OSes complain on such a > lookup with user creds; would that be a problem here? How is the user access to cache is disabled ? And what is the cache itself ? Local filesystem (UFS) that stores your blocks in regular files ? > Of course, we wouldn't have to worry about such things if we just kept the > vnode handy for each cache block file. Maybe it's a price worth paying. Then, you need to take some care of cached vnode lifecircle (e.g., even keeping the vnode vref'ed would not prevent it from being recycled, so you may end with dead vnode). Also, as Robert pointed out in his email, you probably need to decide about MP-safeness of nnpfs. --4Epv4kl9IRBfg3rk Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (FreeBSD) iD8DBQFF1E1WC3+MBN1Mb4gRAjHwAKD3lgPzIH72g875WHdlUurV/1V+6ACfU8Tr kelnnlG00zwmCocrwlHGy70= =xmly -----END PGP SIGNATURE----- --4Epv4kl9IRBfg3rk-- From owner-freebsd-fs@FreeBSD.ORG Thu Feb 15 13:07:27 2007 Return-Path: X-Original-To: freebsd-fs@freebsd.org Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 5F9AB16A400 for ; Thu, 15 Feb 2007 13:07:27 +0000 (UTC) (envelope-from tol@pdc.kth.se) Received: from mx2.kth.se (mx2.kth.se [130.237.48.98]) by mx1.freebsd.org (Postfix) with ESMTP id E239013C46B for ; Thu, 15 Feb 2007 13:07:24 +0000 (UTC) (envelope-from tol@pdc.kth.se) Received: from localhost (localhost.localdomain [127.0.0.1]) by mx2.kth.se (Postfix) with ESMTP id CFF7614080C; Thu, 15 Feb 2007 14:07:22 +0100 (CET) Received: from mx2.kth.se ([127.0.0.1]) by localhost (mx2.kth.se [127.0.0.1]) (amavisd-new, port 10024) with LMTP id 17148-02-57; Thu, 15 Feb 2007 14:07:20 +0100 (CET) Received: from kashyyyk.ite.kth.se (kashyyyk.ite.kth.se [130.237.31.35]) by mx2.kth.se (Postfix) with ESMTP id 11A70140734; Thu, 15 Feb 2007 14:07:20 +0100 (CET) Received: by kashyyyk.ite.kth.se (Postfix, from userid 18404) id EF4F289CA62; Thu, 15 Feb 2007 14:07:19 +0100 (CET) Sender: tol@kashyyyk.ite.kth.se From: Tomas Olsson To: Kostik Belousov References: <20070213192906.U726@chrishome.localnet> <20070214162938.GA96725@keira.kiwi-computer.com> <20070214173211.L1054@chrishome.localnet> <20070214170808.GC96725@keira.kiwi-computer.com> <20070215044707.GA39168@deviant.kiev.zoral.com.ua> <20070215104537.GC39168@deviant.kiev.zoral.com.ua> <20070215120855.GE39168@deviant.kiev.zoral.com.ua> Date: 15 Feb 2007 14:07:19 +0100 In-Reply-To: <20070215120855.GE39168@deviant.kiev.zoral.com.ua> Message-ID: Lines: 53 User-Agent: Gnus/5.09 (Gnus v5.9.0) Emacs/21.3 MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii X-Virus-Scanned: by amavisd-new at kth.se Cc: freebsd-fs@freebsd.org, arla-drinkers@stacken.kth.se Subject: Re: Arla on FreeBSD X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 15 Feb 2007 13:07:27 -0000 Kostik Belousov writes: > On Thu, Feb 15, 2007 at 12:59:04PM +0100, Tomas Olsson wrote: > > Kostik Belousov writes: > > > I made really quick look at the places you mentioned. I have some > > > comment for open_file(). For FreeBSD >= 6.x, the right way to open vnode > > > from the kernel code is to use vn_open() (and then vn_close()) API. > > > > > Great! Sounds reasonable. > > > > We currently open the cache files from nnpfs' VOPs, are there any risks > > (deadlock?) involved if one passes an absolute path to vn_open() in such a > > context? I'd have liked to do use arlad's thread for this, but vput() > There, you already have nnpfs vnode locked. The right lock order for vnodes > is from root down by the tree. As such, you may end up with reversals, that > would result in deadlocks, IMHO. > Ok, vn_open() must be passed curthread so we can't use arlad's thread when in our VOP. And we cannot use an absolute path to the cache. So vn_open() can't be used? > > explicitly uses curthread deep down in namei. Also, users are not normally > > allowed to access the cache files directly so some OSes complain on such a > > lookup with user creds; would that be a problem here? > How is the user access to cache is disabled ? And what is the cache itself ? > Local filesystem (UFS) that stores your blocks in regular files ? > It's just a local dir tree, on UFS or whatever. Currently each node gets it's own subdir, and each "block" (128kB perhaps) a plain file in that dir. The cache is supposed to be readable only by arlad (usually root) and through nnpfs; that's good when handling fancy ACLs etc, so the cache root is chmod:ed to 0700 for root:wheel. > > Of course, we wouldn't have to worry about such things if we just kept the > > vnode handy for each cache block file. Maybe it's a price worth paying. > > Then, you need to take some care of cached vnode lifecircle (e.g., even > keeping the vnode vref'ed would not prevent it from being recycled, so you > may end with dead vnode). > Eep. Tricky. So where does this leave us, is plain lookup() (or VOP_LOOKUP) and VOP_CREATE on the cache a possible way to go? Seems to work on OpenBSD. > Also, as Robert pointed out in his email, you probably need to decide about > MP-safeness of nnpfs. > Well, it's marked "doesn't need giant"; we use our own global lock. I don't trust it 100%, but it seems to work so far. I haven't tried it on any MP FreeBSD boxes though. thanks /t From owner-freebsd-fs@FreeBSD.ORG Thu Feb 15 13:46:26 2007 Return-Path: X-Original-To: freebsd-fs@freebsd.org Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 005C416A402 for ; Thu, 15 Feb 2007 13:46:25 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from relay02.kiev.sovam.com (relay02.kiev.sovam.com [62.64.120.197]) by mx1.freebsd.org (Postfix) with ESMTP id 8D21113C461 for ; Thu, 15 Feb 2007 13:46:25 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from [212.82.216.227] (helo=fw.zoral.com.ua) by relay02.kiev.sovam.com with esmtps (TLSv1:AES256-SHA:256) (Exim 4.60) (envelope-from ) id 1HHgwB-0003LL-Sy for freebsd-fs@freebsd.org; Thu, 15 Feb 2007 15:46:24 +0200 Received: from deviant.kiev.zoral.com.ua (root@deviant.kiev.zoral.com.ua [10.1.1.148]) by fw.zoral.com.ua (8.13.4/8.13.4) with ESMTP id l1FDkAiJ004184 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Thu, 15 Feb 2007 15:46:10 +0200 (EET) (envelope-from kostikbel@gmail.com) Received: from deviant.kiev.zoral.com.ua (kostik@localhost [127.0.0.1]) by deviant.kiev.zoral.com.ua (8.13.8/8.13.8) with ESMTP id l1FDkA5v020572; Thu, 15 Feb 2007 15:46:10 +0200 (EET) (envelope-from kostikbel@gmail.com) Received: (from kostik@localhost) by deviant.kiev.zoral.com.ua (8.13.8/8.13.8/Submit) id l1FDk8us020571; Thu, 15 Feb 2007 15:46:09 +0200 (EET) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: deviant.kiev.zoral.com.ua: kostik set sender to kostikbel@gmail.com using -f Date: Thu, 15 Feb 2007 15:46:08 +0200 From: Kostik Belousov To: Tomas Olsson Message-ID: <20070215134608.GG39168@deviant.kiev.zoral.com.ua> References: <20070214162938.GA96725@keira.kiwi-computer.com> <20070214173211.L1054@chrishome.localnet> <20070214170808.GC96725@keira.kiwi-computer.com> <20070215044707.GA39168@deviant.kiev.zoral.com.ua> <20070215104537.GC39168@deviant.kiev.zoral.com.ua> <20070215120855.GE39168@deviant.kiev.zoral.com.ua> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="7lMq7vMTJT4tNk0a" Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.4.2.2i X-Virus-Scanned: ClamAV version 0.88.7, clamav-milter version 0.88.7 on fw.zoral.com.ua X-Virus-Status: Clean X-Spam-Status: No, score=-0.1 required=5.0 tests=ALL_TRUSTED,SPF_NEUTRAL autolearn=failed version=3.1.7 X-Spam-Checker-Version: SpamAssassin 3.1.7 (2006-10-05) on fw.zoral.com.ua X-Scanner-Signature: 63e2ad49ecdb9688bd8f281dc7b291dd X-DrWeb-checked: yes X-SpamTest-Envelope-From: kostikbel@gmail.com X-SpamTest-Group-ID: 00000000 X-SpamTest-Info: Profiles 776 [Feb 15 2007] X-SpamTest-Info: helo_type=3 X-SpamTest-Info: {received from trusted relay: not dialup} X-SpamTest-Method: none X-SpamTest-Method: Local Lists X-SpamTest-Rate: 0 X-SpamTest-Status: Not detected X-SpamTest-Status-Extended: not_detected X-SpamTest-Version: SMTP-Filter Version 3.0.0 [0255], KAS30/Release Cc: freebsd-fs@freebsd.org, arla-drinkers@stacken.kth.se Subject: Re: Arla on FreeBSD X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 15 Feb 2007 13:46:26 -0000 --7lMq7vMTJT4tNk0a Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Thu, Feb 15, 2007 at 02:07:19PM +0100, Tomas Olsson wrote: > Kostik Belousov writes: > > On Thu, Feb 15, 2007 at 12:59:04PM +0100, Tomas Olsson wrote: > > > Kostik Belousov writes: > > > > I made really quick look at the places you mentioned. I have some > > > > comment for open_file(). For FreeBSD >=3D 6.x, the right way to ope= n vnode > > > > from the kernel code is to use vn_open() (and then vn_close()) API. > > > > > > > Great! Sounds reasonable. > > >=20 > > > We currently open the cache files from nnpfs' VOPs, are there any ris= ks > > > (deadlock?) involved if one passes an absolute path to vn_open() in s= uch a > > > context? I'd have liked to do use arlad's thread for this, but vput() > > There, you already have nnpfs vnode locked. The right lock order for vn= odes > > is from root down by the tree. As such, you may end up with reversals, = that > > would result in deadlocks, IMHO. > > > Ok, vn_open() must be passed curthread so we can't use arlad's thread when > in our VOP. And we cannot use an absolute path to the cache. So vn_open() > can't be used? =20 vn_open() does not need curthread in strong sence, but td is the thread that all locks and sleeps will be performed for. What I said does not exclude neither usage of vn_open() nor pathes, but right locking order shall be ensured to prevent deadlocks. I think that your current strategy would work, but it needs to be checked. >=20 > > > explicitly uses curthread deep down in namei. Also, users are not nor= mally > > > allowed to access the cache files directly so some OSes complain on s= uch a > > > lookup with user creds; would that be a problem here? > > How is the user access to cache is disabled ? And what is the cache its= elf ? > > Local filesystem (UFS) that stores your blocks in regular files ? > > > It's just a local dir tree, on UFS or whatever. Currently each node gets > it's own subdir, and each "block" (128kB perhaps) a plain file in that di= r. > The cache is supposed to be readable only by arlad (usually root) and > through nnpfs; that's good when handling fancy ACLs etc, so the cache root > is chmod:ed to 0700 for root:wheel. >=20 > > > Of course, we wouldn't have to worry about such things if we just kep= t the > > > vnode handy for each cache block file. Maybe it's a price worth payin= g. > >=20 > > Then, you need to take some care of cached vnode lifecircle (e.g., even > > keeping the vnode vref'ed would not prevent it from being recycled, so = you > > may end with dead vnode). > > > Eep. Tricky. >=20 > So where does this leave us, is plain lookup() (or VOP_LOOKUP) and > VOP_CREATE on the cache a possible way to go? Seems to work on OpenBSD. vn_open() is the right way. Otherwise, you would in fact reimplement the code from it. Also, you could look at file handle API, that would save you of path lookups after the vnode is looked up first time (look around for vfs_vptofh and vfs_fhtovp ops). This API is used by NFS server, so it shall work :). Instead of caching vnode, you would save file handle for future accesses. >=20 > > Also, as Robert pointed out in his email, you probably need to decide a= bout > > MP-safeness of nnpfs. > > > Well, it's marked "doesn't need giant"; we use our own global lock. I don= 't > trust it 100%, but it seems to work so far. I haven't tried it on any MP > FreeBSD boxes though. Again, this could lead to lock order reversals with vnode lock. For instance, when lookup() traverses tree, it has the parent directory locked and calls fs-provided VOP_LOOKUP(). This method shall return the leaf vnode locked. Since, according to you claim, some alra-specific global lock is taken between these two vnode locks, it could LOR between vnode locks and alra lock (imagine ".." lookup in one thread, and direct lookup in another). --7lMq7vMTJT4tNk0a Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (FreeBSD) iD8DBQFF1GQgC3+MBN1Mb4gRAuVZAJ9pW94fsxUtY2y63VlMJjGl1EeIeQCfeQAW i+g1CzEkYZ7b0xpAWreRho8= =fXJL -----END PGP SIGNATURE----- --7lMq7vMTJT4tNk0a-- From owner-freebsd-fs@FreeBSD.ORG Thu Feb 15 14:21:36 2007 Return-Path: X-Original-To: fs@freebsd.org Delivered-To: freebsd-fs@FreeBSD.ORG Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 96EE216A400; Thu, 15 Feb 2007 14:21:36 +0000 (UTC) (envelope-from tataz@tataz.chchile.org) Received: from postfix1-g20.free.fr (postfix1-g20.free.fr [212.27.60.42]) by mx1.freebsd.org (Postfix) with ESMTP id 5225113C48D; Thu, 15 Feb 2007 14:21:36 +0000 (UTC) (envelope-from tataz@tataz.chchile.org) Received: from smtp5-g19.free.fr (smtp5-g19.free.fr [212.27.42.35]) by postfix1-g20.free.fr (Postfix) with ESMTP id 880F99D2B38; Thu, 15 Feb 2007 14:56:52 +0100 (CET) Received: from tatooine.tataz.chchile.org (tataz.chchile.org [82.233.239.98]) by smtp5-g19.free.fr (Postfix) with ESMTP id 85EBC278DA; Thu, 15 Feb 2007 14:56:50 +0100 (CET) Received: from obiwan.tataz.chchile.org (unknown [192.168.1.25]) by tatooine.tataz.chchile.org (Postfix) with ESMTP id CB7359D41F; Thu, 15 Feb 2007 13:57:50 +0000 (UTC) Received: by obiwan.tataz.chchile.org (Postfix, from userid 1000) id 9D98B405D; Thu, 15 Feb 2007 14:57:50 +0100 (CET) Date: Thu, 15 Feb 2007 14:57:50 +0100 From: Jeremie Le Hen To: Josef Karthauser Message-ID: <20070215135750.GR64768@obiwan.tataz.chchile.org> References: <20070204023711.GA3393@genius.tao.org.uk> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20070204023711.GA3393@genius.tao.org.uk> User-Agent: Mutt/1.5.13 (2006-08-11) Cc: hackers@freebsd.org, fs@freebsd.org Subject: Re: nullfs and named pipes. X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 15 Feb 2007 14:21:36 -0000 Hi Josef, On Sun, Feb 04, 2007 at 02:37:11AM +0000, Josef Karthauser wrote: > There appears to be a lot of confusion on the lists about this point > as many people are trying to do this so as to make a single mysql > server available from within a number of jails, for instance. However > people appear to think that this is a limitation of the jail code, not a > limitation of the null_fs code. Having named pipes work in null_fs > filesystems would be a very handy thing indeed. Note that all processes within a jail can only intefere with processes from another jail or host as if they were on different machines. This means they can communicate through PF_INET for instance but not PF_LOCAL. IOW you have to think your jails as if theey were multiples boxes. You should therefore make them communicate with networking sockets and protect the latter with firewalling rules or tcpwrapper. Regards, -- Jeremie Le Hen < jeremie at le-hen dot org >< ttz at chchile dot org > From owner-freebsd-fs@FreeBSD.ORG Thu Feb 15 14:56:43 2007 Return-Path: X-Original-To: freebsd-fs@freebsd.org Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id B470C16A402 for ; Thu, 15 Feb 2007 14:56:43 +0000 (UTC) (envelope-from tol@pdc.kth.se) Received: from mx2.kth.se (mx2.kth.se [130.237.48.98]) by mx1.freebsd.org (Postfix) with ESMTP id 4252C13C48E for ; Thu, 15 Feb 2007 14:56:43 +0000 (UTC) (envelope-from tol@pdc.kth.se) Received: from localhost (localhost.localdomain [127.0.0.1]) by mx2.kth.se (Postfix) with ESMTP id 2680C14082B; Thu, 15 Feb 2007 15:56:42 +0100 (CET) Received: from mx2.kth.se ([127.0.0.1]) by localhost (mx2.kth.se [127.0.0.1]) (amavisd-new, port 10024) with LMTP id 31130-01-67; Thu, 15 Feb 2007 15:56:39 +0100 (CET) Received: from kashyyyk.ite.kth.se (kashyyyk.ite.kth.se [130.237.31.35]) by mx2.kth.se (Postfix) with ESMTP id 3E54614081E; Thu, 15 Feb 2007 15:56:39 +0100 (CET) Received: by kashyyyk.ite.kth.se (Postfix, from userid 18404) id F0B8489CA62; Thu, 15 Feb 2007 15:56:38 +0100 (CET) Sender: tol@kashyyyk.ite.kth.se From: Tomas Olsson To: Kostik Belousov References: <20070214162938.GA96725@keira.kiwi-computer.com> <20070214173211.L1054@chrishome.localnet> <20070214170808.GC96725@keira.kiwi-computer.com> <20070215044707.GA39168@deviant.kiev.zoral.com.ua> <20070215104537.GC39168@deviant.kiev.zoral.com.ua> <20070215120855.GE39168@deviant.kiev.zoral.com.ua> <20070215134608.GG39168@deviant.kiev.zoral.com.ua> Date: 15 Feb 2007 15:56:38 +0100 In-Reply-To: <20070215134608.GG39168@deviant.kiev.zoral.com.ua> Message-ID: Lines: 48 User-Agent: Gnus/5.09 (Gnus v5.9.0) Emacs/21.3 MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii X-Virus-Scanned: by amavisd-new at kth.se Cc: freebsd-fs@freebsd.org, arla-drinkers@stacken.kth.se Subject: Re: Arla on FreeBSD X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 15 Feb 2007 14:56:43 -0000 Kostik Belousov writes: > On Thu, Feb 15, 2007 at 02:07:19PM +0100, Tomas Olsson wrote: > > Ok, vn_open() must be passed curthread so we can't use arlad's thread when > > in our VOP. And we cannot use an absolute path to the cache. So vn_open() > > can't be used? > vn_open() does not need curthread in strong sence, but td is the thread > that all locks and sleeps will be performed for. > Except in places like vput() and vrele()? struct thread *td = curthread; /* XXX */ > I think that your current strategy would work, but it needs to be checked. > Nice. Possibly there are still a few places where I mix contexts too much as well. And I've probably missed a few mandatory support routines along the lines of vn_start_write(). > > So where does this leave us, is plain lookup() (or VOP_LOOKUP) and > > VOP_CREATE on the cache a possible way to go? Seems to work on OpenBSD. > vn_open() is the right way. Otherwise, you would in fact reimplement the > code from it. > I know, I just don't see how to get around it. > Also, you could look at file handle API, that would save you of path lookups > after the vnode is looked up first time (look around for vfs_vptofh and > vfs_fhtovp ops). This API is used by NFS server, so it shall work :). > That's an idea. We used that before in another context, but then we ran into trouble with too many OSes so we dropped it. Depends on how much code one can share. > > Well, it's marked "doesn't need giant"; we use our own global lock. I don't > > trust it 100%, but it seems to work so far. I haven't tried it on any MP > > FreeBSD boxes though. > > Again, this could lead to lock order reversals with vnode lock. For > instance, when lookup() traverses tree, it has the parent directory > locked and calls fs-provided VOP_LOOKUP(). This method shall return the > leaf vnode locked. Since, according to you claim, some alra-specific > global lock is taken between these two vnode locks, it could LOR between > vnode locks and alra lock > I'm not sure if that's currently a problem for us, but I'll take a look. I'm sure there are other cases. thanks /t From owner-freebsd-fs@FreeBSD.ORG Thu Feb 15 15:04:40 2007 Return-Path: X-Original-To: freebsd-fs@freebsd.org Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id A817616A401 for ; Thu, 15 Feb 2007 15:04:40 +0000 (UTC) (envelope-from rwatson@FreeBSD.org) Received: from cyrus.watson.org (cyrus.watson.org [209.31.154.42]) by mx1.freebsd.org (Postfix) with ESMTP id 5008E13C50D for ; Thu, 15 Feb 2007 15:04:40 +0000 (UTC) (envelope-from rwatson@FreeBSD.org) Received: from fledge.watson.org (fledge.watson.org [209.31.154.41]) by cyrus.watson.org (Postfix) with ESMTP id 8FC73477DC; Thu, 15 Feb 2007 10:04:39 -0500 (EST) Date: Thu, 15 Feb 2007 15:04:39 +0000 (GMT) From: Robert Watson X-X-Sender: robert@fledge.watson.org To: Tomas Olsson In-Reply-To: Message-ID: <20070215150015.O90799@fledge.watson.org> References: <20070214162938.GA96725@keira.kiwi-computer.com> <20070214173211.L1054@chrishome.localnet> <20070214170808.GC96725@keira.kiwi-computer.com> <20070215044707.GA39168@deviant.kiev.zoral.com.ua> <20070215104537.GC39168@deviant.kiev.zoral.com.ua> <20070215120855.GE39168@deviant.kiev.zoral.com.ua> <20070215134608.GG39168@deviant.kiev.zoral.com.ua> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: freebsd-fs@freebsd.org, arla-drinkers@stacken.kth.se Subject: Re: Arla on FreeBSD X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 15 Feb 2007 15:04:40 -0000 On Thu, 15 Feb 2007, Tomas Olsson wrote: >> Also, you could look at file handle API, that would save you of path >> lookups after the vnode is looked up first time (look around for vfs_vptofh >> and vfs_fhtovp ops). This API is used by NFS server, so it shall work :). >> > That's an idea. We used that before in another context, but then we ran into > trouble with too many OSes so we dropped it. Depends on how much code one > can share. It has been ages since I looked at the xfs/nnpfs internals, but I thought it already used file handles to identify cache files to the kernel? I seem to recall several of our fh*() system calls were added by Assar specifically in support of Arla work. With regard to cache files -- historically (i.e., last time I looked closely), there was a one-to-one mapping between cache files and files in the Arla working set, and all files were transfered "whole" into the local cache. This was in contrast to AFS's chunking behavior. I assume the cache file changes to support "lines" in the cache are in support of adding chunking to Arla. Is it the case that one file visible in /afs may now be backed by pages coming from different cache files? How does this interact with the vnode pager? I assume you actually end up with two pagers -- one for the file as visible in /afs, and then satisfy the pager's VOP requests by forwarding them to the right one of the cache vnodes? I trust you don't mmap the cache vnode, which would lead to cache consistency issues between any mappings of the /afs vnode vs the cache vnode? Any chance we could attract you to BSDCan to meet in person to discuss this? :-) Thanks, Robert N M Watson Computer Laboratory University of Cambridge From owner-freebsd-fs@FreeBSD.ORG Thu Feb 15 15:23:08 2007 Return-Path: X-Original-To: fs@freebsd.org Delivered-To: freebsd-fs@FreeBSD.ORG Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 0A03F16A400; Thu, 15 Feb 2007 15:23:08 +0000 (UTC) (envelope-from joe@tao.org.uk) Received: from mailhost.tao.org.uk (transwarp.tao.org.uk [87.74.4.34]) by mx1.freebsd.org (Postfix) with ESMTP id 2EF0B13C4A7; Thu, 15 Feb 2007 15:23:07 +0000 (UTC) (envelope-from joe@tao.org.uk) Received: from genius.tao.org.uk (wireless58.dhcp.tao.org.uk [87.74.4.58]) by mailhost.tao.org.uk (Postfix) with ESMTP id 1EA025D1F; Thu, 15 Feb 2007 15:23:05 +0000 (GMT) Received: by genius.tao.org.uk (Postfix, from userid 1000) id CD2C440D4; Thu, 15 Feb 2007 15:22:59 +0000 (GMT) Date: Thu, 15 Feb 2007 15:22:59 +0000 From: Josef Karthauser To: Jeremie Le Hen Message-ID: <20070215152259.GA2950@genius.tao.org.uk> References: <20070204023711.GA3393@genius.tao.org.uk> <20070215135750.GR64768@obiwan.tataz.chchile.org> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="opJtzjQTFsWo+cga" Content-Disposition: inline In-Reply-To: <20070215135750.GR64768@obiwan.tataz.chchile.org> User-Agent: Mutt/1.5.11 Cc: hackers@freebsd.org, fs@freebsd.org Subject: Re: nullfs and named pipes. X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 15 Feb 2007 15:23:08 -0000 --opJtzjQTFsWo+cga Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Thu, Feb 15, 2007 at 02:57:50PM +0100, Jeremie Le Hen wrote: >=20 > Note that all processes within a jail can only intefere with processes > from another jail or host as if they were on different machines. This > means they can communicate through PF_INET for instance but not > PF_LOCAL. >=20 You might think so! However that's not what's going on here. The named pipe/nullfs issue is nothing to do with jails. It's just that nullfs is broken with respect to named pipes as I've previously reported. However with this patch: cvs diff: Diffing . Index: null_subr.c =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D RCS file: /home/ncvs/src/sys/fs/nullfs/null_subr.c,v retrieving revision 1.48.2.1 diff -u -r1.48.2.1 null_subr.c --- null_subr.c 13 Mar 2006 03:05:17 -0000 1.48.2.1 +++ null_subr.c 14 Feb 2007 00:02:28 -0000 @@ -235,6 +235,8 @@ xp->null_vnode =3D vp; xp->null_lowervp =3D lowervp; vp->v_type =3D lowervp->v_type; + if (vp->v_type =3D=3D VSOCK || vp->v_type =3D=3D VFIFO) + vp->v_un =3D lowervp->v_un; vp->v_data =3D xp; vp->v_vnlock =3D lowervp->v_vnlock; if (vp->v_vnlock =3D=3D NULL) that problem goes away. Now a named pipe created on a lower layer can be spoken to by a process connecting to it on a higher layer, i.e (for demostration purposes only): # ls -ld /tmp/mysql.sock=20 srwxrwxrwx 1 mysql wheel 0 Jan 4 09:26 /tmp/mysql.sock # mount_nullfs /tmp /mnt # ls -ld /mnt/mysql.sock=20 srwxrwxrwx 1 mysql wheel 0 Jan 4 09:26 /mnt/mysql.sock With a stock kernel this fails: % mysql --socket=3D/mnt/mysql.sock=20 ERROR 2002 (HY000): Can't connect to local MySQL server through socket = '/mnt/mysql.sock' (61) but with the patch above it works: % mysql --socket=3D/mnt/mysql.sock=20 ERROR 1045 (28000): Access denied for user 'joe'@'localhost' (using pas= sword: NO) Of course the patch above doesn't work if the socket is created at /mnt/mysql.sock and something wants to talk to it over at /tmp/mysql.sock, however that is not really a problem. So how does this relate to jails? The point of using nullfs is to make a PF_LOCAL socket appear local even in the jail(!). Using the patch above this is indeed the case and as far as the jail is concerned the socket is indeed local, meaning that a process within a jail can talk via it to a process on the host environment with no restrictions. This is crucially important for mysql for instance as there is significant overhead associated with PF_INET connections which can be avoided by talking to PF_LOCAL sockets. > IOW you have to think your jails as if theey were multiples boxes. > You should therefore make them communicate with networking sockets and > protect the latter with firewalling rules or tcpwrapper. Now in terms of protecting the host environment this is trivially done by using a read-only nullfs mount: # mkdir /mysql # mysqld_safe --socket=3D/mysql/mysql.sock & # mount_nullfs -oro /mysql /jail/mysql voila. The database can now be connected to within the jail environment on /mysql/mysql.sock as a local fast connection, but as /mysql is mounted read-only they cannot do anything other than connect to the socket :). Remember tools not policy :). Joe --opJtzjQTFsWo+cga Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.4 (FreeBSD) iEYEARECAAYFAkXUetMACgkQXVIcjOaxUBbbogCfZrNO34uk2z7eps+SUs2YrzOE lokAoNOWAFev63KHT4KCIOXzQhDvxzCQ =joYw -----END PGP SIGNATURE----- --opJtzjQTFsWo+cga-- From owner-freebsd-fs@FreeBSD.ORG Thu Feb 15 15:32:03 2007 Return-Path: X-Original-To: fs@freebsd.org Delivered-To: freebsd-fs@FreeBSD.ORG Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 04B0416A402; Thu, 15 Feb 2007 15:32:03 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from relay01.kiev.sovam.com (relay01.kiev.sovam.com [62.64.120.200]) by mx1.freebsd.org (Postfix) with ESMTP id 8E26A13C441; Thu, 15 Feb 2007 15:32:02 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from [212.82.216.227] (helo=fw.zoral.com.ua) by relay01.kiev.sovam.com with esmtps (TLSv1:AES256-SHA:256) (Exim 4.60) (envelope-from ) id 1HHiaP-000Dcu-5q; Thu, 15 Feb 2007 17:32:01 +0200 Received: from deviant.kiev.zoral.com.ua (root@deviant.kiev.zoral.com.ua [10.1.1.148]) by fw.zoral.com.ua (8.13.4/8.13.4) with ESMTP id l1FFVasu007194 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Thu, 15 Feb 2007 17:31:36 +0200 (EET) (envelope-from kostikbel@gmail.com) Received: from deviant.kiev.zoral.com.ua (kostik@localhost [127.0.0.1]) by deviant.kiev.zoral.com.ua (8.13.8/8.13.8) with ESMTP id l1FFVam6010115; Thu, 15 Feb 2007 17:31:36 +0200 (EET) (envelope-from kostikbel@gmail.com) Received: (from kostik@localhost) by deviant.kiev.zoral.com.ua (8.13.8/8.13.8/Submit) id l1FFVaS3010114; Thu, 15 Feb 2007 17:31:36 +0200 (EET) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: deviant.kiev.zoral.com.ua: kostik set sender to kostikbel@gmail.com using -f Date: Thu, 15 Feb 2007 17:31:35 +0200 From: Kostik Belousov To: Josef Karthauser Message-ID: <20070215153135.GI39168@deviant.kiev.zoral.com.ua> References: <20070204023711.GA3393@genius.tao.org.uk> <20070215135750.GR64768@obiwan.tataz.chchile.org> <20070215152259.GA2950@genius.tao.org.uk> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="o7gdRJTuwFmWapyH" Content-Disposition: inline In-Reply-To: <20070215152259.GA2950@genius.tao.org.uk> User-Agent: Mutt/1.4.2.2i X-Virus-Scanned: ClamAV version 0.88.7, clamav-milter version 0.88.7 on fw.zoral.com.ua X-Virus-Status: Clean X-Spam-Status: No, score=-0.1 required=5.0 tests=ALL_TRUSTED,SPF_NEUTRAL autolearn=failed version=3.1.7 X-Spam-Checker-Version: SpamAssassin 3.1.7 (2006-10-05) on fw.zoral.com.ua X-Scanner-Signature: 5e2df25549de819b25d1bfd51e9f52eb X-DrWeb-checked: yes X-SpamTest-Envelope-From: kostikbel@gmail.com X-SpamTest-Group-ID: 00000000 X-SpamTest-Info: Profiles 776 [Feb 15 2007] X-SpamTest-Info: helo_type=3 X-SpamTest-Info: {received from trusted relay: not dialup} X-SpamTest-Method: none X-SpamTest-Method: Local Lists X-SpamTest-Rate: 0 X-SpamTest-Status: Not detected X-SpamTest-Status-Extended: not_detected X-SpamTest-Version: SMTP-Filter Version 3.0.0 [0255], KAS30/Release Cc: hackers@freebsd.org, Jeremie Le Hen , fs@freebsd.org Subject: Re: nullfs and named pipes. X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 15 Feb 2007 15:32:03 -0000 --o7gdRJTuwFmWapyH Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Thu, Feb 15, 2007 at 03:22:59PM +0000, Josef Karthauser wrote: > On Thu, Feb 15, 2007 at 02:57:50PM +0100, Jeremie Le Hen wrote: > >=20 > > Note that all processes within a jail can only intefere with processes > > from another jail or host as if they were on different machines. This > > means they can communicate through PF_INET for instance but not > > PF_LOCAL. > >=20 >=20 > You might think so! However that's not what's going on here. >=20 > The named pipe/nullfs issue is nothing to do with jails. It's just > that nullfs is broken with respect to named pipes as I've previously > reported. However with this patch: >=20 > cvs diff: Diffing . > Index: null_subr.c > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > RCS file: /home/ncvs/src/sys/fs/nullfs/null_subr.c,v > retrieving revision 1.48.2.1 > diff -u -r1.48.2.1 null_subr.c > --- null_subr.c 13 Mar 2006 03:05:17 -0000 1.48.2.1 > +++ null_subr.c 14 Feb 2007 00:02:28 -0000 > @@ -235,6 +235,8 @@ > xp->null_vnode =3D vp; > xp->null_lowervp =3D lowervp; > vp->v_type =3D lowervp->v_type; > + if (vp->v_type =3D=3D VSOCK || vp->v_type =3D=3D VFIFO) > + vp->v_un =3D lowervp->v_un; I'm wondering is some reference counting needed there ? > vp->v_data =3D xp; > vp->v_vnlock =3D lowervp->v_vnlock; > if (vp->v_vnlock =3D=3D NULL) >=20 --o7gdRJTuwFmWapyH Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (FreeBSD) iD8DBQFF1HzXC3+MBN1Mb4gRAjWqAKDbD0zmXgl9XH84olKNpGzCa4jUvQCg1wTj PS5YUBnN+3ferD6fSsPKfAQ= =5j5c -----END PGP SIGNATURE----- --o7gdRJTuwFmWapyH-- From owner-freebsd-fs@FreeBSD.ORG Thu Feb 15 15:43:08 2007 Return-Path: X-Original-To: freebsd-fs@FreeBSD.org Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 83BB216A401 for ; Thu, 15 Feb 2007 15:43:08 +0000 (UTC) (envelope-from tol@pdc.kth.se) Received: from mx1.kth.se (mx1.kth.se [130.237.32.140]) by mx1.freebsd.org (Postfix) with ESMTP id 2616513C4A8 for ; Thu, 15 Feb 2007 15:43:01 +0000 (UTC) (envelope-from tol@pdc.kth.se) Received: from localhost (localhost.localdomain [127.0.0.1]) by mx1.kth.se (Postfix) with ESMTP id 7DB38140735; Thu, 15 Feb 2007 16:43:00 +0100 (CET) Received: from mx1.kth.se ([127.0.0.1]) by localhost (mx1.kth.se [127.0.0.1]) (amavisd-new, port 10024) with LMTP id 10619-01-15; Thu, 15 Feb 2007 16:42:48 +0100 (CET) Received: from kashyyyk.ite.kth.se (kashyyyk.ite.kth.se [130.237.31.35]) by mx1.kth.se (Postfix) with ESMTP id 33875140763; Thu, 15 Feb 2007 16:42:48 +0100 (CET) Received: by kashyyyk.ite.kth.se (Postfix, from userid 18404) id 2789389CA62; Thu, 15 Feb 2007 16:42:48 +0100 (CET) Sender: tol@kashyyyk.ite.kth.se From: Tomas Olsson To: Robert Watson References: <20070214162938.GA96725@keira.kiwi-computer.com> <20070214173211.L1054@chrishome.localnet> <20070214170808.GC96725@keira.kiwi-computer.com> <20070215044707.GA39168@deviant.kiev.zoral.com.ua> <20070215104537.GC39168@deviant.kiev.zoral.com.ua> <20070215120855.GE39168@deviant.kiev.zoral.com.ua> <20070215134608.GG39168@deviant.kiev.zoral.com.ua> <20070215150015.O90799@fledge.watson.org> Date: 15 Feb 2007 16:42:48 +0100 In-Reply-To: <20070215150015.O90799@fledge.watson.org> Message-ID: Lines: 49 User-Agent: Gnus/5.09 (Gnus v5.9.0) Emacs/21.3 MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii X-Virus-Scanned: by amavisd-new at kth.se Cc: freebsd-fs@FreeBSD.org, arla-drinkers@stacken.kth.se Subject: Re: Arla on FreeBSD X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 15 Feb 2007 15:43:08 -0000 Robert Watson writes: > On Thu, 15 Feb 2007, Tomas Olsson wrote: > >> Also, you could look at file handle API, that would save you of path > >> lookups after the vnode is looked up first time (look around for vfs_vptofh > >> and vfs_fhtovp ops). This API is used by NFS server, so it shall work :). > >> > > That's an idea. We used that before in another context, but then we ran into > > trouble with too many OSes so we dropped it. Depends on how much code one > > can share. > > It has been ages since I looked at the xfs/nnpfs internals, but I thought it > already used file handles to identify cache files to the kernel? I seem to > recall several of our fh*() system calls were added by Assar specifically in > support of Arla work. > Sounds likely. I first saw it from arlad's side, so I've always thought of it as an optimization to avoid lookups in that end without pondering it all that much. > Is it the case that one file visible in /afs may now be backed by pages > coming from different cache files? > Yup. There ware requests for low latency fetching of ID3 tags etc, and working with files larger than the cache. > How does this interact with the vnode pager? I assume you actually end > up with two pagers -- one for the file as visible in /afs, and then > satisfy the pager's VOP requests by forwarding them to the right one of > the cache vnodes? > Exactly. As always, just more fine grained. > I trust you don't mmap the cache vnode, which would lead to cache > consistency issues between any mappings of the /afs vnode vs the cache > vnode? > Me too :-) But this is still a suboptimal world where we end up with double pager layers, and consistency can still be a problem. In the long term it would be nice to have arlad open it's cache nodes through nnpfs, be it small files or just one large VM space. > Any chance we could attract you to BSDCan to meet in person to discuss this? > Chance, not probable; May is beyond my horizon. And I do expect to have the arla working properly on fbsd well before that. /t From owner-freebsd-fs@FreeBSD.ORG Thu Feb 15 15:46:07 2007 Return-Path: X-Original-To: freebsd-fs@freebsd.org Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 343EC16A400 for ; Thu, 15 Feb 2007 15:46:07 +0000 (UTC) (envelope-from hagelstein@shr.cc) Received: from shr.cc (webmail.shr.cc [213.221.88.186]) by mx1.freebsd.org (Postfix) with ESMTP id AC83813C491 for ; Thu, 15 Feb 2007 15:46:06 +0000 (UTC) (envelope-from hagelstein@shr.cc) X-MimeOLE: Produced By Microsoft Exchange V6.5 Content-class: urn:content-classes:message MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Date: Thu, 15 Feb 2007 16:14:08 +0100 Message-ID: <05F5882A865FBB4EB684C22EA83529094A4BAF@mailserver.open.local> X-MS-Has-Attach: X-MS-TNEF-Correlator: Thread-Topic: 3ware 9650SE RAID5 write performance issues Thread-Index: AcdRE/BafM828/CiQM6NIRhrcok4cQ== From: "Nikolas Hagelstein" To: Subject: 3ware 9650SE RAID5 write performance issues X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 15 Feb 2007 15:46:07 -0000 Hi, First of all: sorry for crossposting to hw/fs but since this issue seems To be hw+fs related i thought it would make sense to post it to both lists ;). I am running the 3ware 9650SE(8LPML) in RAID5 with 7x700GB drives attached to it (~4.2 TB at raid5). Since i need one huge "storage area" the filesystem resides at the raw device (i.e. /dev/da0). For some reasons the raids write performance is very slow (2 to 3MB/s): 33554432 bytes transferred in 14.679426 secs (2285814 bytes/sec)=20 The raid is initialized and all disks are ok (double checked that) I tried several different filesystem and raidcontroller options (e.g. blocksize,frag, Slicesice etc.) without any success. Another strange thing: for testing prupose i set the controllerto raid0 mode which resulted in 55MB/s. Though raid0 is faster than raid5 by principle that anyhow seems really strange to me. I am not sure if this is a pure hardware related issue since "raw access" performes quite good: diskinfo -t /dev/da0 /dev/da0 512 # sectorsize 4499932446720 # mediasize in bytes (4.1T) 8788930560 # mediasize in sectors 547085 # Cylinders according to firmware. 255 # Heads according to firmware. 63 # Sectors according to firmware. .... Transfer rates: outside: 102400 kbytes in 1.052566 sec =3D 97286 kbytes/sec middle: 102400 kbytes in 1.065493 sec =3D 96106 kbytes/sec inside: 102400 kbytes in 1.067073 sec =3D 95963 kbytes/sec I attached any needed information: FreeBSD 6.2-RELEASE Dmesg snippet(a full version can be find here:http://phpfi.com/203614): --- 3ware device driver for 9000 series storage controllers, version: 3.60.03.005 twa0: <3ware 9000 series Storage Controller> port 0xe800-0xe8ff mem 0xde000000-0xdfffffff,0xbffff000-0xbfffffff irq 16 at device 0.0 on pci4 twa0: [GIANT-LOCKED] twa0: INFO: (0x15: 0x1300): Controller details:: Model 9650SE-8LPML, 8 ports, Firmware FE9X 3.06.00.005, BIOS BE9X 3.06.00.002 da0 at twa0 bus 0 target 0 lun 0 da0: Fixed Direct Access SCSI-5 device da0: 100.000MB/s transfers da0: 4291470MB (8788930560 512 byte sectors: 255H 63S/T 547085C) --- --- tunefs -p /dev/da0 tunefs: ACLs: (-a) disabled tunefs: MAC multilabel: (-l) disabled tunefs: soft updates: (-n) disabled tunefs: gjournal: (-J) disabled tunefs: maximum blocks per file in a cylinder group: (-e) 2048 tunefs: average file size: (-f) 16384 tunefs: average number of files in a directory: (-s) 64 tunefs: minimum percentage of free space: (-m) 8% tunefs: optimization preference: (-o) time tunefs: volume label: (-L) --- --- Ctl Model Ports Drives Units NotOpt RRate VRate BBU ------------------------------------------------------------------------ c0 9650SE-8LPML 8 7 1 0 1 1 - //backup> /c0 show Unit UnitType Status %RCmpl %V/I/M Stripe Size(GB) Cache AVrfy ------------------------------------------------------------------------ ------ u0 RAID-5 OK - - 256K 4190.89 OFF OFF Port Status Unit Size Blocks Serial --------------------------------------------------------------- p0 OK u0 698.63 GB 1465149168 3QD0Q4N6 p1 OK u0 698.63 GB 1465149168 3QD0RYEP p2 OK u0 698.63 GB 1465149168 3QD0HEZM p3 OK u0 698.63 GB 1465149168 3QD0PZK7 p4 OK u0 698.63 GB 1465149168 3QD07Q7G p5 OK u0 698.63 GB 1465149168 3QD0PX0J p6 OK u0 698.63 GB 1465149168 3QD0M5VQ p7 NOT-PRESENT - - - - --- Cheers, Nikolas From owner-freebsd-fs@FreeBSD.ORG Thu Feb 15 16:34:19 2007 Return-Path: X-Original-To: fs@freebsd.org Delivered-To: freebsd-fs@FreeBSD.ORG Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id DBDE716A400; Thu, 15 Feb 2007 16:34:19 +0000 (UTC) (envelope-from joe@tao.org.uk) Received: from mailhost.tao.org.uk (transwarp.tao.org.uk [87.74.4.34]) by mx1.freebsd.org (Postfix) with ESMTP id 9767113C48E; Thu, 15 Feb 2007 16:34:19 +0000 (UTC) (envelope-from joe@tao.org.uk) Received: from genius.tao.org.uk (wireless58.dhcp.tao.org.uk [87.74.4.58]) by mailhost.tao.org.uk (Postfix) with ESMTP id 6C9B665C5; Thu, 15 Feb 2007 16:34:18 +0000 (GMT) Received: by genius.tao.org.uk (Postfix, from userid 1000) id 28B2A40D4; Thu, 15 Feb 2007 16:34:13 +0000 (GMT) Date: Thu, 15 Feb 2007 16:34:12 +0000 From: Josef Karthauser To: Kostik Belousov Message-ID: <20070215163412.GC3439@genius.tao.org.uk> References: <20070204023711.GA3393@genius.tao.org.uk> <20070215135750.GR64768@obiwan.tataz.chchile.org> <20070215152259.GA2950@genius.tao.org.uk> <20070215153135.GI39168@deviant.kiev.zoral.com.ua> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="+xNpyl7Qekk2NvDX" Content-Disposition: inline In-Reply-To: <20070215153135.GI39168@deviant.kiev.zoral.com.ua> User-Agent: Mutt/1.5.11 Cc: hackers@freebsd.org, Jeremie Le Hen , fs@freebsd.org Subject: Re: nullfs and named pipes. X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 15 Feb 2007 16:34:20 -0000 --+xNpyl7Qekk2NvDX Content-Type: text/plain; charset=us-ascii Content-Disposition: inline On Thu, Feb 15, 2007 at 05:31:35PM +0200, Kostik Belousov wrote: > > --- null_subr.c 13 Mar 2006 03:05:17 -0000 1.48.2.1 > > +++ null_subr.c 14 Feb 2007 00:02:28 -0000 > > @@ -235,6 +235,8 @@ > > xp->null_vnode = vp; > > xp->null_lowervp = lowervp; > > vp->v_type = lowervp->v_type; > > + if (vp->v_type == VSOCK || vp->v_type == VFIFO) > > + vp->v_un = lowervp->v_un; > I'm wondering is some reference counting needed there ? Maybe. I'm not at all familiar with the filesystem code. I was hoping someone with a bit of knowledge could fix it up. Joe --+xNpyl7Qekk2NvDX Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.4 (FreeBSD) iEYEARECAAYFAkXUi4QACgkQXVIcjOaxUBb7xwCfUi+KfPEgFryt/ox89Jn3CMbJ omUAnjBx3Qk5XHGwWe7mfKgRyjPx8hbg =E9Xn -----END PGP SIGNATURE----- --+xNpyl7Qekk2NvDX-- From owner-freebsd-fs@FreeBSD.ORG Thu Feb 15 18:12:04 2007 Return-Path: X-Original-To: fs@freebsd.org Delivered-To: freebsd-fs@FreeBSD.ORG Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id A691D16A46F for ; Thu, 15 Feb 2007 18:12:04 +0000 (UTC) (envelope-from julian@elischer.org) Received: from outU.internet-mail-service.net (outU.internet-mail-service.net [216.240.47.244]) by mx1.freebsd.org (Postfix) with ESMTP id 1E32313C4BC for ; Thu, 15 Feb 2007 18:12:03 +0000 (UTC) (envelope-from julian@elischer.org) Received: from mx0.idiom.com (HELO idiom.com) (216.240.32.160) by out.internet-mail-service.net (qpsmtpd/0.32) with ESMTP; Thu, 15 Feb 2007 09:35:31 -0800 Received: from [192.168.2.5] (home.elischer.org [216.240.48.38]) by idiom.com (Postfix) with ESMTP id 018C8125B2B; Thu, 15 Feb 2007 09:58:59 -0800 (PST) Message-ID: <45D49F63.6090709@elischer.org> Date: Thu, 15 Feb 2007 09:58:59 -0800 From: Julian Elischer User-Agent: Thunderbird 1.5.0.9 (Macintosh/20061207) MIME-Version: 1.0 To: Josef Karthauser References: <20070204023711.GA3393@genius.tao.org.uk> <20070215135750.GR64768@obiwan.tataz.chchile.org> <20070215152259.GA2950@genius.tao.org.uk> In-Reply-To: <20070215152259.GA2950@genius.tao.org.uk> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: hackers@freebsd.org, Jeremie Le Hen , fs@freebsd.org Subject: Re: nullfs and named pipes. X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 15 Feb 2007 18:12:04 -0000 Josef Karthauser wrote: > On Thu, Feb 15, 2007 at 02:57:50PM +0100, Jeremie Le Hen wrote: >> Note that all processes within a jail can only intefere with processes >> from another jail or host as if they were on different machines. This >> means they can communicate through PF_INET for instance but not >> PF_LOCAL. >> > > You might think so! However that's not what's going on here. > Yes I tried to do this once before and failed.. I was trying to have the same named pipes available in two chroots (not jails) and it failed as you say. I would certainly like to have this fixed. Does anyone know if our nullfs rewrite is related at all to the\ dragonfly nullfs rewrite? They say lots of good things about the dragonfly rewrite.... > The named pipe/nullfs issue is nothing to do with jails. It's just > that nullfs is broken with respect to named pipes as I've previously > reported. However with this patch: > > cvs diff: Diffing . > Index: null_subr.c > =================================================================== > RCS file: /home/ncvs/src/sys/fs/nullfs/null_subr.c,v > retrieving revision 1.48.2.1 > diff -u -r1.48.2.1 null_subr.c > --- null_subr.c 13 Mar 2006 03:05:17 -0000 1.48.2.1 > +++ null_subr.c 14 Feb 2007 00:02:28 -0000 > @@ -235,6 +235,8 @@ > xp->null_vnode = vp; > xp->null_lowervp = lowervp; > vp->v_type = lowervp->v_type; > + if (vp->v_type == VSOCK || vp->v_type == VFIFO) > + vp->v_un = lowervp->v_un; > vp->v_data = xp; > vp->v_vnlock = lowervp->v_vnlock; > if (vp->v_vnlock == NULL) > > that problem goes away. Now a named pipe created on a lower layer > can be spoken to by a process connecting to it on a higher layer, > i.e (for demostration purposes only): > > # ls -ld /tmp/mysql.sock > srwxrwxrwx 1 mysql wheel 0 Jan 4 09:26 /tmp/mysql.sock > # mount_nullfs /tmp /mnt > # ls -ld /mnt/mysql.sock > srwxrwxrwx 1 mysql wheel 0 Jan 4 09:26 /mnt/mysql.sock > > With a stock kernel this fails: > > % mysql --socket=/mnt/mysql.sock > ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/mnt/mysql.sock' (61) > > but with the patch above it works: > > % mysql --socket=/mnt/mysql.sock > ERROR 1045 (28000): Access denied for user 'joe'@'localhost' (using password: NO) > > Of course the patch above doesn't work if the socket is created at > /mnt/mysql.sock and something wants to talk to it over at > /tmp/mysql.sock, however that is not really a problem. > > > So how does this relate to jails? > > The point of using nullfs is to make a PF_LOCAL socket appear local > even in the jail(!). Using the patch above this is indeed the case > and as far as the jail is concerned the socket is indeed local, > meaning that a process within a jail can talk via it to a process > on the host environment with no restrictions. This is crucially > important for mysql for instance as there is significant overhead > associated with PF_INET connections which can be avoided by talking > to PF_LOCAL sockets. this is something I need to be able to do. > >> IOW you have to think your jails as if theey were multiples boxes. >> You should therefore make them communicate with networking sockets and >> protect the latter with firewalling rules or tcpwrapper. > > Now in terms of protecting the host environment this is trivially done > by using a read-only nullfs mount: > > # mkdir /mysql > # mysqld_safe --socket=/mysql/mysql.sock & > > # mount_nullfs -oro /mysql /jail/mysql > > voila. The database can now be connected to within the jail environment > on /mysql/mysql.sock as a local fast connection, but as /mysql is > mounted read-only they cannot do anything other than connect to the > socket :). > yep > > Remember tools not policy :). > > Joe From owner-freebsd-fs@FreeBSD.ORG Thu Feb 15 20:40:29 2007 Return-Path: X-Original-To: freebsd-fs@freebsd.org Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 5CF9E16A401 for ; Thu, 15 Feb 2007 20:40:29 +0000 (UTC) (envelope-from wollman@khavrinen.csail.mit.edu) Received: from khavrinen.csail.mit.edu (khavrinen.csail.mit.edu [128.30.28.20]) by mx1.freebsd.org (Postfix) with ESMTP id 1DB8913C46B for ; Thu, 15 Feb 2007 20:40:28 +0000 (UTC) (envelope-from wollman@khavrinen.csail.mit.edu) Received: from khavrinen.csail.mit.edu (localhost.csail.mit.edu [127.0.0.1]) by khavrinen.csail.mit.edu (8.13.6/8.13.6) with ESMTP id l1FK8D8K033689 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=OK CN=khavrinen.csail.mit.edu issuer=Client+20CA); Thu, 15 Feb 2007 15:08:14 -0500 (EST) (envelope-from wollman@khavrinen.csail.mit.edu) Received: (from wollman@localhost) by khavrinen.csail.mit.edu (8.13.6/8.13.6/Submit) id l1FK8Cn0033686; Thu, 15 Feb 2007 15:08:12 -0500 (EST) (envelope-from wollman) From: Garrett Wollman MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Message-ID: <17876.48556.909232.678959@khavrinen.csail.mit.edu> Date: Thu, 15 Feb 2007 15:08:12 -0500 To: Tomas Olsson In-Reply-To: References: <6FC9F9894A9F8C49A722CF9F2132FC2204C9DAB6@ms05.mailstreet2003.net> <45D1F30A.6080403@freebsd.org> <20070213192906.U726@chrishome.localnet> <20070214162938.GA96725@keira.kiwi-computer.com> <20070214173211.L1054@chrishome.localnet> <20070214170808.GC96725@keira.kiwi-computer.com> <20070215044707.GA39168@deviant.kiev.zoral.com.ua> <20070215104537.GC39168@deviant.kiev.zoral.com.ua> X-Mailer: VM 7.17 under 21.4 (patch 20) "Double Solitaire" XEmacs Lucid X-Greylist: Sender DNS name whitelisted, not delayed by milter-greylist-3.0 (khavrinen.csail.mit.edu [127.0.0.1]); Thu, 15 Feb 2007 15:08:14 -0500 (EST) Cc: freebsd-fs@freebsd.org, arla-drinkers@stacken.kth.se Subject: Re: Arla on FreeBSD X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 15 Feb 2007 20:40:29 -0000 < said: > We currently open the cache files from nnpfs' VOPs, are there any risks > (deadlock?) involved if one passes an absolute path to vn_open() in such a > context? A possible compromise (although I don't think vn_open() can handle it) is to skip the naming layer altogether and just have nnpfs and arlad pass around file handles (of the getfh()/VFS_FHTOVP() variety). Then you don't have to worry about naming since you're just accessing the cache files by inode, and the kernel takes care of managing the vnodes for you. -GAWollman From owner-freebsd-fs@FreeBSD.ORG Thu Feb 15 23:21:34 2007 Return-Path: X-Original-To: freebsd-fs@freebsd.org Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id E9B0316A400 for ; Thu, 15 Feb 2007 23:21:34 +0000 (UTC) (envelope-from freebsd-fs@m.gmane.org) Received: from ciao.gmane.org (main.gmane.org [80.91.229.2]) by mx1.freebsd.org (Postfix) with ESMTP id 7987A13C471 for ; Thu, 15 Feb 2007 23:21:34 +0000 (UTC) (envelope-from freebsd-fs@m.gmane.org) Received: from list by ciao.gmane.org with local (Exim 4.43) id 1HHpuj-0002iR-5p for freebsd-fs@freebsd.org; Fri, 16 Feb 2007 00:21:21 +0100 Received: from 89-172-53-203.adsl.net.t-com.hr ([89.172.53.203]) by main.gmane.org with esmtp (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Fri, 16 Feb 2007 00:21:21 +0100 Received: from ivoras by 89-172-53-203.adsl.net.t-com.hr with local (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Fri, 16 Feb 2007 00:21:21 +0100 X-Injected-Via-Gmane: http://gmane.org/ To: freebsd-fs@freebsd.org From: Ivan Voras Date: Fri, 16 Feb 2007 00:20:55 +0100 Lines: 40 Message-ID: References: <45D1F30A.6080403@freebsd.org> <20070213192906.U726@chrishome.localnet> <20070214162938.GA96725@keira.kiwi-computer.com> <20070214173211.L1054@chrishome.localnet> <20070214170808.GC96725@keira.kiwi-computer.com> <20070215011402.A6280@chrishome.localnet> <20070215010017.GA9175@keira.kiwi-computer.com> <20070215020643.B6280@chrishome.localnet> <20070215012221.GA9607@keira.kiwi-computer.com> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="------------enigE34BEC2EE961718B4C0E2964" X-Complaints-To: usenet@sea.gmane.org X-Gmane-NNTP-Posting-Host: 89-172-53-203.adsl.net.t-com.hr User-Agent: Thunderbird 1.5.0.9 (Windows/20061207) In-Reply-To: <20070215012221.GA9607@keira.kiwi-computer.com> X-Enigmail-Version: 0.94.1.2 Sender: news Cc: arla-drinkers@stacken.kth.se Subject: Re: UFS2 with SAN X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 15 Feb 2007 23:21:35 -0000 This is an OpenPGP/MIME signed message (RFC 2440 and 3156) --------------enigE34BEC2EE961718B4C0E2964 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Rick C. Petty wrote: > I certainly hope someone can futz around with getting arla to work with= the > post-nerfed VFS changes in 6.x. Maybe that same person would be willin= g to > help me get HFS+ working too. I gave up on both these projects because= the > APIs kept shifting while I was trying to get them to work, and I just d= on't > have the time to relearn everything and familiarize myself with the new= > locking mechanisms. I propose we politely ask Pawel for some digested documentation about the new VFS world order, since he's the one whose hands are most dirty with it at the moment :) --------------enigE34BEC2EE961718B4C0E2964 Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.4 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFF1OrdldnAQVacBcgRAmzgAJ9QwbyJv+T8hT1aTCdjtpFZJTOHfACfcHby tnFdOhcT1PNb/ncQ85zNpVk= =bN5T -----END PGP SIGNATURE----- --------------enigE34BEC2EE961718B4C0E2964-- From owner-freebsd-fs@FreeBSD.ORG Fri Feb 16 10:31:16 2007 Return-Path: X-Original-To: fs@freebsd.org Delivered-To: freebsd-fs@FreeBSD.ORG Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id D3A1016A400; Fri, 16 Feb 2007 10:31:16 +0000 (UTC) (envelope-from tataz@tataz.chchile.org) Received: from smtp2-g19.free.fr (smtp2-g19.free.fr [212.27.42.28]) by mx1.freebsd.org (Postfix) with ESMTP id 9459A13C4B4; Fri, 16 Feb 2007 10:31:16 +0000 (UTC) (envelope-from tataz@tataz.chchile.org) Received: from tatooine.tataz.chchile.org (tataz.chchile.org [82.233.239.98]) by smtp2-g19.free.fr (Postfix) with ESMTP id 774877CCE; Fri, 16 Feb 2007 11:31:15 +0100 (CET) Received: from obiwan.tataz.chchile.org (unknown [192.168.1.25]) by tatooine.tataz.chchile.org (Postfix) with ESMTP id 0CDE39D41F; Fri, 16 Feb 2007 10:32:15 +0000 (UTC) Received: by obiwan.tataz.chchile.org (Postfix, from userid 1000) id ECF43405D; Fri, 16 Feb 2007 11:32:14 +0100 (CET) Date: Fri, 16 Feb 2007 11:32:14 +0100 From: Jeremie Le Hen To: Josef Karthauser Message-ID: <20070216103214.GW64768@obiwan.tataz.chchile.org> References: <20070204023711.GA3393@genius.tao.org.uk> <20070215135750.GR64768@obiwan.tataz.chchile.org> <20070215152259.GA2950@genius.tao.org.uk> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20070215152259.GA2950@genius.tao.org.uk> User-Agent: Mutt/1.5.13 (2006-08-11) Cc: hackers@freebsd.org, fs@freebsd.org Subject: Re: nullfs and named pipes. X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 16 Feb 2007 10:31:16 -0000 Josef, On Thu, Feb 15, 2007 at 03:22:59PM +0000, Josef Karthauser wrote: > On Thu, Feb 15, 2007 at 02:57:50PM +0100, Jeremie Le Hen wrote: > > > > Note that all processes within a jail can only intefere with processes > > from another jail or host as if they were on different machines. This > > means they can communicate through PF_INET for instance but not > > PF_LOCAL. > > > > [...] > > So how does this relate to jails? > > The point of using nullfs is to make a PF_LOCAL socket appear local > even in the jail(!). Using the patch above this is indeed the case > and as far as the jail is concerned the socket is indeed local, > meaning that a process within a jail can talk via it to a process > on the host environment with no restrictions. This is crucially > important for mysql for instance as there is significant overhead > associated with PF_INET connections which can be avoided by talking > to PF_LOCAL sockets. I was wrong, you are right. I was pretty sure the kernel retained the credentials of the listening process and that trying to connect to the latter using a process that has a mismatching jail ID would fail. On term #1: % jarjarbinks:~:103# nc -U -l /usr/space/chroot/tmp/mysock On term #2: % jarjarbinks:/usr/src:102# echo "I won't speak before testing" | jail /usr/space/chroot test 192.168.1.3 /usr/bin/nc -U /tmp/mysock On term #1! % I won't speak before testing Sorry for the noise. At least, I rekindled the thread :-). Regards, -- Jeremie Le Hen < jeremie at le-hen dot org >< ttz at chchile dot org > From owner-freebsd-fs@FreeBSD.ORG Fri Feb 16 13:25:50 2007 Return-Path: X-Original-To: fs@freebsd.org Delivered-To: freebsd-fs@FreeBSD.ORG Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 2110216A401; Fri, 16 Feb 2007 13:25:50 +0000 (UTC) (envelope-from rwatson@FreeBSD.org) Received: from cyrus.watson.org (cyrus.watson.org [209.31.154.42]) by mx1.freebsd.org (Postfix) with ESMTP id E2C7413C494; Fri, 16 Feb 2007 13:25:49 +0000 (UTC) (envelope-from rwatson@FreeBSD.org) Received: from fledge.watson.org (fledge.watson.org [209.31.154.41]) by cyrus.watson.org (Postfix) with ESMTP id 40A1D46CCA; Fri, 16 Feb 2007 07:54:39 -0500 (EST) Date: Fri, 16 Feb 2007 12:54:39 +0000 (GMT) From: Robert Watson X-X-Sender: robert@fledge.watson.org To: Kostik Belousov In-Reply-To: <20070215153135.GI39168@deviant.kiev.zoral.com.ua> Message-ID: <20070216125007.D38234@fledge.watson.org> References: <20070204023711.GA3393@genius.tao.org.uk> <20070215135750.GR64768@obiwan.tataz.chchile.org> <20070215152259.GA2950@genius.tao.org.uk> <20070215153135.GI39168@deviant.kiev.zoral.com.ua> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: Josef Karthauser , hackers@freebsd.org, Jeremie Le Hen , fs@freebsd.org Subject: Re: nullfs and named pipes. X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 16 Feb 2007 13:25:50 -0000 On Thu, 15 Feb 2007, Kostik Belousov wrote: > On Thu, Feb 15, 2007 at 03:22:59PM +0000, Josef Karthauser wrote: >> On Thu, Feb 15, 2007 at 02:57:50PM +0100, Jeremie Le Hen wrote: >>> >>> Note that all processes within a jail can only intefere with processes >>> from another jail or host as if they were on different machines. This >>> means they can communicate through PF_INET for instance but not PF_LOCAL. >> >> You might think so! However that's not what's going on here. >> >> The named pipe/nullfs issue is nothing to do with jails. It's just that >> nullfs is broken with respect to named pipes as I've previously reported. >> However with this patch: >> >> cvs diff: Diffing . >> Index: null_subr.c >> =================================================================== >> RCS file: /home/ncvs/src/sys/fs/nullfs/null_subr.c,v >> retrieving revision 1.48.2.1 >> diff -u -r1.48.2.1 null_subr.c >> --- null_subr.c 13 Mar 2006 03:05:17 -0000 1.48.2.1 >> +++ null_subr.c 14 Feb 2007 00:02:28 -0000 >> @@ -235,6 +235,8 @@ >> xp->null_vnode = vp; >> xp->null_lowervp = lowervp; >> vp->v_type = lowervp->v_type; >> + if (vp->v_type == VSOCK || vp->v_type == VFIFO) >> + vp->v_un = lowervp->v_un; > > I'm wondering is some reference counting needed there ? Yes, I find this a bit worrying also, but I don't know enough about how nullfs works to reason about it. What happens when a vnode in the bottom layer has its on-disk reference count drop to zero -- is the vnode in the top layer invalidated somehow? Robert N M Watson Computer Laboratory University of Cambridge From owner-freebsd-fs@FreeBSD.ORG Fri Feb 16 14:37:13 2007 Return-Path: X-Original-To: fs@freebsd.org Delivered-To: freebsd-fs@FreeBSD.ORG Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 3B95916A402; Fri, 16 Feb 2007 14:37:13 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from relay01.kiev.sovam.com (relay01.kiev.sovam.com [62.64.120.200]) by mx1.freebsd.org (Postfix) with ESMTP id BC70713C467; Fri, 16 Feb 2007 14:37:12 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from [212.82.216.227] (helo=fw.zoral.com.ua) by relay01.kiev.sovam.com with esmtps (TLSv1:AES256-SHA:256) (Exim 4.60) (envelope-from ) id 1HI4Ct-0004Fy-5m; Fri, 16 Feb 2007 16:37:11 +0200 Received: from deviant.kiev.zoral.com.ua (root@deviant.kiev.zoral.com.ua [10.1.1.148]) by fw.zoral.com.ua (8.13.4/8.13.4) with ESMTP id l1GEauKq043680 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Fri, 16 Feb 2007 16:36:56 +0200 (EET) (envelope-from kostikbel@gmail.com) Received: from deviant.kiev.zoral.com.ua (kostik@localhost [127.0.0.1]) by deviant.kiev.zoral.com.ua (8.13.8/8.13.8) with ESMTP id l1GEauVl019393; Fri, 16 Feb 2007 16:36:56 +0200 (EET) (envelope-from kostikbel@gmail.com) Received: (from kostik@localhost) by deviant.kiev.zoral.com.ua (8.13.8/8.13.8/Submit) id l1GEauSJ019392; Fri, 16 Feb 2007 16:36:56 +0200 (EET) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: deviant.kiev.zoral.com.ua: kostik set sender to kostikbel@gmail.com using -f Date: Fri, 16 Feb 2007 16:36:56 +0200 From: Kostik Belousov To: Robert Watson Message-ID: <20070216143656.GM39168@deviant.kiev.zoral.com.ua> References: <20070204023711.GA3393@genius.tao.org.uk> <20070215135750.GR64768@obiwan.tataz.chchile.org> <20070215152259.GA2950@genius.tao.org.uk> <20070215153135.GI39168@deviant.kiev.zoral.com.ua> <20070216125007.D38234@fledge.watson.org> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="NZtAI5QFBF0GmLcW" Content-Disposition: inline In-Reply-To: <20070216125007.D38234@fledge.watson.org> User-Agent: Mutt/1.4.2.2i X-Virus-Scanned: ClamAV version 0.88.7, clamav-milter version 0.88.7 on fw.zoral.com.ua X-Virus-Status: Clean X-Scanner-Signature: b11a0dd960b728cbee92fea3b4ac1d10 X-DrWeb-checked: yes X-SpamTest-Envelope-From: kostikbel@gmail.com X-SpamTest-Group-ID: 00000000 X-SpamTest-Info: Profiles 782 [Feb 16 2007] X-SpamTest-Info: helo_type=3 X-SpamTest-Info: {received from trusted relay: not dialup} X-SpamTest-Method: none X-SpamTest-Method: Local Lists X-SpamTest-Rate: 0 X-SpamTest-Status: Not detected X-SpamTest-Status-Extended: not_detected X-SpamTest-Version: SMTP-Filter Version 3.0.0 [0255], KAS30/Release Cc: Josef Karthauser , hackers@freebsd.org, Jeremie Le Hen , fs@freebsd.org Subject: Re: nullfs and named pipes. X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 16 Feb 2007 14:37:13 -0000 --NZtAI5QFBF0GmLcW Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Fri, Feb 16, 2007 at 12:54:39PM +0000, Robert Watson wrote: >=20 > On Thu, 15 Feb 2007, Kostik Belousov wrote: >=20 > >On Thu, Feb 15, 2007 at 03:22:59PM +0000, Josef Karthauser wrote: > >>On Thu, Feb 15, 2007 at 02:57:50PM +0100, Jeremie Le Hen wrote: > >>> > >>>Note that all processes within a jail can only intefere with processes= =20 > >>>from another jail or host as if they were on different machines. This= =20 > >>>means they can communicate through PF_INET for instance but not PF_LOC= AL. > >> > >>You might think so! However that's not what's going on here. > >> > >>The named pipe/nullfs issue is nothing to do with jails. It's just tha= t=20 > >>nullfs is broken with respect to named pipes as I've previously reporte= d.=20 > >>However with this patch: > >> > >> cvs diff: Diffing . > >> Index: null_subr.c > >> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > >> RCS file: /home/ncvs/src/sys/fs/nullfs/null_subr.c,v > >> retrieving revision 1.48.2.1 > >> diff -u -r1.48.2.1 null_subr.c > >> --- null_subr.c 13 Mar 2006 03:05:17 -0000 1.48.2.1 > >> +++ null_subr.c 14 Feb 2007 00:02:28 -0000 > >> @@ -235,6 +235,8 @@ > >> xp->null_vnode =3D vp; > >> xp->null_lowervp =3D lowervp; > >> vp->v_type =3D lowervp->v_type; > >> + if (vp->v_type =3D=3D VSOCK || vp->v_type =3D=3D VFIFO) > >> + vp->v_un =3D lowervp->v_un; > > > >I'm wondering is some reference counting needed there ? >=20 > Yes, I find this a bit worrying also, but I don't know enough about how= =20 > nullfs works to reason about it. What happens when a vnode in the bottom= =20 > layer has its on-disk reference count drop to zero -- is the vnode in the= =20 > top layer invalidated somehow? Vnode reclamation from lower layer cannot do anithing for corresponding nul= lfs vnode, but that vnode has reference from nullfs vnode. On the other hand, can forced unmount proceed for lower layer ? --NZtAI5QFBF0GmLcW Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (FreeBSD) iD8DBQFF1cGHC3+MBN1Mb4gRAmbnAJ4rZL3YjmoJjaBDku4KTaq8fBPSbQCcDAsr S6l/UoJLOLDXyGST6j7MVjM= =1gZf -----END PGP SIGNATURE----- --NZtAI5QFBF0GmLcW-- From owner-freebsd-fs@FreeBSD.ORG Sat Feb 17 05:56:40 2007 Return-Path: X-Original-To: freebsd-fs@freebsd.org Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 4B3E916A406 for ; Sat, 17 Feb 2007 05:56:40 +0000 (UTC) (envelope-from anderson@freebsd.org) Received: from mh1.centtech.com (moat3.centtech.com [64.129.166.50]) by mx1.freebsd.org (Postfix) with ESMTP id 25DD113C4A8 for ; Sat, 17 Feb 2007 05:56:40 +0000 (UTC) (envelope-from anderson@freebsd.org) Received: from [192.168.42.21] (andersonbox1.centtech.com [192.168.42.21]) by mh1.centtech.com (8.13.8/8.13.8) with ESMTP id l1H5ucnQ090945 for ; Fri, 16 Feb 2007 23:56:39 -0600 (CST) (envelope-from anderson@freebsd.org) Message-ID: <45D69918.3000008@freebsd.org> Date: Fri, 16 Feb 2007 23:56:40 -0600 From: Eric Anderson User-Agent: Thunderbird 1.5.0.9 (X11/20070204) MIME-Version: 1.0 To: "freebsd-fs@freebsd.org" Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Virus-Scanned: ClamAV 0.88.4/2586/Fri Feb 16 21:52:45 2007 on mh1.centtech.com X-Virus-Status: Clean X-Spam-Status: No, score=-2.6 required=8.0 tests=AWL,BAYES_00 autolearn=ham version=3.1.6 X-Spam-Checker-Version: SpamAssassin 3.1.6 (2006-10-03) on mh1.centtech.com Subject: question about vfs_lookup X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 17 Feb 2007 05:56:40 -0000 I'm just curious about how this block (starting at line 712) could possibly ever get executed, since right before it there is a condition that would cause a panic. Can anyone explain this please? In sys/kern/vfs_lookup.c: 710 KASSERT((cnp->cn_flags & ISLASTCN) || *ndp->ni_next == '/', 711 ("lookup: invalid path state.")); 712 if (*ndp->ni_next == '/') { 713 cnp->cn_nameptr = ndp->ni_next; 714 while (*cnp->cn_nameptr == '/') { 715 cnp->cn_nameptr++; 716 ndp->ni_pathlen--; 717 } 718 if (ndp->ni_dvp != dp) 719 vput(ndp->ni_dvp); 720 else 721 vrele(ndp->ni_dvp); 722 VFS_UNLOCK_GIANT(dvfslocked); 723 dvfslocked = vfslocked; /* dp becomes dvp in dirloop */ 724 vfslocked = 0; 725 goto dirloop; 726 } Thanks! Eric From owner-freebsd-fs@FreeBSD.ORG Sat Feb 17 19:53:17 2007 Return-Path: X-Original-To: freebsd-fs@freebsd.org Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 653DD16A402 for ; Sat, 17 Feb 2007 19:53:17 +0000 (UTC) (envelope-from adamsch1@yahoo.com) Received: from web31809.mail.mud.yahoo.com (web31809.mail.mud.yahoo.com [68.142.207.72]) by mx1.freebsd.org (Postfix) with SMTP id 1321513C4A3 for ; Sat, 17 Feb 2007 19:53:16 +0000 (UTC) (envelope-from adamsch1@yahoo.com) Received: (qmail 80987 invoked by uid 60001); 17 Feb 2007 19:26:35 -0000 DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=s1024; d=yahoo.com; h=X-YMail-OSG:Received:X-Mailer:Date:From:Subject:To:MIME-Version:Content-Type:Content-Transfer-Encoding:Message-ID; b=XSDAYzp2fZW3ZUM5XnT6rW395AUYoZbJCB9zBUkMGpkTLcqP4xYdniOfV3tEJCrvZCKEw0PsJtPe2hteoshsU8UOLy2qY8a6sfZ+O056cpvrOBkYaLby7fXQPCM1Rvx0jhhkHpB9I3eFN/4ClvLokvVTf542/20ZoesdP29+TMM=; X-YMail-OSG: 8sbudvcVM1lIsq_lGSncx8W19auHbMYpk7xFuwCcYzLrUcxEnPIrP3WDO5Aqe7zvgQ-- Received: from [69.236.88.234] by web31809.mail.mud.yahoo.com via HTTP; Sat, 17 Feb 2007 11:26:35 PST X-Mailer: YahooMailRC/463 YahooMailWebService/0.6.132.8 Date: Sat, 17 Feb 2007 11:26:35 -0800 (PST) From: Shane Adams To: "freebsd-fs@freebsd.org" MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: quoted-printable Message-ID: <401753.71633.qm@web31809.mail.mud.yahoo.com> Subject: Advice on runtime directories ala .snapshot X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 17 Feb 2007 19:53:17 -0000 Hi,=0A=0AI've been thinking about how I could add a "virtual" directory ent= ry similar to netapp, namely a .snapshot. Obviously I'd want to do this at= runtime, but I'm having trouble attacking this problem.=0A=0AI've been loo= king at the ufs_lookup routines, seems thats the only place to tackle such = a feature? Or possibly inject a .snapshot entry as the last entry read in = a call to ufs_readdir ?=0A=0AI believe doing a ls -la on a netapp will not = return the .snapshot directory, only explicitly nameing the directory will = achieve the effects you want. This would be somewhere beneith namei ultima= tly in ufs_lookup?=0A=0ASorry I'm a bit rambling!=0A=0ACheers,=0A Shane=0A= =0A=0A From owner-freebsd-fs@FreeBSD.ORG Sat Feb 17 20:11:52 2007 Return-Path: X-Original-To: freebsd-fs@freebsd.org Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 53D0816A407 for ; Sat, 17 Feb 2007 20:11:52 +0000 (UTC) (envelope-from freebsd-fs@m.gmane.org) Received: from ciao.gmane.org (main.gmane.org [80.91.229.2]) by mx1.freebsd.org (Postfix) with ESMTP id 0DB8813C48E for ; Sat, 17 Feb 2007 20:11:51 +0000 (UTC) (envelope-from freebsd-fs@m.gmane.org) Received: from list by ciao.gmane.org with local (Exim 4.43) id 1HIVuJ-00079n-Ux for freebsd-fs@freebsd.org; Sat, 17 Feb 2007 21:11:43 +0100 Received: from 89-172-50-63.adsl.net.t-com.hr ([89.172.50.63]) by main.gmane.org with esmtp (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Sat, 17 Feb 2007 21:11:43 +0100 Received: from ivoras by 89-172-50-63.adsl.net.t-com.hr with local (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Sat, 17 Feb 2007 21:11:43 +0100 X-Injected-Via-Gmane: http://gmane.org/ To: freebsd-fs@freebsd.org From: Ivan Voras Date: Sat, 17 Feb 2007 21:11:22 +0100 Lines: 29 Message-ID: References: <401753.71633.qm@web31809.mail.mud.yahoo.com> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="------------enig6F64D38D7CCC35DF5EDC2008" X-Complaints-To: usenet@sea.gmane.org X-Gmane-NNTP-Posting-Host: 89-172-50-63.adsl.net.t-com.hr User-Agent: Thunderbird 1.5.0.9 (Windows/20061207) In-Reply-To: <401753.71633.qm@web31809.mail.mud.yahoo.com> X-Enigmail-Version: 0.94.1.2 Sender: news Subject: Re: Advice on runtime directories ala .snapshot X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 17 Feb 2007 20:11:52 -0000 This is an OpenPGP/MIME signed message (RFC 2440 and 3156) --------------enig6F64D38D7CCC35DF5EDC2008 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Shane Adams wrote: > I believe doing a ls -la on a netapp will not return the .snapshot dire= ctory, only explicitly nameing the directory will achieve the effects you= want. This would be somewhere beneith namei ultimatly in ufs_lookup? Wouldn't it be easier to modify ls not to show it? :) --------------enig6F64D38D7CCC35DF5EDC2008 Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.4 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFF12FvldnAQVacBcgRAnZ9AKD1e+NSw2JdoIN81iVhvi2BUZY9PACfQ6ns 9uVPbXJ/mkcsMnQshSCbbzI= =unqh -----END PGP SIGNATURE----- --------------enig6F64D38D7CCC35DF5EDC2008-- From owner-freebsd-fs@FreeBSD.ORG Sat Feb 17 21:04:43 2007 Return-Path: X-Original-To: freebsd-fs@freebsd.org Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id D709A16A406; Sat, 17 Feb 2007 21:04:43 +0000 (UTC) (envelope-from pjd@garage.freebsd.pl) Received: from mail.garage.freebsd.pl (arm132.internetdsl.tpnet.pl [83.17.198.132]) by mx1.freebsd.org (Postfix) with ESMTP id 73DF513C4A7; Sat, 17 Feb 2007 21:04:43 +0000 (UTC) (envelope-from pjd@garage.freebsd.pl) Received: by mail.garage.freebsd.pl (Postfix, from userid 65534) id 391B9456B1; Sat, 17 Feb 2007 22:04:42 +0100 (CET) Received: from localhost (154.81.datacomsa.pl [195.34.81.154]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mail.garage.freebsd.pl (Postfix) with ESMTP id 2EE2545681; Sat, 17 Feb 2007 22:04:36 +0100 (CET) Date: Sat, 17 Feb 2007 22:03:22 +0100 From: Pawel Jakub Dawidek To: Eric Anderson Message-ID: <20070217210322.GB64936@garage.freebsd.pl> References: <45D69918.3000008@freebsd.org> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="61jdw2sOBCFtR2d/" Content-Disposition: inline In-Reply-To: <45D69918.3000008@freebsd.org> X-PGP-Key-URL: http://people.freebsd.org/~pjd/pjd.asc X-OS: FreeBSD 7.0-CURRENT i386 User-Agent: mutt-ng/devel-r804 (FreeBSD) X-Spam-Checker-Version: SpamAssassin 3.0.4 (2005-06-05) on mail.garage.freebsd.pl X-Spam-Level: X-Spam-Status: No, score=-2.6 required=3.0 tests=BAYES_00 autolearn=ham version=3.0.4 Cc: "freebsd-fs@freebsd.org" Subject: Re: question about vfs_lookup X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 17 Feb 2007 21:04:43 -0000 --61jdw2sOBCFtR2d/ Content-Type: text/plain; charset=iso-8859-2 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Fri, Feb 16, 2007 at 11:56:40PM -0600, Eric Anderson wrote: > I'm just curious about how this block (starting at line 712) could possib= ly ever get executed, since right before it there is a condition that would= cause a panic. >=20 > Can anyone explain this please? You read KASSERT(9) in a wrong way. The condition used in assertion says "don't panic if the condition is true". Here we panic when flag ISLASTCN is not set and *ndp->ni_next is not equal to '/'. > In sys/kern/vfs_lookup.c: > 710 KASSERT((cnp->cn_flags & ISLASTCN) || *ndp->ni_next =3D=3D '/= ', > 711 ("lookup: invalid path state.")); > 712 if (*ndp->ni_next =3D=3D '/') { > 713 cnp->cn_nameptr =3D ndp->ni_next; > 714 while (*cnp->cn_nameptr =3D=3D '/') { > 715 cnp->cn_nameptr++; > 716 ndp->ni_pathlen--; > 717 } > 718 if (ndp->ni_dvp !=3D dp) > 719 vput(ndp->ni_dvp); > 720 else > 721 vrele(ndp->ni_dvp); > 722 VFS_UNLOCK_GIANT(dvfslocked); > 723 dvfslocked =3D vfslocked; /* dp becomes dvp in dirloo= p */ > 724 vfslocked =3D 0; > 725 goto dirloop; > 726 } --=20 Pawel Jakub Dawidek http://www.wheel.pl pjd@FreeBSD.org http://www.FreeBSD.org FreeBSD committer Am I Evil? Yes, I Am! --61jdw2sOBCFtR2d/ Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (FreeBSD) iD8DBQFF122aForvXbEpPzQRAuPXAJ0TObEF1Cb23L+EbDSzYIuVZTriFgCfWZMj Ks8fpKFxdmwyNrj+g+iYDTE= =FHlZ -----END PGP SIGNATURE----- --61jdw2sOBCFtR2d/--