From owner-freebsd-cluster Mon Dec 9 3:52:23 2002 Delivered-To: freebsd-cluster@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 13BF137B401 for ; Mon, 9 Dec 2002 03:52:21 -0800 (PST) Received: from gate.nentec.de (gate2.nentec.de [194.25.215.66]) by mx1.FreeBSD.org (Postfix) with ESMTP id 71A7B43EA9 for ; Mon, 9 Dec 2002 03:52:19 -0800 (PST) (envelope-from sporner@nentec.de) Received: from nenny.nentec.de (root@nenny.nentec.de [153.92.64.1]) by gate.nentec.de (8.11.3/8.9.3) with ESMTP id gB9BqGP17528; Mon, 9 Dec 2002 12:52:16 +0100 Received: from nentec.de (andromeda.nentec.de [153.92.64.34]) by nenny.nentec.de (8.11.3/8.11.3) with ESMTP id gB9BqAt13765; Mon, 9 Dec 2002 12:52:11 +0100 Message-ID: <3DF483EA.3040207@nentec.de> Date: Mon, 09 Dec 2002 12:52:10 +0100 From: Andy Sporner User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.2a) Gecko/20020910 X-Accept-Language: en-us, en MIME-Version: 1.0 To: Julio Cesar Pinto , freebsd-cluster Subject: Re: How to set cluster References: <1039183737.3225.9.camel@marley.ifx.net.co> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Virus-Scanned: by AMaViS-perl11-milter (http://amavis.org/) Sender: owner-freebsd-cluster@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG Hi, For high availability you can use my tools. There are a few people out there that have told me that they are using it and it works well (though NetBSD users). For load balancing, I think there are some other solutions. I have heard of somebody using the NAT in the kernel to do this with some code. Perhaps this would be a good thing for the cluster software I offer to do. It will be needed later anyways. For now I suggest you look at some of the other offerings. Regards Andy PS. FYI to the list at large. A big release is coming at the start of the year including more documentation. Julio Cesar Pinto wrote: >Hi, > >I'm looking. >High-Avability and Load-Balanced. > >The idea is run MRTG for consultations SNMP, it's over PERL. > >What do your think about this. > >Thanks, > >On Fri, 2002-12-06 at 07:17, Marc Perisa wrote: > > >>Julio Cesar Pinto wrote: >> >> >>>Hello. >>> >>>I.m very new in this subject. Can somebody send me a URL with >>>information about how to setting cluster in free. >>> >>>Thanks for yor information. >>> >>>Regards, >>> >>> >>Hi Julio, >> >>which kind of cluster you are looking for? >> >>High-Avability? >>Load-Balanced? >>Number-Cruncher? >>Transparent Computing (Like LOCUS) ? >> >>And what applications do you want to use? >> >>Without that I can not hint you to any URL - because there are major >>differences in the toolchains available. >> >>Hope that helps >> >>Marc >> >> >> >> >> To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-cluster" in the body of the message From owner-freebsd-cluster Mon Dec 9 3:53:26 2002 Delivered-To: freebsd-cluster@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 6BBF437B401 for ; Mon, 9 Dec 2002 03:53:26 -0800 (PST) Received: from gate.nentec.de (gate2.nentec.de [194.25.215.66]) by mx1.FreeBSD.org (Postfix) with ESMTP id 3736143EA9 for ; Mon, 9 Dec 2002 03:53:24 -0800 (PST) (envelope-from sporner@nentec.de) Received: from nenny.nentec.de (root@nenny.nentec.de [153.92.64.1]) by gate.nentec.de (8.11.3/8.9.3) with ESMTP id gB9BrMP17544 for ; Mon, 9 Dec 2002 12:53:22 +0100 Received: from nentec.de (andromeda.nentec.de [153.92.64.34]) by nenny.nentec.de (8.11.3/8.11.3) with ESMTP id gB9BrHt13831 for ; Mon, 9 Dec 2002 12:53:17 +0100 Message-ID: <3DF4842D.30707@nentec.de> Date: Mon, 09 Dec 2002 12:53:17 +0100 From: Andy Sporner User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.2a) Gecko/20020910 X-Accept-Language: en-us, en MIME-Version: 1.0 To: freebsd-cluster@FreeBSD.ORG Subject: Forgot the URL Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit X-Virus-Scanned: by AMaViS-perl11-milter (http://amavis.org/) Sender: owner-freebsd-cluster@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG http://www.sporner.com/bsdclusters To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-cluster" in the body of the message From owner-freebsd-cluster Mon Dec 9 15: 0:42 2002 Delivered-To: freebsd-cluster@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id D3E8837B401 for ; Mon, 9 Dec 2002 15:00:41 -0800 (PST) Received: from grant.org (grant.org [206.190.164.98]) by mx1.FreeBSD.org (Postfix) with ESMTP id 50DE043EC5 for ; Mon, 9 Dec 2002 15:00:41 -0800 (PST) (envelope-from mgrant@splat.grant.org) Received: from splat.grant.org (mgrant@splat.grant.org [213.39.2.177]) by grant.org (8.12.6/8.12.6) with ESMTP id gB9N0XVM059755 for ; Mon, 9 Dec 2002 18:00:34 -0500 (EST) (envelope-from mgrant@splat.grant.org) Received: (from mgrant@localhost) by splat.grant.org (8.11.6+Sun/8.11.6) id gB9Mw5l08130; Mon, 9 Dec 2002 23:58:05 +0100 (MET) Date: Mon, 9 Dec 2002 23:58:05 +0100 (MET) Message-Id: <200212092258.gB9Mw5l08130@splat.grant.org> From: Michael Grant To: freebsd-cluster@freebsd.org Subject: sharing files within a cluster Sender: owner-freebsd-cluster@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG I'm still stymied by the problem of how to share files across machines without having a third (and forth!) machine acting as a disk server. I looked into the Coda file system. It's beta code and fairly complicated to set up and it's unclear to me whether it will do what's necessary. Let's say I have 2 boxes sharing the load. Each box runs a mailer. A message could arrive at either box. How should these two mailers share a single user's mailbox? Similar problem with users who log in to their shell account, how can they get at their files from either box? Again, a similar problem with web stuff that accesses a database like an mldbm file. And for easier management, certain conf files are identical across the machines, when modified, the one on the other machine should also get modified. The password and aliases files comes to mind as examples, there are pleanty more. The more I think about this problem, the more I become convinced that there's definitely a need for some shared writable storage. But how? Michael Grant To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-cluster" in the body of the message From owner-freebsd-cluster Mon Dec 9 15:44:36 2002 Delivered-To: freebsd-cluster@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 56F7337B401 for ; Mon, 9 Dec 2002 15:44:34 -0800 (PST) Received: from gpo.schools.net.au (gpo.schools.net.au [203.2.135.1]) by mx1.FreeBSD.org (Postfix) with ESMTP id 5C0B843EA9 for ; Mon, 9 Dec 2002 15:44:33 -0800 (PST) (envelope-from joshua@shallow.net) Received: from yello.shallow.net (yello.shallow.net [203.18.243.120]) by gpo.schools.net.au (Postfix) with ESMTP id D912EAF5C9; Tue, 10 Dec 2002 10:44:30 +1100 (EST) Received: by yello.shallow.net (Postfix, from userid 1001) id B99E72A6B; Tue, 10 Dec 2002 10:44:29 +1100 (EST) Date: Tue, 10 Dec 2002 10:44:28 +1100 From: Joshua Goodall To: Michael Grant Cc: freebsd-cluster@freebsd.org Subject: Re: sharing files within a cluster Message-ID: <20021209234428.GE98967@roughtrade.net> References: <200212092258.gB9Mw5l08130@splat.grant.org> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <200212092258.gB9Mw5l08130@splat.grant.org> User-Agent: Mutt/1.5.1i Sender: owner-freebsd-cluster@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG On Mon, Dec 09, 2002 at 11:58:05PM +0100, Michael Grant wrote: > I'm still stymied by the problem of how to share files across > machines without having a third (and forth!) machine acting as a > disk server. Conceptually, if you have two machines, they could act as servers for each other. There are some nasty failure modes associated with doing this, epecially if you sharing via NFS. Coda offers a more disconnected consistency model that should alleviate that somewhat. It may be beta code, but Coda itself is in production use at some large sites. > Let's say I have 2 boxes sharing the load. Each box runs a mailer. A > message could arrive at either box. How should these two mailers > share a single user's mailbox? With shared NFS mounts, the obvious answer is: use Maildirs. They are multiple-writer-safe and NFS-safe. I suggest experimenting with Coda in this scenario; particularly try an "unplug" test during delivery. > Similar problem with users who log in to their shell account, how can > they get at their files from either box? NFS cross-mounts again will do it. > Again, a similar problem with web stuff that accesses a database like > an mldbm file. Here you have a problem with locking. In the absence of a distributed lock manager, shared databases are Hard. > And for easier management, certain conf files are identical across the > machines, when modified, the one on the other machine should also get > modified. The password and aliases files comes to mind as examples, > there are pleanty more. > > The more I think about this problem, the more I become convinced that > there's definitely a need for some shared writable storage. But how? You are wanting a single-system-image cluster. Not many *nix vendors have got this right (Tru64 quite highly rated in this regard) and none of the free Unices AFAIK. Joshua -- Joshua Goodall joshua@roughtrade.net "Your byte hit ratio is weak, old man" "If you cache me now, I will dump more core than you can possibly imagine" To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-cluster" in the body of the message From owner-freebsd-cluster Mon Dec 9 16:18:25 2002 Delivered-To: freebsd-cluster@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 276B637B401 for ; Mon, 9 Dec 2002 16:18:24 -0800 (PST) Received: from fubar.adept.org (fubar.adept.org [63.147.172.249]) by mx1.FreeBSD.org (Postfix) with ESMTP id CC5B243ED4 for ; Mon, 9 Dec 2002 16:18:23 -0800 (PST) (envelope-from mike@adept.org) Received: by fubar.adept.org (Postfix, from userid 1001) id 1858715247; Mon, 9 Dec 2002 16:15:19 -0800 (PST) Received: from localhost (localhost [127.0.0.1]) by fubar.adept.org (Postfix) with ESMTP id 160BB15213 for ; Mon, 9 Dec 2002 16:15:19 -0800 (PST) Date: Mon, 9 Dec 2002 16:15:19 -0800 (PST) From: Mike Hoskins To: freebsd-cluster@freebsd.org Subject: Re: sharing files within a cluster In-Reply-To: <20021209234428.GE98967@roughtrade.net> Message-ID: <20021209160251.E10322-100000@fubar.adept.org> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-freebsd-cluster@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG On Tue, 10 Dec 2002, Joshua Goodall wrote: > On Mon, Dec 09, 2002 at 11:58:05PM +0100, Michael Grant wrote: > > I'm still stymied by the problem of how to share files across > > machines without having a third (and forth!) machine acting as a > > disk server. > Conceptually, if you have two machines, they could act as servers > for each other. Some would argue that having a single fileserver, with RAID, and one set of RAID config files, etc. to maintain... Could be a good thing. Of course it depends on your requirements. Some complexity could be mitigated with configuration management. (And, agreeably, any significantly sized cluster will likely have working config mgmt.) However, both software and hardware RAID have came far, and with filers like Netapp's, keeping individual nodes as minimal as possible (flash disks?) may be a good thing in larger environments. (Thinking large amounts of data, ala snap-mirror for backups/DR.) > There are some nasty failure modes associated with doing this, > epecially if you sharing via NFS. Coda offers a more disconnected > consistency model that should alleviate that somewhat. It may be > beta code, but Coda itself is in production use at some large sites. Does anyone have non-NDA-protected sketches/diagrams/notes/etc. of working (or not!) FreeBSD clusters they'd care to share? I'm just starting to experiment myself, and it would be good to form a sort of roadmap of "what's worked" and "what hasn't." > You are wanting a single-system-image cluster. Not many *nix vendors > have got this right (Tru64 quite highly rated in this regard) and none > of the free Unices AFAIK. Sounds like we need to bribe some Tru64 folks. ;) -- Mike Hoskins This message is RFC 1855 compliant, mike@adept.org www.adept.org/~mike/pub/rfcs/rfc1855.html To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-cluster" in the body of the message From owner-freebsd-cluster Mon Dec 9 16:40:13 2002 Delivered-To: freebsd-cluster@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id D83E837B401 for ; Mon, 9 Dec 2002 16:40:12 -0800 (PST) Received: from vps.vitalit.com (vps.vitalit.com [64.105.194.69]) by mx1.FreeBSD.org (Postfix) with ESMTP id 33D8143ED8 for ; Mon, 9 Dec 2002 16:40:12 -0800 (PST) (envelope-from car@vitalit.com) Received: from LAPTOP (gso167-138-145.triad.rr.com [24.167.138.145]) by vps.vitalit.com (8.12.3/8.12.3) with SMTP id gBA0e5hQ032022 for ; Mon, 9 Dec 2002 19:40:06 -0500 (EST) (envelope-from car@vitalit.com) Message-ID: <006101c29fe4$a13fad60$0301000a@LAPTOP> From: "Rouzer, Charles A (Chuck)" To: References: <20021209160251.E10322-100000@fubar.adept.org> Subject: Re: sharing files within a cluster Date: Mon, 9 Dec 2002 19:39:26 -0500 MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2720.3000 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2600.0000 Sender: owner-freebsd-cluster@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG Shared SCSI would probably be the way to go in this situation. iSCSI (not sure of driver status) and vinum should enable software RAID mirroring of file systems over a network providing you fail-over capabilities without shared SCSI. > > You are wanting a single-system-image cluster. Not many *nix vendors > > have got this right (Tru64 quite highly rated in this regard) and none > > of the free Unices AFAIK. > > Sounds like we need to bribe some Tru64 folks. ;) Compaq/DEC/HP is porting some of their single-system-image clustering code to Linux. http://ssic-linux.sf.net We can only hope FreeBSD will offer single-system-image clustering one day, but who knows. To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-cluster" in the body of the message From owner-freebsd-cluster Mon Dec 9 20:11:47 2002 Delivered-To: freebsd-cluster@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id A17EA37B401 for ; Mon, 9 Dec 2002 20:11:45 -0800 (PST) Received: from spf1.us.outblaze.com (205-158-62-139.outblaze.com [205.158.62.139]) by mx1.FreeBSD.org (Postfix) with SMTP id 0C65E43EC5 for ; Mon, 9 Dec 2002 20:11:41 -0800 (PST) (envelope-from click46@operamail.com) Received: (qmail 10164 invoked from network); 9 Dec 2002 21:13:06 -0000 Received: from unknown (205.158.62.68) by 205-158-62-139.outblaze.com with QMQP; 9 Dec 2002 21:13:06 -0000 Received: (qmail 56114 invoked from network); 10 Dec 2002 03:30:29 -0000 Received: from unknown (HELO ws5-1.us4.outblaze.com) (205.158.62.131) by 205-158-62-153.outblaze.com with SMTP; 10 Dec 2002 03:30:29 -0000 Received: (qmail 4731 invoked by uid 1001); 10 Dec 2002 03:30:27 -0000 Message-ID: <20021210033027.4730.qmail@operamail.com> Content-Type: text/plain; charset="iso-8859-1" Content-Disposition: inline Content-Transfer-Encoding: 7bit MIME-Version: 1.0 X-Mailer: MIME-tools 5.41 (Entity 5.404) Received: from [159.115.213.190] by ws5-1.us4.outblaze.com with http for click46@operamail.com; Tue, 10 Dec 2002 11:30:27 +0800 From: "aaron g" To: "Rouzer, Charles A (Chuck)" Cc: freebsd-cluster@freebsd.org Date: Tue, 10 Dec 2002 11:30:27 +0800 Subject: Re: sharing files within a cluster X-Originating-Ip: 159.115.213.190 X-Originating-Server: ws5-1.us4.outblaze.com Sender: owner-freebsd-cluster@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG the iSCSI drivers, as far as I can tell, are non-existant. While there are one or two projects in the works, nothing is publicly available or usable. Both Cisco and Intel have iSCSI drivers only for linux. Not even Adaptec, the one that likes to tout its support of the BSD's, has an iSCSI driver for its hardware cards. someone knowledgable needs to get on this pronto; Fibre is such an expensive alternative :-\ -aarong ----- Original Message ----- From: "Rouzer, Charles A (Chuck)" Date: Mon, 9 Dec 2002 19:39:26 -0500 To: Subject: Re: sharing files within a cluster > > Shared SCSI would probably be the way to go in this situation. iSCSI > (not sure of driver status) and vinum should enable software RAID mirroring > of file systems over a network providing you fail-over capabilities without > shared SCSI. > > > > You are wanting a single-system-image cluster. Not many *nix vendors > > > have got this right (Tru64 quite highly rated in this regard) and none > > > of the free Unices AFAIK. > > > > Sounds like we need to bribe some Tru64 folks. ;) > > Compaq/DEC/HP is porting some of their single-system-image clustering > code to Linux. > http://ssic-linux.sf.net > We can only hope FreeBSD will offer single-system-image clustering one > day, but who knows. > > > > To Unsubscribe: send mail to majordomo@FreeBSD.org > with "unsubscribe freebsd-cluster" in the body of the message > -- _______________________________________________ Get your free email from http://mymail.operamail.com Powered by Outblaze To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-cluster" in the body of the message From owner-freebsd-cluster Tue Dec 10 0:48:58 2002 Delivered-To: freebsd-cluster@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 049C937B401 for ; Tue, 10 Dec 2002 00:48:57 -0800 (PST) Received: from grant.org (grant.org [206.190.164.98]) by mx1.FreeBSD.org (Postfix) with ESMTP id 700EE43EB2 for ; Tue, 10 Dec 2002 00:48:56 -0800 (PST) (envelope-from mgrant@splat.grant.org) Received: from splat.grant.org (mgrant@splat.grant.org [213.39.2.177]) by grant.org (8.12.6/8.12.6) with ESMTP id gBA8mhVM076119; Tue, 10 Dec 2002 03:48:44 -0500 (EST) (envelope-from mgrant@splat.grant.org) Received: (from mgrant@localhost) by splat.grant.org (8.11.6+Sun/8.11.6) id gBA8kBO08833; Tue, 10 Dec 2002 09:46:11 +0100 (MET) Date: Tue, 10 Dec 2002 09:46:11 +0100 (MET) Message-Id: <200212100846.gBA8kBO08833@splat.grant.org> From: Michael Grant To: joshua@roughtrade.net Subject: Re: sharing files within a cluster Cc: freebsd-cluster@freebsd.org Sender: owner-freebsd-cluster@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG Joshua Goodall wrote: > Conceptually, if you have two machines, they could act as servers > for each other. ... > NFS cross-mounts again will do I've never heard of this, could you please explain perhaps by example what you're talking about? To my knowledge, one cannot cross-mount with nfs. > With shared NFS mounts, the obvious answer is: use Maildirs. They > are multiple-writer-safe and NFS-safe. I suggest experimenting with > Coda in this scenario; particularly try an "unplug" test during > delivery. I think that with proper locking or rather I should say *consistent* locking, a single mbox file should work with both servers writing to the same file. Just need to use a type of locking that's going to work across coda or whatever, for example the type of locking that creates a .lock file or uses a lock manager. Creating a .lock file can be dangerous since it can get stuck and have to be deleted manually if the box which is locking that mail file goes down. Anyone know if when a machine goes down in coda if the it's style of locks are automatically released? Michael Grant To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-cluster" in the body of the message From owner-freebsd-cluster Tue Dec 10 0:56:38 2002 Delivered-To: freebsd-cluster@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id EF6FB37B401 for ; Tue, 10 Dec 2002 00:56:36 -0800 (PST) Received: from grant.org (grant.org [206.190.164.98]) by mx1.FreeBSD.org (Postfix) with ESMTP id 7AABC43E4A for ; Tue, 10 Dec 2002 00:56:36 -0800 (PST) (envelope-from mgrant@splat.grant.org) Received: from splat.grant.org (mgrant@splat.grant.org [213.39.2.177]) by grant.org (8.12.6/8.12.6) with ESMTP id gBA8uYVM076312; Tue, 10 Dec 2002 03:56:34 -0500 (EST) (envelope-from mgrant@splat.grant.org) Received: (from mgrant@localhost) by splat.grant.org (8.11.6+Sun/8.11.6) id gBA8s6508847; Tue, 10 Dec 2002 09:54:06 +0100 (MET) Date: Tue, 10 Dec 2002 09:54:06 +0100 (MET) Message-Id: <200212100854.gBA8s6508847@splat.grant.org> From: Michael Grant To: car@vitalit.com Cc: freebsd-cluster@FreeBSD.ORG Subject: Re: sharing fules within a cluster Sender: owner-freebsd-cluster@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG Rouzer, Charles A (Chuck) wrote: > Shared SCSI would probably be the way to go in this situation. iSCSI > (not sure of driver status) and vinum should enable software RAID mirroring > of file systems over a network providing you fail-over capabilities without > shared SCSI. I emailed the author of vinum, Greg Lehey, a few weeks about about this. He says that there are problems with this due to caching: ----Forwarded message---- Date: Wed, 13 Nov 2002 11:44:39 +1030 From: "Greg 'groggy' Lehey" Subject: Re: multi master vinum over a lan? On Tuesday, 12 November 2002 at 13:54:03 +0100, Michael Grant wrote: > I'm trying to add some redundency to my box by adding a second box > creating a sort of cluster. > > I would like /home to be available on both boxes and be in sync, i.e. > mirrored over a lan or via double-ended scsi. Is it possible to use > vinum to do this? > > +---------+ +---------+ >>| box 1 | | box 2 | >>| disk1a |---scsi-cable---| disk1b | >>| disk2a | | disk2b | > +---------+ +---------+ > > run vinum on box 1 to create a mirror using disk1a and disk2a. on box > 2, same thing, have a mirror between disk1a and disk2a. Then mount r/w > the vinum partition on box1 and box2 simultaneously. > > It seems dangerous to have 2 machines mount the same disk(s) read-write. > But maybe this is a solved problem. Not really. The problem is that each system caches disk data locally, and they don't know about each other. There are solutions, but not in the space we're talking about here. > If vinum can't cope with this, do you have any suggestions? That's what NFS is for. Greg To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-cluster" in the body of the message From owner-freebsd-cluster Tue Dec 10 1: 0: 3 2002 Delivered-To: freebsd-cluster@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 3950637B401 for ; Tue, 10 Dec 2002 01:00:01 -0800 (PST) Received: from gate.nentec.de (gate2.nentec.de [194.25.215.66]) by mx1.FreeBSD.org (Postfix) with ESMTP id 791B743EA9 for ; Tue, 10 Dec 2002 00:59:59 -0800 (PST) (envelope-from sporner@nentec.de) Received: from nenny.nentec.de (root@nenny.nentec.de [153.92.64.1]) by gate.nentec.de (8.11.3/8.9.3) with ESMTP id gBA8xvP20475 for ; Tue, 10 Dec 2002 09:59:57 +0100 Received: from nentec.de (andromeda.nentec.de [153.92.64.34]) by nenny.nentec.de (8.11.3/8.11.3) with ESMTP id gBA8xqt13988 for ; Tue, 10 Dec 2002 09:59:52 +0100 Message-ID: <3DF5AD08.4040105@nentec.de> Date: Tue, 10 Dec 2002 09:59:52 +0100 From: Andy Sporner User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.2a) Gecko/20020910 X-Accept-Language: en-us, en MIME-Version: 1.0 To: freebsd-cluster Subject: Re: sharing files within a cluster References: <20021209160251.E10322-100000@fubar.adept.org> <006101c29fe4$a13fad60$0301000a@LAPTOP> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Virus-Scanned: by AMaViS-perl11-milter (http://amavis.org/) Sender: owner-freebsd-cluster@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG Greetings, I have looking into the porting efforts of the project. They are very close to the beginning of developement, rather than near the end. I have said a few times that a goal of the development I am doing (with welcomes to any volunteers!) would be to this end. However I believe architecturally that the compaq solution is bloated and offers far too many opportunitys for disaster. I still regard the KISS principle as the best one. I think that there are *many* nice things within FreeBSD that would not be properly utilized if it was just a straight port. Not only that there are some significan pieces missing that I have already implemented in this code. There was once MOSIX and it is more on the model that I am looking at, with some dynamic features added. I am nearing (finally!) the completion of the architectural overview that I will be releasing around the first of the year to lay out a concrete plan on how to arrive at this point we all would like to see. Hopefully then there will be some volunteers that would want to help to this goal, otherwise people will still be knashing their teeth and wailing outside of the walls. Let's here from some volunteers now ;-) Andy Rouzer, Charles A (Chuck) wrote: > Shared SCSI would probably be the way to go in this situation. iSCSI >(not sure of driver status) and vinum should enable software RAID mirroring >of file systems over a network providing you fail-over capabilities without >shared SCSI. > > > >>>You are wanting a single-system-image cluster. Not many *nix vendors >>>have got this right (Tru64 quite highly rated in this regard) and none >>>of the free Unices AFAIK. >>> >>> >>Sounds like we need to bribe some Tru64 folks. ;) >> >> > > Compaq/DEC/HP is porting some of their single-system-image clustering >code to Linux. > http://ssic-linux.sf.net > We can only hope FreeBSD will offer single-system-image clustering one >day, but who knows. > > > >To Unsubscribe: send mail to majordomo@FreeBSD.org >with "unsubscribe freebsd-cluster" in the body of the message > > > > To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-cluster" in the body of the message From owner-freebsd-cluster Tue Dec 10 1: 2:55 2002 Delivered-To: freebsd-cluster@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 3944737B401 for ; Tue, 10 Dec 2002 01:02:54 -0800 (PST) Received: from gate.nentec.de (gate2.nentec.de [194.25.215.66]) by mx1.FreeBSD.org (Postfix) with ESMTP id 0851843E4A for ; Tue, 10 Dec 2002 01:02:53 -0800 (PST) (envelope-from sporner@nentec.de) Received: from nenny.nentec.de (root@nenny.nentec.de [153.92.64.1]) by gate.nentec.de (8.11.3/8.9.3) with ESMTP id gBA92oP24997 for ; Tue, 10 Dec 2002 10:02:50 +0100 Received: from nentec.de (andromeda.nentec.de [153.92.64.34]) by nenny.nentec.de (8.11.3/8.11.3) with ESMTP id gBA92jt14237 for ; Tue, 10 Dec 2002 10:02:45 +0100 Message-ID: <3DF5ADB5.9060908@nentec.de> Date: Tue, 10 Dec 2002 10:02:45 +0100 From: Andy Sporner User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.2a) Gecko/20020910 X-Accept-Language: en-us, en MIME-Version: 1.0 To: freebsd-cluster Subject: Re: sharing fules within a cluster References: <200212100854.gBA8s6508847@splat.grant.org> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Virus-Scanned: by AMaViS-perl11-milter (http://amavis.org/) Sender: owner-freebsd-cluster@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG Michael Grant wrote: >Rouzer, Charles A (Chuck) wrote: > > >> Shared SCSI would probably be the way to go in this situation. iSCSI >>(not sure of driver status) and vinum should enable software RAID mirroring >>of file systems over a network providing you fail-over capabilities without >>shared SCSI. >> >> > >I emailed the author of vinum, Greg Lehey, a few weeks about about >this. He says that there are problems with this due to caching: > > > >Not really. The problem is that each system caches disk data locally, >and they don't know about each other. There are solutions, but not in >the space we're talking about here. > > However in a true failover environment this problem would not exist. Check out what I have implemented. People are using it--it works! http://www.sporner.com/bsdclusters To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-cluster" in the body of the message From owner-freebsd-cluster Tue Dec 10 1:50: 1 2002 Delivered-To: freebsd-cluster@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 7772637B401 for ; Tue, 10 Dec 2002 01:50:00 -0800 (PST) Received: from gidgate.gid.co.uk (gid.co.uk [194.32.164.225]) by mx1.FreeBSD.org (Postfix) with ESMTP id 95C9843EBE for ; Tue, 10 Dec 2002 01:49:58 -0800 (PST) (envelope-from rb@gid.co.uk) Received: (from rb@localhost) by gidgate.gid.co.uk (8.11.6/8.11.6) id gBA9njx03512; Tue, 10 Dec 2002 09:49:45 GMT (envelope-from rb) Message-Id: <4.3.2.7.2.20021210094729.02055bd0@localhost> X-Sender: rbmail@localhost X-Mailer: QUALCOMM Windows Eudora Version 4.3.2 Date: Tue, 10 Dec 2002 09:49:38 +0000 To: Michael Grant From: Bob Bishop Subject: Re: sharing files within a cluster Cc: freebsd-cluster@FreeBSD.ORG In-Reply-To: <200212092258.gB9Mw5l08130@splat.grant.org> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; format=flowed Sender: owner-freebsd-cluster@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG At 22:58 9/12/02, Michael Grant wrote: >I'm still stymied by the problem of how to share files across >machines without having a third (and forth!) machine acting as a >disk server. For another alternative approach, have a look at http://www.complang.tuwien.ac.at/reisner/drbd/ Linux only at the moment, but lighter-weight than Coda and directly addresses this problem. -- Bob Bishop +44 (0)118 977 4017 rb@gid.co.uk fax +44 (0)118 989 4254 To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-cluster" in the body of the message From owner-freebsd-cluster Tue Dec 10 1:53:30 2002 Delivered-To: freebsd-cluster@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 0115437B401 for ; Tue, 10 Dec 2002 01:53:29 -0800 (PST) Received: from vps.vitalit.com (vps.vitalit.com [64.105.194.69]) by mx1.FreeBSD.org (Postfix) with ESMTP id 4378F43EA9 for ; Tue, 10 Dec 2002 01:53:28 -0800 (PST) (envelope-from car@vitalit.com) Received: from LAPTOP (gso167-138-145.triad.rr.com [24.167.138.145]) by vps.vitalit.com (8.12.3/8.12.3) with SMTP id gBA9rGhQ062372 for ; Tue, 10 Dec 2002 04:53:17 -0500 (EST) (envelope-from car@vitalit.com) Message-ID: <00c601c2a031$e527aee0$0301000a@LAPTOP> From: "Rouzer, Charles A (Chuck)" To: References: <200212100854.gBA8s6508847@splat.grant.org> Subject: Re: sharing fules within a cluster Date: Tue, 10 Dec 2002 04:52:46 -0500 MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2720.3000 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2600.0000 Sender: owner-freebsd-cluster@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG The configuration you provided is not like the configuration I discussed with Greg in regards to using Vinum to mirror between machines. He agreed this should work using Vinum as a software mirror for iSCSI and I assume local shared SCSI: HostA harddrive1 is mirrored with harddrive1 on HostB. HostB does not mount harddrive1 and start HostAs applications until HostA is determined down hard. HostB harddrive2 is mirrored with harddrive2 on HostA. HostA does not mount harddrive2 and start HostBs applications until HostB is determined down hard. At the moment I have not investigated mirror syncing and recovery procedures for this type of fail-over cluster. > On Tuesday, 12 November 2002 at 13:54:03 +0100, Michael Grant wrote: > > I'm trying to add some redundency to my box by adding a second box > > creating a sort of cluster. > > > > I would like /home to be available on both boxes and be in sync, i.e. > > mirrored over a lan or via double-ended scsi. Is it possible to use > > vinum to do this? > > > > +---------+ +---------+ > >>| box 1 | | box 2 | > >>| disk1a |---scsi-cable---| disk1b | > >>| disk2a | | disk2b | > > +---------+ +---------+ > > > > run vinum on box 1 to create a mirror using disk1a and disk2a. on box > > 2, same thing, have a mirror between disk1a and disk2a. Then mount r/w > > the vinum partition on box1 and box2 simultaneously. > > > > It seems dangerous to have 2 machines mount the same disk(s) read-write. > > But maybe this is a solved problem. > > Not really. The problem is that each system caches disk data locally, > and they don't know about each other. There are solutions, but not in > the space we're talking about here. > > > If vinum can't cope with this, do you have any suggestions? > > That's what NFS is for. > > Greg > > To Unsubscribe: send mail to majordomo@FreeBSD.org > with "unsubscribe freebsd-cluster" in the body of the message > To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-cluster" in the body of the message From owner-freebsd-cluster Tue Dec 10 5: 0:41 2002 Delivered-To: freebsd-cluster@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id F05D437B401 for ; Tue, 10 Dec 2002 05:00:40 -0800 (PST) Received: from grant.org (grant.org [206.190.164.98]) by mx1.FreeBSD.org (Postfix) with ESMTP id 5175F43E4A for ; Tue, 10 Dec 2002 05:00:40 -0800 (PST) (envelope-from mgrant@splat.grant.org) Received: from splat.grant.org (mgrant@splat.grant.org [213.39.2.177]) by grant.org (8.12.6/8.12.6) with ESMTP id gBAD0RVM081835; Tue, 10 Dec 2002 08:00:28 -0500 (EST) (envelope-from mgrant@splat.grant.org) Received: (from mgrant@localhost) by splat.grant.org (8.11.6+Sun/8.11.6) id gBACvv609153; Tue, 10 Dec 2002 13:57:57 +0100 (MET) Date: Tue, 10 Dec 2002 13:57:57 +0100 (MET) Message-Id: <200212101257.gBACvv609153@splat.grant.org> From: Michael Grant To: rb@gid.co.uk Subject: Re: sharing files within a cluster Cc: freebsd-cluster@FreeBSD.ORG Sender: owner-freebsd-cluster@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG Bob Bishop wrote: > For another alternative approach, have a look at > http://www.complang.tuwien.ac.at/reisner/drbd/ > Linux only at the moment, but lighter-weight than Coda and directly > addresses this problem. Good try, but no dice, this is a fail-over solution only. You can only mount the disk read-write on one of the boxes at a time. However, there's a link on the page you pointed me at to GFS the Global File System by Sistina: http://www.sistina.com/products_gfs.htm This seems to be similar to Coda but for Linux. I hate to say this but it does look like Linux has superior support for clustering. Michael Grant To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-cluster" in the body of the message From owner-freebsd-cluster Tue Dec 10 5:26:44 2002 Delivered-To: freebsd-cluster@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id D4C3837B401 for ; Tue, 10 Dec 2002 05:26:42 -0800 (PST) Received: from gate.nentec.de (gate2.nentec.de [194.25.215.66]) by mx1.FreeBSD.org (Postfix) with ESMTP id 14C8543EC2 for ; Tue, 10 Dec 2002 05:26:41 -0800 (PST) (envelope-from sporner@nentec.de) Received: from nenny.nentec.de (root@nenny.nentec.de [153.92.64.1]) by gate.nentec.de (8.11.3/8.9.3) with ESMTP id gBADQcP04700 for ; Tue, 10 Dec 2002 14:26:38 +0100 Received: from nentec.de (andromeda.nentec.de [153.92.64.34]) by nenny.nentec.de (8.11.3/8.11.3) with ESMTP id gBADQWt30394 for ; Tue, 10 Dec 2002 14:26:32 +0100 Message-ID: <3DF5EB88.9090409@nentec.de> Date: Tue, 10 Dec 2002 14:26:32 +0100 From: Andy Sporner User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.2a) Gecko/20020910 X-Accept-Language: en-us, en MIME-Version: 1.0 Cc: freebsd-cluster@FreeBSD.ORG Subject: Re: sharing files within a cluster References: <200212101257.gBACvv609153@splat.grant.org> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Virus-Scanned: by AMaViS-perl11-milter (http://amavis.org/) Sender: owner-freebsd-cluster@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG Michael Grant wrote: >Bob Bishop wrote: > > >>For another alternative approach, have a look at >>http://www.complang.tuwien.ac.at/reisner/drbd/ >>Linux only at the moment, but lighter-weight than Coda and directly >>addresses this problem. >> >> > >Good try, but no dice, this is a fail-over solution only. You can >only mount the disk read-write on one of the boxes at a time. > >However, there's a link on the page you pointed me at to GFS the >Global File System by Sistina: > >http://www.sistina.com/products_gfs.htm > >This seems to be similar to Coda but for Linux. I hate to say this >but it does look like Linux has superior support for clustering. > >Michael Grant > >To Unsubscribe: send mail to majordomo@FreeBSD.org >with "unsubscribe freebsd-cluster" in the body of the message > > > > However, what's wrong with this if you also failover NFS??? I wrote this to another recipient privately, but I put this here too as a possible scenario... I am also looking into the porting of the drbd code that Bob Bishop mentioned earlier. I am not a real "Fan" of the idea of shared SCSI. I have seen it work and have worked on such systems, but it has some very significant drawbacks. I like the idea of a network distributed collection of raw devices that can be brought together in a VINUM kind of way so that redundancy is assured. I have no problem with NFS as long as the underlying backing store is redundant. Then the NFS server can fail to anywhere in the cluster. Take this example. Node 1, 2, 3 and 4 have a 2 GB slice of raw space that is network exported (via DRBD). These slices are present on all of the cluster nodes via the network interface. (I don't know Vinum terms so I use Sequent ones (varitas)). We create a subdisk on all or part of these raw slices. Now we create two plexes by concatenating the subdisks on nodes 1 and 3. We create a corresponding plex on nodes 2 and 4. Now we create a mirrored volume by attaching both plexes. Now Node 1 becomes the NFS server. He has half of the volume local and the other half remote. He can serve any file to any other node in the cluster and assure that the locking paradymns work correctly. If he should die, than any of the other nodes can recover the NFS server and continue to operate. For instance if Node 2 takes it over it has the mirrored half of what was on node 1 and the other half is available over the network. The NFS server than is configured on a virtual IP address that fails around the cluster. With a journaled filesystem underlying the volume, it is coherent if not all of the mirroring was completed when node 1 took the fall. To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-cluster" in the body of the message From owner-freebsd-cluster Tue Dec 10 7: 8:38 2002 Delivered-To: freebsd-cluster@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 88C7A37B401 for ; Tue, 10 Dec 2002 07:08:36 -0800 (PST) Received: from heaven.gigo.com (heaven.gigo.com [64.57.102.22]) by mx1.FreeBSD.org (Postfix) with ESMTP id DFA7043EC5 for ; Tue, 10 Dec 2002 07:08:35 -0800 (PST) (envelope-from lioux@brturbo.com) Received: from 200-193-224-224-bsace7003.dsl.telebrasilia.net.br (200-193-224-224-bsace7003.dsl.telebrasilia.net.br [200.193.224.224]) by heaven.gigo.com (Postfix) with ESMTP id 2F786B916 for ; Tue, 10 Dec 2002 07:08:23 -0800 (PST) Received: (qmail 20976 invoked by uid 1001); 10 Dec 2002 14:56:15 -0000 Message-ID: <20021210145615.20975.qmail@exxodus.fedaykin.here> Date: Tue, 10 Dec 2002 12:55:53 -0200 From: Mario Sergio Fujikawa Ferreira To: Andy Sporner Cc: freebsd-cluster@FreeBSD.ORG Subject: Re: sharing files within a cluster References: <200212101257.gBACvv609153@splat.grant.org> <3DF5EB88.9090409@nentec.de> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <3DF5EB88.9090409@nentec.de> User-Agent: Mutt/1.4i X-Operating-System: FreeBSD 4.7-STABLE X-Disclaimer: I hope you find what you are looking for... in life :) Sender: owner-freebsd-cluster@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG On Tue, Dec 10, 2002 at 02:26:10PM +0100, Andy Sporner wrote: > Michael Grant wrote: > > >Bob Bishop wrote: > > However, what's wrong with this if you also failover NFS??? I wrote > this to another > recipient privately, but I put this here too as a possible scenario... > > I am also looking into the porting of the drbd code that Bob Bishop > mentioned earlier. I am not a real "Fan" of the idea of shared SCSI. > I have seen it work and have worked on such systems, but it has some > very significant drawbacks. I like the idea of a network distributed > collection of raw devices that can be brought together in a VINUM > kind of way so that redundancy is assured. I have no problem with > NFS as long as the underlying backing store is redundant. Then the > NFS server can fail to anywhere in the cluster. Take this example. I might be saying something foolish but have any of you investigated GEOM? It might be possible to write an abstraction layer for it that will enable some of the necessary features: - localization transparency - distribution - disconnected mode - blabla... unfortunaly, FreeBSD based only as GEOM is only supported for FreeBSD for now - GEOM's Author Page http://www.freebsd.org/~phk/Geom/ - Status Report on GEOM http://www.freebsd.org/news/status/report-sept-2002-oct-2002.html#GEOM---generalized-block-storage-manipulation - FreeBSD GEOM man page http://www.FreeBSD.org/cgi/man.cgi?query=geom&apropos=0&sektion=0&manpath=FreeBSD+5.0-current&format=html Regards, -- Mario S F Ferreira - DF - Brazil - "I guess this is a signature." Computer Science Undergraduate | FreeBSD Committer | CS Developer flames to beloved devnull@someotherworldbeloworabove.org feature, n: a documented bug | bug, n: an undocumented feature To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-cluster" in the body of the message From owner-freebsd-cluster Tue Dec 10 7:25:42 2002 Delivered-To: freebsd-cluster@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id E276B37B401; Tue, 10 Dec 2002 07:25:40 -0800 (PST) Received: from gate.nentec.de (gate2.nentec.de [194.25.215.66]) by mx1.FreeBSD.org (Postfix) with ESMTP id 3EEF043E4A; Tue, 10 Dec 2002 07:25:37 -0800 (PST) (envelope-from sporner@nentec.de) Received: from nenny.nentec.de (root@nenny.nentec.de [153.92.64.1]) by gate.nentec.de (8.11.3/8.9.3) with ESMTP id gBAFPYP26641; Tue, 10 Dec 2002 16:25:34 +0100 Received: from nentec.de (andromeda.nentec.de [153.92.64.34]) by nenny.nentec.de (8.11.3/8.11.3) with ESMTP id gBAFPSt08605; Tue, 10 Dec 2002 16:25:28 +0100 Message-ID: <3DF60768.7040407@nentec.de> Date: Tue, 10 Dec 2002 16:25:28 +0100 From: Andy Sporner User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.2a) Gecko/20020910 X-Accept-Language: en-us, en MIME-Version: 1.0 To: Mario Sergio Fujikawa Ferreira Cc: freebsd-cluster@FreeBSD.org Subject: Re: sharing files within a cluster References: <200212101257.gBACvv609153@splat.grant.org> <3DF5EB88.9090409@nentec.de> <20021210145615.20975.qmail@exxodus.fedaykin.here> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Virus-Scanned: by AMaViS-perl11-milter (http://amavis.org/) Sender: owner-freebsd-cluster@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG Hi, Mario Sergio Fujikawa Ferreira wrote: > I might be saying something foolish but have any of you investigated >GEOM? It might be possible to write an abstraction layer for it that will >enable some of the necessary features: > > - localization transparency > - distribution > - disconnected mode > - blabla... unfortunaly, FreeBSD based only as GEOM is only > supported for FreeBSD for now > > This is the main 'gotcha'. The secondary one is that this guy has never replied to *any* of my emails and with all the difficulties related to time, I simply don't have any more to deal with that as well... :-( I am trying to make this a *bsd neutral environment. This way everybody wins. But thanks for your opinion... Andy To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-cluster" in the body of the message From owner-freebsd-cluster Tue Dec 10 7:40:22 2002 Delivered-To: freebsd-cluster@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 57CDF37B401 for ; Tue, 10 Dec 2002 07:40:21 -0800 (PST) Received: from mailrelay2.lanl.gov (mailrelay2.lanl.gov [128.165.4.103]) by mx1.FreeBSD.org (Postfix) with ESMTP id 8522A43EB2 for ; Tue, 10 Dec 2002 07:40:20 -0800 (PST) (envelope-from rminnich@lanl.gov) Received: from ccs.lanl.gov (localhost.localdomain [127.0.0.1]) by mailrelay2.lanl.gov (8.12.3/8.12.3/(ccn-5)) with SMTP id gBAFPggp000820 for ; Tue, 10 Dec 2002 08:25:43 -0700 Received: (qmail 778 invoked from network); 10 Dec 2002 08:25:42 -0700 Received: from unknown (HELO carotid.ccs.lanl.gov) (128.165.148.162) by 128.165.148.1 with SMTP; 10 Dec 2002 08:25:42 -0700 Received: (qmail 30717 invoked by uid 3499); 10 Dec 2002 08:25:42 -0700 Received: from localhost (sendmail-bs@127.0.0.1) by localhost with SMTP; 10 Dec 2002 08:25:42 -0700 Date: Tue, 10 Dec 2002 08:25:42 -0700 (MST) From: "Ronald G. Minnich" X-X-Sender: rminnich@carotid.ccs.lanl.gov To: Michael Grant Cc: joshua@roughtrade.net, Subject: Re: sharing files within a cluster In-Reply-To: <200212100846.gBA8kBO08833@splat.grant.org> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-freebsd-cluster@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG On Tue, 10 Dec 2002, Michael Grant wrote: > > NFS cross-mounts again will do no they won't. Old idea, bad idea, don't do it. NFS is not robust enough for this. ron > what you're talking about? To my knowledge, one cannot cross-mount > with nfs. oh yeah you can. It's just very bad to do this. ron To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-cluster" in the body of the message From owner-freebsd-cluster Tue Dec 10 8:29:32 2002 Delivered-To: freebsd-cluster@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id CE91637B401 for ; Tue, 10 Dec 2002 08:29:30 -0800 (PST) Received: from mgr3.xmission.com (mgr3.xmission.com [198.60.22.203]) by mx1.FreeBSD.org (Postfix) with ESMTP id 8FF6143EC5 for ; Tue, 10 Dec 2002 08:29:29 -0800 (PST) (envelope-from glewis@eyesbeyond.com) Received: from mail by mgr3.xmission.com with spam-scanned (Exim 3.35 #1) id 18LnG7-0006k0-03; Tue, 10 Dec 2002 09:29:24 -0700 Received: from [207.135.128.145] (helo=misty.eyesbeyond.com) by mgr3.xmission.com with esmtp (Exim 3.35 #1) id 18LnG6-0006iz-03; Tue, 10 Dec 2002 09:29:23 -0700 Received: (from glewis@localhost) by misty.eyesbeyond.com (8.11.6/8.11.6) id gBAGTHl09131; Wed, 11 Dec 2002 02:59:17 +1030 (CST) (envelope-from glewis@eyesbeyond.com) X-Authentication-Warning: misty.eyesbeyond.com: glewis set sender to glewis@eyesbeyond.com using -f Date: Wed, 11 Dec 2002 02:59:17 +1030 From: Greg Lewis To: Michael Grant Cc: rb@gid.co.uk, freebsd-cluster@FreeBSD.ORG Subject: Re: sharing files within a cluster Message-ID: <20021211025917.A9059@misty.eyesbeyond.com> References: <200212101257.gBACvv609153@splat.grant.org> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.2.5i In-Reply-To: <200212101257.gBACvv609153@splat.grant.org>; from mg-fbsd3@grant.org on Tue, Dec 10, 2002 at 01:57:57PM +0100 X-Spam-Status: No, hits=-2.1 required=8.0 tests=IN_REP_TO,PORN_4,QUOTED_EMAIL_TEXT,REFERENCES, SIGNATURE_SHORT_DENSE,SPAM_PHRASE_00_01,USER_AGENT, USER_AGENT_MUTT,X_AUTH_WARNING version=2.43 X-Spam-Level: Sender: owner-freebsd-cluster@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG On Tue, Dec 10, 2002 at 01:57:57PM +0100, Michael Grant wrote: > However, there's a link on the page you pointed me at to GFS the > Global File System by Sistina: > > http://www.sistina.com/products_gfs.htm > > This seems to be similar to Coda but for Linux. I hate to say this > but it does look like Linux has superior support for clustering. Be aware that GFS is a commercial product though. There are a number of other Linux clustering filesystems in this area if you're looking to port one though. Ignoring GFS and Coda (since they have both been mentioned already), a brief (and incomplete) list would be something like: . PVFS http://www.parl.clemson.edu/pvfs/ . InterMezzo http://inter-mezzo.org/ . GPFS http://www-1.ibm.com/servers/eserver/clusters/software/gpfs.html . Lustre http://www.lustre.org/ Unfortunately I'd have to agree that Linux has better clustering support at the moment. That is basically because more people (particularly the corporations and national labs) are working on cluster support for Linux. -- Greg Lewis Email : glewis@eyesbeyond.com Eyes Beyond Web : http://www.eyesbeyond.com Information Technology FreeBSD : glewis@FreeBSD.org To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-cluster" in the body of the message From owner-freebsd-cluster Tue Dec 10 13:44: 0 2002 Delivered-To: freebsd-cluster@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 8F5AA37B401 for ; Tue, 10 Dec 2002 13:43:58 -0800 (PST) Received: from fubar.adept.org (fubar.adept.org [63.147.172.249]) by mx1.FreeBSD.org (Postfix) with ESMTP id 4937243EA9 for ; Tue, 10 Dec 2002 13:43:58 -0800 (PST) (envelope-from mike@adept.org) Received: by fubar.adept.org (Postfix, from userid 1001) id 5B99215247; Tue, 10 Dec 2002 13:40:47 -0800 (PST) Received: from localhost (localhost [127.0.0.1]) by fubar.adept.org (Postfix) with ESMTP id 5955915213 for ; Tue, 10 Dec 2002 13:40:47 -0800 (PST) Date: Tue, 10 Dec 2002 13:40:47 -0800 (PST) From: Mike Hoskins To: freebsd-cluster@FreeBSD.ORG Subject: Re: sharing files within a cluster In-Reply-To: <20021211025917.A9059@misty.eyesbeyond.com> Message-ID: <20021210132642.N80252-100000@fubar.adept.org> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-freebsd-cluster@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG On Wed, 11 Dec 2002, Greg Lewis wrote: > . PVFS http://www.parl.clemson.edu/pvfs/ > . InterMezzo http://inter-mezzo.org/ Thanks for the reading material. > . GPFS http://www-1.ibm.com/servers/eserver/clusters/software/gpfs.html Anyone used this with Linux? I tend to have faith in Big Blue's engineering practices, in general they hire smart people. > . Lustre http://www.lustre.org/ "The central target in this project is the development of Lustre, a next-generation cluster file system which can serve clusters with 10,000's of nodes, petabytes of storage, move 100's of GB/sec with state of the art security and management infrastructure. The 1.0 release of Lustre will happen early 2002 and will target clusters up to 1,000 nodes with 100'TB's of storage." Quite the claim(s), anyone actually seen it work? > Unfortunately I'd have to agree that Linux has better clustering support > at the moment. That is basically because more people (particularly the > corporations and national labs) are working on cluster support for Linux. Paraphrasing Jossie and the Pussycats (excellent pop culture commentary ;), "Linux is the new Windows." So, indeed, no surprise here. I don't think there's any 'pretty' FreeBSD clustering solution today... Pretty meaning a combination of managability, stability, robustness and correctness. So, moving forward, can we reach a consensus on 'what's best to try to port?' I have to admit I'll be implementing a mini-itx cluster in my garage this summer as a side project and may experiment with NFS, Coda, etc... However, those solutions would not be feasible in my production envrionments. However, once the cluster is up, I'll be more than happy to help some person or group implement and test better solutions. -- Mike Hoskins This message is RFC 1855 compliant, mike@adept.org www.adept.org/~mike/pub/rfcs/rfc1855.html To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-cluster" in the body of the message From owner-freebsd-cluster Tue Dec 10 14:22:31 2002 Delivered-To: freebsd-cluster@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 0167837B401 for ; Tue, 10 Dec 2002 14:22:30 -0800 (PST) Received: from gpo.schools.net.au (gpo.schools.net.au [203.2.135.1]) by mx1.FreeBSD.org (Postfix) with ESMTP id D00F943ED4 for ; Tue, 10 Dec 2002 14:22:26 -0800 (PST) (envelope-from joshua@shallow.net) Received: from yello.shallow.net (yello.shallow.net [203.18.243.120]) by gpo.schools.net.au (Postfix) with ESMTP id 042BAAF6C5; Wed, 11 Dec 2002 09:22:13 +1100 (EST) Received: by yello.shallow.net (Postfix, from userid 1001) id 037842A6B; Wed, 11 Dec 2002 09:22:10 +1100 (EST) Date: Wed, 11 Dec 2002 09:22:10 +1100 From: Joshua Goodall To: "Ronald G. Minnich" Cc: Michael Grant , freebsd-cluster@FreeBSD.ORG Subject: Re: sharing files within a cluster Message-ID: <20021210222210.GG98967@roughtrade.net> References: <200212100846.gBA8kBO08833@splat.grant.org> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.1i Sender: owner-freebsd-cluster@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG On Tue, Dec 10, 2002 at 08:25:42AM -0700, Ronald G. Minnich wrote: > On Tue, 10 Dec 2002, Michael Grant wrote: > > > > NFS cross-mounts again will do > > no they won't. Old idea, bad idea, don't do it. NFS is not robust enough > for this. * Michael Grant didn't write that. I did. * You've left out the statement that followed it: "There are some nasty failure modes associated with doing this ... " * NFS probably is robust enough for some environments. A suggestion I was given to remember with cross-mounts is to use soft/interruptible, and always login as root directly (rather than trying to sudo from a user account that may have just become unavailable in /home). > ron > > what you're talking about? To my knowledge, one cannot cross-mount > > with nfs. > > oh yeah you can. It's just very bad to do this. -- Joshua Goodall joshua@roughtrade.net "Your byte hit ratio is weak, old man" "If you cache me now, I will dump more core than you can possibly imagine" To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-cluster" in the body of the message From owner-freebsd-cluster Tue Dec 10 14:28:40 2002 Delivered-To: freebsd-cluster@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 95CE537B401 for ; Tue, 10 Dec 2002 14:28:38 -0800 (PST) Received: from vps.vitalit.com (vps.vitalit.com [64.105.194.69]) by mx1.FreeBSD.org (Postfix) with ESMTP id D836843EA9 for ; Tue, 10 Dec 2002 14:28:37 -0800 (PST) (envelope-from car@vitalit.com) Received: from LAPTOP (gso167-138-145.triad.rr.com [24.167.138.145]) by vps.vitalit.com (8.12.3/8.12.3) with SMTP id gBAMSQhQ089367 for ; Tue, 10 Dec 2002 17:28:26 -0500 (EST) (envelope-from car@vitalit.com) Message-ID: <003501c2a09b$5f5116c0$0301000a@LAPTOP> From: "Rouzer, Charles A (Chuck)" To: References: <20021210132642.N80252-100000@fubar.adept.org> Subject: Re: sharing files within a cluster Date: Tue, 10 Dec 2002 17:27:43 -0500 MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2720.3000 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2600.0000 Sender: owner-freebsd-cluster@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG > > Unfortunately I'd have to agree that Linux has better clustering support > > at the moment. That is basically because more people (particularly the > > corporations and national labs) are working on cluster support for Linux. > > Paraphrasing Jossie and the Pussycats (excellent pop culture commentary > ;), "Linux is the new Windows." So, indeed, no surprise here. Lets not put Linux down like that. ;-)) I like "Linux is the new Unix." better. ;-) > I don't think there's any 'pretty' FreeBSD clustering solution today... > Pretty meaning a combination of managability, stability, robustness and > correctness. So, moving forward, can we reach a consensus on 'what's best > to try to port?' I agree. I think investigating Single-System-Image-Clustering Linux and GlobalFS for potential ports would be a good start. I don't know anyone who is willing and capable to begin a port. It could also be that both projects are not mature enough to begin porting or better off creating a "branch" dedicated to FreeBSD. http://opengfs.sourceforge.net/ http://ssic-linux.sourceforge.net/ I would love to see FreeBSD get the following features most of which are currently being developed in the above projects or available (in order of importance): - iSCSI for storage area networks (LAN or WAN) (probably coming sooner than later) - Process migration, cluster wide /proc, GlobalFS or Other (ClusterFS?), - Cluster wide IP addresses. - Scale to 100s of nodes. - Cluster wide shared memory. > I have to admit I'll be implementing a mini-itx cluster in my garage this > summer as a side project and may experiment with NFS, Coda, etc... > However, those solutions would not be feasible in my production > envrionments. However, once the cluster is up, I'll be more than happy to > help some person or group implement and test better solutions. I am in the (slow) process of putting together a working SSIC-Linux here at home. Chuck To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-cluster" in the body of the message From owner-freebsd-cluster Tue Dec 10 14:30:59 2002 Delivered-To: freebsd-cluster@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 9ECF937B401 for ; Tue, 10 Dec 2002 14:30:58 -0800 (PST) Received: from mailrelay2.lanl.gov (mailrelay2.lanl.gov [128.165.4.103]) by mx1.FreeBSD.org (Postfix) with ESMTP id 1619F43EDA for ; Tue, 10 Dec 2002 14:30:57 -0800 (PST) (envelope-from rminnich@lanl.gov) Received: from ccs.lanl.gov (localhost.localdomain [127.0.0.1]) by mailrelay2.lanl.gov (8.12.3/8.12.3/(ccn-5)) with SMTP id gBAMUugp028453 for ; Tue, 10 Dec 2002 15:30:56 -0700 Received: (qmail 5230 invoked from network); 10 Dec 2002 15:30:55 -0700 Received: from unknown (HELO carotid.ccs.lanl.gov) (128.165.148.162) by 128.165.148.1 with SMTP; 10 Dec 2002 15:30:55 -0700 Received: (qmail 1462 invoked by uid 3499); 10 Dec 2002 15:30:55 -0700 Received: from localhost (sendmail-bs@127.0.0.1) by localhost with SMTP; 10 Dec 2002 15:30:55 -0700 Date: Tue, 10 Dec 2002 15:30:55 -0700 (MST) From: "Ronald G. Minnich" X-X-Sender: rminnich@carotid.ccs.lanl.gov To: Joshua Goodall Cc: Michael Grant , Subject: Re: sharing files within a cluster In-Reply-To: <20021210222210.GG98967@roughtrade.net> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-freebsd-cluster@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG On Wed, 11 Dec 2002, Joshua Goodall wrote: > * NFS probably is robust enough for some environments. A suggestion > I was given to remember with cross-mounts is to use soft/interruptible, and > always login as root directly (rather than trying to sudo from a user > account that may have just become unavailable in /home). soft/interruptible: bad. Leads to large blocks of zeros in files. Spongy: better. ron To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-cluster" in the body of the message From owner-freebsd-cluster Tue Dec 10 14:34:27 2002 Delivered-To: freebsd-cluster@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 0DA8737B40C for ; Tue, 10 Dec 2002 14:34:26 -0800 (PST) Received: from mailrelay2.lanl.gov (mailrelay2.lanl.gov [128.165.4.103]) by mx1.FreeBSD.org (Postfix) with ESMTP id 8693C43EC2 for ; Tue, 10 Dec 2002 14:34:24 -0800 (PST) (envelope-from rminnich@lanl.gov) Received: from ccs.lanl.gov (localhost.localdomain [127.0.0.1]) by mailrelay2.lanl.gov (8.12.3/8.12.3/(ccn-5)) with SMTP id gBAMYOgp029784 for ; Tue, 10 Dec 2002 15:34:24 -0700 Received: (qmail 5246 invoked from network); 10 Dec 2002 15:34:23 -0700 Received: from unknown (HELO carotid.ccs.lanl.gov) (128.165.148.162) by 128.165.148.1 with SMTP; 10 Dec 2002 15:34:23 -0700 Received: (qmail 1470 invoked by uid 3499); 10 Dec 2002 15:34:23 -0700 Received: from localhost (sendmail-bs@127.0.0.1) by localhost with SMTP; 10 Dec 2002 15:34:23 -0700 Date: Tue, 10 Dec 2002 15:34:23 -0700 (MST) From: "Ronald G. Minnich" X-X-Sender: rminnich@carotid.ccs.lanl.gov To: "Rouzer, Charles A (Chuck)" Cc: freebsd-cluster@FreeBSD.ORG Subject: Re: sharing files within a cluster In-Reply-To: <003501c2a09b$5f5116c0$0301000a@LAPTOP> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-freebsd-cluster@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG On Tue, 10 Dec 2002, Rouzer, Charles A (Chuck) wrote: > Lets not put Linux down like that. ;-)) I like "Linux is the new Unix." > better. ;-) I think FreeBSD would run all these Linux things just as well as Linux does. I sure wish somebody would look at bproc for FreeBSD. > GlobalFS for potential ports would be a good start. I don't know anyone who "GlobalFS" in general is bad. global anything is bad. Consider Private Name Spaces. > - Process migration, cluster wide /proc, GlobalFS or Other (ClusterFS?), no, please check out Plan 9, cluster-wide /proc is not a good thing. At most you want the bproc-style proc. Anything that involves global consensus is going to give you headaches. > - Scale to 100s of nodes. start at 1024. > - Cluster wide shared memory. eek. ron To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-cluster" in the body of the message From owner-freebsd-cluster Tue Dec 10 14:48:55 2002 Delivered-To: freebsd-cluster@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 8F21937B404 for ; Tue, 10 Dec 2002 14:48:52 -0800 (PST) Received: from mgr2.xmission.com (mgr2.xmission.com [198.60.22.202]) by mx1.FreeBSD.org (Postfix) with ESMTP id 600C643E4A for ; Tue, 10 Dec 2002 14:48:51 -0800 (PST) (envelope-from glewis@eyesbeyond.com) Received: from mail by mgr2.xmission.com with spam-scanned (Exim 3.35 #1) id 18LtAl-0003ND-02 for freebsd-cluster@freebsd.org; Tue, 10 Dec 2002 15:48:15 -0700 Received: from [207.135.128.145] (helo=misty.eyesbeyond.com) by mgr2.xmission.com with esmtp (Exim 3.35 #1) id 18LtAZ-0002q0-02; Tue, 10 Dec 2002 15:48:04 -0700 Received: (from glewis@localhost) by misty.eyesbeyond.com (8.11.6/8.11.6) id gBAMlxX10957; Wed, 11 Dec 2002 09:17:59 +1030 (CST) (envelope-from glewis@eyesbeyond.com) X-Authentication-Warning: misty.eyesbeyond.com: glewis set sender to glewis@eyesbeyond.com using -f Date: Wed, 11 Dec 2002 09:17:58 +1030 From: Greg Lewis To: Mike Hoskins Cc: freebsd-cluster@FreeBSD.ORG Subject: Re: sharing files within a cluster Message-ID: <20021211091758.A10814@misty.eyesbeyond.com> References: <20021211025917.A9059@misty.eyesbeyond.com> <20021210132642.N80252-100000@fubar.adept.org> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.2.5i In-Reply-To: <20021210132642.N80252-100000@fubar.adept.org>; from mike@adept.org on Tue, Dec 10, 2002 at 01:40:47PM -0800 X-Spam-Status: No, hits=-2.1 required=8.0 tests=IN_REP_TO,PORN_4,QUOTED_EMAIL_TEXT,REFERENCES, SIGNATURE_SHORT_DENSE,SPAM_PHRASE_00_01,USER_AGENT, USER_AGENT_MUTT,X_AUTH_WARNING version=2.43 X-Spam-Level: Sender: owner-freebsd-cluster@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG On Tue, Dec 10, 2002 at 01:40:47PM -0800, Mike Hoskins wrote: > > . GPFS http://www-1.ibm.com/servers/eserver/clusters/software/gpfs.html > > Anyone used this with Linux? I tend to have faith in Big Blue's > engineering practices, in general they hire smart people. Interestingly, the last paragraph on this page also appears to say that its BSD licensed, making it a good fit at least license wise. I think technical considerations should come first, but if people it want it in the mainstream FreeBSD code, licensing is a consideration. > > . Lustre http://www.lustre.org/ > > "The central target in this project is the development of Lustre, a > next-generation cluster file system which can serve clusters with 10,000's > of nodes, petabytes of storage, move 100's of GB/sec with state of the art > security and management infrastructure. The 1.0 release of Lustre will > happen early 2002 and will target clusters up to 1,000 nodes with 100'TB's > of storage." > > Quite the claim(s), anyone actually seen it work? Lustre is in use on the MCR cluster (http://www.llnl.gov/linux/mcr/) that comes in at 5 on the latest top 500 list (http://www.top500.org/) of super computers. I'm pretty sure that is the biggest cluster currently using it. MCR certainly satisfies the 1000 nodes (1152) and 100 TBs (110) of disk space parts of their statements regarding 1.0 (which isn't even out). > I don't think there's any 'pretty' FreeBSD clustering solution today... > Pretty meaning a combination of managability, stability, robustness and > correctness. So, moving forward, can we reach a consensus on 'what's best > to try to port?' I think there can certainly be a worthwhile discussion of what people think would be the best to port and why. Ultimately the people who volunteer to do the work get to pick though :). -- Greg Lewis Email : glewis@eyesbeyond.com Eyes Beyond Web : http://www.eyesbeyond.com Information Technology FreeBSD : glewis@FreeBSD.org To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-cluster" in the body of the message From owner-freebsd-cluster Tue Dec 10 14:51:29 2002 Delivered-To: freebsd-cluster@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 8BD4637B401 for ; Tue, 10 Dec 2002 14:51:27 -0800 (PST) Received: from fubar.adept.org (fubar.adept.org [63.147.172.249]) by mx1.FreeBSD.org (Postfix) with ESMTP id 14FF543EC5 for ; Tue, 10 Dec 2002 14:51:27 -0800 (PST) (envelope-from mike@adept.org) Received: by fubar.adept.org (Postfix, from userid 1001) id A428215247; Tue, 10 Dec 2002 14:48:16 -0800 (PST) Received: from localhost (localhost [127.0.0.1]) by fubar.adept.org (Postfix) with ESMTP id A1E4415213 for ; Tue, 10 Dec 2002 14:48:16 -0800 (PST) Date: Tue, 10 Dec 2002 14:48:16 -0800 (PST) From: Mike Hoskins To: freebsd-cluster@FreeBSD.ORG Subject: Re: sharing files within a cluster In-Reply-To: Message-ID: <20021210144140.F80524-100000@fubar.adept.org> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-freebsd-cluster@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG On Tue, 10 Dec 2002, Ronald G. Minnich wrote: > > Lets not put Linux down like that. ;-)) I like "Linux is the new Unix." > > better. ;-) While it is more and more a matter of semantics, Linux is not Unix. That was the design idea behind it, and something I hope any true Linux fan (!me) can at least appreciate. > I think FreeBSD would run all these Linux things just as well as Linux > does. I sure wish somebody would look at bproc for FreeBSD. OK, I'm searching for bproc now. ;) In general though, the current problem is there's plenty of people saying "I wish someone would do X" and not many people doing... anything. I agree it's beneficial to first choose a direction... But I think we must be careful to avoid talking too much and coding too little. ;) > no, please check out Plan 9, cluster-wide /proc is not a good thing. At > most you want the bproc-style proc. Did you mean "At most..." or "At least..." ? I ask, because if bproc is the maximal desired implementation, what is a minimal implementation IYO? You seem knowledgeable here, and I certainly am not... (I'm just trying to get a firm grasp and what's needed, then maybe I can start attempting to understand what's desired.) > > - Scale to 100s of nodes. > start at 1024. It seems that's about where lustre.org's efforts are. I'm going to try to get a better understanding of their architecture. > > - Cluster wide shared memory. > eek. Eek indeed! Let's do everything else right first. ;) LOTS of complications here. -- Mike Hoskins This message is RFC 1855 compliant, mike@adept.org www.adept.org/~mike/pub/rfcs/rfc1855.html To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-cluster" in the body of the message From owner-freebsd-cluster Tue Dec 10 14:52:59 2002 Delivered-To: freebsd-cluster@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 84A1437B401 for ; Tue, 10 Dec 2002 14:52:57 -0800 (PST) Received: from vps.vitalit.com (vps.vitalit.com [64.105.194.69]) by mx1.FreeBSD.org (Postfix) with ESMTP id DC0F243EC2 for ; Tue, 10 Dec 2002 14:52:56 -0800 (PST) (envelope-from car@vitalit.com) Received: from LAPTOP (gso167-138-145.triad.rr.com [24.167.138.145]) by vps.vitalit.com (8.12.3/8.12.3) with SMTP id gBAMqthQ090240 for ; Tue, 10 Dec 2002 17:52:56 -0500 (EST) (envelope-from car@vitalit.com) Message-ID: <004901c2a09e$cae92870$0301000a@LAPTOP> From: "Rouzer, Charles A (Chuck)" To: References: Subject: Re: sharing files within a cluster Date: Tue, 10 Dec 2002 17:52:18 -0500 MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2720.3000 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2600.0000 Sender: owner-freebsd-cluster@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG > > Lets not put Linux down like that. ;-)) I like "Linux is the new Unix." > > better. ;-) > > I think FreeBSD would run all these Linux things just as well as Linux > does. I sure wish somebody would look at bproc for FreeBSD. Sure, FreeBSD runs great and runs Linux apps great, but unfortunately Linux is killing it (everything server wise?) as far as market share, mind share, developers, etc. There are still some impressive leading edge solutions being developed for proprietary unix platforms, but we can see that with the impressive market share and active development of leading edge solutions for Linux, it is clearly the new Unix IMO. I don't want to get into an argument about this I just wanted to emphasize the meaning behind my comment. As recently stated on newsforge.com ... "Enterprise-level software vendors can no longer ignore Linux" We can hope this helps FreeBSD along the way. > > GlobalFS for potential ports would be a good start. I don't know anyone who > > "GlobalFS" in general is bad. global anything is bad. Consider Private > Name Spaces. Are we refering to the same thing? I'm refering to the named Global File System originally created by Sistina and also available for Linux only by The OpenGFS Project. What would be a cluster capable file system you might suggest? > > - Process migration, cluster wide /proc, GlobalFS or Other (ClusterFS?), > > no, please check out Plan 9, cluster-wide /proc is not a good thing. At > most you want the bproc-style proc. Why? > > - Scale to 100s of nodes. > > start at 1024. I just want to start with 2. hehe > > - Cluster wide shared memory. > > eek. I would think that being able to suspend and move processes between machines would be a good thing. Chuck To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-cluster" in the body of the message From owner-freebsd-cluster Tue Dec 10 15: 6:14 2002 Delivered-To: freebsd-cluster@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id EE5F937B401 for ; Tue, 10 Dec 2002 15:06:12 -0800 (PST) Received: from mgr2.xmission.com (mgr2.xmission.com [198.60.22.202]) by mx1.FreeBSD.org (Postfix) with ESMTP id 7671943EB2 for ; Tue, 10 Dec 2002 15:06:12 -0800 (PST) (envelope-from glewis@eyesbeyond.com) Received: from mail by mgr2.xmission.com with spam-scanned (Exim 3.35 #1) id 18LtQA-0005uZ-02 for freebsd-cluster@freebsd.org; Tue, 10 Dec 2002 16:04:10 -0700 Received: from [207.135.128.145] (helo=misty.eyesbeyond.com) by mgr2.xmission.com with esmtp (Exim 3.35 #1) id 18LtPJ-00059a-02; Tue, 10 Dec 2002 16:03:19 -0700 Received: (from glewis@localhost) by misty.eyesbeyond.com (8.11.6/8.11.6) id gBAN3DI11094; Wed, 11 Dec 2002 09:33:13 +1030 (CST) (envelope-from glewis@eyesbeyond.com) X-Authentication-Warning: misty.eyesbeyond.com: glewis set sender to glewis@eyesbeyond.com using -f Date: Wed, 11 Dec 2002 09:33:13 +1030 From: Greg Lewis To: Mike Hoskins Cc: freebsd-cluster@FreeBSD.ORG Subject: Re: sharing files within a cluster Message-ID: <20021211093313.A11028@misty.eyesbeyond.com> References: <20021210144140.F80524-100000@fubar.adept.org> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.2.5i In-Reply-To: <20021210144140.F80524-100000@fubar.adept.org>; from mike@adept.org on Tue, Dec 10, 2002 at 02:48:16PM -0800 X-Spam-Status: No, hits=-3.5 required=8.0 tests=IN_REP_TO,QUOTED_EMAIL_TEXT,REFERENCES, SIGNATURE_SHORT_DENSE,SPAM_PHRASE_00_01,USER_AGENT, USER_AGENT_MUTT,X_AUTH_WARNING version=2.43 X-Spam-Level: Sender: owner-freebsd-cluster@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG On Tue, Dec 10, 2002 at 02:48:16PM -0800, Mike Hoskins wrote: > > I think FreeBSD would run all these Linux things just as well as Linux > > does. I sure wish somebody would look at bproc for FreeBSD. > > OK, I'm searching for bproc now. ;) In general though, the current > problem is there's plenty of people saying "I wish someone would do X" and > not many people doing... anything. http://sourceforge.net/projects/bproc Also consider things like Open MOSIX in this space (http://www.openmosix.org/). > > > - Scale to 100s of nodes. > > start at 1024. > > It seems that's about where lustre.org's efforts are. I'm going to try to > get a better understanding of their architecture. Actually, for anything new, you should aim at scaling to at least 10,000 nodes IMO. The 1,000 node mark is already here, the future is only going to bring higher node counts. -- Greg Lewis Email : glewis@eyesbeyond.com Eyes Beyond Web : http://www.eyesbeyond.com Information Technology FreeBSD : glewis@FreeBSD.org To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-cluster" in the body of the message From owner-freebsd-cluster Tue Dec 10 15: 7:28 2002 Delivered-To: freebsd-cluster@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 506C737B401 for ; Tue, 10 Dec 2002 15:07:27 -0800 (PST) Received: from mgr4.xmission.com (mgr4.xmission.com [198.60.22.204]) by mx1.FreeBSD.org (Postfix) with ESMTP id 7C0A543EC2 for ; Tue, 10 Dec 2002 15:07:26 -0800 (PST) (envelope-from glewis@eyesbeyond.com) Received: from mail by mgr4.xmission.com with spam-scanned (Exim 3.35 #1) id 18LtTK-0000cu-04 for freebsd-cluster@freebsd.org; Tue, 10 Dec 2002 16:07:26 -0700 Received: from [207.135.128.145] (helo=misty.eyesbeyond.com) by mgr4.xmission.com with esmtp (Exim 3.35 #1) id 18LtTJ-0000cp-00; Tue, 10 Dec 2002 16:07:25 -0700 Received: (from glewis@localhost) by misty.eyesbeyond.com (8.11.6/8.11.6) id gBAN7Ld11113; Wed, 11 Dec 2002 09:37:21 +1030 (CST) (envelope-from glewis@eyesbeyond.com) X-Authentication-Warning: misty.eyesbeyond.com: glewis set sender to glewis@eyesbeyond.com using -f Date: Wed, 11 Dec 2002 09:37:20 +1030 From: Greg Lewis To: "Rouzer, Charles A (Chuck)" Cc: freebsd-cluster@FreeBSD.ORG Subject: Re: sharing files within a cluster Message-ID: <20021211093720.B11028@misty.eyesbeyond.com> References: <004901c2a09e$cae92870$0301000a@LAPTOP> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.2.5i In-Reply-To: <004901c2a09e$cae92870$0301000a@LAPTOP>; from car@vitalit.com on Tue, Dec 10, 2002 at 05:52:18PM -0500 X-Spam-Status: No, hits=-3.5 required=8.0 tests=IN_REP_TO,QUOTED_EMAIL_TEXT,REFERENCES, SIGNATURE_SHORT_DENSE,SPAM_PHRASE_00_01,USER_AGENT, USER_AGENT_MUTT,X_AUTH_WARNING version=2.43 X-Spam-Level: Sender: owner-freebsd-cluster@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG On Tue, Dec 10, 2002 at 05:52:18PM -0500, Rouzer, Charles A (Chuck) wrote: > > > - Cluster wide shared memory. > > > > eek. > > I would think that being able to suspend and move processes between > machines would be a good thing. You don't need shared memory to do this. Look at technologies like BProc and Open MOSIX. The holy grail here is not only process migration but process checkpoint and restarting. -- Greg Lewis Email : glewis@eyesbeyond.com Eyes Beyond Web : http://www.eyesbeyond.com Information Technology FreeBSD : glewis@FreeBSD.org To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-cluster" in the body of the message From owner-freebsd-cluster Tue Dec 10 15:14:21 2002 Delivered-To: freebsd-cluster@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 18CF237B401 for ; Tue, 10 Dec 2002 15:14:21 -0800 (PST) Received: from mailrelay1.lanl.gov (mailrelay1.lanl.gov [128.165.4.101]) by mx1.FreeBSD.org (Postfix) with ESMTP id 6F40A43EC5 for ; Tue, 10 Dec 2002 15:14:20 -0800 (PST) (envelope-from rminnich@lanl.gov) Received: from ccs.lanl.gov (localhost.localdomain [127.0.0.1]) by mailrelay1.lanl.gov (8.12.3/8.12.3/(ccn-5)) with SMTP id gBANEJ9i017704 for ; Tue, 10 Dec 2002 16:14:19 -0700 Received: (qmail 5722 invoked from network); 10 Dec 2002 16:14:19 -0700 Received: from unknown (HELO carotid.ccs.lanl.gov) (128.165.148.162) by 128.165.148.1 with SMTP; 10 Dec 2002 16:14:19 -0700 Received: (qmail 1851 invoked by uid 3499); 10 Dec 2002 16:14:19 -0700 Received: from localhost (sendmail-bs@127.0.0.1) by localhost with SMTP; 10 Dec 2002 16:14:19 -0700 Date: Tue, 10 Dec 2002 16:14:19 -0700 (MST) From: "Ronald G. Minnich" X-X-Sender: rminnich@carotid.ccs.lanl.gov To: Greg Lewis Cc: Mike Hoskins , Subject: Re: sharing files within a cluster In-Reply-To: <20021211091758.A10814@misty.eyesbeyond.com> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-freebsd-cluster@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG On Wed, 11 Dec 2002, Greg Lewis wrote: > Lustre is in use on the MCR cluster (http://www.llnl.gov/linux/mcr/) that > comes in at 5 on the latest top 500 list (http://www.top500.org/) of super > computers. I'm pretty sure that is the biggest cluster currently using > it. we're using it here too when our 1024 comes up. It's good stuff and ought to be capable of working on freebsd. ron To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-cluster" in the body of the message From owner-freebsd-cluster Tue Dec 10 15:15:19 2002 Delivered-To: freebsd-cluster@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 8665637B401 for ; Tue, 10 Dec 2002 15:15:18 -0800 (PST) Received: from mailrelay2.lanl.gov (mailrelay2.lanl.gov [128.165.4.103]) by mx1.FreeBSD.org (Postfix) with ESMTP id C9CC043EC5 for ; Tue, 10 Dec 2002 15:15:17 -0800 (PST) (envelope-from rminnich@lanl.gov) Received: from ccs.lanl.gov (localhost.localdomain [127.0.0.1]) by mailrelay2.lanl.gov (8.12.3/8.12.3/(ccn-5)) with SMTP id gBANFHgp014517 for ; Tue, 10 Dec 2002 16:15:17 -0700 Received: (qmail 5743 invoked from network); 10 Dec 2002 16:15:17 -0700 Received: from unknown (HELO carotid.ccs.lanl.gov) (128.165.148.162) by 128.165.148.1 with SMTP; 10 Dec 2002 16:15:17 -0700 Received: (qmail 1858 invoked by uid 3499); 10 Dec 2002 16:15:16 -0700 Received: from localhost (sendmail-bs@127.0.0.1) by localhost with SMTP; 10 Dec 2002 16:15:16 -0700 Date: Tue, 10 Dec 2002 16:15:16 -0700 (MST) From: "Ronald G. Minnich" X-X-Sender: rminnich@carotid.ccs.lanl.gov To: Mike Hoskins Cc: freebsd-cluster@FreeBSD.ORG Subject: Re: sharing files within a cluster In-Reply-To: <20021210144140.F80524-100000@fubar.adept.org> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-freebsd-cluster@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG On Tue, 10 Dec 2002, Mike Hoskins wrote: > > no, please check out Plan 9, cluster-wide /proc is not a good thing. At > > most you want the bproc-style proc. > > Did you mean "At most..." or "At least..." ? I ask, because if bproc is > the maximal desired implementation, what is a minimal implementation IYO? I mean at most. I like Plan 9 somewhat better. ron To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-cluster" in the body of the message From owner-freebsd-cluster Tue Dec 10 15:20:12 2002 Delivered-To: freebsd-cluster@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 7C3C837B401 for ; Tue, 10 Dec 2002 15:20:11 -0800 (PST) Received: from mailrelay2.lanl.gov (mailrelay2.lanl.gov [128.165.4.103]) by mx1.FreeBSD.org (Postfix) with ESMTP id CFE9D43ED1 for ; Tue, 10 Dec 2002 15:20:10 -0800 (PST) (envelope-from rminnich@lanl.gov) Received: from ccs.lanl.gov (localhost.localdomain [127.0.0.1]) by mailrelay2.lanl.gov (8.12.3/8.12.3/(ccn-5)) with SMTP id gBANKAgp016598 for ; Tue, 10 Dec 2002 16:20:10 -0700 Received: (qmail 5775 invoked from network); 10 Dec 2002 16:20:10 -0700 Received: from unknown (HELO carotid.ccs.lanl.gov) (128.165.148.162) by 128.165.148.1 with SMTP; 10 Dec 2002 16:20:10 -0700 Received: (qmail 1867 invoked by uid 3499); 10 Dec 2002 16:20:09 -0700 Received: from localhost (sendmail-bs@127.0.0.1) by localhost with SMTP; 10 Dec 2002 16:20:09 -0700 Date: Tue, 10 Dec 2002 16:20:09 -0700 (MST) From: "Ronald G. Minnich" X-X-Sender: rminnich@carotid.ccs.lanl.gov To: "Rouzer, Charles A (Chuck)" Cc: freebsd-cluster@FreeBSD.ORG Subject: Re: sharing files within a cluster In-Reply-To: <004901c2a09e$cae92870$0301000a@LAPTOP> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-freebsd-cluster@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG On Tue, 10 Dec 2002, Rouzer, Charles A (Chuck) wrote: > System originally created by Sistina and also available for Linux only by > The OpenGFS Project. What would be a cluster capable file system you might > suggest? > Lustre. I don't really think GFS qualifies as a file system in spite of the name. > > no, please check out Plan 9, cluster-wide /proc is not a good thing. At > > most you want the bproc-style proc. > > Why? global is bad. Global requires consensus, and DLMs and all that bad stuff. Please check out what Plan 9 does, it's really pretty neat. Way more than I can describe in an email. > > start at 1024. > > I just want to start with 2. hehe yeah but if you don't think in large terms you'll build things that won't go, e.g. OpenSSI. > I would think that being able to suspend and move processes between > machines would be a good thing. That is unrelated to global shared memory in clusters. thanks ron p.s. Just hoping that FreeBSD can get in this game too. To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-cluster" in the body of the message From owner-freebsd-cluster Tue Dec 10 15:31:12 2002 Delivered-To: freebsd-cluster@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 5F9CB37B401 for ; Tue, 10 Dec 2002 15:31:11 -0800 (PST) Received: from vps.vitalit.com (vps.vitalit.com [64.105.194.69]) by mx1.FreeBSD.org (Postfix) with ESMTP id 96BEB43ED1 for ; Tue, 10 Dec 2002 15:31:10 -0800 (PST) (envelope-from car@vitalit.com) Received: from LAPTOP (gso167-138-145.triad.rr.com [24.167.138.145]) by vps.vitalit.com (8.12.3/8.12.3) with SMTP id gBANV8hQ091540 for ; Tue, 10 Dec 2002 18:31:09 -0500 (EST) (envelope-from car@vitalit.com) Message-ID: <007601c2a0a4$218c2ab0$0301000a@LAPTOP> From: "Rouzer, Charles A (Chuck)" To: Subject: Clustermatic Date: Tue, 10 Dec 2002 18:30:25 -0500 MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2720.3000 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2600.0000 Sender: owner-freebsd-cluster@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG I'm not sure how I missed this. This project looks very similar to some of the comments mentioned today (non-global file system using private name spaces like Plan9). I'm looking forward to giving it a shot. Maybe this is a good candidate to port to FreeBSD and Debian. http://www.clustermatic.org http://www.acl.lanl.gov/cluster/ -- Chuck Rouzer Vital Information Technology, Inc. http://www.vitalit.com http://www.vitalserver.com To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-cluster" in the body of the message From owner-freebsd-cluster Tue Dec 10 15:35:24 2002 Delivered-To: freebsd-cluster@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 04E8B37B401 for ; Tue, 10 Dec 2002 15:35:24 -0800 (PST) Received: from spf1.us.outblaze.com (205-158-62-139.outblaze.com [205.158.62.139]) by mx1.FreeBSD.org (Postfix) with SMTP id 65DC143EA9 for ; Tue, 10 Dec 2002 15:35:23 -0800 (PST) (envelope-from derekbarrett@graffiti.net) Received: (qmail 22755 invoked from network); 10 Dec 2002 16:36:38 -0000 Received: from unknown (205.158.62.68) by 205-158-62-139.outblaze.com with QMQP; 10 Dec 2002 16:36:38 -0000 Received: (qmail 45684 invoked from network); 10 Dec 2002 23:23:49 -0000 Received: from unknown (HELO ws3.hk5.outblaze.com) (202.77.181.90) by 205-158-62-153.outblaze.com with SMTP; 10 Dec 2002 23:23:48 -0000 Received: (qmail 20222 invoked by uid 1001); 10 Dec 2002 23:23:47 -0000 Message-ID: <20021210232347.20221.qmail@graffiti.net> Content-Type: text/plain; charset="iso-8859-1" Content-Disposition: inline Content-Transfer-Encoding: 7bit MIME-Version: 1.0 X-Mailer: MIME-tools 5.41 (Entity 5.404) Received: from [66.51.217.108] by ws3.hk5.outblaze.com with http for derekbarrett@graffiti.net; Tue, 10 Dec 2002 15:23:47 -0800 From: "Derek Barrett" To: freebsd-cluster@FreeBSD.ORG Date: Tue, 10 Dec 2002 15:23:47 -0800 Subject: Wal-Mart pcs for $200 X-Originating-Ip: 66.51.217.108 X-Originating-Server: ws3.hk5.outblaze.com Sender: owner-freebsd-cluster@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG Well Wal-Mart is selling pcs with Linux on them for $200. They are 800mhz but are powered by a Via processor. I've never heard of Via. Who makes that? Does FreeBSD run on Via processors? Anyway, you could get yourself a cluster for $1000 but then again there are probably better sources that cost less for used equipment running Intel processors. http://www.walmart.com/catalog/product.gsp?product_id=2009643&cat=3951&type=19&dept=3944&path=0%3A3944%3A3951 -- _______________________________________________ Get your free email from http://www.graffiti.net Powered by Outblaze To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-cluster" in the body of the message From owner-freebsd-cluster Tue Dec 10 15:38: 4 2002 Delivered-To: freebsd-cluster@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 602BE37B401 for ; Tue, 10 Dec 2002 15:38:03 -0800 (PST) Received: from mailrelay2.lanl.gov (mailrelay2.lanl.gov [128.165.4.103]) by mx1.FreeBSD.org (Postfix) with ESMTP id 930C443E4A for ; Tue, 10 Dec 2002 15:38:02 -0800 (PST) (envelope-from rminnich@lanl.gov) Received: from ccs.lanl.gov (localhost.localdomain [127.0.0.1]) by mailrelay2.lanl.gov (8.12.3/8.12.3/(ccn-5)) with SMTP id gBANbwgp023435 for ; Tue, 10 Dec 2002 16:37:58 -0700 Received: (qmail 5931 invoked from network); 10 Dec 2002 16:37:57 -0700 Received: from unknown (HELO carotid.ccs.lanl.gov) (128.165.148.162) by 128.165.148.1 with SMTP; 10 Dec 2002 16:37:57 -0700 Received: (qmail 1921 invoked by uid 3499); 10 Dec 2002 16:37:57 -0700 Received: from localhost (sendmail-bs@127.0.0.1) by localhost with SMTP; 10 Dec 2002 16:37:57 -0700 Date: Tue, 10 Dec 2002 16:37:57 -0700 (MST) From: "Ronald G. Minnich" X-X-Sender: rminnich@carotid.ccs.lanl.gov To: "Rouzer, Charles A (Chuck)" Cc: freebsd-cluster@FreeBSD.ORG Subject: Re: Clustermatic In-Reply-To: <007601c2a0a4$218c2ab0$0301000a@LAPTOP> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-freebsd-cluster@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG On Tue, 10 Dec 2002, Rouzer, Charles A (Chuck) wrote: > http://www.clustermatic.org yup, that's us. I.e. our stuff, and I am happy you are interested. Debian already works AFAIK. ron To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-cluster" in the body of the message From owner-freebsd-cluster Tue Dec 10 15:44:21 2002 Delivered-To: freebsd-cluster@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id E121937B401 for ; Tue, 10 Dec 2002 15:44:20 -0800 (PST) Received: from fubar.adept.org (fubar.adept.org [63.147.172.249]) by mx1.FreeBSD.org (Postfix) with ESMTP id 8A78743EC2 for ; Tue, 10 Dec 2002 15:44:20 -0800 (PST) (envelope-from mike@adept.org) Received: by fubar.adept.org (Postfix, from userid 1001) id 0102915247; Tue, 10 Dec 2002 15:41:09 -0800 (PST) Received: from localhost (localhost [127.0.0.1]) by fubar.adept.org (Postfix) with ESMTP id 0045115213 for ; Tue, 10 Dec 2002 15:41:09 -0800 (PST) Date: Tue, 10 Dec 2002 15:41:09 -0800 (PST) From: Mike Hoskins To: freebsd-cluster@FreeBSD.ORG Subject: Re: Wal-Mart pcs for $200 In-Reply-To: <20021210232347.20221.qmail@graffiti.net> Message-ID: <20021210153956.P80524-100000@fubar.adept.org> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-freebsd-cluster@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG On Tue, 10 Dec 2002, Derek Barrett wrote: > Does FreeBSD run on Via processors? This was discussed on freebsd-questions: http://www.freebsd.org/cgi/search.cgi?words=mini-itx&max=25&sort=score&index=recent&source=freebsd-questions -- Mike Hoskins This message is RFC 1855 compliant, mike@adept.org www.adept.org/~mike/pub/rfcs/rfc1855.html To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-cluster" in the body of the message From owner-freebsd-cluster Tue Dec 10 16:10:28 2002 Delivered-To: freebsd-cluster@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id C10CC37B401 for ; Tue, 10 Dec 2002 16:10:27 -0800 (PST) Received: from mgr2.xmission.com (mgr2.xmission.com [198.60.22.202]) by mx1.FreeBSD.org (Postfix) with ESMTP id 449D343EA9 for ; Tue, 10 Dec 2002 16:10:27 -0800 (PST) (envelope-from glewis@eyesbeyond.com) Received: from mail by mgr2.xmission.com with spam-scanned (Exim 3.35 #1) id 18LuPY-0007eN-02 for freebsd-cluster@freebsd.org; Tue, 10 Dec 2002 17:07:36 -0700 Received: from [207.135.128.145] (helo=misty.eyesbeyond.com) by mgr2.xmission.com with esmtp (Exim 3.35 #1) id 18LuO1-0006eV-02; Tue, 10 Dec 2002 17:06:01 -0700 Received: (from glewis@localhost) by misty.eyesbeyond.com (8.11.6/8.11.6) id gBB05us11432; Wed, 11 Dec 2002 10:35:56 +1030 (CST) (envelope-from glewis@eyesbeyond.com) X-Authentication-Warning: misty.eyesbeyond.com: glewis set sender to glewis@eyesbeyond.com using -f Date: Wed, 11 Dec 2002 10:35:56 +1030 From: Greg Lewis To: "Ronald G. Minnich" Cc: freebsd-cluster@FreeBSD.ORG Subject: Re: sharing files within a cluster Message-ID: <20021211103556.A11403@misty.eyesbeyond.com> References: <20021211091758.A10814@misty.eyesbeyond.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.2.5i In-Reply-To: ; from rminnich@lanl.gov on Tue, Dec 10, 2002 at 04:14:19PM -0700 X-Spam-Status: No, hits=-3.5 required=8.0 tests=IN_REP_TO,QUOTED_EMAIL_TEXT,REFERENCES, SIGNATURE_SHORT_DENSE,SPAM_PHRASE_00_01,USER_AGENT, USER_AGENT_MUTT,X_AUTH_WARNING version=2.43 X-Spam-Level: Sender: owner-freebsd-cluster@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG On Tue, Dec 10, 2002 at 04:14:19PM -0700, Ronald G. Minnich wrote: > On Wed, 11 Dec 2002, Greg Lewis wrote: > > Lustre is in use on the MCR cluster (http://www.llnl.gov/linux/mcr/) that > > comes in at 5 on the latest top 500 list (http://www.top500.org/) of super > > computers. I'm pretty sure that is the biggest cluster currently using > > it. > > we're using it here too when our 1024 comes up. I assume you mean ASCI Pink? > It's good stuff and ought to be capable of working on freebsd. Sure, it just needs someone with time to port it. I see you were also looking into what needs to done to get LinuxBIOS booting FreeBSD on freebsd-hackers. It would be nice to get that, a cluster filesystem and a BProc/MOSIX type piece of functionality ported to FreeBSD. I'm assuming I need some assembly skills and familiarity with BIOS calls to look at the LinuxBIOS piece? Or do you have a good handle on the problem and just need to railroad the changes through? -- Greg Lewis Email : glewis@eyesbeyond.com Eyes Beyond Web : http://www.eyesbeyond.com Information Technology FreeBSD : glewis@FreeBSD.org To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-cluster" in the body of the message From owner-freebsd-cluster Tue Dec 10 16:54:45 2002 Delivered-To: freebsd-cluster@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 3929D37B401 for ; Tue, 10 Dec 2002 16:54:44 -0800 (PST) Received: from fubar.adept.org (fubar.adept.org [63.147.172.249]) by mx1.FreeBSD.org (Postfix) with ESMTP id EBA1C43EA9 for ; Tue, 10 Dec 2002 16:54:43 -0800 (PST) (envelope-from mike@adept.org) Received: by fubar.adept.org (Postfix, from userid 1001) id CFA5A15247; Tue, 10 Dec 2002 16:51:32 -0800 (PST) Received: from localhost (localhost [127.0.0.1]) by fubar.adept.org (Postfix) with ESMTP id CD6DD15213 for ; Tue, 10 Dec 2002 16:51:32 -0800 (PST) Date: Tue, 10 Dec 2002 16:51:32 -0800 (PST) From: Mike Hoskins To: freebsd-cluster@FreeBSD.ORG Subject: Re: sharing files within a cluster In-Reply-To: <20021211103556.A11403@misty.eyesbeyond.com> Message-ID: <20021210164505.H80831-100000@fubar.adept.org> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-freebsd-cluster@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG On Wed, 11 Dec 2002, Greg Lewis wrote: > I'm assuming I need some assembly skills and familiarity with BIOS calls > to look at the LinuxBIOS piece? Or do you have a good handle on the > problem and just need to railroad the changes through? I'm sure I'm missing something... What about, http://www.openfirmware.org/ It already works with Sun, LinuxPPC (where I liked what I saw back around '95/96) and some BSDs. Why start a new project with Linux in the name when a good solution is already underway? It seems like it would have benefitted everyone if the LinuxBIOS folks worked with the OpenFirmware folks. -- Mike Hoskins This message is RFC 1855 compliant, mike@adept.org www.adept.org/pub/rfcs/rfc1855.html To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-cluster" in the body of the message From owner-freebsd-cluster Tue Dec 10 17:48: 0 2002 Delivered-To: freebsd-cluster@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 2ADCA37B401 for ; Tue, 10 Dec 2002 17:47:59 -0800 (PST) Received: from mgr1.xmission.com (mgr1.xmission.com [198.60.22.201]) by mx1.FreeBSD.org (Postfix) with ESMTP id 2BA1E43E4A for ; Tue, 10 Dec 2002 17:47:58 -0800 (PST) (envelope-from glewis@eyesbeyond.com) Received: from mail by mgr1.xmission.com with spam-scanned (Exim 3.35 #1) id 18Lvtm-0005tN-01 for freebsd-cluster@freebsd.org; Tue, 10 Dec 2002 18:42:54 -0700 Received: from [207.135.128.145] (helo=misty.eyesbeyond.com) by mgr1.xmission.com with esmtp (Exim 3.35 #1) id 18LvZy-00013E-01; Tue, 10 Dec 2002 18:22:26 -0700 Received: (from glewis@localhost) by misty.eyesbeyond.com (8.11.6/8.11.6) id gBB1MMg11674; Wed, 11 Dec 2002 11:52:22 +1030 (CST) (envelope-from glewis@eyesbeyond.com) X-Authentication-Warning: misty.eyesbeyond.com: glewis set sender to glewis@eyesbeyond.com using -f Date: Wed, 11 Dec 2002 11:52:21 +1030 From: Greg Lewis To: Mike Hoskins Cc: freebsd-cluster@FreeBSD.ORG Subject: Re: sharing files within a cluster Message-ID: <20021211115221.A11632@misty.eyesbeyond.com> References: <20021211103556.A11403@misty.eyesbeyond.com> <20021210164505.H80831-100000@fubar.adept.org> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.2.5i In-Reply-To: <20021210164505.H80831-100000@fubar.adept.org>; from mike@adept.org on Tue, Dec 10, 2002 at 04:51:32PM -0800 X-Spam-Status: No, hits=-3.5 required=8.0 tests=IN_REP_TO,QUOTED_EMAIL_TEXT,REFERENCES, SIGNATURE_SHORT_DENSE,SPAM_PHRASE_00_01,USER_AGENT, USER_AGENT_MUTT,X_AUTH_WARNING version=2.43 X-Spam-Level: Sender: owner-freebsd-cluster@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG On Tue, Dec 10, 2002 at 04:51:32PM -0800, Mike Hoskins wrote: > On Wed, 11 Dec 2002, Greg Lewis wrote: > > I'm assuming I need some assembly skills and familiarity with BIOS calls > > to look at the LinuxBIOS piece? Or do you have a good handle on the > > problem and just need to railroad the changes through? > > I'm sure I'm missing something... What about, > > http://www.openfirmware.org/ > > It already works with Sun, LinuxPPC (where I liked what I saw back > around '95/96) and some BSDs. Why start a new project with Linux in > the name when a good solution is already underway? It seems like it would > have benefitted everyone if the LinuxBIOS folks worked with the > OpenFirmware folks. Actually, I happen to like OpenFirmware :). However its not the problem I'd like to see solved at the moment. OpenFirmware is: 1. Only available on Sparc and PowerPC machines. 2. Generally a proprietary product. Yes, it does have an open standard describing it, but there is no accompanying source code. So, you would need to start from scratch and write something for all the x86 motherboards you wanted to target that met the OpenFirmware standard. Sorry, thats a project for somebody a bit more ambitious :). In comparison, LinuxBIOS exists and is designed for booting nodes in a cluster (among other things). All I'd like to see is to add the necessary changes so it can boot FreeBSD. Its code is available, so is the FreeBSD booting code. They just need to be taught to cooperate with each other :). Apart from that, consider that LinuxBIOS is in use on production clusters right now and is responsible for several million dollars of Linux cluster sales this year. The relatively simple task of making it boot FreeBSD should take a relatively short time (for someone with the know how) and would be a helpful boost to FreeBSD's "clusterability". IMHO of course :). -- Greg Lewis Email : glewis@eyesbeyond.com Eyes Beyond Web : http://www.eyesbeyond.com Information Technology FreeBSD : glewis@FreeBSD.org To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-cluster" in the body of the message From owner-freebsd-cluster Tue Dec 10 20:52: 6 2002 Delivered-To: freebsd-cluster@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 3631C37B401 for ; Tue, 10 Dec 2002 20:52:05 -0800 (PST) Received: from mailrelay1.lanl.gov (mailrelay1.lanl.gov [128.165.4.101]) by mx1.FreeBSD.org (Postfix) with ESMTP id 785B743EC2 for ; Tue, 10 Dec 2002 20:52:04 -0800 (PST) (envelope-from rminnich@lanl.gov) Received: from ccs.lanl.gov (localhost.localdomain [127.0.0.1]) by mailrelay1.lanl.gov (8.12.3/8.12.3/(ccn-5)) with SMTP id gBB4q39i019598 for ; Tue, 10 Dec 2002 21:52:03 -0700 Received: (qmail 8140 invoked from network); 10 Dec 2002 21:52:03 -0700 Received: from unknown (HELO carotid.ccs.lanl.gov) (128.165.148.162) by 128.165.148.1 with SMTP; 10 Dec 2002 21:52:03 -0700 Received: (qmail 3869 invoked by uid 3499); 10 Dec 2002 21:52:03 -0700 Received: from localhost (sendmail-bs@127.0.0.1) by localhost with SMTP; 10 Dec 2002 21:52:03 -0700 Date: Tue, 10 Dec 2002 21:52:03 -0700 (MST) From: "Ronald G. Minnich" X-X-Sender: rminnich@carotid.ccs.lanl.gov To: Greg Lewis Cc: freebsd-cluster@FreeBSD.ORG Subject: Re: sharing files within a cluster In-Reply-To: <20021211103556.A11403@misty.eyesbeyond.com> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-freebsd-cluster@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG On Wed, 11 Dec 2002, Greg Lewis wrote: > I assume you mean ASCI Pink? oh goodness don't call it that :-) I doubt any of my ASCI friends want somebody calling Pink an ASCI machine :-) BTW if anybody knows Pink we want her to come autograph it. We failed to get the cabs repainted Pink however. > I'm assuming I need some assembly skills and familiarity with BIOS calls > to look at the LinuxBIOS piece? Or do you have a good handle on the > problem and just need to railroad the changes through? No, actually, all you need to do is this: figure out what BIOS calls freebsd is making, then figure out how we can get freebsd not to make them, and see what we need to do with the tables linuxbios provides so that freebsd can get its bios-provided info via the tables. So the questions are: - what calls is freebsd making? - why? thanks ron To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-cluster" in the body of the message From owner-freebsd-cluster Tue Dec 10 20:55:21 2002 Delivered-To: freebsd-cluster@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id AFBD537B401 for ; Tue, 10 Dec 2002 20:55:20 -0800 (PST) Received: from mailrelay1.lanl.gov (mailrelay1.lanl.gov [128.165.4.101]) by mx1.FreeBSD.org (Postfix) with ESMTP id 15F0B43EC5 for ; Tue, 10 Dec 2002 20:55:20 -0800 (PST) (envelope-from rminnich@lanl.gov) Received: from ccs.lanl.gov (localhost.localdomain [127.0.0.1]) by mailrelay1.lanl.gov (8.12.3/8.12.3/(ccn-5)) with SMTP id gBB4tJ9i020132 for ; Tue, 10 Dec 2002 21:55:19 -0700 Received: (qmail 8176 invoked from network); 10 Dec 2002 21:55:19 -0700 Received: from unknown (HELO carotid.ccs.lanl.gov) (128.165.148.162) by 128.165.148.1 with SMTP; 10 Dec 2002 21:55:19 -0700 Received: (qmail 3887 invoked by uid 3499); 10 Dec 2002 21:55:19 -0700 Received: from localhost (sendmail-bs@127.0.0.1) by localhost with SMTP; 10 Dec 2002 21:55:19 -0700 Date: Tue, 10 Dec 2002 21:55:19 -0700 (MST) From: "Ronald G. Minnich" X-X-Sender: rminnich@carotid.ccs.lanl.gov To: Mike Hoskins Cc: freebsd-cluster@FreeBSD.ORG Subject: Re: sharing files within a cluster In-Reply-To: <20021210164505.H80831-100000@fubar.adept.org> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-freebsd-cluster@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG On Tue, 10 Dec 2002, Mike Hoskins wrote: > http://www.openfirmware.org/ > > It already works with Sun, LinuxPPC (where I liked what I saw back > around '95/96) and some BSDs. Why start a new project with Linux in > the name when a good solution is already underway? It seems like it would > have benefitted everyone if the LinuxBIOS folks worked with the > OpenFirmware folks. well, cause we do different things. I'm not going to get into that argument. ron To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-cluster" in the body of the message From owner-freebsd-cluster Tue Dec 10 21:24:45 2002 Delivered-To: freebsd-cluster@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 6C8B737B401 for ; Tue, 10 Dec 2002 21:24:44 -0800 (PST) Received: from mailrelay2.lanl.gov (mailrelay2.lanl.gov [128.165.4.103]) by mx1.FreeBSD.org (Postfix) with ESMTP id C6DAE43EB2 for ; Tue, 10 Dec 2002 21:24:43 -0800 (PST) (envelope-from rminnich@lanl.gov) Received: from ccs.lanl.gov (localhost.localdomain [127.0.0.1]) by mailrelay2.lanl.gov (8.12.3/8.12.3/(ccn-5)) with SMTP id gBB5Ohgp021713 for ; Tue, 10 Dec 2002 22:24:43 -0700 Received: (qmail 8665 invoked from network); 10 Dec 2002 22:24:42 -0700 Received: from unknown (HELO carotid.ccs.lanl.gov) (128.165.148.162) by 128.165.148.1 with SMTP; 10 Dec 2002 22:24:42 -0700 Received: (qmail 3990 invoked by uid 3499); 10 Dec 2002 22:24:42 -0700 Received: from localhost (sendmail-bs@127.0.0.1) by localhost with SMTP; 10 Dec 2002 22:24:42 -0700 Date: Tue, 10 Dec 2002 22:24:42 -0700 (MST) From: "Ronald G. Minnich" X-X-Sender: rminnich@carotid.ccs.lanl.gov To: Derek Barrett Cc: freebsd-cluster@FreeBSD.ORG Subject: Re: Wal-Mart pcs for $200 In-Reply-To: <20021210232347.20221.qmail@graffiti.net> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-freebsd-cluster@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG On Tue, 10 Dec 2002, Derek Barrett wrote: > Well Wal-Mart is selling pcs with Linux on them > for $200. They are 800mhz but are powered by a Via processor. > I've never heard of Via. Who makes that? works fine. VIA is the maker of something like 50% of the chipsets sold in the world. A very interesting company. You can get the boards alone from cwlinux.com for $105. AMAZING for a pIII/800 equivalent. (you have to supply memory) And, they will soon support linuxbios, so for clusters, no hard drive etc. needed :-) Runs clustermatic just fine. ron To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-cluster" in the body of the message From owner-freebsd-cluster Wed Dec 11 2:34:19 2002 Delivered-To: freebsd-cluster@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 8812B37B401 for ; Wed, 11 Dec 2002 02:34:17 -0800 (PST) Received: from gate.nentec.de (gate2.nentec.de [194.25.215.66]) by mx1.FreeBSD.org (Postfix) with ESMTP id D6FD543EA9 for ; Wed, 11 Dec 2002 02:34:15 -0800 (PST) (envelope-from sporner@nentec.de) Received: from nenny.nentec.de (root@nenny.nentec.de [153.92.64.1]) by gate.nentec.de (8.11.3/8.9.3) with ESMTP id gBBAY4P13069; Wed, 11 Dec 2002 11:34:05 +0100 Received: from nentec.de (andromeda.nentec.de [153.92.64.34]) by nenny.nentec.de (8.11.3/8.11.3) with ESMTP id gBBAXut29629; Wed, 11 Dec 2002 11:33:56 +0100 Message-ID: <3DF71494.7040701@nentec.de> Date: Wed, 11 Dec 2002 11:33:56 +0100 From: Andy Sporner User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.2a) Gecko/20020910 X-Accept-Language: en-us, en MIME-Version: 1.0 To: "Ronald G. Minnich" Cc: "Rouzer, Charles A (Chuck)" , freebsd-cluster@FreeBSD.ORG Subject: Re: sharing files within a cluster References: Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Virus-Scanned: by AMaViS-perl11-milter (http://amavis.org/) Sender: owner-freebsd-cluster@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG I think here there needs to be made a distinction between farm computing such as Ron is talking about and he is then of course correct. However in a single-image environment these concepts are actually sort of desirable. Example if I have a process that is attached to shared memory, how can it migrate. It should be able to! The answer is to map the memory pages in a way so as to allow them to be swappable from node to node at the VM layer. Sequent used a Numa concept on their new machines (before IBM killed them) and I am very familiar with this approach. This allowed standard applications to run without modifications (such as Oracle)... Andy Ronald G. Minnich wrote: >On Tue, 10 Dec 2002, Rouzer, Charles A (Chuck) wrote: > > > >> Lets not put Linux down like that. ;-)) I like "Linux is the new Unix." >>better. ;-) >> >> > >I think FreeBSD would run all these Linux things just as well as Linux >does. I sure wish somebody would look at bproc for FreeBSD. > > > >>GlobalFS for potential ports would be a good start. I don't know anyone who >> >> > >"GlobalFS" in general is bad. global anything is bad. Consider Private >Name Spaces. > > > > >>- Process migration, cluster wide /proc, GlobalFS or Other (ClusterFS?), >> >> > >no, please check out Plan 9, cluster-wide /proc is not a good thing. At >most you want the bproc-style proc. > >Anything that involves global consensus is going to give you headaches. > > > >>- Scale to 100s of nodes. >> >> > >start at 1024. > > > >>- Cluster wide shared memory. >> >> > >eek. > >ron > > >To Unsubscribe: send mail to majordomo@FreeBSD.org >with "unsubscribe freebsd-cluster" in the body of the message > > > > To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-cluster" in the body of the message From owner-freebsd-cluster Wed Dec 11 13:44:44 2002 Delivered-To: freebsd-cluster@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id B32CF37B401 for ; Wed, 11 Dec 2002 13:44:43 -0800 (PST) Received: from fubar.adept.org (fubar.adept.org [63.147.172.249]) by mx1.FreeBSD.org (Postfix) with ESMTP id 6B74543EC5 for ; Wed, 11 Dec 2002 13:44:43 -0800 (PST) (envelope-from mike@adept.org) Received: by fubar.adept.org (Postfix, from userid 1001) id 0A09615247; Wed, 11 Dec 2002 13:41:27 -0800 (PST) Received: from localhost (localhost [127.0.0.1]) by fubar.adept.org (Postfix) with ESMTP id 0943315213 for ; Wed, 11 Dec 2002 13:41:27 -0800 (PST) Date: Wed, 11 Dec 2002 13:41:27 -0800 (PST) From: Mike Hoskins To: freebsd-cluster@FreeBSD.ORG Subject: Re: sharing files within a cluster In-Reply-To: <20021211115221.A11632@misty.eyesbeyond.com> Message-ID: <20021211133703.C83245-100000@fubar.adept.org> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-freebsd-cluster@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG On Wed, 11 Dec 2002, Greg Lewis wrote: > Apart from that, consider that LinuxBIOS is in use on production clusters > right now and is responsible for several million dollars of Linux cluster > sales this year. The relatively simple task of making it boot FreeBSD > should take a relatively short time (for someone with the know how) and > would be a helpful boost to FreeBSD's "clusterability". Sure would be nice if Linux wasn't in the name. Maybe it's just me, but it'd be nice of some "open" things were actually more "open." What next? RedHatSamba? Samba just isn't good enough... Gotta have the OS name/distribution in there for it to be "cool." Ahh, OpenBIOS... Time to work on a new "port" or version I guess. Something that any OS vendor/user can get behind and implement without being associated with Linux. (Yes, some of us have had very bad encounters with Linux and do anything to avoid it.) -- Mike Hoskins This message is RFC 1855 compliant, mike@adept.org www.adept.org/pub/rfcs/rfc1855.html To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-cluster" in the body of the message From owner-freebsd-cluster Wed Dec 11 14:28:26 2002 Delivered-To: freebsd-cluster@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 039F237B401 for ; Wed, 11 Dec 2002 14:28:26 -0800 (PST) Received: from grant.org (grant.org [206.190.164.98]) by mx1.FreeBSD.org (Postfix) with ESMTP id 5FBA343E4A for ; Wed, 11 Dec 2002 14:28:25 -0800 (PST) (envelope-from mgrant@splat.grant.org) Received: from splat.grant.org (mgrant@splat.grant.org [213.39.2.177]) by grant.org (8.12.6/8.12.6) with ESMTP id gBBMSIVM062969 for ; Wed, 11 Dec 2002 17:28:18 -0500 (EST) (envelope-from mgrant@splat.grant.org) Received: (from mgrant@localhost) by splat.grant.org (8.11.6+Sun/8.11.6) id gBBMPi411534; Wed, 11 Dec 2002 23:25:44 +0100 (MET) Date: Wed, 11 Dec 2002 23:25:44 +0100 (MET) Message-Id: <200212112225.gBBMPi411534@splat.grant.org> From: Michael Grant To: freebsd-cluster@FreeBSD.ORG Subject: Re: sharing files within a cluster Sender: owner-freebsd-cluster@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG Anyone know about this "single image linux cluster"? How's that done? It seems to me that there would need to be something local to each box, minimally /etc, no? Andy, I agree with you that having a common process space and the ability to migrate processies across machines would be a big win. If that could be done on a freebsd cluster along with a single image, that would definitely be a reason to use a freebsd cluster over a linux cluster. Realistically, how close are you to this? Michael Grant To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-cluster" in the body of the message From owner-freebsd-cluster Wed Dec 11 14:34:22 2002 Delivered-To: freebsd-cluster@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 4318137B401 for ; Wed, 11 Dec 2002 14:34:21 -0800 (PST) Received: from vps.vitalit.com (vps.vitalit.com [64.105.194.69]) by mx1.FreeBSD.org (Postfix) with ESMTP id 7ABF943E4A for ; Wed, 11 Dec 2002 14:34:20 -0800 (PST) (envelope-from car@vitalit.com) Received: from LAPTOP (gso167-138-145.triad.rr.com [24.167.138.145]) by vps.vitalit.com (8.12.3/8.12.3) with SMTP id gBBMY9hQ046436 for ; Wed, 11 Dec 2002 17:34:09 -0500 (EST) (envelope-from car@vitalit.com) Message-ID: <000c01c2a165$4d67c3d0$0301000a@LAPTOP> From: "Rouzer, Charles A (Chuck)" To: References: <200212112225.gBBMPi411534@splat.grant.org> Subject: Re: ssic-linux etc network configuration Date: Wed, 11 Dec 2002 17:33:12 -0500 MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2720.3000 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2600.0000 Sender: owner-freebsd-cluster@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG They use a method of "overmounting". Meaning they use the linux-mount bind option (IIRC) to mount the nodes network configuration directory "on top" of the configuration directory expected by the init scripts. This will result in the node booting with the proper network configuration. Its kinda a strange work around but seems to work. Chuck > Anyone know about this "single image linux cluster"? How's that done? > It seems to me that there would need to be something local to each > box, minimally /etc, no? To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-cluster" in the body of the message From owner-freebsd-cluster Wed Dec 11 14:40:59 2002 Delivered-To: freebsd-cluster@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 22F2C37B401 for ; Wed, 11 Dec 2002 14:40:58 -0800 (PST) Received: from mailrelay2.lanl.gov (mailrelay2.lanl.gov [128.165.4.103]) by mx1.FreeBSD.org (Postfix) with ESMTP id 64A4F43EA9 for ; Wed, 11 Dec 2002 14:40:57 -0800 (PST) (envelope-from rminnich@lanl.gov) Received: from ccs.lanl.gov (localhost.localdomain [127.0.0.1]) by mailrelay2.lanl.gov (8.12.3/8.12.3/(ccn-5)) with SMTP id gBBMeugp006418 for ; Wed, 11 Dec 2002 15:40:56 -0700 Received: (qmail 18144 invoked from network); 11 Dec 2002 15:40:56 -0700 Received: from unknown (HELO carotid.ccs.lanl.gov) (128.165.148.162) by 128.165.148.1 with SMTP; 11 Dec 2002 15:40:56 -0700 Received: (qmail 11069 invoked by uid 3499); 11 Dec 2002 15:40:56 -0700 Received: from localhost (sendmail-bs@127.0.0.1) by localhost with SMTP; 11 Dec 2002 15:40:56 -0700 Date: Wed, 11 Dec 2002 15:40:56 -0700 (MST) From: "Ronald G. Minnich" X-X-Sender: rminnich@carotid.ccs.lanl.gov To: Michael Grant Cc: freebsd-cluster@FreeBSD.ORG Subject: Re: sharing files within a cluster In-Reply-To: <200212112225.gBBMPi411534@splat.grant.org> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-freebsd-cluster@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG On Wed, 11 Dec 2002, Michael Grant wrote: > Anyone know about this "single image linux cluster"? How's that done? don't know. But we have one of 1007 nodes up as of this second. [rminnich@pink rminnich]$ bpstat -t allup 1007 Sorry but it is kind of slow to do 1007 date commands. [rminnich@pink rminnich]$ time bpsh allup date > /dev/null real 0m5.252s user 0m0.360s sys 0m10.790s 10 seconds, we want faster. ron To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-cluster" in the body of the message From owner-freebsd-cluster Wed Dec 11 14:44:20 2002 Delivered-To: freebsd-cluster@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 6458F37B401 for ; Wed, 11 Dec 2002 14:44:19 -0800 (PST) Received: from mailrelay2.lanl.gov (mailrelay2.lanl.gov [128.165.4.103]) by mx1.FreeBSD.org (Postfix) with ESMTP id BD16443E4A for ; Wed, 11 Dec 2002 14:44:18 -0800 (PST) (envelope-from rminnich@lanl.gov) Received: from ccs.lanl.gov (localhost.localdomain [127.0.0.1]) by mailrelay2.lanl.gov (8.12.3/8.12.3/(ccn-5)) with SMTP id gBBMiIgp008167 for ; Wed, 11 Dec 2002 15:44:18 -0700 Received: (qmail 18198 invoked from network); 11 Dec 2002 15:44:18 -0700 Received: from unknown (HELO carotid.ccs.lanl.gov) (128.165.148.162) by 128.165.148.1 with SMTP; 11 Dec 2002 15:44:18 -0700 Received: (qmail 11336 invoked by uid 3499); 11 Dec 2002 15:44:17 -0700 Received: from localhost (sendmail-bs@127.0.0.1) by localhost with SMTP; 11 Dec 2002 15:44:17 -0700 Date: Wed, 11 Dec 2002 15:44:17 -0700 (MST) From: "Ronald G. Minnich" X-X-Sender: rminnich@carotid.ccs.lanl.gov To: Mike Hoskins Cc: freebsd-cluster@FreeBSD.ORG Subject: Re: sharing files within a cluster In-Reply-To: <20021211133703.C83245-100000@fubar.adept.org> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-freebsd-cluster@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG On Wed, 11 Dec 2002, Mike Hoskins wrote: > What next? RedHatSamba? Samba just isn't good enough... Gotta have the > OS name/distribution in there for it to be "cool." who cares? You maybe ought to find out why linux is in the name before spouting. Did you even bother reading the web page or docs? I doubt it. Your note is the sort of typical nonsense you find on mailing lists though. So I'll stop here. ron To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-cluster" in the body of the message From owner-freebsd-cluster Wed Dec 11 17:27:25 2002 Delivered-To: freebsd-cluster@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id CD6EA37B401 for ; Wed, 11 Dec 2002 17:27:23 -0800 (PST) Received: from malasada.lava.net (malasada.lava.net [64.65.64.17]) by mx1.FreeBSD.org (Postfix) with ESMTP id 42A5843EA9 for ; Wed, 11 Dec 2002 17:27:23 -0800 (PST) (envelope-from cliftonr@lava.net) Received: from localhost (3547 bytes) by malasada.lava.net; Wed, 11 Dec 2002 15:27:14 -1000 (HST) via sendmail [stdio] id for Date: Wed, 11 Dec 2002 15:27:14 -1000 From: Clifton Royston To: "Ronald G. Minnich" Cc: Joshua Goodall , Michael Grant , freebsd-cluster@FreeBSD.ORG Subject: Re: sharing files within a cluster Message-ID: <20021211152714.A25854@lava.net> Mail-Followup-To: "Ronald G. Minnich" , Joshua Goodall , Michael Grant , freebsd-cluster@FreeBSD.ORG References: <20021210222210.GG98967@roughtrade.net> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.2.5.1i In-Reply-To: ; from rminnich@lanl.gov on Tue, Dec 10, 2002 at 03:30:55PM -0700 Sender: owner-freebsd-cluster@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG On Tue, Dec 10, 2002 at 03:30:55PM -0700, Ronald G. Minnich wrote: > On Wed, 11 Dec 2002, Joshua Goodall wrote: > > > * NFS probably is robust enough for some environments. A suggestion > > I was given to remember with cross-mounts is to use soft/interruptible, and The most critical thing with NFS cross-mounts is that all servers involved need to have their boot sequences carefully checked for dependencies, and preferably should have two separate phases - one which gets the servers up to the point that they can act reliably as a *server* for all NFS volumes they export, and only start trying to mount NFS volumes from other servers after the first phase is completed successfully, If you make a mistake in this - or if your default rc scripts do - you can end up configuring a situation where your entire network is unbootable if more than one server goes down, because you have NFS volumes listed in multiple servers' fstabs, and the servers then deadlock trying to mount each other at startup. (This happened to me some years back at a previous job where a lot of random cross-mounts were set up on various HP-UX servers.) > > always login as root directly (rather than trying to sudo from a user > > account that may have just become unavailable in /home). > > soft/interruptible: bad. Leads to large blocks of zeros in files. Interruptible is quite different from soft. Soft = bad ("emulates a broken hard drive".) Interruptible = GOOD! (avoids creating processes that are effectively unkillable until the NFS server comes back up.) Note that this doesn't lead to misread/miswritten files, unless that could normally result from a process being killed with an interrupt signal. man mount_nfs ... -i Make the mount interruptible, which implies that file system calls that are delayed due to an unresponsive server will fail with EINTR when a termination signal is posted for the process. ... > Spongy: > better. I've seen this referred to, but had not seen a system that actually implements spongy NFS mounts. Does FreeBSD-stable or -current actually support spongy mounts? Have I missed this in the docs? -- Clifton -- Clifton Royston -- LavaNet Systems Architect -- cliftonr@lava.net "If you ride fast enough, the Specialist can't catch you." "What's the Specialist?" Samantha says. "The Specialist wears a hat," says the babysitter. "The hat makes noises." She doesn't say anything else. Kelly Link, _The Specialist's Hat_ To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-cluster" in the body of the message From owner-freebsd-cluster Wed Dec 11 17:32:14 2002 Delivered-To: freebsd-cluster@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 94F4E37B401; Wed, 11 Dec 2002 17:32:12 -0800 (PST) Received: from malasada.lava.net (malasada.lava.net [64.65.64.17]) by mx1.FreeBSD.org (Postfix) with ESMTP id 290BA43EA9; Wed, 11 Dec 2002 17:32:12 -0800 (PST) (envelope-from cliftonr@lava.net) Received: from localhost (2545 bytes) by malasada.lava.net; Wed, 11 Dec 2002 15:32:10 -1000 (HST) via sendmail [stdio] id for Date: Wed, 11 Dec 2002 15:32:09 -1000 From: Clifton Royston To: Andy Sporner Cc: Mario Sergio Fujikawa Ferreira , freebsd-cluster@FreeBSD.org Subject: Re: sharing files within a cluster Message-ID: <20021211153209.B25854@lava.net> Mail-Followup-To: Andy Sporner , Mario Sergio Fujikawa Ferreira , freebsd-cluster@FreeBSD.org References: <200212101257.gBACvv609153@splat.grant.org> <3DF5EB88.9090409@nentec.de> <20021210145615.20975.qmail@exxodus.fedaykin.here> <3DF60768.7040407@nentec.de> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.2.5.1i In-Reply-To: <3DF60768.7040407@nentec.de>; from sporner@nentec.de on Tue, Dec 10, 2002 at 04:25:28PM +0100 Sender: owner-freebsd-cluster@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG On Tue, Dec 10, 2002 at 04:25:28PM +0100, Andy Sporner wrote: > > I might be saying something foolish but have any of you investigated > >GEOM? It might be possible to write an abstraction layer for it that will > >enable some of the necessary features: > > > > - localization transparency > > - distribution > > - disconnected mode > > - blabla... unfortunaly, FreeBSD based only as GEOM is only > > supported for FreeBSD for now > > > > > This is the main 'gotcha'. The secondary one is that this guy has > never replied to *any* of my emails and with all the difficulties > related to time, I simply don't have any more to deal with that as > well... :-( He's a busy guy too. Read the freebsd-current list if you want to see what's going on there; PHK is one of the frequent posters, as GEOM is now a fundamental component of FreeBSD 5.x. > I am trying to make this a *bsd neutral environment. This way everybody > wins. But thanks > for your opinion... I don't see other OSes necessarily adopting GEOM, even though it seems like a sound design. It's an "architectural-underpinnings" type of software, and nobody is going to switch to it until they start feeling that what they have is seriously broken or deficient. All IMHO. -- Clifton -- Clifton Royston -- LavaNet Systems Architect -- cliftonr@lava.net "If you ride fast enough, the Specialist can't catch you." "What's the Specialist?" Samantha says. "The Specialist wears a hat," says the babysitter. "The hat makes noises." She doesn't say anything else. Kelly Link, _The Specialist's Hat_ To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-cluster" in the body of the message From owner-freebsd-cluster Thu Dec 12 1:14:30 2002 Delivered-To: freebsd-cluster@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id ADD5837B401 for ; Thu, 12 Dec 2002 01:14:28 -0800 (PST) Received: from gate.nentec.de (gate2.nentec.de [194.25.215.66]) by mx1.FreeBSD.org (Postfix) with ESMTP id 1482943EC5 for ; Thu, 12 Dec 2002 01:14:27 -0800 (PST) (envelope-from sporner@nentec.de) Received: from nenny.nentec.de (root@nenny.nentec.de [153.92.64.1]) by gate.nentec.de (8.11.3/8.9.3) with ESMTP id gBC9EOP32005; Thu, 12 Dec 2002 10:14:24 +0100 Received: from nentec.de (andromeda.nentec.de [153.92.64.34]) by nenny.nentec.de (8.11.3/8.11.3) with ESMTP id gBC9EJt04291; Thu, 12 Dec 2002 10:14:19 +0100 Message-ID: <3DF8536B.6040908@nentec.de> Date: Thu, 12 Dec 2002 10:14:19 +0100 From: Andy Sporner User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.2a) Gecko/20020910 X-Accept-Language: en-us, en MIME-Version: 1.0 To: Mike Hoskins , freebsd-cluster Subject: Re: sharing files within a cluster References: <20021211115133.B82994-100000@fubar.adept.org> Content-Type: text/plain; charset=US-ASCII; format=flowed Content-Transfer-Encoding: 7bit X-Virus-Scanned: by AMaViS-perl11-milter (http://amavis.org/) Sender: owner-freebsd-cluster@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG Mike Hoskins wrote: >On Wed, 11 Dec 2002, Andy Sporner wrote: > > >>Sequent used a Numa concept on their new machines (before IBM killed >>them) and I am very familiar with this approach. This allowed standard >>applications to run without modifications (such as Oracle)... >> >> > >NUMA is nice... > >I'm still looking into IBM's work and the stuff at lustre.org. I'll >probably start hacking as soon as I understand the architecture a bit >better, but between work and my outdated coding skills I'm not expecting >anything to actually work. ;) > > I have lost track since I have been here. I had a friend who was looking to get me into Base OS Group at Sequent before I came here (Hey is Kevin Smallwood listening here???) But I have no idea since then. IMHO single image machines are more usefull to the majority of the people doing clustering. I wish we could all afford to be about to buy 1000 machines, but I think this is out of the question. The plan to publish a direction paper for a single image cluster is still on target for the end of the year. Andy To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-cluster" in the body of the message From owner-freebsd-cluster Thu Dec 12 1:50:24 2002 Delivered-To: freebsd-cluster@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id B770037B401 for ; Thu, 12 Dec 2002 01:50:21 -0800 (PST) Received: from gate.nentec.de (gate2.nentec.de [194.25.215.66]) by mx1.FreeBSD.org (Postfix) with ESMTP id 5BD6D43EC5 for ; Thu, 12 Dec 2002 01:50:20 -0800 (PST) (envelope-from sporner@nentec.de) Received: from nenny.nentec.de (root@nenny.nentec.de [153.92.64.1]) by gate.nentec.de (8.11.3/8.9.3) with ESMTP id gBC9oIP05092; Thu, 12 Dec 2002 10:50:18 +0100 Received: from nentec.de (andromeda.nentec.de [153.92.64.34]) by nenny.nentec.de (8.11.3/8.11.3) with ESMTP id gBC9oCt05933; Thu, 12 Dec 2002 10:50:13 +0100 Message-ID: <3DF85BD4.1050200@nentec.de> Date: Thu, 12 Dec 2002 10:50:12 +0100 From: Andy Sporner User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.2a) Gecko/20020910 X-Accept-Language: en-us, en MIME-Version: 1.0 To: Michael Grant Cc: freebsd-cluster@FreeBSD.ORG Subject: Re: sharing files within a cluster References: <200212112225.gBBMPi411534@splat.grant.org> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Virus-Scanned: by AMaViS-perl11-milter (http://amavis.org/) Sender: owner-freebsd-cluster@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG Michael Grant wrote: >Anyone know about this "single image linux cluster"? How's that done? >It seems to me that there would need to be something local to each >box, minimally /etc, no? > > Don't know much about the Linux single image stuff. I see dependancies on RPC's and I get scared... In a nutshell, what I am looking for in a single image is very NUMA like with the exception that instead of being a single image (one OS) it is a cooperative image(many kernels, one execution space). The only thing that is "single image" is the VM that each machine exports to the single image. I think it is possible architecturally to do this while minimizing the amount of work that has to be done by the master node. IE: when a process starts on a node, the PID is a multiple of the node number, same thing with file handles, sockets and such. >Andy, I agree with you that having a common process space and the >ability to migrate processies across machines would be a big win. If >that could be done on a freebsd cluster along with a single image, >that would definitely be a reason to use a freebsd cluster over a >linux cluster. >\ > Thanks! I agree as well. >Realistically, how close are you to this? > It's really a matter of motivation ;-) If I am having fun, the time is short, if I am not, things tend to take an eternity. I have my main fun at work developing high speed networking hardware and I have little time or tolerance for strange people--especially those who don't like to reply to email, where I could normally expect one from. I don't mean necessarily our own superstars. Lately I have come to the conclusion that organized computing has no future because of the influx of newbies (which are not bad--it's just now that if you are new on a list the veterans regard you as a newbie and this is a barrier to normal operation and cooperation). I make some offers only once and if there are no takers, I don't regard it as a worthwhile thing to do. I had intended to look into porting BPROC at some time, but it is now off the radar screen. (though not far enough off to learn how they swap processes ;-)) With that little flame out of the way ;-) Here is the rough list of things that must be accomplished to realize the goal. Naturally the more people working on it, the faster it goes because I only spend what spare time I have on this. Here is what has been done: 1. Cluster configuration and monitor (build 114) for failover. (supports distrubuted process table views. IE: there is NO master process table). Here is what must be done: 1. Front end load balancing (later this is the gateway for network based processes that have to move arround the cluster with their sockets.) Current plans are Dec 31 (will use initially IPFILTER). 2. The message infrastructure. I have so far studied BPROC and DRDB. I think it would be a weeks worth of work to make a messaging system that takes both requirements in account. I would like to use the embedded TCP stack that I wrote for my current project, but that's Intellectual Property :-( It's just the kind of thing we need (event driven with lot's of hooks for callouts). But there is access to the stack pretty good inside. Current Plans Jan 15. 3. Process swapping engine. I would like to extend the page swapping architecture (thank god FreeBSD has made this very modular!) I regard this as a better mechanism than what I have seen in for instance BPROC. That way shared memory can be supported too. This is the most difficult part and hard to put a time estimate on. It would be very helpful to team up with people who know well the VM architecture good. It could be as early as March or April for this one. 4. Much testing! That's I think a distributed affair so this can be very short perhaps. Assumptions: 1. Use of some sort of shared filesystem for the machines that are part of the domain since this would be a distributed image. 2. Networked based (sockets) must pass through a front-end device to be directed to the node that owns the process. By nature of the Hi-AV failover stuff this would NOT be a single point of failure. Executive Summary: probably early summer. But I think it would take that long anyways to port the other stuff that is also available. To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-cluster" in the body of the message From owner-freebsd-cluster Thu Dec 12 2: 0:45 2002 Delivered-To: freebsd-cluster@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 6348B37B401 for ; Thu, 12 Dec 2002 02:00:43 -0800 (PST) Received: from gate.nentec.de (gate2.nentec.de [194.25.215.66]) by mx1.FreeBSD.org (Postfix) with ESMTP id 151A643E4A for ; Thu, 12 Dec 2002 02:00:42 -0800 (PST) (envelope-from sporner@nentec.de) Received: from nenny.nentec.de (root@nenny.nentec.de [153.92.64.1]) by gate.nentec.de (8.11.3/8.9.3) with ESMTP id gBCA0cP05232; Thu, 12 Dec 2002 11:00:38 +0100 Received: from nentec.de (andromeda.nentec.de [153.92.64.34]) by nenny.nentec.de (8.11.3/8.11.3) with ESMTP id gBCA0Xt06433; Thu, 12 Dec 2002 11:00:34 +0100 Message-ID: <3DF85E41.7090202@nentec.de> Date: Thu, 12 Dec 2002 11:00:33 +0100 From: Andy Sporner User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.2a) Gecko/20020910 X-Accept-Language: en-us, en MIME-Version: 1.0 To: Clifton Royston , freebsd-cluster Subject: Re: sharing files within a cluster References: <200212101257.gBACvv609153@splat.grant.org> <3DF5EB88.9090409@nentec.de> <20021210145615.20975.qmail@exxodus.fedaykin.here> <3DF60768.7040407@nentec.de> <20021211153209.B25854@lava.net> Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit X-Virus-Scanned: by AMaViS-perl11-milter (http://amavis.org/) Sender: owner-freebsd-cluster@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG Clifton Royston wrote: >On Tue, Dec 10, 2002 at 04:25:28PM +0100, Andy Sporner wrote: > > >>> I might be saying something foolish but have any of you investigated >>>GEOM? It might be possible to write an abstraction layer for it that will >>>enable some of the necessary features: >>> >>> - localization transparency >>> - distribution >>> - disconnected mode >>> - blabla... unfortunaly, FreeBSD based only as GEOM is only >>> supported for FreeBSD for now >>> >>> >>> >>> >>This is the main 'gotcha'. The secondary one is that this guy has >>never replied to *any* of my emails and with all the difficulties >>related to time, I simply don't have any more to deal with that as >>well... :-( >> >> > > He's a busy guy too. Read the freebsd-current list if you want to >see what's going on there; PHK is one of the frequent posters, as GEOM >is now a fundamental component of FreeBSD 5.x. > > I can understand. It's hard sometimes to tell the difference between people who are genuinely busy or those that have depreciated list members because of some narrow interpretation of what somebody might have said. I don't proofread my emails and often there are fragments missing that would change the meaning of things ;-) Normally if people follow the complete conversation they get the right idea. > > >>I am trying to make this a *bsd neutral environment. This way everybody >>wins. But thanks >>for your opinion... >> >> > > I don't see other OSes necessarily adopting GEOM, even though it seems >like a sound design. It's an "architectural-underpinnings" type of >software, and nobody is going to switch to it until they start feeling >that what they have is seriously broken or deficient. All IMHO. > Though I might be speaking sacrilege here, I rather like the way the windows/NT has a uniform disk access layer. Though they sort of blew it on their "uniform" network layer. Abstraction layers always are nice to make platform independant things. But they have to be on all platforms in order to be a platform independant solution. As you say, it isn't. I suppose if Intel was to get behind it (like ACPIO) it would be a standard. With all of the historic in-fighting within the various BSD's this is a bigger chalange then to get a driver source adopted by Linus for the Linux Kernel. also IMHO. --Andy To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-cluster" in the body of the message From owner-freebsd-cluster Thu Dec 12 2:10:36 2002 Delivered-To: freebsd-cluster@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 8EFDE37B401 for ; Thu, 12 Dec 2002 02:10:35 -0800 (PST) Received: from gate.nentec.de (gate2.nentec.de [194.25.215.66]) by mx1.FreeBSD.org (Postfix) with ESMTP id 588E443EA9 for ; Thu, 12 Dec 2002 02:10:34 -0800 (PST) (envelope-from sporner@nentec.de) Received: from nenny.nentec.de (root@nenny.nentec.de [153.92.64.1]) by gate.nentec.de (8.11.3/8.9.3) with ESMTP id gBCAAWP10482 for ; Thu, 12 Dec 2002 11:10:32 +0100 Received: from nentec.de (andromeda.nentec.de [153.92.64.34]) by nenny.nentec.de (8.11.3/8.11.3) with ESMTP id gBCAARt06809 for ; Thu, 12 Dec 2002 11:10:27 +0100 Message-ID: <3DF86093.1080901@nentec.de> Date: Thu, 12 Dec 2002 11:10:27 +0100 From: Andy Sporner User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.2a) Gecko/20020910 X-Accept-Language: en-us, en MIME-Version: 1.0 To: freebsd-cluster Subject: Re: sharing files within a cluster References: <20021211133703.C83245-100000@fubar.adept.org> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Virus-Scanned: by AMaViS-perl11-milter (http://amavis.org/) Sender: owner-freebsd-cluster@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG This is a multi-part message in MIME format. To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-cluster" in the body of the message From owner-freebsd-cluster Thu Dec 12 2:12:36 2002 Delivered-To: freebsd-cluster@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 5E61237B401 for ; Thu, 12 Dec 2002 02:12:36 -0800 (PST) Received: from gate.nentec.de (gate2.nentec.de [194.25.215.66]) by mx1.FreeBSD.org (Postfix) with ESMTP id 29D1643E4A for ; Thu, 12 Dec 2002 02:12:35 -0800 (PST) (envelope-from sporner@nentec.de) Received: from nenny.nentec.de (root@nenny.nentec.de [153.92.64.1]) by gate.nentec.de (8.11.3/8.9.3) with ESMTP id gBCACWP10494 for ; Thu, 12 Dec 2002 11:12:33 +0100 Received: from nentec.de (andromeda.nentec.de [153.92.64.34]) by nenny.nentec.de (8.11.3/8.11.3) with ESMTP id gBCACSt06863 for ; Thu, 12 Dec 2002 11:12:28 +0100 Message-ID: <3DF8610C.8050703@nentec.de> Date: Thu, 12 Dec 2002 11:12:28 +0100 From: Andy Sporner User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.2a) Gecko/20020910 X-Accept-Language: en-us, en MIME-Version: 1.0 To: freebsd-cluster Subject: Boot bios and such... Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Virus-Scanned: by AMaViS-perl11-milter (http://amavis.org/) Sender: owner-freebsd-cluster@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG This is a multi-part message in MIME format. To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-cluster" in the body of the message From owner-freebsd-cluster Thu Dec 12 2:16: 8 2002 Delivered-To: freebsd-cluster@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 58F6137B401 for ; Thu, 12 Dec 2002 02:16:07 -0800 (PST) Received: from gate.nentec.de (gate2.nentec.de [194.25.215.66]) by mx1.FreeBSD.org (Postfix) with ESMTP id 1927F43ED4 for ; Thu, 12 Dec 2002 02:16:06 -0800 (PST) (envelope-from sporner@nentec.de) Received: from nenny.nentec.de (root@nenny.nentec.de [153.92.64.1]) by gate.nentec.de (8.11.3/8.9.3) with ESMTP id gBCAG4P10569 for ; Thu, 12 Dec 2002 11:16:04 +0100 Received: from nentec.de (andromeda.nentec.de [153.92.64.34]) by nenny.nentec.de (8.11.3/8.11.3) with ESMTP id gBCAFxt07117 for ; Thu, 12 Dec 2002 11:15:59 +0100 Message-ID: <3DF861DF.9000704@nentec.de> Date: Thu, 12 Dec 2002 11:15:59 +0100 From: Andy Sporner User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.2a) Gecko/20020910 X-Accept-Language: en-us, en MIME-Version: 1.0 Cc: freebsd-cluster@FreeBSD.ORG Subject: Re: sharing files within a cluster References: <20021211133703.C83245-100000@fubar.adept.org> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Virus-Scanned: by AMaViS-perl11-milter (http://amavis.org/) Sender: owner-freebsd-cluster@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG Mike Hoskins wrote: >On Wed, 11 Dec 2002, Greg Lewis wrote: > > >>Apart from that, consider that LinuxBIOS is in use on production clusters >>right now and is responsible for several million dollars of Linux cluster >>sales this year. The relatively simple task of making it boot FreeBSD >>should take a relatively short time (for someone with the know how) and >>would be a helpful boost to FreeBSD's "clusterability". >> >> > >Sure would be nice if Linux wasn't in the name. Maybe it's just me, but >it'd be nice of some "open" things were actually more "open." > > >Ahh, OpenBIOS... Time to work on a new "port" or version I guess. >Something that any OS vendor/user can get behind and implement without >being associated with Linux. (Yes, some of us have had very bad >encounters with Linux and do anything to avoid it.) > > > I think this is called PXE. Pre-XEcution boot environment from Intel. When you select Network Boot, it hits a DHCP server for an IP address and then a PXE server for a floppy image. Then it kicks off the image just like a FD. The sizes can be any standard size and can even me larger, though I think this could run into problems with some systems. I know for sure that Linux has a problem this way because the code used by Intel to do the networking stuff lies right on top of where the second stage loader did it's thing. The end result is a crash on boot. This was a few years ago so perhaps by now it is fixed. Most modern servers have this feature and for those who have an aversion to certain names this is a solution. If one does not like Intel, well then I guess there is a bigger problem To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-cluster" in the body of the message From owner-freebsd-cluster Thu Dec 12 2:17:53 2002 Delivered-To: freebsd-cluster@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 7B64C37B401 for ; Thu, 12 Dec 2002 02:17:52 -0800 (PST) Received: from gate.nentec.de (gate2.nentec.de [194.25.215.66]) by mx1.FreeBSD.org (Postfix) with ESMTP id 4559743EC5 for ; Thu, 12 Dec 2002 02:17:51 -0800 (PST) (envelope-from sporner@nentec.de) Received: from nenny.nentec.de (root@nenny.nentec.de [153.92.64.1]) by gate.nentec.de (8.11.3/8.9.3) with ESMTP id gBCAHnP10582 for ; Thu, 12 Dec 2002 11:17:49 +0100 Received: from nentec.de (andromeda.nentec.de [153.92.64.34]) by nenny.nentec.de (8.11.3/8.11.3) with ESMTP id gBCAHit07169 for ; Thu, 12 Dec 2002 11:17:44 +0100 Message-ID: <3DF86248.3050603@nentec.de> Date: Thu, 12 Dec 2002 11:17:44 +0100 From: Andy Sporner User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.2a) Gecko/20020910 X-Accept-Language: en-us, en MIME-Version: 1.0 To: freebsd-cluster Subject: Please disregard messages with heading "Bios boot and such" from me Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit X-Virus-Scanned: by AMaViS-perl11-milter (http://amavis.org/) Sender: owner-freebsd-cluster@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG I had some difficulty trying to convince Mozzilla just to send my replies inline rather than as an attachment! :-) Thanks again Andy To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-cluster" in the body of the message From owner-freebsd-cluster Thu Dec 12 7:58:20 2002 Delivered-To: freebsd-cluster@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 9CB9937B401 for ; Thu, 12 Dec 2002 07:58:19 -0800 (PST) Received: from mailrelay1.lanl.gov (mailrelay1.lanl.gov [128.165.4.101]) by mx1.FreeBSD.org (Postfix) with ESMTP id DC3BF43EB2 for ; Thu, 12 Dec 2002 07:58:18 -0800 (PST) (envelope-from rminnich@lanl.gov) Received: from ccs.lanl.gov (localhost.localdomain [127.0.0.1]) by mailrelay1.lanl.gov (8.12.3/8.12.3/(ccn-5)) with SMTP id gBCFwH9i025576 for ; Thu, 12 Dec 2002 08:58:17 -0700 Received: (qmail 26993 invoked from network); 12 Dec 2002 08:58:17 -0700 Received: from unknown (HELO carotid.ccs.lanl.gov) (128.165.148.162) by 128.165.148.1 with SMTP; 12 Dec 2002 08:58:17 -0700 Received: (qmail 17500 invoked by uid 3499); 12 Dec 2002 08:58:17 -0700 Received: from localhost (sendmail-bs@127.0.0.1) by localhost with SMTP; 12 Dec 2002 08:58:17 -0700 Date: Thu, 12 Dec 2002 08:58:17 -0700 (MST) From: "Ronald G. Minnich" X-X-Sender: rminnich@carotid.ccs.lanl.gov To: Andy Sporner Cc: Michael Grant , Subject: Re: sharing files within a cluster In-Reply-To: <3DF85BD4.1050200@nentec.de> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-freebsd-cluster@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG On Thu, 12 Dec 2002, Andy Sporner wrote: > Don't know much about the Linux single image stuff. I see dependancies > on RPC's and I get scared... best bet: - read and learn - then get scared, but only if necessary. Your comment, to me, makes not a lot of sense. No disrespect intended, this is a great bunch of people and freebsd is a great os, but we're plowing a lot of already-planted ground here. ron To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-cluster" in the body of the message From owner-freebsd-cluster Thu Dec 12 7:59:47 2002 Delivered-To: freebsd-cluster@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 516FD37B401 for ; Thu, 12 Dec 2002 07:59:46 -0800 (PST) Received: from mailrelay1.lanl.gov (mailrelay1.lanl.gov [128.165.4.101]) by mx1.FreeBSD.org (Postfix) with ESMTP id A23FF43EA9 for ; Thu, 12 Dec 2002 07:59:45 -0800 (PST) (envelope-from rminnich@lanl.gov) Received: from ccs.lanl.gov (localhost.localdomain [127.0.0.1]) by mailrelay1.lanl.gov (8.12.3/8.12.3/(ccn-5)) with SMTP id gBCFxj9i026218 for ; Thu, 12 Dec 2002 08:59:45 -0700 Received: (qmail 27012 invoked from network); 12 Dec 2002 08:59:44 -0700 Received: from unknown (HELO carotid.ccs.lanl.gov) (128.165.148.162) by 128.165.148.1 with SMTP; 12 Dec 2002 08:59:44 -0700 Received: (qmail 17505 invoked by uid 3499); 12 Dec 2002 08:59:44 -0700 Received: from localhost (sendmail-bs@127.0.0.1) by localhost with SMTP; 12 Dec 2002 08:59:44 -0700 Date: Thu, 12 Dec 2002 08:59:44 -0700 (MST) From: "Ronald G. Minnich" X-X-Sender: rminnich@carotid.ccs.lanl.gov To: Andy Sporner Cc: freebsd-cluster@FreeBSD.ORG Subject: Re: sharing files within a cluster In-Reply-To: <3DF861DF.9000704@nentec.de> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-freebsd-cluster@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG On Thu, 12 Dec 2002, Andy Sporner wrote: > I think this is called PXE. Pre-XEcution boot environment from Intel. > When you select Network Boot, it hits a DHCP server for an IP address > and then a PXE server for a floppy image. Then it kicks off the image > just like a FD. The systems I've used have a 32KB limit. So you use PXE to boot the thing that boots the thing. It's stupid. Also can't boot over things it doesn't know, e.g. myrinet. We boot over myrinet. ron To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-cluster" in the body of the message From owner-freebsd-cluster Thu Dec 12 8: 7: 1 2002 Delivered-To: freebsd-cluster@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 68FA537B401 for ; Thu, 12 Dec 2002 08:07:00 -0800 (PST) Received: from gate.nentec.de (gate2.nentec.de [194.25.215.66]) by mx1.FreeBSD.org (Postfix) with ESMTP id ABA0543EA9 for ; Thu, 12 Dec 2002 08:06:55 -0800 (PST) (envelope-from sporner@nentec.de) Received: from nenny.nentec.de (root@nenny.nentec.de [153.92.64.1]) by gate.nentec.de (8.11.3/8.9.3) with ESMTP id gBCG6qP11167; Thu, 12 Dec 2002 17:06:52 +0100 Received: from nentec.de (andromeda.nentec.de [153.92.64.34]) by nenny.nentec.de (8.11.3/8.11.3) with ESMTP id gBCG6lt32414; Thu, 12 Dec 2002 17:06:47 +0100 Message-ID: <3DF8B417.9040509@nentec.de> Date: Thu, 12 Dec 2002 17:06:47 +0100 From: Andy Sporner User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.2a) Gecko/20020910 X-Accept-Language: en-us, en MIME-Version: 1.0 To: "Ronald G. Minnich" , freebsd-cluster@freebsd.org Subject: Re: sharing files within a cluster References: Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Virus-Scanned: by AMaViS-perl11-milter (http://amavis.org/) Sender: owner-freebsd-cluster@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG Ronald G. Minnich wrote: >On Thu, 12 Dec 2002, Andy Sporner wrote: > > > >>I think this is called PXE. Pre-XEcution boot environment from Intel. >>When you select Network Boot, it hits a DHCP server for an IP address >>and then a PXE server for a floppy image. Then it kicks off the image >>just like a FD. >> >> > >The systems I've used have a 32KB limit. So you use PXE to boot the thing >that boots the thing. It's stupid. > >Also can't boot over things it doesn't know, e.g. myrinet. We boot over >myrinet. > > I am not sure about the rest of the folks here, but a little civility is *ALWAYS* a good thing. Most "real" servers this works perfectly. I know because I was involved in building them! My suggestion to you is that if you can't say something constructive is that you should say nothing. That's the problem with academic people they just don't know when to just keep quiet. Out of the last 10 posts of yours most of them were negative -- LIGHTEN UP! Thing *were* a lot more civil here! -Andy To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-cluster" in the body of the message From owner-freebsd-cluster Thu Dec 12 9:44:35 2002 Delivered-To: freebsd-cluster@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 517D937B401 for ; Thu, 12 Dec 2002 09:44:33 -0800 (PST) Received: from mgr3.xmission.com (mgr3.xmission.com [198.60.22.203]) by mx1.FreeBSD.org (Postfix) with ESMTP id DE41943E4A for ; Thu, 12 Dec 2002 09:44:31 -0800 (PST) (envelope-from glewis@eyesbeyond.com) Received: from mail by mgr3.xmission.com with spam-scanned (Exim 3.35 #1) id 18MXNv-0005FB-03; Thu, 12 Dec 2002 10:44:31 -0700 Received: from [207.135.128.145] (helo=misty.eyesbeyond.com) by mgr3.xmission.com with esmtp (Exim 3.35 #1) id 18MXNt-0005Ef-03; Thu, 12 Dec 2002 10:44:30 -0700 Received: (from glewis@localhost) by misty.eyesbeyond.com (8.11.6/8.11.6) id gBCHiO819072; Fri, 13 Dec 2002 04:14:24 +1030 (CST) (envelope-from glewis@eyesbeyond.com) X-Authentication-Warning: misty.eyesbeyond.com: glewis set sender to glewis@eyesbeyond.com using -f Date: Fri, 13 Dec 2002 04:14:24 +1030 From: Greg Lewis To: Andy Sporner Cc: "Ronald G. Minnich" , freebsd-cluster@FreeBSD.ORG Subject: Re: sharing files within a cluster Message-ID: <20021213041424.A18994@misty.eyesbeyond.com> References: <3DF8B417.9040509@nentec.de> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.2.5i In-Reply-To: <3DF8B417.9040509@nentec.de>; from sporner@nentec.de on Thu, Dec 12, 2002 at 05:06:47PM +0100 X-Spam-Status: No, hits=-3.2 required=8.0 tests=IN_REP_TO,QUOTED_EMAIL_TEXT,REFERENCES, SIGNATURE_SHORT_DENSE,SPAM_PHRASE_03_05,USER_AGENT, USER_AGENT_MUTT,X_AUTH_WARNING version=2.43 X-Spam-Level: Sender: owner-freebsd-cluster@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG On Thu, Dec 12, 2002 at 05:06:47PM +0100, Andy Sporner wrote: > Ronald G. Minnich wrote: > >On Thu, 12 Dec 2002, Andy Sporner wrote: > >>I think this is called PXE. Pre-XEcution boot environment from Intel. > >>When you select Network Boot, it hits a DHCP server for an IP address > >>and then a PXE server for a floppy image. Then it kicks off the image > >>just like a FD. > >> > > > >The systems I've used have a 32KB limit. So you use PXE to boot the thing > >that boots the thing. It's stupid. > > > >Also can't boot over things it doesn't know, e.g. myrinet. We boot over > >myrinet. > > Most "real" servers this works perfectly. I know because > I was involved in building them! > > My suggestion to you is that if you can't say something > constructive is that you should say nothing. That's the > problem with academic people they just don't know when > to just keep quiet. Ron is making a real world point though, so his posting is both constructive and relevant. His postings may be brief, but they are to the point and draw on a lot of experience with HPC clusters. In this case he is spot on. PXE is useful, but limited. PXE clients that I've seen put arbitrary limits on both the size of the image and the download time. I've seen at least one PXE client that will time out _in the middle of downloading the image_. Thats right, its getting the image but it will still time out. So, we use PXE to download etherboot and boot into that and then use that to do the actual booting since it doesn't have these limitations. As Ron points out, this is stupid. PXE also has no facilities for multicast that I'm aware of so you have to tftp your image. The etherboot client we have has a multicast client built in, another advantage on big clusters where tftp doesn't scale. As a result we have a preference for NICs with etherboot flashed onto the ROM, not PXE NICs. So, yes PXE is useful. But get a big enough cluster and it falls in a screaming heap. -- Greg Lewis Email : glewis@eyesbeyond.com Eyes Beyond Web : http://www.eyesbeyond.com Information Technology FreeBSD : glewis@FreeBSD.org To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-cluster" in the body of the message From owner-freebsd-cluster Thu Dec 12 9:50:52 2002 Delivered-To: freebsd-cluster@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 7CFF437B401 for ; Thu, 12 Dec 2002 09:50:50 -0800 (PST) Received: from mailrelay1.lanl.gov (mailrelay1.lanl.gov [128.165.4.101]) by mx1.FreeBSD.org (Postfix) with ESMTP id 689D543ED8 for ; Thu, 12 Dec 2002 09:50:49 -0800 (PST) (envelope-from rminnich@lanl.gov) Received: from ccs.lanl.gov (localhost.localdomain [127.0.0.1]) by mailrelay1.lanl.gov (8.12.3/8.12.3/(ccn-5)) with SMTP id gBCHom9i013359 for ; Thu, 12 Dec 2002 10:50:48 -0700 Received: (qmail 28159 invoked from network); 12 Dec 2002 10:50:48 -0700 Received: from unknown (HELO carotid.ccs.lanl.gov) (128.165.148.162) by 128.165.148.1 with SMTP; 12 Dec 2002 10:50:48 -0700 Received: (qmail 18331 invoked by uid 3499); 12 Dec 2002 10:50:48 -0700 Received: from localhost (sendmail-bs@127.0.0.1) by localhost with SMTP; 12 Dec 2002 10:50:48 -0700 Date: Thu, 12 Dec 2002 10:50:48 -0700 (MST) From: "Ronald G. Minnich" X-X-Sender: rminnich@carotid.ccs.lanl.gov To: Andy Sporner Cc: freebsd-cluster@FreeBSD.ORG Subject: Re: sharing files within a cluster In-Reply-To: <3DF8B537.70908@nentec.de> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-freebsd-cluster@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG On Thu, 12 Dec 2002, Andy Sporner wrote: > Well let's review here. RPC's have many dependencies for correct > operation (portmap, etc) regular socket I/O does not--so which one is > more reliable??? Actually, that's SunRPC, one variation of RPC, that requires all that stuff. RPC is a very simple concept, and what you typically see is people implementing things that are actually RPC but calling it socket I/O. Pretty much any protocol running over sockets that does a request with a "procedure number" and expects a response to that request is RPC. You can even do async RPC, sources on my web page to a modified SunRPC that does this. Performance is pretty good to lots of nodes ... works on FreeBSD, or used to. My apologies, the "linuxbios naming" thing really teed me off. I'm also sorry to be so darned tactless, a lot of those messages I was just pounding out between Pink bringing firefighting or early in the morning when I should sleep instead of write email. I apologize to this list. > Your "read and learn" comment is just the kind of pompousity I expect > from the academic world. I gotta watch that. I'm being terse and I guess I should not be. Esp. since I have not been in the academic world for 14 years, I've been building HPC systems for various gov't and commercial entities instead. I'm getting really frustrated with this list, however, since people seem to be refusing to learn from linux because it's "evil", and at the same time proposing stuff that's been tried years ago on linux and found not to work. I think it is really important that the FreeBSD community break out into something new and really neat. At a minimum, however, I'd like to see FreeBSD have equal capability to Linux clustering for HPC. And I think before we condemn technologies ("RPC") it's best to make sure it's the technology, and not a specific implementation of it, that are the issue. I would really encourage people to get at the literature of SSI, going back to Farber's original DCS paper in '72, and try to find an angle that is somehow new and uniquely suited to FreeBSD. And "Linux sucks, FreeBSD rules" is not going to be the thing that does it. What about FreeBSD can somehow make it much better for clustering? That's the problem I've never been able to answer -- maybe one of you can. This list is sporadically active, but there have not been tons of new ideas crossing it -- seems like we see the same stuff over and over again. > If this was such well plowed ground, why does it not already exists in > FreeBSD? Because, and as a former FreeBSD cluster builder I hate to say this, Linux won. And the mindshare is in Linux, as are the compilers, 3rd party apps, and all the big vendors. There are many 10s of millions of dollars of clustering money being spent by IBM, HP, Intel, etc., etc. and absent something really new and innovative I don't see FreeBSD breaking in. The closed-minded nature of the FreeBSD core has not helped. I wrote a DSM for FreeBSD 2.0.5 ca. 1994, which was based on simple mods to NFS -- called MNFS (really it was a port from a SunOS version I also wrote). It required a simple extension to the VM system (it required the VM layer to tell the VFS layer if a page fault was for read or read/write -- that's what SunOS does) and I could not get the FreeBSD core to add this simple additional parameter -- it violated "information hiding between layers", which is good in theory but in practice the more info the better. (this code is still on my web page along with the MNFS papers). 8 years ago both Sandia/Livermore and I were building freebsd clusters. It's arguably the better OS. Is it so much better or different from Linux that it is inherently better for clustering? Probably not. So we got clobbered by the rest of the world. What we got steamrolled by was the availability of so many 3rd part apps. > As for comments to you, I think it is pretty clear that when I made an offer > to port the very thing you asked for a reply is in order. I'm glad to hear it, sorry for lack of reply. Please get the clustermatic ISO at www.clustermatic.org and then we can talk about porting. I would LOVE to see this happen. You can find an initial private name space VFS implementation for FreeBSD, called v9fs, I think on my web page. Sorry, Andy. I'll try to behave. ron To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-cluster" in the body of the message From owner-freebsd-cluster Thu Dec 12 10:54:12 2002 Delivered-To: freebsd-cluster@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 3294537B401 for ; Thu, 12 Dec 2002 10:54:11 -0800 (PST) Received: from vps.vitalit.com (vps.vitalit.com [64.105.194.69]) by mx1.FreeBSD.org (Postfix) with ESMTP id 7AD7B43EC2 for ; Thu, 12 Dec 2002 10:54:10 -0800 (PST) (envelope-from car@vitalit.com) Received: from LAPTOP (gso167-138-145.triad.rr.com [24.167.138.145]) by vps.vitalit.com (8.12.3/8.12.3) with SMTP id gBCIs5hQ089575 for ; Thu, 12 Dec 2002 13:54:06 -0500 (EST) (envelope-from car@vitalit.com) Message-ID: <003d01c2a20f$b2e35fd0$0301000a@LAPTOP> From: "Rouzer, Charles A (Chuck)" To: References: <3DF8B417.9040509@nentec.de> Subject: Re: sharing files within a cluster Date: Thu, 12 Dec 2002 13:52:55 -0500 MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2720.3000 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2600.0000 Sender: owner-freebsd-cluster@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG I appreciate Rons comments and you should too. More often than not you will be ignored for not investigating/researching the current available solutions prior to commenting on them, asking questions, or related discussions. This is often why so many of the newbie questions go unanswered. In our case, no one cares to waste time treading in the same old water or explaining to someone what is currently available when a search engine could do just as good. Just my $0.02. Chuck > I am not sure about the rest of the folks here, but a little > civility is *ALWAYS* a good thing. > > Most "real" servers this works perfectly. I know because > I was involved in building them! > > My suggestion to you is that if you can't say something > constructive is that you should say nothing. That's the > problem with academic people they just don't know when > to just keep quiet. > > Out of the last 10 posts of yours most of them were negative > > -- LIGHTEN UP! Thing *were* a lot more civil here! > > > -Andy > > > > To Unsubscribe: send mail to majordomo@FreeBSD.org > with "unsubscribe freebsd-cluster" in the body of the message > To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-cluster" in the body of the message From owner-freebsd-cluster Thu Dec 12 12: 3:56 2002 Delivered-To: freebsd-cluster@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 19E7837B401 for ; Thu, 12 Dec 2002 12:03:56 -0800 (PST) Received: from vps.vitalit.com (vps.vitalit.com [64.105.194.69]) by mx1.FreeBSD.org (Postfix) with ESMTP id 1573443EA9 for ; Thu, 12 Dec 2002 12:03:55 -0800 (PST) (envelope-from car@vitalit.com) Received: from LAPTOP (gso167-138-145.triad.rr.com [24.167.138.145]) by vps.vitalit.com (8.12.3/8.12.3) with SMTP id gBCK3rhQ092017 for ; Thu, 12 Dec 2002 15:03:53 -0500 (EST) (envelope-from car@vitalit.com) Message-ID: <006c01c2a219$72938db0$0301000a@LAPTOP> From: "Rouzer, Charles A (Chuck)" To: Subject: IRC anyone? Date: Thu, 12 Dec 2002 15:02:43 -0500 MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2720.3000 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2600.0000 Sender: owner-freebsd-cluster@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG I'm just making a general inquiry if anyone is interested in creating a freebsd-cluster IRC channel to hash out ideas in real-time. It works well for some projects. I'll assume we could use the freenode network (www.freenode.net). This doesn't work for everyone time wise, but its great for sessions (scheduled or otherwise). One project I am following is GNUenterprise.org which uses IRC heavily and goes so far to provide kernel cousins from IRC logs. -- Chuck Rouzer Vital Information Technology, Inc. http://www.vitalit.com http://www.vitalserver.com To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-cluster" in the body of the message From owner-freebsd-cluster Thu Dec 12 12:39:33 2002 Delivered-To: freebsd-cluster@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id D345237B401 for ; Thu, 12 Dec 2002 12:39:31 -0800 (PST) Received: from vps.vitalit.com (vps.vitalit.com [64.105.194.69]) by mx1.FreeBSD.org (Postfix) with ESMTP id 287EE43ED1 for ; Thu, 12 Dec 2002 12:39:31 -0800 (PST) (envelope-from car@vitalit.com) Received: from LAPTOP (gso167-138-145.triad.rr.com [24.167.138.145]) by vps.vitalit.com (8.12.3/8.12.3) with SMTP id gBCKdOhQ093237 for ; Thu, 12 Dec 2002 15:39:25 -0500 (EST) (envelope-from car@vitalit.com) Message-ID: <007b01c2a21e$68b360e0$0301000a@LAPTOP> From: "Rouzer, Charles A (Chuck)" To: References: <006c01c2a219$72938db0$0301000a@LAPTOP> Subject: Re: IRC anyone? Date: Thu, 12 Dec 2002 15:38:20 -0500 MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2720.3000 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2600.0000 Sender: owner-freebsd-cluster@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG Well I'll idle in #freebsd-cluster on freenode.net network if anyone is interested. I usually connect to whatever irc.debian.org is aliased to. Chuck ----- Original Message ----- From: "Rouzer, Charles A (Chuck)" To: Sent: Thursday, December 12, 2002 3:02 PM Subject: IRC anyone? > > I'm just making a general inquiry if anyone is interested in > creating a freebsd-cluster IRC channel to hash out ideas in real-time. It > works well for some projects. I'll assume we could use the freenode network > (www.freenode.net). This doesn't work for everyone time wise, but its great > for sessions (scheduled or otherwise). One project I am following is > GNUenterprise.org which uses IRC heavily and goes so far to provide kernel > cousins from IRC logs. > > -- > Chuck Rouzer > Vital Information Technology, Inc. > http://www.vitalit.com > http://www.vitalserver.com > > > To Unsubscribe: send mail to majordomo@FreeBSD.org > with "unsubscribe freebsd-cluster" in the body of the message > To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-cluster" in the body of the message From owner-freebsd-cluster Thu Dec 12 14:19:43 2002 Delivered-To: freebsd-cluster@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 4010837B401 for ; Thu, 12 Dec 2002 14:19:39 -0800 (PST) Received: from fubar.adept.org (fubar.adept.org [63.147.172.249]) by mx1.FreeBSD.org (Postfix) with ESMTP id BEF8E43EA9 for ; Thu, 12 Dec 2002 14:19:38 -0800 (PST) (envelope-from mike@adept.org) Received: by fubar.adept.org (Postfix, from userid 1001) id 0B93815247; Thu, 12 Dec 2002 14:19:33 -0800 (PST) Received: from localhost (localhost [127.0.0.1]) by fubar.adept.org (Postfix) with ESMTP id 08F7F15213 for ; Thu, 12 Dec 2002 14:19:33 -0800 (PST) Date: Thu, 12 Dec 2002 14:19:33 -0800 (PST) From: Mike Hoskins To: freebsd-cluster@FreeBSD.ORG Subject: (long, warm & fuzzy) Re: sharing files within a cluster In-Reply-To: Message-ID: <20021212133117.D2430-100000@fubar.adept.org> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-freebsd-cluster@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG On Thu, 12 Dec 2002, Ronald G. Minnich wrote: > My apologies, the "linuxbios naming" thing really teed me off. Please understand that this state of "teed off" is, for the most part, what BSD advocates feel like all the time... Linux has advantages (at least it's not M$!), and it's gained mass appeal... But "mass appeal" didn't cause technical people to love M$... So some of us have the same sort of "you've only won because the people with the money made the decission and not the people with the engineering degrees" mentality. At least I do. I'm just trying to give some personal background... To (hopefully) help you understand that (like you) I can often post heated arguments to mailing lists when I should just get up and pass out or drink more coffee. :) So please don't take things personally. I have strong opinions, and the almost 50/50 German/Irish blood in my veins doesn't typically promote good discussion/interaction skills. Trust me, I pay for my temper and impatience every day. ;) > I'm also > sorry to be so darned tactless, a lot of those messages I was just > pounding out between Pink bringing firefighting or early in the morning > when I should sleep instead of write email. I apologize to this list. And I'm just another dot-com guy that hasn't lost his job yet (sometimes wish I would, but not really when there's bills to pay)... So we're all posting in states of relatively-high mental stress. :) Don't apologize for tactlessness, I honestly think we could use a lot less tact in the world. For the most part, yell and shout at me, do whatever you need to do to convince me of something if you know (or believe) it's right. You have my word I won't get offended. > > Your "read and learn" comment is just the kind of pompousity I expect > > from the academic world. > I gotta watch that. I'm being terse and I guess I should not be. We all do. I apologize for any unfounded terseness on my part. (Trust me, search the archives, you'll find plenty of it.) That said, I know (I don't doubt for a minute) that we all want the same thing... A robust, reliable, working solution for *BSD. (My dream would be to help all the BSDs, not just Free*... But from your page it looks like Free is lagging the farthest behind wrt PIRQ.) Unfortuneately, I sometimes feel like a technical posistion in today's economic environment can be more taxing than ever... But we have to be careful not to start viewing each other as enemies... Otherwise, who do we have as friends? > since I have not been in the academic world for 14 years, I've been > building HPC systems for various gov't and commercial entities instead. You obviouslly have the most experience among us, so I thank you for bearing with my shortcomings and continuing to keep and open mind, offer assistance, etc. > I would really encourage people to get at the literature of SSI, going > back to Farber's original DCS paper in '72, and try to find an angle that > is somehow new and uniquely suited to FreeBSD. I think some of the rendering farms that have been BSD based, while possibly not showing an area uniquely suited to BSD, do clearly show that there would be plenty of demand for BSD clusters. I remember when Linux wasn't so popular. (Anyone else spend all night getting the chat script just right so you could dialup in Slackware 1.0?) When I got my for Unix admin job, I was almost laughed out of the building when I revealed I used Slackware at home and not some BSD... When it did start gaining momentum, a lot of Linux folks (friends, coworkers, IRC channels, mailing lists, etc.) adopted the mentality "we're winning, because we have more users" -- most *BSD folks, however, adopted the much more founded (IMO!) "uh, windows has lots of users too, with lots of money behind it... that doesn't mean it's great" train of thought. Competition is a funny thing... I think we all hate M$ ("butchering standards for fun and profit")... But when it comes to working together as a community (the opensource community, let's say)... We instead focus on our differences. And, most certainly, I am no exception. Some of these biases (Linux folks against BSD, BSD against Linux, Free against Open, RH against Debian, etc.) have formed based upon real world experience... so it's hard to get the sort of mindshare we really need across camps. I don't know if there's an easy solution, because it really just boils down to everyone swallowing their pride... and that's something that takes human beings awhile. > And "Linux sucks, FreeBSD > rules" is not going to be the thing that does it. What about FreeBSD can > somehow make it much better for clustering? That's the problem I've never > been able to answer -- maybe one of you can. This could maybe be reduced to "What makes FreeBSD better than Linux?" That's a loaded question, and will certainly provide very different answers based upon whom you ask. For me, just being able to use FreeBSD... The OS that years of admin work have taught me to trust... is enough of an advantage for me to join this list rather than using existing solutions. > This list is sporadically active, but there have not been tons of new > ideas crossing it -- seems like we see the same stuff over and over again. I don't think that's necessarily as bad as it may sound. For the most part, humans are prone to error... And discussion and peer review is rarely bad in technical circles. > Because, and as a former FreeBSD cluster builder I hate to say this, Linux > won. And the mindshare is in Linux, as are the compilers, 3rd party apps, > and all the big vendors. There are many 10s of millions of dollars of > clustering money being spent by IBM, HP, Intel, etc., etc. and absent > something really new and innovative I don't see FreeBSD breaking in. The same was said for M$ when Linux was born. :) It's funny that many of the arguments that are now made for Linux have historically been made for Windows. I really hope M$ doesn't decide to muck around in the Linux world as they've implied... They could only make things worse. (Granted, there would be lots of money invested.) > The closed-minded nature of the FreeBSD core has not helped. I wrote a DSM > for FreeBSD 2.0.5 ca. 1994, which was based on simple mods to NFS -- > called MNFS (really it was a port from a SunOS version I also wrote). It > required a simple extension to the VM system (it required the VM layer to > tell the VFS layer if a page fault was for read or read/write -- that's > what SunOS does) and I could not get the FreeBSD core to add this simple > additional parameter -- it violated "information hiding between layers", > which is good in theory but in practice the more info the better. (this > code is still on my web page along with the MNFS papers). You call them close-minded, I call them the guardians of my personal interests. A simple VM change to you, if found so objectionable to an entire team of developers (who obviouslly maintain a great OS), should at least me more thoroughly reviewed IMCO. I am not saying there were any technical (or other) problems with your extension, but I think the core team has to be bullish. If your extension ends up causing other people trouble... Fingers will point at core. (That's why they're there.) This more structured development process is just one of the many reasons I've migrated to BSD and not Linux. Numerous driver, kernel, VM, etc. issues have bitten me in the past on various Linux platforms. Most people I know that deploy Linux servers, then migrate to some BSD... Typically say it's due to stablity. Performance, for the most part, has greatly equalized. (Assuming gurus from both sides tune the respective machines, something many past tests made obvious was not the case.) > clobbered by the rest of the world. What we got steamrolled by was the > availability of so many 3rd part apps. That's a Windows argument if I've ever heard one. Not sure how we can combat that, other than selling out (yes, I view it that way...) like Linux has. (More users, more patches... Not necessarily more clue or working patches, ala RedHat, but patches nontheless!) > Sorry, Andy. I'll try to behave. I'll do the same. It's often hard to talk about things calmly and rationaly, even when you know you should, when you've been personally affected by related issues. Now I just have to go explain to my Linux-using friends why I'm runing LinuxBIOS if BSD's so great. ;) -- Mike Hoskins This message is RFC 1855 compliant, mike@adept.org www.adept.org/pub/rfcs/rfc1855.html To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-cluster" in the body of the message From owner-freebsd-cluster Thu Dec 12 21:23:58 2002 Delivered-To: freebsd-cluster@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 1B5BF37B401 for ; Thu, 12 Dec 2002 21:23:57 -0800 (PST) Received: from mailrelay2.lanl.gov (mailrelay2.lanl.gov [128.165.4.103]) by mx1.FreeBSD.org (Postfix) with ESMTP id C03B443ED8 for ; Thu, 12 Dec 2002 21:23:54 -0800 (PST) (envelope-from rminnich@lanl.gov) Received: from ccs.lanl.gov (localhost.localdomain [127.0.0.1]) by mailrelay2.lanl.gov (8.12.3/8.12.3/(ccn-5)) with SMTP id gBD5Nsgp010543 for ; Thu, 12 Dec 2002 22:23:54 -0700 Received: (qmail 1810 invoked from network); 12 Dec 2002 22:23:53 -0700 Received: from unknown (HELO carotid.ccs.lanl.gov) (128.165.148.162) by 128.165.148.1 with SMTP; 12 Dec 2002 22:23:53 -0700 Received: (qmail 22725 invoked by uid 3499); 12 Dec 2002 22:23:53 -0700 Received: from localhost (sendmail-bs@127.0.0.1) by localhost with SMTP; 12 Dec 2002 22:23:53 -0700 Date: Thu, 12 Dec 2002 22:23:53 -0700 (MST) From: "Ronald G. Minnich" X-X-Sender: rminnich@carotid.ccs.lanl.gov To: Mike Hoskins Cc: freebsd-cluster@FreeBSD.ORG Subject: Re: (long, warm & fuzzy) Re: sharing files within a cluster In-Reply-To: <20021212133117.D2430-100000@fubar.adept.org> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-freebsd-cluster@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG On Thu, 12 Dec 2002, Mike Hoskins wrote: > Please understand that this state of "teed off" is, for the most part, > what BSD advocates feel like all the time... I'm familiar with it. I was one. > I think some of the rendering farms that have been BSD based, while > possibly not showing an area uniquely suited to BSD, do clearly show that > there would be plenty of demand for BSD clusters. that's not so convincing, because if there would be plenty of demand we would see it. We should be seeing it. Again, you have to look at where those third-party apps are -- compilers and so on. And, sadly, it's not on freebsd. > The same was said for M$ when Linux was born. :) It's funny that many of > the arguments that are now made for Linux have historically been made for > Windows. I really hope M$ doesn't decide to muck around in the Linux > world as they've implied... They could only make things worse. (Granted, > there would be lots of money invested.) To a joe average user, the guy who buys the boxes, the difference between freebsd and linux is ... well, what precisely. They're both pretty good. So, the key question: what's the compelling advantage of freebsd for the guy who writes the checks. How would running freebsd make life better? The move to "some open source Unix" from "some really expensive Unix" was a clear win ten years ago. But the move from "some open source Unix" to "some other open source Unix" is a really hard sell. So, what would freebsd do to make a cluster person's life better, given that they will give up a lot? That's what we would have to answer. ron To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-cluster" in the body of the message From owner-freebsd-cluster Fri Dec 13 6:13:12 2002 Delivered-To: freebsd-cluster@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id A87CC37B401 for ; Fri, 13 Dec 2002 06:13:07 -0800 (PST) Received: from gate.nentec.de (gate2.nentec.de [194.25.215.66]) by mx1.FreeBSD.org (Postfix) with ESMTP id BB81743EB2 for ; Fri, 13 Dec 2002 06:13:05 -0800 (PST) (envelope-from sporner@nentec.de) Received: from nenny.nentec.de (root@nenny.nentec.de [153.92.64.1]) by gate.nentec.de (8.11.3/8.9.3) with ESMTP id gBDED2P24330; Fri, 13 Dec 2002 15:13:02 +0100 Received: from nentec.de (andromeda.nentec.de [153.92.64.34]) by nenny.nentec.de (8.11.3/8.11.3) with ESMTP id gBDECut08441; Fri, 13 Dec 2002 15:12:57 +0100 Message-ID: <3DF9EAE8.9040704@nentec.de> Date: Fri, 13 Dec 2002 15:12:56 +0100 From: Andy Sporner User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.2a) Gecko/20020910 X-Accept-Language: en-us, en MIME-Version: 1.0 To: "Ronald G. Minnich" , freebsd-cluster Subject: Re: sharing files within a cluster References: Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Virus-Scanned: by AMaViS-perl11-milter (http://amavis.org/) Sender: owner-freebsd-cluster@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG Hi Ron, No harm done! :-) Sometimes this can be a very passionate subject. Ronald G. Minnich wrote: >On Thu, 12 Dec 2002, Andy Sporner wrote: > > > >>Well let's review here. RPC's have many dependencies for correct >>operation (portmap, etc) regular socket I/O does not--so which one is >>more reliable??? >> >> > >Actually, that's SunRPC, one variation of RPC, that requires all that >stuff. RPC is a very simple concept, and what you typically see is people >implementing things that are actually RPC but calling it socket I/O. >Pretty much any protocol running over sockets that does a request with a >"procedure number" and expects a response to that request is RPC. > Point taken, then I would have to agree. It would be simpler. I had planned to put some of this in the kernel (though I normally hate this) The justification is that these are proc related things and why go into user space when all the action is already happening in the kernel. >My apologies, the "linuxbios naming" thing really teed me off. I'm also >sorry to be so darned tactless, a lot of those messages I was just >pounding out between Pink bringing firefighting or early in the morning >when I should sleep instead of write email. I apologize to this list. > > I think it takes a better person to acknowledge this. Most people in the past I have seen don't so this is a very welcome thing! >I'm getting really frustrated with this list, however, since people seem >to be refusing to learn from linux because it's "evil", and at the same >time proposing stuff that's been tried years ago on linux and found not to >work. I think it is really important that the FreeBSD community break out >into something new and really neat. At a minimum, however, I'd like to see >FreeBSD have equal capability to Linux clustering for HPC. And I think >before we condemn technologies ("RPC") it's best to make sure it's the >technology, and not a specific implementation of it, that are the issue. > > Agreed. As i said in my "small flame" point I wanted to port the Bproc as a learning exercise, and then to expound on that to do the kind of VM stuff I had spoken about. I was not aware of the differences, mainly because having been stung once (and it was also at a Government lab--Fermi-labs "CPS project" that I saw over-featurisms cause problems--mainly because of RPC's. Since the socket interface isn't too complicated and required very few times, I developed my habits away from RPC's. In the old days when I worked with SYSV systems they weren't available. These were 3B2 systems and Pyramid computer (Ok, they had RPC's :-)) >I would really encourage people to get at the literature of SSI, going >back to Farber's original DCS paper in '72, and try to find an angle that >is somehow new and uniquely suited to FreeBSD. And "Linux sucks, FreeBSD >rules" is not going to be the thing that does it. What about FreeBSD can >somehow make it much better for clustering? That's the problem I've never >been able to answer -- maybe one of you can. > > I am not sure that there is such an aswer. Religous wars (either computer or theological NEVER solved anything--nor will they). I do believe that when one feels a strength in a system, one tends to use what is more comfortable. I in face use Linux on my desktop (as I write this email) because I have more tools at my disposal (VMware for instance). For embedded systems I prefer FreeBSD because it is easy to work in kernel space compared to Linux. We have a project in the house that is using Linux and the developer who was on my project has been swearing against it because a misplaced printf statement crashes the kernel. Where he could have freely put them in Freebsd kernel and got away with it. But to the point. IMHO, Digital had it right with Vaxclusters and it is to that goal that I am trying on my efforts to achieve for FreeBSD. So in terms of plowing new ground, I am under no illusions at all! :-) I think even the former NOW project even had a lot of this in it. I found many others too in my early investigation in this. With the time I have left for such projects, I would have to be committed if I was going to do this just out of pride. >This list is sporadically active, but there have not been tons of new >ideas crossing it -- seems like we see the same stuff over and over again. > > New people come and go all the time :-) I think though we are at the point of critical mass though (or at least near to it). > > >>If this was such well plowed ground, why does it not already exists in >>FreeBSD? >> >> > >Because, and as a former FreeBSD cluster builder I hate to say this, Linux >won. And the mindshare is in Linux, as are the compilers, 3rd party apps, >and all the big vendors. There are many 10s of millions of dollars of >clustering money being spent by IBM, HP, Intel, etc., etc. and absent >something really new and innovative I don't see FreeBSD breaking in. > >The closed-minded nature of the FreeBSD core has not helped. I wrote a DSM >for FreeBSD 2.0.5 ca. 1994, which was based on simple mods to NFS -- >called MNFS (really it was a port from a SunOS version I also wrote). It >required a simple extension to the VM system (it required the VM layer to >tell the VFS layer if a page fault was for read or read/write -- that's >what SunOS does) and I could not get the FreeBSD core to add this simple >additional parameter -- it violated "information hiding between layers", >which is good in theory but in practice the more info the better. (this >code is still on my web page along with the MNFS papers). > > I have had my share of issues with some of these people as well. This is the only list that I have not removed myself from because I can say that people here are generally polite to one another and I have yet to see a major flame. I must also apologize for my response. Fighting fire with fire is also never a solution and I felt sort of bad having said the things that I did. So I must also ask your indulgence. >8 years ago both Sandia/Livermore and I were building freebsd clusters. >It's arguably the better OS. Is it so much better or different from Linux >that it is inherently better for clustering? Probably not. So we got >clobbered by the rest of the world. What we got steamrolled by was the >availability of so many 3rd part apps. > > Totally agreed. My biggest objection to Linux isn't about Linux it's about some of the marketing engines that are pushing it. I think even if you were to ask this one who complained about RedHat samba, I think it was more of a complaint about Redhat than anything else. I used to work with Redhat before it was traded on the stock market and I can really say that their attitude has changed. I used to work for a company that also went out about the same time. It is important for the stock prices to make announcements periodically to bolster the price. So if they announce RedHat Samba for instance, there is a small increase. Does this justify the development of a separate version (if I understand this correctly), yes and no. I generally don't agree with doing this unless it can help the body public first. I would say only this about the debate so far. That being that a project or initiative must be viable by itself without consideration of "one-upmanship". For my part I work now primarily with FreeBSD because it is comfortable and that is the only reason. Some years back I had my old company called BSCsoft (put bscsoft linux in google and you will see it). and we did load balancing for Linux. We had then a minor offering for BSDi, but at that time it was the black sheep. >I'm glad to hear it, sorry for lack of reply. Please get the clustermatic >ISO at www.clustermatic.org and then we can talk about porting. I would >LOVE to see this happen. > > You have a deal... >You can find an initial private name space VFS implementation for FreeBSD, >called v9fs, I think on my web page. > >Sorry, Andy. I'll try to behave. > > So will I ;-) Best wishes and welcome back... Andy To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-cluster" in the body of the message From owner-freebsd-cluster Fri Dec 13 18:59:30 2002 Delivered-To: freebsd-cluster@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 75F0D37B401 for ; Fri, 13 Dec 2002 18:59:29 -0800 (PST) Received: from hotmail.com (f25.pav0.hotmail.com [64.4.32.209]) by mx1.FreeBSD.org (Postfix) with ESMTP id 2F1FC43EA9 for ; Fri, 13 Dec 2002 18:59:29 -0800 (PST) (envelope-from oykai@msn.com) Received: from mail pickup service by hotmail.com with Microsoft SMTPSVC; Fri, 13 Dec 2002 18:59:28 -0800 Received: from 210.74.136.33 by pv0fd.pav0.hotmail.msn.com with HTTP; Sat, 14 Dec 2002 02:59:28 GMT X-Originating-IP: [210.74.136.33] From: "ouyang kai" To: cluster@FreeBSD.org Subject: Hi, I have some questions! Date: Sat, 14 Dec 2002 10:59:28 +0800 Mime-Version: 1.0 Content-Type: text/plain; format=flowed Message-ID: X-OriginalArrivalTime: 14 Dec 2002 02:59:28.0971 (UTC) FILETIME=[D18A51B0:01C2A31C] Sender: owner-freebsd-cluster@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG Hi, everybody, This is my first time to come into the 'cluster' family. :) I have 2+ years experience about FreeBSD. My interestings are FreeBSD Kernel, Storage & Network based FreeBSD and cluster. I heared about DAFS based FreeBSD. http://www.cs.duke.edu/ari/dafs/ http://www.eecs.harvard.edu/~vino/fs-perf/dafs/dafs-server-release-dec8-2001.html But I find the two university have no more study about that. I do not know the reason. And The implementation of DAFS based on FreeBSD must based specail hardware(Emulex cLAN). Another, I do not know whether there are a simulator in FreeBSD like M-VIA in Linux. the M-VIA supports both software and hardware VIA. I want to know whether whether there are softwares to support VIA based normal Giga NIC or 100base NIC. I have not a clear idea about that. I want to research something about that, but I do not know how and where I should start. Thank you very much!! Best Regards Ouyang Kai _________________________________________________________________ MSN 8 helps eliminate e-mail viruses. Get 2 months FREE*. http://join.msn.com/?page=features/virus To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-cluster" in the body of the message