From owner-freebsd-questions@FreeBSD.ORG Tue Jun 17 22:49:03 2003 Return-Path: Delivered-To: freebsd-questions@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 5707737B401 for ; Tue, 17 Jun 2003 22:49:03 -0700 (PDT) Received: from corp.intelius.com (ns1.intelius.com [65.73.232.181]) by mx1.FreeBSD.org (Postfix) with ESMTP id BD95243FAF for ; Tue, 17 Jun 2003 22:49:02 -0700 (PDT) (envelope-from kmarcus@corp.intelius.com) Received: from corp.intelius.com (localhost.intelius.com [127.0.0.1]) by corp.intelius.com (8.12.3/8.12.3) with ESMTP id h5I5gWOr087354 for ; Tue, 17 Jun 2003 22:42:32 -0700 (PDT) (envelope-from kmarcus@corp.intelius.com) Received: from localhost (kmarcus@localhost)h5I5gWv1087351 for ; Tue, 17 Jun 2003 22:42:32 -0700 (PDT) Date: Tue, 17 Jun 2003 22:42:32 -0700 (PDT) From: Kevin Marcus To: freebsd-questions@freebsd.org Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Subject: Re: 3+ TB Storage... CCD, growfs, etc... X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 18 Jun 2003 05:49:03 -0000 > Hi all, > > I am looking at the promise ultratrak RM 15000 > > (http://www.promise.com/product/product_detail_eng.asp?productId=109&familyI > d=6) Raid appliance with a 3TB disk configuration. This box connects > to the host with a SCSI 160 interface which is no problem, and as I > understand it UFS2 is 64 bit so I am not constrained by a 2TB > filesystem limit. The smallest size file on this box will be 11GB, and > there will be lots of them. > > My questions are. > > 1) What is the maximum filesystem size with UFS2? > Are there any special tuning parameters that I should be aware of that > will better optimize the disk? > > 2) How much CPU/Ram would be suggested per TB of disk attached? > 3) If I wanted to eventually strip two+ of these external boxes what > would I need to do? Given this configuration would Vinum or CCD be > better? Why? > > Oh... and this will be running Samba to serve these files to windows > pc's over 1Gb copper ethernet. I can't speak to the exact hardware you're using, although I can speak to my experiences with large filesystems. I primarily use Adaptec RAID card directly connected to their disks. This is also somewhat nice since for the freebsd 4.x series, adaptec also has a utility ('raidutil') which allows you to monkey around with the raid right from the os instead of the post/bios. These utilities do not work on freebsd 5.0 or 5.1, even if you have the proper shared libs so that is moot. So to try and answer these questions: 1) You will likely have problems with fdisk and disklabel as soon as you try to get filesystems over 2TB because of int32 overflows. I dont even want to think of what fsck would do on a non 64 bit system. 2) Uh, as much as possible? There are various limits around the sizes of a single block of memory that can be allocated although these are somewhat easy to tweak. Older versions of FreeBSD could only use 2-4G of RAM with some kernel hacking *AND* disabling swapping all together. I am not sure of the current state of affairs there - matt dillon could probably tell you though. 3) can't help you there I have more recently become concerned with freebsd's ability to simply have mroe than 2TB of disk actually attached. I recently acquired a system with about 2.1TB of disk and I have tried again, various versions all of which have the same problem as shown below: --------------------- ad0: 190782MB [387621/16/63] at ata0-master UDMA100 da1 at asr1 bus 0 target 0 lun 0 da1: Fixed Direct Access SCSI-2 device da1: Tagged Queueing Enabled da1: 840086MB (1720496128 512 byte sectors: 255H 63S/T 41559C) da2 at asr1 bus 0 target 10 lun 0 da2: Fixed Direct Access SCSI-2 device da2: Tagged Queueing Enabled da2: 560057MB (1146996736 512 byte sectors: 255H 63S/T 5861C) da0 at asr0 bus 0 target 0 lun 0 da0: Fixed Direct Access SCSI-2 device da0: Tagged Queueing Enabled da0: 572346MB (1172164608 512 byte sectors: 255H 63S/T 7427C) Mounting root from ufs:/dev/ad0s1a da2: cannot find label (no disk label) da2s1: cannot find label (no disk label) da2: cannot find label (no disk label) da2s1: cannot find label (no disk label) da2: cannot find label (no disk label) da2s1: cannot find label (no disk label) da2: cannot find label (no disk label) --------------------- No matter what I do and no matter how hard I try, I am simply unable to get the da2 filesystem to function. Since this system is brand new I have tried rebuilding the raid (which I originally had simply as a single large 1.3+TB filesystem which didn't work at all. Anyway, I've rearranged things in various orders and tried making them individual disks, smaller systems, etc. but as soon as the actual attached storage goes over 2TB, I start to get these types of errors. interestingly, fdisk and disklabel also really hose the da2 system as well. You can't even rewrite the partition table through fdisk -i on the system. So I would be very very very weary before you attempt to add storage beyond 2tb to a single system and would be cautious on any single filesystem between 1-2tb.