From owner-freebsd-hardware@FreeBSD.ORG Mon Oct 11 01:19:13 2004 Return-Path: Delivered-To: freebsd-hardware@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 5BEA216A4CE for ; Mon, 11 Oct 2004 01:19:13 +0000 (GMT) Received: from mproxy.gmail.com (rproxy.gmail.com [64.233.170.202]) by mx1.FreeBSD.org (Postfix) with ESMTP id 178F143D31 for ; Mon, 11 Oct 2004 01:19:12 +0000 (GMT) (envelope-from Roisin.Murphy@gmail.com) Received: by mproxy.gmail.com with SMTP id 77so407627rnl for ; Sun, 10 Oct 2004 18:19:11 -0700 (PDT) Received: by 10.38.74.32 with SMTP id w32mr1343116rna; Sun, 10 Oct 2004 18:19:11 -0700 (PDT) Received: by 10.38.171.45 with HTTP; Sun, 10 Oct 2004 18:19:11 -0700 (PDT) Message-ID: Date: Sun, 10 Oct 2004 18:19:11 -0700 From: Roisin Murphy To: freebsd-hardware@freebsd.org Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Subject: sata raid & write cache state X-BeenThere: freebsd-hardware@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list Reply-To: Roisin Murphy List-Id: General discussion of FreeBSD hardware List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 11 Oct 2004 01:19:13 -0000 hi I'm thinking of setting up a raid array with sata disks so far this intel controller seems to be winning :) http://www.intel.com/design/servers/RAID/SRCS16/ the handbook says: 11.12.1.5 hw.ata.wc FreeBSD 4.3 flirted with turning off IDE write caching. This reduced write bandwidth to IDE disks but was considered necessary due to serious data consistency issues introduced by hard drive vendors. The problem is that IDE drives lie about when a write completes. With IDE write caching turned on, IDE hard drives not only write data to disk out of order, but will sometimes delay writing some blocks indefinitely when under heavy disk loads. A crash or power failure may cause serious file system corruption. FreeBSD's default was changed to be safe. Unfortunately, the result was such a huge performance loss that we changed write caching back to on by default after the release. You should check the default on your system by observing the hw.ata.wc sysctl variable. If IDE write caching is turned off, you can turn it back on by setting the kernel variable back to 1. This must be done from the boot loader at boot time. Attempting to do it after the kernel boots will have no effect. well, this is what i also heard from a friend of mine, that he has seen too many dead ide raid setups, because of that ata command set, that has no way to tell the state of the write cache content. 1. now, the question is, is this the same with the SATA command set? or is sata more like scsi in this respect? 2. i haven't read much about raid controllers yet, but i would think that with a proper hardware raid5 controller, there's no need for write cache being enabled on the actual disks, as the controller with its cache could optimize the disk writes. Is this so? does a proper hardware raid controller switch the cache off on its disks? 3. Is this the case with scsi also? if the disk could fully report write cache state, the array couldn't mess up/die like they report it with ide raid setups, right? is the disk write cache enabled on scsi raid5 setups? 4. with that intel SRCS16 controller, would hw.ata.wc sysctl work? could i turn the cache off on my sata disks like that? or do i need that manufacturer DOS floppy with utilities to turn the defaults on/off on my disks? (ideally the controller would take care of this) 5. also that intel SRCS16 controller should support 'online capacity expansion' < that means if i start with 3 disks, i can add more disk if i need more storage, without having to recreate the array from scratch? thanks From owner-freebsd-hardware@FreeBSD.ORG Mon Oct 11 04:35:10 2004 Return-Path: Delivered-To: freebsd-hardware@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 27C6F16A4CE for ; Mon, 11 Oct 2004 04:35:10 +0000 (GMT) Received: from nargothrond.kdm.org (nargothrond.kdm.org [70.56.43.81]) by mx1.FreeBSD.org (Postfix) with ESMTP id A2A1343D1F for ; Mon, 11 Oct 2004 04:35:09 +0000 (GMT) (envelope-from ken@nargothrond.kdm.org) Received: from nargothrond.kdm.org (localhost [127.0.0.1]) by nargothrond.kdm.org (8.12.11/8.12.11) with ESMTP id i9B4Z8mu072303; Sun, 10 Oct 2004 22:35:08 -0600 (MDT) (envelope-from ken@nargothrond.kdm.org) Received: (from ken@localhost) by nargothrond.kdm.org (8.12.11/8.12.5/Submit) id i9B4Z8O6072302; Sun, 10 Oct 2004 22:35:08 -0600 (MDT) (envelope-from ken) Date: Sun, 10 Oct 2004 22:35:08 -0600 From: "Kenneth D. Merry" To: Roisin Murphy Message-ID: <20041011043508.GA72113@nargothrond.kdm.org> References: Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.4.2i X-Virus-Scanned: clamd / ClamAV version 0.75.1, clamav-milter version 0.75c on nargothrond.kdm.org X-Virus-Status: Clean cc: freebsd-hardware@freebsd.org Subject: Re: sata raid & write cache state X-BeenThere: freebsd-hardware@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: General discussion of FreeBSD hardware List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 11 Oct 2004 04:35:10 -0000 On Sun, Oct 10, 2004 at 18:19:11 -0700, Roisin Murphy wrote: > hi > > I'm thinking of setting up a raid array with sata disks > so far this intel controller seems to be winning :) > http://www.intel.com/design/servers/RAID/SRCS16/ > > the handbook says: 11.12.1.5 hw.ata.wc > FreeBSD 4.3 flirted with turning off IDE write caching. This reduced > write bandwidth to IDE disks but was considered necessary due to > serious data consistency issues introduced by hard drive vendors. The > problem is that IDE drives lie about when a write completes. With IDE > write caching turned on, IDE hard drives not only write data to disk > out of order, but will sometimes delay writing some blocks > indefinitely when under heavy disk loads. A crash or power failure may > cause serious file system corruption. FreeBSD's default was changed to > be safe. Unfortunately, the result was such a huge performance loss > that we changed write caching back to on by default after the release. > You should check the default on your system by observing the hw.ata.wc > sysctl variable. If IDE write caching is turned off, you can turn it > back on by setting the kernel variable back to 1. This must be done > from the boot loader at boot time. Attempting to do it after the > kernel boots will have no effect. > > well, this is what i also heard from a friend of mine, that he has > seen too many dead ide raid setups, because of that ata command set, > that has no way to tell the state of the write cache content. > > 1. now, the question is, is this the same with the SATA command set? > or is sata more like scsi in this respect? Things are mostly the same with SATA disks, especially if the disk doesn't support tagged queueing. > 2. i haven't read much about raid controllers yet, but i would think > that with a proper hardware raid5 controller, there's no need for > write cache being enabled on the actual disks, as the controller with > its cache could optimize the disk writes. Is this so? does a proper > hardware raid controller switch the cache off on its disks? A proper hardware RAID controller does switch the cache off on its disks, but I seriously doubt you will find an ATA or SATA RAID controller that does that. You basically get incredibly lousy performance without write caching turned on. There are two reasons for this. One is obvious, the other I'm not certain about, you'll need to check the specs: 1. Many (most) ATA and SATA drives do not support tagged queueing. Because of this, you can only have one command outstanding to the drive at a time. A write cache masks this, since it will immediately tell you it has completed the write command when in fact the data for that command has just gone into cache. That's the cool thing that a write cache does for you -- it boosts throughput without tagged queueing. (Note that many Hitachi (formerly IBM) disks do support queueing. Some other vendors may as well, I don't know for sure.) 2. The way tagged queueing in ATA (and SATA) works is that the status phase for a given command must come directly after the data phase. I'm not sure if that is because they don't include a tag number in the status phase or just what. Anyway, this is the part I haven't verified for myself in the specs. If that is true, what it means is that even if your disk supports tagged queueing, you won't be able to submit data for a bunch of different write commands and then get separate status back. As soon as you submit data for one write, you have to wait for status to come back. So it's about as bad as having a drive that can't do tagged queueing. The only thing a scheme like that would give you is a bit better performance on random reads. > 3. Is this the case with scsi also? if the disk could fully report > write cache state, the array couldn't mess up/die like they report it > with ide raid setups, right? is the disk write cache enabled on scsi > raid5 setups? Well, any decent SCSI RAID controller will either just disable the write cache altogether, or it will give the user the option of disabling the write cache. With SCSI disks, you don't have the same performance problem that you do with ATA disks doing tagged queueing (or not), because: 1. All modern SCSI disks can do tagged queueing. 2. With SCSI disks, the status phase is completely decoupled from the data phase. There is no ordering constraint. Since a tag number comes along with the status, the controller knows which command completed. Also, keep in mind that with any RAID controller that does RAID-5 or RAID-1, you should get a battery backed cache. It may be an option, but you should get it. This will protect you from the RAID-5/RAID-1 write hole. That is, when you have a crash, you don't know: 1. What writes you have outstanding. 2. Whether all, part, or none of those writes got committed. So without a battery backed cache, you will have to scrub your entire array to make sure the parity is consistent, and you still will not know whether some of your data was corrupted. All you can really do is sync the parity. Of course a battery backed cache is useless if write caching is turned on on your drives. So it will be a useless feature with most ATA or SATA RAID controllers, because it's unlikely that they would want to tank their performance badly by disabling write caching. With that sort of setup, you should just run with a UPS, and make sure your machine shuts down cleanly in the event of a power outage. Don't get me wrong, I'm not bashing ATA, SATA or SATA RAID controllers. The disks are much cheaper, and make a lot of sense for some applications. As long as you know the limitations, you can use that sort of hardware successfully. (FWIW, my day job has a lot to do with SATA RAID.) > 4. with that intel SRCS16 controller, would hw.ata.wc sysctl work? > could i turn the cache off on my sata disks like that? or do i need > that manufacturer DOS floppy with utilities to turn the defaults > on/off on my disks? (ideally the controller would take care of this) Nope, the sysctl wouldn't work, if it is a processor based RAID controller. > 5. also that intel SRCS16 controller should support 'online capacity > expansion' < that means if i start with 3 disks, i can add more disk > if i need more storage, without having to recreate the array from > scratch? If it does support online capacity expansion, then yes, you would be able to add capacity to the array. What they probably do is just re-stripe/create the array on the fly. With that, combined with growfs(8) you could add space to your system. (You could also just add another partition if that's more convenient.) Ken -- Kenneth Merry ken@FreeBSD.org From owner-freebsd-hardware@FreeBSD.ORG Mon Oct 11 14:05:09 2004 Return-Path: Delivered-To: freebsd-hardware@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 2C7B316A4CF for ; Mon, 11 Oct 2004 14:05:09 +0000 (GMT) Received: from mproxy.gmail.com (rproxy.gmail.com [64.233.170.201]) by mx1.FreeBSD.org (Postfix) with ESMTP id D152B43D49 for ; Mon, 11 Oct 2004 14:05:08 +0000 (GMT) (envelope-from Roisin.Murphy@gmail.com) Received: by mproxy.gmail.com with SMTP id 77so504156rnl for ; Mon, 11 Oct 2004 07:05:05 -0700 (PDT) Received: by 10.38.74.32 with SMTP id w32mr1626395rna; Mon, 11 Oct 2004 07:05:05 -0700 (PDT) Received: by 10.38.171.45 with HTTP; Mon, 11 Oct 2004 07:05:05 -0700 (PDT) Message-ID: Date: Mon, 11 Oct 2004 07:05:05 -0700 From: Roisin Murphy To: freebsd-hardware@freebsd.org In-Reply-To: <20041011043508.GA72113@nargothrond.kdm.org> Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit References: <20041011043508.GA72113@nargothrond.kdm.org> Subject: sata raid & write cache state&In-Reply-To=b21e6cca041010181932879aeb@mail.gmail.com X-BeenThere: freebsd-hardware@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list Reply-To: Roisin Murphy List-Id: General discussion of FreeBSD hardware List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 11 Oct 2004 14:05:09 -0000 thanks for your response well, i'm not really concerned with read/write performance or with loosing little bit of cached data that the controller didn't manage to write down in a case of a power outage or anything breaking in the box. I don't plan to store anything critical. It will be used to dump lots of uncompressed pictures and video, but still the performance doesn't matter. The only thing i want to avoid is an inconsistent state of the array (=dead raid), if anything breaks in the box, and the actual disks&parity are not in sync. From what ken mentioned, it seems like ata/sata is a loose specification, and even if some sata disks do support tagged queueing, the controller most likely won't use it. So my best bet would be to get those floppy manufacturer utilities and set the write cache default to off on all the disks. > Well, any decent SCSI RAID controller will either just disable the write > cache altogether, or it will give the user the option of disabling the > write cache. > Of course a battery backed cache is useless if write caching is turned on > on your drives. So it will be a useless feature with most ATA or SATA RAID > controllers, because it's unlikely that they would want to tank their > performance badly by disabling write caching. 1. So if scsi controllers most likely switch the disk caches off, and optimize the writes with it's own on-board cache, why wouldn't a better/smarter ata/sata controller be able to do the same? I mean, how does a sata disk with cache off differ from a scsi disk with it's cache off? > Also, keep in mind that with any RAID controller that does RAID-5 or > RAID-1, you should get a battery backed cache. It may be an option, but > you should get it. This will protect you from the RAID-5/RAID-1 write > hole. That is, when you have a crash, you don't know: > 1. What writes you have outstanding. > 2. Whether all, part, or none of those writes got committed. 2. So the backup battery (the intel raid card has that option too), is that primarily meant to save the data from the controller's cache? Without the backup battery, could you still end up with an inconsistent/dead array? > So without a battery backed cache, you will have to scrub your entire > array to make sure the parity is consistent, and you still will not know > whether some of your data was corrupted. All you can really do is sync the > parity. 3. I would assume that this parity syncing will be automatic, or not necessary at all. Isn't the controller acting kind of like a transactional db, so even if the machine crashes in any way, the array still survives in a consistent state, only loosing the last active writes? So yes, you could end up with a corrupted file or two, but not with an inconsistent parity/dead array? and with battery backed up controller, you could avoid it altogether, unless the actual controller breaks? 4. Is any software raid solution able to deliver a safe raid5 setup? cache off on all the disks is a must of course. Anybody has a successful story with software solutions, proven by simulated crashes? :) All i keep hearing about vinum is 'not for production use'/'horror stories' From owner-freebsd-hardware@FreeBSD.ORG Mon Oct 11 21:03:04 2004 Return-Path: Delivered-To: freebsd-hardware@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id CF1B316A4CE for ; Mon, 11 Oct 2004 21:03:04 +0000 (GMT) Received: from nargothrond.kdm.org (nargothrond.kdm.org [70.56.43.81]) by mx1.FreeBSD.org (Postfix) with ESMTP id 61B4543D4C for ; Mon, 11 Oct 2004 21:03:04 +0000 (GMT) (envelope-from ken@nargothrond.kdm.org) Received: from nargothrond.kdm.org (localhost [127.0.0.1]) by nargothrond.kdm.org (8.12.11/8.12.11) with ESMTP id i9BL33V9078712; Mon, 11 Oct 2004 15:03:03 -0600 (MDT) (envelope-from ken@nargothrond.kdm.org) Received: (from ken@localhost) by nargothrond.kdm.org (8.12.11/8.12.5/Submit) id i9BL33Lm078711; Mon, 11 Oct 2004 15:03:03 -0600 (MDT) (envelope-from ken) Date: Mon, 11 Oct 2004 15:03:03 -0600 From: "Kenneth D. Merry" To: Roisin Murphy Message-ID: <20041011210303.GA78436@nargothrond.kdm.org> References: <20041011043508.GA72113@nargothrond.kdm.org> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.4.2i X-Virus-Scanned: clamd / ClamAV version 0.75.1, clamav-milter version 0.75c on nargothrond.kdm.org X-Virus-Status: Clean cc: freebsd-hardware@freebsd.org Subject: Re: sata raid & write cache state X-BeenThere: freebsd-hardware@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: General discussion of FreeBSD hardware List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 11 Oct 2004 21:03:05 -0000 On Mon, Oct 11, 2004 at 09:08:01 -0700, Roisin Murphy wrote: > thanks for your response > > well, i'm not really concerned with read/write performance or with > loosing little bit of cached data that the controller didn't manage to > write down in a case of a power outage or anything breaking in the > box. I don't plan to store anything critical. It will be used to dump > lots of uncompressed pictures and video, but still the performance > doesn't matter. The only thing i want to avoid is an inconsistent > state of the array (=dead raid), if anything breaks in the box, and > the actual disks&parity are not in sync. From what ken mentioned, it > seems like ata/sata is a loose specification, and even if some sata > disks do support tagged queueing, the controller most likely won't use > it. So my best bet would be to get those floppy manufacturer utilities > and set the write cache default to off on all the disks. > > > Well, any decent SCSI RAID controller will either just disable the write > > cache altogether, or it will give the user the option of disabling the > > write cache. > > Of course a battery backed cache is useless if write caching is turned on > > on your drives. So it will be a useless feature with most ATA or SATA RAID > > controllers, because it's unlikely that they would want to tank their > > performance badly by disabling write caching. > > 1. So if scsi controllers most likely switch the disk caches off, and > optimize the writes with it's own on-board cache, why wouldn't a > better/smarter ata/sata controller be able to do the same? I mean, how > does a sata disk with cache off differ from a scsi disk with it's > cache off? See my previous mail. SATA disks differ in two ways: 1. Many don't support tagged queueing. 2. If the SATA disk does support tagged queueing, there is still a fundamental problem with the queueing model in SATA (and probably ATA, not sure). According to a coworker of mine (hardware engineer) who is a SATA expert, the status phase on the bus is the same phase as the data phase. So you basically have to send all the data to the drive on a write and the drive has to send the status back before the drive can accept any more data for another queued write command. So that limits you, effectively, to writing data for one command at a time. My coworker also mentioned that in order to figure this out, you have to go look at some of the state diagrams in the SATA spec. The reason you can get good performance with SCSI disks by doing caching in the RAID controller is that SCSI disks do tagged queueing, and their queueing model isn't broken. > > Also, keep in mind that with any RAID controller that does RAID-5 or > > RAID-1, you should get a battery backed cache. It may be an option, but > > you should get it. This will protect you from the RAID-5/RAID-1 write > > hole. That is, when you have a crash, you don't know: > > 1. What writes you have outstanding. > > 2. Whether all, part, or none of those writes got committed. > > 2. So the backup battery (the intel raid card has that option too), is > that primarily meant to save the data from the controller's cache? Yes. > Without the backup battery, could you still end up with an > inconsistent/dead array? You could end up with an inconsistent array, yes. A "dead" array implies that you have two failed disks. Lack of a battery won't cause that. > > So without a battery backed cache, you will have to scrub your entire > > array to make sure the parity is consistent, and you still will not know > > whether some of your data was corrupted. All you can really do is sync the > > parity. > > 3. I would assume that this parity syncing will be automatic, or not > necessary at all. Isn't the controller acting kind of like a > transactional db, so even if the machine crashes in any way, the array > still survives in a consistent state, only loosing the last active > writes? So yes, you could end up with a corrupted file or two, but not > with an inconsistent parity/dead array? and with battery backed up > controller, you could avoid it altogether, unless the actual > controller breaks? The thing to realize about a RAID controller is that it will generally ack writes as soon as it DMAs them into its cache. i.e. most RAID controllers run in write back caching mode. So the OS (filesystem, really) thinks that the data has been committed to media, but it hasn't. As for the RAID controller, the writes it does for any given I/O are not atomic. Without a battery backed cache, or some sort of (very slow) I/O log on the disk to tell it what writes are outstanding, it will not know what I/O was active at the time of a crash. So it will not know whether all, part, or none of those I/Os made it to disk. This is called the RAID-5/RAID-1 write hole. So if you don't have a battery backed cache, and therefore you don't know which stripes on your array had I/O outstanding, you have to scrub the entire array. (This is assuming RAID-5.) That means you read all the data blocks and recompute the parity for every data block. That takes quite a while. With a battery backed cache, you get two pieces of information: 1. What I/Os were outstanding to the disk. 2. The actual data for those I/Os. So since you know what I/Os were outstanding to disk, and you have the actual data, all you have to do is replay those I/Os and your array is completely consistent. So you don't have to do a (very time consuming) scrub of the entire array, and you won't have any data corruption. Because of the RAID-5 write hole, you can not only give the user back old data if a particular piece of data isn't written around the time of a crash, but you can also give the user back corrupt data in the event of a disk failure. (Because your parity isn't consistent any more. Even scrubbing will only make the parity consistent, but won't solve the partial write problem.) > 4. Is any software raid solution able to deliver a safe raid5 setup? > cache off on all the disks is a must of course. Anybody has a > successful story with software solutions, proven by simulated crashes? > :) All i keep hearing about vinum is 'not for production use'/'horror > stories' Well, with (S)ATA disks, I don't think you'll get good performance with the write cache off, with software or hardware RAID. With a software RAID solution, you won't be able to cover the RAID-5/RAID-1 write hole because you don't have a battery backed cache for the RAID controller. (The RAID controller is now the host OS.) If you want the cheapest solution, and something somewhat reliable, you could go with software RAID on SATA disks, and put the machine on a UPS. With a UPS, you could run with write caching turned on on the drives, and without a battery backed RAID controller cache and not worry about it too much. As long as your UPS has enough power to shut down the machine, you'll be fine. In any case, software RAID will actually be faster than most hardware RAID controllers, because your Pentium 4/Opteron/etc. is much faster than the processor on hardware RAID cards. The only problem with pure software RAID is that you generally can't boot off of anything other than a RAID-1. (Even then you may have issues.) So your boot disk won't have any protection. That's why various vendors (Promise, Broadcom, Adaptec, maybe Intel, etc.) have come out with controllers that are basically standard SATA controllers with a special BIOS that lets you boot from a RAID-0, RAID-5, etc. Software RAID takes over once the OS boots. Ken -- Kenneth Merry ken@FreeBSD.ORG From owner-freebsd-hardware@FreeBSD.ORG Tue Oct 12 00:50:49 2004 Return-Path: Delivered-To: freebsd-hardware@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id B3E2C16A4CE for ; Tue, 12 Oct 2004 00:50:49 +0000 (GMT) Received: from mproxy.gmail.com (rproxy.gmail.com [64.233.170.199]) by mx1.FreeBSD.org (Postfix) with ESMTP id 631AB43D3F for ; Tue, 12 Oct 2004 00:50:49 +0000 (GMT) (envelope-from Roisin.Murphy@gmail.com) Received: by mproxy.gmail.com with SMTP id 79so326976rnk for ; Mon, 11 Oct 2004 17:50:49 -0700 (PDT) Received: by 10.38.126.23 with SMTP id y23mr1950281rnc; Mon, 11 Oct 2004 17:50:48 -0700 (PDT) Received: by 10.38.171.45 with HTTP; Mon, 11 Oct 2004 17:50:48 -0700 (PDT) Message-ID: Date: Mon, 11 Oct 2004 17:50:48 -0700 From: Roisin Murphy To: freebsd-hardware@freebsd.org Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Subject: sata raid & write cache state X-BeenThere: freebsd-hardware@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list Reply-To: Roisin Murphy List-Id: General discussion of FreeBSD hardware List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 12 Oct 2004 00:50:49 -0000 thanks, i hope i'm not running in circles now :) > most RAID controllers run in write back caching mode. 3. Do you think the raid controller could also keep the info on how the data is spread across the disk platters? From what i understand, the write back cache on disks, is there to avoid the disk heads from having to jump from sector 1 to sector 500 to sector 2, but instead allow it to go from 1 to 2 to 500. But if the controller would know how the data is spread, it could use its cache to send the bits in a preferable order. So the controller itself is like another layer, so there really isn't any need for cache on the disks. And thats what i was asking, as you yourself mentioned that you think scsi controllers usually turn the cache off on scsi disks anyway. So therefor i asked, what the difference was between scsi and (s)ata disks, when both have disks write cache off. I do understand the tagged queueing concept now, but if the cache is off on the scsi disk, and the controller's cache is used, it shouldn't matter. > A "dead" array implies that you have two failed disks. 2. well, i have to ask that guy i know, who does a lot of incident response type of work, what exactly happened to all those 'messed up' (s)ata raid arrays. I didn't know, you could recover from a power outage situation, where you end up with inconsistent parity. 3. As i said, i wont be running any critical transactional db, that would end up in an inconsistent state if the power goes out, and the disks reported all the writes (but instead they were pending in the disk cache). So the db finished the transaction, but after the crash&reboot, some tables will be modified and some won't. ALL I'm concerned about is, that my whole raid setup could die because of an inconsistent parity. I don't really mind loosing/corrupting few individual files. > If you want the cheapest solution, and something somewhat reliable, you > could go with software RAID on SATA disks, and put the machine on a UPS. > With a UPS, you could run with write caching turned on on the drives, and > without a battery backed RAID controller cache and not worry about it too > much. As long as your UPS has enough power to shut down the machine, > you'll be fine. 4. In my experience, my PSU or any flaky hardare in my box breaks more often then there are power outages in my area :) > The only problem with pure software RAID is that you generally can't boot > off of anything other than a RAID-1. (Even then you may have issues.) So > your boot disk won't have any protection. 5. the boot disks usually don't matter, as long as you backup your configuration, when its changed... 6. anyway, thanks for taking time to explain stuff in detail :) From owner-freebsd-hardware@FreeBSD.ORG Tue Oct 12 14:34:49 2004 Return-Path: Delivered-To: freebsd-hardware@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 1907C16A4CE for ; Tue, 12 Oct 2004 14:34:49 +0000 (GMT) Received: from somic.vox-mundi.net (somic.vox-mundi.net [80.253.170.35]) by mx1.FreeBSD.org (Postfix) with ESMTP id 950B543D49 for ; Tue, 12 Oct 2004 14:34:47 +0000 (GMT) (envelope-from gordan@vox-mundi.net) Received: from VMPresario ([80.253.170.106]) by somic.vox-mundi.net (8.11.6/8.11.6) with SMTP id i9CEYir26182 for ; Tue, 12 Oct 2004 16:34:44 +0200 From: "Gordan Remus (Vox Mundi)" To: Date: Tue, 12 Oct 2004 16:32:04 +0200 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.6604 (9.0.2911.0) Importance: Normal X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.2180 Subject: FreeBSD 4.X and Intel E7520 chipset problems X-BeenThere: freebsd-hardware@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: General discussion of FreeBSD hardware List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 12 Oct 2004 14:34:49 -0000 Recently we have purchased a Supermicro X6DHR-IG2 motherboard based on Intel E7520 chipset with 2x3.0GHz Xeons and are unable to install FreeBSD 4.X on it. It seems that FreeBSD 4.X is not recognizing the chipset and therefore it's unable to recognize Mylex AcceleRAID 160 controller. Everything is working fine on FreeBSD 5.X, but unfortunately the application we need to run on top of that is made specifically for FreeBSD 4.X. Is there any chance that Intel E7520 chipset will be supported on some future 4.X releases or is there some kind of patch/workaround ? Best Regards, Gordan Remus Technical Department Vox Mundi From owner-freebsd-hardware@FreeBSD.ORG Tue Oct 12 15:33:24 2004 Return-Path: Delivered-To: freebsd-hardware@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 4336916A4CE; Tue, 12 Oct 2004 15:33:24 +0000 (GMT) Received: from ox.eicat.ca (ox.eicat.ca [66.96.30.35]) by mx1.FreeBSD.org (Postfix) with ESMTP id 0057743D1F; Tue, 12 Oct 2004 15:33:24 +0000 (GMT) (envelope-from dgilbert@daveg.ca) Received: by ox.eicat.ca (Postfix, from userid 66) id 1A306C745; Tue, 12 Oct 2004 11:33:23 -0400 (EDT) Received: by canoe.dclg.ca (Postfix, from userid 101) id 866981D2195; Tue, 12 Oct 2004 11:33:15 -0400 (EDT) From: David Gilbert MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Message-ID: <16747.63803.470649.921882@canoe.dclg.ca> Date: Tue, 12 Oct 2004 11:33:15 -0400 To: "Kenneth D. Merry" In-Reply-To: <20041011210303.GA78436@nargothrond.kdm.org> References: <20041011043508.GA72113@nargothrond.kdm.org> <20041011210303.GA78436@nargothrond.kdm.org> X-Mailer: VM 7.17 under 21.5 (beta17) "chayote" (+CVS-20040321) XEmacs Lucid cc: Roisin Murphy cc: freebsd-hardware@freebsd.org Subject: Re: sata raid & write cache state X-BeenThere: freebsd-hardware@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: General discussion of FreeBSD hardware List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 12 Oct 2004 15:33:24 -0000 >>>>> "Kenneth" == Kenneth D Merry writes: Kenneth> See my previous mail. SATA disks differ in two ways: Kenneth> 1. Many don't support tagged queueing. I'd like to see more information on this. I was under the impression that SATA required some form of command queueing in all drives. Kenneth> 2. If the SATA disk does support tagged queueing, there is Kenneth> still a fundamental problem with the queueing model in SATA Kenneth> (and probably ATA, not sure). According to a coworker of Kenneth> mine (hardware engineer) who is a SATA expert, the status Kenneth> phase on the bus is the same phase as the data phase. So you Kenneth> basically have to send all the data to the drive on a write Kenneth> and the drive has to send the status back before the drive Kenneth> can accept any more data for another queued write command. Kenneth> So that limits you, effectively, to writing data for one Kenneth> command at a time. It would appear that the SATA folks are introducing 'NCQ' (Native Command Queueing) ... which does supply out-of-order returns among other things. Dave. -- ============================================================================ |David Gilbert, Independent Contractor. | Two things can only be | |Mail: dave@daveg.ca | equal if and only if they | |http://daveg.ca | are precisely opposite. | =========================================================GLO================ From owner-freebsd-hardware@FreeBSD.ORG Tue Oct 12 15:45:40 2004 Return-Path: Delivered-To: freebsd-hardware@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id DC95616A4DA for ; Tue, 12 Oct 2004 15:45:40 +0000 (GMT) Received: from mail2.speakeasy.net (mail2.speakeasy.net [216.254.0.202]) by mx1.FreeBSD.org (Postfix) with ESMTP id B8F1143D2D for ; Tue, 12 Oct 2004 15:45:40 +0000 (GMT) (envelope-from jhb@FreeBSD.org) Received: (qmail 11343 invoked from network); 12 Oct 2004 15:45:40 -0000 Received: from dsl027-160-063.atl1.dsl.speakeasy.net (HELO server.baldwin.cx) ([216.27.160.63]) (envelope-sender ) encrypted SMTP for ; 12 Oct 2004 15:45:40 -0000 Received: from [10.50.40.210] (gw1.twc.weather.com [216.133.140.1]) (authenticated bits=0) by server.baldwin.cx (8.12.11/8.12.11) with ESMTP id i9CFjSRS091976; Tue, 12 Oct 2004 11:45:36 -0400 (EDT) (envelope-from jhb@FreeBSD.org) From: John Baldwin To: freebsd-hardware@FreeBSD.org Date: Tue, 12 Oct 2004 11:19:27 -0400 User-Agent: KMail/1.6.2 References: In-Reply-To: MIME-Version: 1.0 Content-Disposition: inline Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit Message-Id: <200410121119.27359.jhb@FreeBSD.org> X-Spam-Checker-Version: SpamAssassin 2.63 (2004-01-11) on server.baldwin.cx cc: "Gordan Remus \(Vox Mundi\)" Subject: Re: FreeBSD 4.X and Intel E7520 chipset problems X-BeenThere: freebsd-hardware@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: General discussion of FreeBSD hardware List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 12 Oct 2004 15:45:41 -0000 On Tuesday 12 October 2004 10:32 am, Gordan Remus (Vox Mundi) wrote: > Recently we have purchased a Supermicro X6DHR-IG2 motherboard > based on Intel E7520 chipset with 2x3.0GHz Xeons and are unable > to install FreeBSD 4.X on it. > It seems that FreeBSD 4.X is not recognizing the chipset > and therefore it's unable to recognize Mylex AcceleRAID 160 controller. > > Everything is working fine on FreeBSD 5.X, but unfortunately > the application we need to run on top of that is made specifically > for FreeBSD 4.X. > > Is there any chance that Intel E7520 chipset will be supported > on some future 4.X releases or is there some kind of patch/workaround ? What specifically does 4.x not recognize? Does it only see one host-PCI bridge when there are more than one and the missing devices are on the other host bridges? If so, can you install 5.x on it and get an acpidump output that I can look at? -- John Baldwin <>< http://www.FreeBSD.org/~jhb/ "Power Users Use the Power to Serve" = http://www.FreeBSD.org From owner-freebsd-hardware@FreeBSD.ORG Tue Oct 12 16:50:33 2004 Return-Path: Delivered-To: freebsd-hardware@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 3512116A4CE for ; Tue, 12 Oct 2004 16:50:33 +0000 (GMT) Received: from nargothrond.kdm.org (nargothrond.kdm.org [70.56.43.81]) by mx1.FreeBSD.org (Postfix) with ESMTP id B707643D3F for ; Tue, 12 Oct 2004 16:50:32 +0000 (GMT) (envelope-from ken@nargothrond.kdm.org) Received: from nargothrond.kdm.org (localhost [127.0.0.1]) by nargothrond.kdm.org (8.12.11/8.12.11) with ESMTP id i9CGoTGO086768; Tue, 12 Oct 2004 10:50:29 -0600 (MDT) (envelope-from ken@nargothrond.kdm.org) Received: (from ken@localhost) by nargothrond.kdm.org (8.12.11/8.12.5/Submit) id i9CGoTH4086767; Tue, 12 Oct 2004 10:50:29 -0600 (MDT) (envelope-from ken) Date: Tue, 12 Oct 2004 10:50:29 -0600 From: "Kenneth D. Merry" To: David Gilbert Message-ID: <20041012165029.GA86646@nargothrond.kdm.org> References: <20041011043508.GA72113@nargothrond.kdm.org> <20041011210303.GA78436@nargothrond.kdm.org> <16747.63803.470649.921882@canoe.dclg.ca> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <16747.63803.470649.921882@canoe.dclg.ca> User-Agent: Mutt/1.4.2i X-Virus-Scanned: clamd / ClamAV version 0.75.1, clamav-milter version 0.75c on nargothrond.kdm.org X-Virus-Status: Clean cc: Roisin Murphy cc: freebsd-hardware@freebsd.org Subject: Re: sata raid & write cache state X-BeenThere: freebsd-hardware@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: General discussion of FreeBSD hardware List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 12 Oct 2004 16:50:33 -0000 On Tue, Oct 12, 2004 at 11:33:15 -0400, David Gilbert wrote: > >>>>> "Kenneth" == Kenneth D Merry writes: > > Kenneth> See my previous mail. SATA disks differ in two ways: > > Kenneth> 1. Many don't support tagged queueing. > > I'd like to see more information on this. I was under the impression > that SATA required some form of command queueing in all drives. Some do, some don't. Take a look at the vendors' specs and they'll tell you whether the drive has support for queueing. e.g. from a quick look at Maxtor's site, the DiamondMax Plus 9 doesn't support queueing, and the DiamondMax 10 claims to support NCQ. > Kenneth> 2. If the SATA disk does support tagged queueing, there is > Kenneth> still a fundamental problem with the queueing model in SATA > Kenneth> (and probably ATA, not sure). According to a coworker of > Kenneth> mine (hardware engineer) who is a SATA expert, the status > Kenneth> phase on the bus is the same phase as the data phase. So you > Kenneth> basically have to send all the data to the drive on a write > Kenneth> and the drive has to send the status back before the drive > Kenneth> can accept any more data for another queued write command. > Kenneth> So that limits you, effectively, to writing data for one > Kenneth> command at a time. > > It would appear that the SATA folks are introducing 'NCQ' (Native > Command Queueing) ... which does supply out-of-order returns among > other things. Hopefully it'll be better than the current generation queueing support. Note that even the current generation queueing support can do out of order completion on reads. That's because it's pretty easy to return the data and status at the same time on a read. :) The problem is on writes. If you have write caching turned off, the drive will have to commit the write to media before it can send status back. So you can't effectively send data for the next write command before the current write data is committed. It's entirely possible that SATA II has fixed things, I don't know. Ken -- Kenneth Merry ken@FreeBSD.ORG From owner-freebsd-hardware@FreeBSD.ORG Tue Oct 12 16:57:49 2004 Return-Path: Delivered-To: freebsd-hardware@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 7E90516A4CE; Tue, 12 Oct 2004 16:57:49 +0000 (GMT) Received: from ox.eicat.ca (ox.eicat.ca [66.96.30.35]) by mx1.FreeBSD.org (Postfix) with ESMTP id 56BCF43D39; Tue, 12 Oct 2004 16:57:49 +0000 (GMT) (envelope-from dgilbert@daveg.ca) Received: by ox.eicat.ca (Postfix, from userid 66) id AF848C681; Tue, 12 Oct 2004 12:57:48 -0400 (EDT) Received: by canoe.dclg.ca (Postfix, from userid 101) id 308141D1DD5; Tue, 12 Oct 2004 12:57:41 -0400 (EDT) From: David Gilbert MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Message-ID: <16748.3333.136486.42182@canoe.dclg.ca> Date: Tue, 12 Oct 2004 12:57:41 -0400 To: "Kenneth D. Merry" In-Reply-To: <20041012165029.GA86646@nargothrond.kdm.org> References: <20041011043508.GA72113@nargothrond.kdm.org> <20041011210303.GA78436@nargothrond.kdm.org> <16747.63803.470649.921882@canoe.dclg.ca> <20041012165029.GA86646@nargothrond.kdm.org> X-Mailer: VM 7.17 under 21.5 (beta17) "chayote" (+CVS-20040321) XEmacs Lucid cc: Roisin Murphy cc: freebsd-hardware@freebsd.org cc: David Gilbert Subject: Re: sata raid & write cache state X-BeenThere: freebsd-hardware@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: General discussion of FreeBSD hardware List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 12 Oct 2004 16:57:49 -0000 >>>>> "Kenneth" == Kenneth D Merry writes: Kenneth> e.g. from a quick look at Maxtor's site, the DiamondMax Plus Kenneth> 9 doesn't support queueing, and the DiamondMax 10 claims to Kenneth> support NCQ. I suppose the more important question is "do we support NCQ?", but I thought that there was another form of queueing that was mandatory in SATA. Dave. -- ============================================================================ |David Gilbert, Independent Contractor. | Two things can only be | |Mail: dave@daveg.ca | equal if and only if they | |http://daveg.ca | are precisely opposite. | =========================================================GLO================ From owner-freebsd-hardware@FreeBSD.ORG Tue Oct 12 17:11:07 2004 Return-Path: Delivered-To: freebsd-hardware@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id D8DFC16A4CE for ; Tue, 12 Oct 2004 17:11:07 +0000 (GMT) Received: from nargothrond.kdm.org (nargothrond.kdm.org [70.56.43.81]) by mx1.FreeBSD.org (Postfix) with ESMTP id 69C7943D3F for ; Tue, 12 Oct 2004 17:11:07 +0000 (GMT) (envelope-from ken@nargothrond.kdm.org) Received: from nargothrond.kdm.org (localhost [127.0.0.1]) by nargothrond.kdm.org (8.12.11/8.12.11) with ESMTP id i9CHB6Xm086988; Tue, 12 Oct 2004 11:11:06 -0600 (MDT) (envelope-from ken@nargothrond.kdm.org) Received: (from ken@localhost) by nargothrond.kdm.org (8.12.11/8.12.5/Submit) id i9CHB6we086987; Tue, 12 Oct 2004 11:11:06 -0600 (MDT) (envelope-from ken) Date: Tue, 12 Oct 2004 11:11:06 -0600 From: "Kenneth D. Merry" To: Roisin Murphy Message-ID: <20041012171106.GB86646@nargothrond.kdm.org> References: <20041011043508.GA72113@nargothrond.kdm.org> <20041011210303.GA78436@nargothrond.kdm.org> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.4.2i X-Virus-Scanned: clamd / ClamAV version 0.75.1, clamav-milter version 0.75c on nargothrond.kdm.org X-Virus-Status: Clean cc: freebsd-hardware@freebsd.org Subject: Re: sata raid & write cache state X-BeenThere: freebsd-hardware@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: General discussion of FreeBSD hardware List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 12 Oct 2004 17:11:08 -0000 [ your mail client is broken -- you replied separately to me and the list.] On Mon, Oct 11, 2004 at 17:42:55 -0700, Roisin Murphy wrote: > thanks, i hope i'm not running in circles now :) > > > most RAID controllers run in write back caching mode. > > 3. Do you think the raid controller could also keep the info on how > the data is spread across the disk platters? From what i understand, > the write back cache on disks, is there to avoid the disk heads from > having to jump from sector 1 to sector 500 to sector 2, but instead > allow it to go from 1 to 2 to 500. But if the controller would know > how the data is spread, it could use its cache to send the bits in a > preferable order. So the controller itself is like another layer, so > there really isn't any need for cache on the disks. And thats what i > was asking, as you yourself mentioned that you think scsi controllers > usually turn the cache off on scsi disks anyway. So therefor i asked, > what the difference was between scsi and (s)ata disks, when both have > disks write cache off. I do understand the tagged queueing concept > now, but if the cache is off on the scsi disk, and the controller's > cache is used, it shouldn't matter. The main point of write back caching is to give the user faster turnaround for their commands without the application having to support deep queueing. As for the RAID controller, the best it can do is re-order commands by LBA. The RAID controller won't have any knowledge of the physical layout of the disk. Even with write caching turned off, though, a disk can re-order commands with a 'simple' tag attribute. (For SCSI, you can have simple, ordered, head of queue, etc.) So the drive can still re-order the commands that it has queued so it can execute them more efficiently. It just waits to send status to the RAID controller until the command has completed. > > A "dead" array implies that you have two failed disks. > > 2. well, i have to ask that guy i know, who does a lot of incident > response type of work, what exactly happened to all those 'messed up' > (s)ata raid arrays. I didn't know, you could recover from a power > outage situation, where you end up with inconsistent parity. Sure, you can recover from a power outage, you just have to scrub the array when you come back up. You also won't know for sure whether you have some data corruption. > 3. As i said, i wont be running any critical transactional db, that > would end up in an inconsistent state if the power goes out, and the > disks reported all the writes (but instead they were pending in the > disk cache). So the db finished the transaction, but after the > crash&reboot, some tables will be modified and some won't. ALL I'm > concerned about is, that my whole raid setup could die because of an > inconsistent parity. I don't really mind loosing/corrupting few > individual files. No, your whole RAID setup won't be dead due to inconsistent parity. I'd recommend getting a UPS if you're going to do a SATA RAID setup. It'll probably cost as much as one or two of the drives. > > If you want the cheapest solution, and something somewhat reliable, you > > could go with software RAID on SATA disks, and put the machine on a UPS. > > With a UPS, you could run with write caching turned on on the drives, and > > without a battery backed RAID controller cache and not worry about it too > > much. As long as your UPS has enough power to shut down the machine, > > you'll be fine. > > 4. In my experience, my PSU or any flaky hardare in my box breaks more > often then there are power outages in my area :) > > > The only problem with pure software RAID is that you generally can't boot > > off of anything other than a RAID-1. (Even then you may have issues.) So > > your boot disk won't have any protection. > > 5. the boot disks usually don't matter, as long as you backup your > configuration, when its changed... > > 6. anyway, thanks for taking time to explain stuff in detail :) Ken -- Kenneth Merry ken@FreeBSD.ORG From owner-freebsd-hardware@FreeBSD.ORG Tue Oct 12 18:44:40 2004 Return-Path: Delivered-To: freebsd-hardware@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id AD76016A4CE for ; Tue, 12 Oct 2004 18:44:40 +0000 (GMT) Received: from mproxy.gmail.com (rproxy.gmail.com [64.233.170.207]) by mx1.FreeBSD.org (Postfix) with ESMTP id 54ADD43D45 for ; Tue, 12 Oct 2004 18:44:40 +0000 (GMT) (envelope-from Roisin.Murphy@gmail.com) Received: by mproxy.gmail.com with SMTP id 79so419459rnk for ; Tue, 12 Oct 2004 11:44:39 -0700 (PDT) Received: by 10.38.96.30 with SMTP id t30mr2433838rnb; Tue, 12 Oct 2004 11:44:39 -0700 (PDT) Received: by 10.38.171.45 with HTTP; Tue, 12 Oct 2004 11:44:39 -0700 (PDT) Message-ID: Date: Tue, 12 Oct 2004 11:44:39 -0700 From: Roisin Murphy To: "Kenneth D. Merry" In-Reply-To: <20041012171106.GB86646@nargothrond.kdm.org> Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit References: <20041011043508.GA72113@nargothrond.kdm.org> <20041011210303.GA78436@nargothrond.kdm.org> <20041012171106.GB86646@nargothrond.kdm.org> cc: freebsd-hardware@freebsd.org Subject: Re: sata raid & write cache state X-BeenThere: freebsd-hardware@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list Reply-To: Roisin Murphy List-Id: General discussion of FreeBSD hardware List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 12 Oct 2004 18:44:40 -0000 > The main point of write back caching is to give the user faster turnaround > for their commands without the application having to support deep queueing. 1. that is now taken care of by the 'write back' cache on the raid controller. So (s)ata raid setups can still perform similarly to scsi setups. But enough about that :) anyway, enough of that :) > Sure, you can recover from a power outage, you just have to scrub the array > when you come back up. You also won't know for sure whether you have some > data corruption. 2. how do you usually 'scrub' the array? is this automatically done by the controller when it boots up? or does this usually require the user issuing commands via those raid CLIs? 3. one more thing about the battery back up option, now that i thought about it, it has to be like a mini PSU that has enough power to keep all the disks spinning for couple seconds after your main psu/power dies, so now i understand why it cost $100+ From owner-freebsd-hardware@FreeBSD.ORG Tue Oct 12 21:03:50 2004 Return-Path: Delivered-To: freebsd-hardware@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 047F516A4D1 for ; Tue, 12 Oct 2004 21:03:50 +0000 (GMT) Received: from nargothrond.kdm.org (nargothrond.kdm.org [70.56.43.81]) by mx1.FreeBSD.org (Postfix) with ESMTP id 6EF4743D45 for ; Tue, 12 Oct 2004 21:03:49 +0000 (GMT) (envelope-from ken@nargothrond.kdm.org) Received: from nargothrond.kdm.org (localhost [127.0.0.1]) by nargothrond.kdm.org (8.12.11/8.12.11) with ESMTP id i9CL3kMH088913; Tue, 12 Oct 2004 15:03:46 -0600 (MDT) (envelope-from ken@nargothrond.kdm.org) Received: (from ken@localhost) by nargothrond.kdm.org (8.12.11/8.12.5/Submit) id i9CL3krU088912; Tue, 12 Oct 2004 15:03:46 -0600 (MDT) (envelope-from ken) Date: Tue, 12 Oct 2004 15:03:46 -0600 From: "Kenneth D. Merry" To: David Gilbert Message-ID: <20041012210346.GA88834@nargothrond.kdm.org> References: <20041011043508.GA72113@nargothrond.kdm.org> <20041011210303.GA78436@nargothrond.kdm.org> <16747.63803.470649.921882@canoe.dclg.ca> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <16747.63803.470649.921882@canoe.dclg.ca> User-Agent: Mutt/1.4.2i X-Virus-Scanned: clamd / ClamAV version 0.75.1, clamav-milter version 0.75c on nargothrond.kdm.org X-Virus-Status: Clean cc: Roisin Murphy cc: freebsd-hardware@freebsd.org Subject: Re: sata raid & write cache state X-BeenThere: freebsd-hardware@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: General discussion of FreeBSD hardware List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 12 Oct 2004 21:03:50 -0000 On Tue, Oct 12, 2004 at 11:33:15 -0400, David Gilbert wrote: > >>>>> "Kenneth" == Kenneth D Merry writes: > > Kenneth> See my previous mail. SATA disks differ in two ways: > > Kenneth> 1. Many don't support tagged queueing. > > I'd like to see more information on this. I was under the impression > that SATA required some form of command queueing in all drives. > > Kenneth> 2. If the SATA disk does support tagged queueing, there is > Kenneth> still a fundamental problem with the queueing model in SATA > Kenneth> (and probably ATA, not sure). According to a coworker of > Kenneth> mine (hardware engineer) who is a SATA expert, the status > Kenneth> phase on the bus is the same phase as the data phase. So you > Kenneth> basically have to send all the data to the drive on a write > Kenneth> and the drive has to send the status back before the drive > Kenneth> can accept any more data for another queued write command. > Kenneth> So that limits you, effectively, to writing data for one > Kenneth> command at a time. > > It would appear that the SATA folks are introducing 'NCQ' (Native > Command Queueing) ... which does supply out-of-order returns among > other things. >From talking to my co-worker again, it sounds like the SATA II spec does allow breaking up the data and status phases for a command. (i.e. out of order command completion) The key is that even if the drive supports it, the controller has to enable that feature on the drive. So the bottom line is that with a drive and a controller that support SATA II NCQ and out of order completion, you could disable write caching on the drive and still (in theory) get reasonable performance out of it. Ken -- Kenneth Merry ken@FreeBSD.ORG From owner-freebsd-hardware@FreeBSD.ORG Tue Oct 12 21:59:53 2004 Return-Path: Delivered-To: freebsd-hardware@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 583A216A4CE for ; Tue, 12 Oct 2004 21:59:53 +0000 (GMT) Received: from nargothrond.kdm.org (nargothrond.kdm.org [70.56.43.81]) by mx1.FreeBSD.org (Postfix) with ESMTP id C032543D3F for ; Tue, 12 Oct 2004 21:59:52 +0000 (GMT) (envelope-from ken@nargothrond.kdm.org) Received: from nargothrond.kdm.org (localhost [127.0.0.1]) by nargothrond.kdm.org (8.12.11/8.12.11) with ESMTP id i9CLxqS1089462; Tue, 12 Oct 2004 15:59:52 -0600 (MDT) (envelope-from ken@nargothrond.kdm.org) Received: (from ken@localhost) by nargothrond.kdm.org (8.12.11/8.12.5/Submit) id i9CLxqeo089461; Tue, 12 Oct 2004 15:59:52 -0600 (MDT) (envelope-from ken) Date: Tue, 12 Oct 2004 15:59:51 -0600 From: "Kenneth D. Merry" To: Roisin Murphy Message-ID: <20041012215951.GA89335@nargothrond.kdm.org> References: <20041011043508.GA72113@nargothrond.kdm.org> <20041011210303.GA78436@nargothrond.kdm.org> <20041012171106.GB86646@nargothrond.kdm.org> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.4.2i X-Virus-Scanned: clamd / ClamAV version 0.75.1, clamav-milter version 0.75c on nargothrond.kdm.org X-Virus-Status: Clean cc: freebsd-hardware@freebsd.org Subject: Re: sata raid & write cache state X-BeenThere: freebsd-hardware@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: General discussion of FreeBSD hardware List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 12 Oct 2004 21:59:53 -0000 On Tue, Oct 12, 2004 at 11:44:39 -0700, Roisin Murphy wrote: > > The main point of write back caching is to give the user faster turnaround > > for their commands without the application having to support deep queueing. > > 1. that is now taken care of by the 'write back' cache on the raid > controller. So (s)ata raid setups can still perform similarly to scsi > setups. But enough about that :) > anyway, enough of that :) They can only perform similarly if they use a non-broken queueing model. See my other mail. Basically it'll require a drive and RAID controller that support SATA II Native Command Queueing, out of order completion, etc. > > Sure, you can recover from a power outage, you just have to scrub the array > > when you come back up. You also won't know for sure whether you have some > > data corruption. > > 2. how do you usually 'scrub' the array? is this automatically done by > the controller when it boots up? or does this usually require the user > issuing commands via those raid CLIs? A "scrub" means that the controller regenerates the parity on the array. So it goes through, reads the data chunks for each stripe, recalculates the parity, and writes the parity back out. The main difference between a scrub and a verify on a RAID-5 array is that the controller will generally tell you if there was an inconsistency on a verify. (So it has to read back the parity that is already on disk and compare it with the parity it calculated. I suppose it could also avoid writing the parity again on a verify if the old and new parity match.) If your machine crashes, the controller will generally automatically initiate a scrub if you don't have battery backed RAM or if your RAM isn't valid. With some controllers, you can issue commands via a CLI. (e.g. you can do it with Adaptec controllers that use the aac(4) driver.) > 3. one more thing about the battery back up option, now that i thought > about it, it has to be like a mini PSU that has enough power to keep > all the disks spinning for couple seconds after your main psu/power > dies, so now i understand why it cost $100+ Head to www.apc.com and look at the UPSes they have there. They're basically lead acid batteries with inverters to put out A/C power when your main power feed goes out. Ken -- Kenneth Merry ken@FreeBSD.ORG From owner-freebsd-hardware@FreeBSD.ORG Tue Oct 12 22:10:45 2004 Return-Path: Delivered-To: freebsd-hardware@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 349A316A4CE for ; Tue, 12 Oct 2004 22:10:45 +0000 (GMT) Received: from mproxy.gmail.com (rproxy.gmail.com [64.233.170.206]) by mx1.FreeBSD.org (Postfix) with ESMTP id D4D3143D31 for ; Tue, 12 Oct 2004 22:10:44 +0000 (GMT) (envelope-from Roisin.Murphy@gmail.com) Received: by mproxy.gmail.com with SMTP id 79so444725rnk for ; Tue, 12 Oct 2004 15:10:39 -0700 (PDT) Received: by 10.38.209.30 with SMTP id h30mr427875rng; Tue, 12 Oct 2004 15:10:38 -0700 (PDT) Received: by 10.38.171.45 with HTTP; Tue, 12 Oct 2004 15:10:38 -0700 (PDT) Message-ID: Date: Tue, 12 Oct 2004 15:10:38 -0700 From: Roisin Murphy To: "Kenneth D. Merry" In-Reply-To: <20041012215951.GA89335@nargothrond.kdm.org> Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit References: <20041011043508.GA72113@nargothrond.kdm.org> <20041011210303.GA78436@nargothrond.kdm.org> <20041012171106.GB86646@nargothrond.kdm.org> <20041012215951.GA89335@nargothrond.kdm.org> cc: freebsd-hardware@freebsd.org Subject: Re: sata raid & write cache state X-BeenThere: freebsd-hardware@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list Reply-To: Roisin Murphy List-Id: General discussion of FreeBSD hardware List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 12 Oct 2004 22:10:45 -0000 > Head to www.apc.com and look at the UPSes they have there. They're > basically lead acid batteries with inverters to put out A/C power when your > main power feed goes out. i meant the battery option they sell with some raid controllers, is this and extra little PSU, that also powers the disks for a while, or is this just meant to keep the raid controller's cache powered till the machine boots up again, so it can finish its cache flush? From owner-freebsd-hardware@FreeBSD.ORG Tue Oct 12 22:13:55 2004 Return-Path: Delivered-To: freebsd-hardware@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id CE66D16A4CE for ; Tue, 12 Oct 2004 22:13:55 +0000 (GMT) Received: from nargothrond.kdm.org (nargothrond.kdm.org [70.56.43.81]) by mx1.FreeBSD.org (Postfix) with ESMTP id 6EA3D43D2F for ; Tue, 12 Oct 2004 22:13:55 +0000 (GMT) (envelope-from ken@nargothrond.kdm.org) Received: from nargothrond.kdm.org (localhost [127.0.0.1]) by nargothrond.kdm.org (8.12.11/8.12.11) with ESMTP id i9CMDo7e089611; Tue, 12 Oct 2004 16:13:50 -0600 (MDT) (envelope-from ken@nargothrond.kdm.org) Received: (from ken@localhost) by nargothrond.kdm.org (8.12.11/8.12.5/Submit) id i9CMDktg089610; Tue, 12 Oct 2004 16:13:46 -0600 (MDT) (envelope-from ken) Date: Tue, 12 Oct 2004 16:13:46 -0600 From: "Kenneth D. Merry" To: Roisin Murphy Message-ID: <20041012221346.GA89589@nargothrond.kdm.org> References: <20041011043508.GA72113@nargothrond.kdm.org> <20041011210303.GA78436@nargothrond.kdm.org> <20041012171106.GB86646@nargothrond.kdm.org> <20041012215951.GA89335@nargothrond.kdm.org> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.4.2i X-Virus-Scanned: clamd / ClamAV version 0.75.1, clamav-milter version 0.75c on nargothrond.kdm.org X-Virus-Status: Clean cc: freebsd-hardware@freebsd.org Subject: Re: sata raid & write cache state X-BeenThere: freebsd-hardware@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: General discussion of FreeBSD hardware List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 12 Oct 2004 22:13:55 -0000 On Tue, Oct 12, 2004 at 15:10:38 -0700, Roisin Murphy wrote: > > Head to www.apc.com and look at the UPSes they have there. They're > > basically lead acid batteries with inverters to put out A/C power when your > > main power feed goes out. > > i meant the battery option they sell with some raid controllers, is > this and extra little PSU, that also powers the disks for a while, or > is this just meant to keep the raid controller's cache powered till > the machine boots up again, so it can finish its cache flush? It just powers the cache until the machine boots again and the controller can flush the cache out. Ken -- Kenneth Merry ken@FreeBSD.ORG From owner-freebsd-hardware@FreeBSD.ORG Wed Oct 13 16:10:50 2004 Return-Path: Delivered-To: freebsd-hardware@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id A4AEA16A4CE for ; Wed, 13 Oct 2004 16:10:50 +0000 (GMT) Received: from muon.mono.org (muon.mono.org [217.206.161.163]) by mx1.FreeBSD.org (Postfix) with ESMTP id 8AF0243D48 for ; Wed, 13 Oct 2004 16:10:49 +0000 (GMT) (envelope-from ben@mono.org) Received: from ben (helo=localhost) by muon.mono.org with local-esmtp (Exim 4.22) id 1CHli5-0001cj-Rc for freebsd-hardware@freebsd.org; Wed, 13 Oct 2004 17:10:41 +0100 Date: Wed, 13 Oct 2004 17:10:41 +0100 (BST) From: Ben To: freebsd-hardware@freebsd.org Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Subject: FDX310 X-BeenThere: freebsd-hardware@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: General discussion of FreeBSD hardware List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 13 Oct 2004 16:10:50 -0000 Has anyone got te Fujitsu FDX310 modem working on FreeBSD? I've got the drivers from http://dsl-linux.tripod.com/index/ but there seems to be a file missing or something. I'm lost. Ben. From owner-freebsd-hardware@FreeBSD.ORG Wed Oct 13 18:09:48 2004 Return-Path: Delivered-To: freebsd-hardware@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 5655F16A4CE for ; Wed, 13 Oct 2004 18:09:48 +0000 (GMT) Received: from kang.dnsalias.com (cpc2-nwrk1-5-0-cust174.nott.cable.ntl.com [81.111.105.174]) by mx1.FreeBSD.org (Postfix) with ESMTP id 36CFB43D31 for ; Wed, 13 Oct 2004 18:09:47 +0000 (GMT) (envelope-from freebsd-hardware@akers-online.co.uk) Received: from gowron.qonos.com (akers-online.co.uk [192.168.123.251]) id i9DI9Fq14306 for ; Wed, 13 Oct 2004 19:09:15 +0100 To: freebsd-hardware@freebsd.org MIME-Version: 1.0 X-Mailer: Lotus Notes Release 6.5.2 June 01, 2004 Message-ID: From: Sean Akers Date: Wed, 13 Oct 2004 19:09:13 +0100 X-MIMETrack: Serialize by Router on gowron/Qonos(Release 6.5.2|June 01, 2004) at 13/10/2004 19:09:15, Serialize complete at 13/10/2004 19:09:15 Content-Type: text/plain; charset="US-ASCII" X-Qonos-MailScanner-Information: Please contact the ISP for more information X-Qonos-MailScanner: Not scanned: please contact your Internet E-Mail Service Provider for details X-Qonos-MailScanner-SpamCheck: not spam (whitelisted), SpamAssassin (score=-2.82, required 4.5, autolearn=not spam, ALL_TRUSTED -2.82) Subject: FreeBSD on Shuttle Zen ST62K XPC ? X-BeenThere: freebsd-hardware@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: General discussion of FreeBSD hardware List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 13 Oct 2004 18:09:48 -0000 I'm wanting to build a new server on which I'd like put FreeBSD. The main criteria hardware-wise is that it must be small and quiet as it must be on 24/7 in the room next to our bedroom. I'm thinking that a Shuttle Zen ST62K XPC will do just fine. Does FreeBSD work OK on these machines ? I'll be running X but don't need accelerated 3D graphics. As long as I can get 1280x1024 24bit I'll be happy. The integrated NIC working correctly is important but I'm not bothered by on-board sound, TV-out and the like. Haven't managed to find any reviews on this machine with FreeBSD on the net yet to I thought I'd ask here. Any info you have would be much appreciated. Cheers, Sean From owner-freebsd-hardware@FreeBSD.ORG Thu Oct 14 03:10:36 2004 Return-Path: Delivered-To: freebsd-hardware@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 7BBCA16A4CE for ; Thu, 14 Oct 2004 03:10:36 +0000 (GMT) Received: from cydem.org (S0106000103ce4c9c.ed.shawcable.net [68.149.254.167]) by mx1.FreeBSD.org (Postfix) with ESMTP id F110D43D2F for ; Thu, 14 Oct 2004 03:10:21 +0000 (GMT) (envelope-from soralx@cydem.org) Received: from S01060020ed3972ba.ed.shawcable.net (S01060020ed3972ba.ed.shawcable.net [68.149.254.42]) by cydem.org (Postfix/FreeBSD) with ESMTP id 779F639563 for ; Wed, 13 Oct 2004 21:10:21 -0600 (MDT) From: To: freebsd-hardware@freebsd.org Date: Wed, 13 Oct 2004 21:10:09 -0600 User-Agent: KMail/1.5.4 MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Content-Disposition: inline Message-Id: <200410132110.09915.soralx@cydem.org> Subject: Linksys PCM200 X-BeenThere: freebsd-hardware@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: General discussion of FreeBSD hardware List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 14 Oct 2004 03:10:36 -0000 Good localtime I've got a Linksys EhterFast PCM200 CardBus ethernet adapter recently, and now I'm trying to make it work with FreeBSD 5.2.1-R, on a Sony PCG-748 notebook. Seems like it is based on a DEC 21143 chipset, specifically ADMtek Centaur-C. After the changes I made to dc(4) (see diffs below), the OS recognizes the card, and is able to attach it. I can assign an IP to dc0, and when I connect it to a switch, the link type [100baseTX, FD] is properly recognized. All 3 LEDs on the card ("Link/Act", "10/100", "FD/COL") appear to work as intended. It even seems arp packets can be sent and recieved (while in promisc mode - monitoring with 'tcpdump'). However, if I try to run dhclient, ping, etc - nothing else works: "dc0: watchdog timeout" Any thoughts? `dmesg`: (booted without ACPI because otherwise the OS halts after a while) Copyright (c) 1992-2004 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD 5.2.1-RELEASE #7: Wed Oct 13 20:31:48 MDT 2004 root@:/usr/src/sys/i386/compile/SORALX Preloaded elf kernel "/boot/kernel/kernel" at 0xc09e0000. Timecounter "i8254" frequency 1193182 Hz quality 0 CPU: Pentium/P55C (quarter-micron) (265.26-MHz 586-class CPU) Origin = "GenuineIntel" Id = 0x581 Stepping = 1 Features=0x8001bf real memory = 167706624 (159 MB) avail memory = 153288704 (146 MB) Intel Pentium detected, installing workaround for F00F bug npx0: [FAST] npx0: on motherboard npx0: INT 16 interface pcibios: BIOS version 2.10 Using $PIR table, 4 entries at 0xc00fdf80 pcib0: at pcibus 0 on motherboard pci0: on pcib0 pci_cfgintr: 0:7 INTD BIOS irq 9 pci_cfgintr: 0:8 INTA BIOS irq 9 isab0: at device 7.0 on pci0 isa0: on isab0 atapci0: port 0xfcd0-0xfcdf at device 7.1 on pci0 ata0: at 0x1f0 irq 14 on atapci0 ata0: [MPSAFE] ata1: at 0x170 irq 15 on atapci0 ata1: [MPSAFE] pci0: at device 7.2 (no driver attached) piix0: port 0x2180-0x218f at device 7.3 on pci0 Timecounter "PIIX" frequency 3579545 Hz quality 0 pci0: at device 8.0 (no driver attached) cbb0: at device 10.0 on pci0 cardbus0: on cbb0 pccard0: <16-bit PCCard bus> on cbb0 pci_cfgintr: 0:10 INTA routed to irq 9 cbb0: [MPSAFE] cbb1: at device 10.1 on pci0 cardbus1: on cbb1 pccard1: <16-bit PCCard bus> on cbb1 pci_cfgintr: 0:10 INTB routed to irq 9 cbb1: [MPSAFE] orm0: