From owner-freebsd-stable Tue Jul 13 19:43:16 1999 Delivered-To: freebsd-stable@freebsd.org Received: from feral.com (feral.com [192.67.166.1]) by hub.freebsd.org (Postfix) with ESMTP id 9D84914D13; Tue, 13 Jul 1999 19:43:02 -0700 (PDT) (envelope-from mjacob@feral.com) Received: from semuta.feral.com (semuta [192.67.166.70]) by feral.com (8.8.7/8.8.7) with ESMTP id TAA07743; Tue, 13 Jul 1999 19:41:01 -0700 Date: Tue, 13 Jul 1999 19:40:56 -0700 (PDT) From: Matthew Jacob Reply-To: mjacob@feral.com To: "Robert J. Adams" Cc: hardware@FreeBSD.ORG, stable@FreeBSD.ORG Subject: Re: Fibre Channel Controller [ LONG RESPONSE ] In-Reply-To: <121f01becd68$46c41120$3102fbd1@siscom.net> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-freebsd-stable@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.ORG On Tue, 13 Jul 1999, Robert J. Adams wrote: > Have you used these? A lot. Not only for FreeBSD, but Solaris, Linux, OpenBSD, NetBSD, NT, etc.. > If so, how does the FC route compare to say U2W SCSI? > Something about a little piece of fiber running my drives makes me feel all > warm and fuzzy. I'd have to say it's a mixed experience at this time. The data rates for U2W SCSI busses (and U3W which I'm getting a sample card for in about a month), what with PCI latencies, are comparable to what you get for 1024Mbit Fibre Channel (the difference between 40 Mhz x 16 bits and 100MB is not a lot, really). The drives for U2W or Fibre Channel are the same (but with different interfaces), so if you're running a small setup, U2W is probably a better choice (see below). Now, that's data rates. What turns out to be somewhat significant (particularly for larger bus configurations) is that the non-data transfer phases stuff on SCSI is the same data rate as it's always been (async mode, varying between 2 && 8MB/s depending on bus length)- Additionally, Request Sense is a completely separate command. In Fibre Channel SCSI, command and device control functions go out as a single FC packet, data moves as multiple packets, and the response that includes status *and* sense data (as needed) goes as one packet, and they all go at the same rate. This gives more headroom that you don't get in parallel SCSI. A classic amusement to me about this is an arrangement at NASA/Ames where differential SCSI is carried to a fibre optic repeater and then over fibre optics to a building about a kilometer away to tape drives in a silo (for offsite secondaries)- the tape drives really do take the data bursts at a healthy 10Mhz, 16 bits wide, but because command, mesage and status bytes, and inter-burst times, all operate at async data rates, the effective data rate is about 5MB/s. Whuf. What you really get with Fibre Channel that you don't get with U2W is a very large address space that configures a little bit more sanely than parallel SCSI. With private loop (that's the best tested at this time), you get 126 loop addresses, so you can put a lot of devices on one loop. You can ask (as the FreeBSD driver does do) for a specific loop address, but you'll just get what's available after arbitration. No more of this "oops, two devices have the same ID, we're hosed!". Don't. Even. Think. About. Mentioning. SCAM. Thank you. On the other hand, the code that's currently in FreeBSD that trys to manage loop address rearbitration where it's possible to come back after a loop initialization ('coz joe bob rebooted his workstation) is not very well tested yet. The Fabric support (which allows you to have, in theory, an arbitrarily large disk address space, but in practice for the way the current fabric management is being done allows only another 127 disks, has a lot more wringing out to be done too). You also get a bit more tolerant interconnect technology (whether you're using FC Optical or FC copper) where cable distance pretty much becomes a non-issue. But you also pay for this. For U2W you can expect to pay a tad more for the disks && the controllers and LVD cabling, and whatever your normal SCSI disk case costs are, but for Fibre Channel you should expect to budget for FC hubs (at a minimum 2k$ for a ~10 port hub), beaucoup more if you get a switch (for fabrics). The FC adapters are also more expensive than SCSI, but FC disks are in fact often *cheaper* than LVD or Ultra disks (why? I dunno- probably because the FC connector is a 40 pin SCA rather than an 80 pin SCA or 68 pin high density.... and the actual bus interface technology basically only has to manage 4 twisted pairs plus grounds). The real killer here is that boxes to put your fibre channel disks in are not only expensive, they're harder than hell to find, particularly in unit 1 quantities. So, to get back to your question: I'd have to say that FC is cool, and will really be awesome over time, but it's just *barely* at the level you could put it together in your own shop with. Caveat Emptor. Man, it's painful to admit this, but it's true. -matt To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-stable" in the body of the message