From owner-freebsd-fs@FreeBSD.ORG Sat Mar 26 05:16:47 2011 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 001871065670 for ; Sat, 26 Mar 2011 05:16:46 +0000 (UTC) (envelope-from jdc@koitsu.dyndns.org) Received: from qmta12.westchester.pa.mail.comcast.net (qmta12.westchester.pa.mail.comcast.net [76.96.59.227]) by mx1.freebsd.org (Postfix) with ESMTP id 9FC3B8FC19 for ; Sat, 26 Mar 2011 05:16:46 +0000 (UTC) Received: from omta22.westchester.pa.mail.comcast.net ([76.96.62.73]) by qmta12.westchester.pa.mail.comcast.net with comcast id PhDY1g0031ap0As5ChGmB3; Sat, 26 Mar 2011 05:16:46 +0000 Received: from koitsu.dyndns.org ([67.180.84.87]) by omta22.westchester.pa.mail.comcast.net with comcast id PhGl1g0081t3BNj3ihGl7o; Sat, 26 Mar 2011 05:16:46 +0000 Received: by icarus.home.lan (Postfix, from userid 1000) id DE4259B429; Fri, 25 Mar 2011 22:16:43 -0700 (PDT) Date: Fri, 25 Mar 2011 22:16:43 -0700 From: Jeremy Chadwick To: Michael DeMan Message-ID: <20110326051643.GA43364@icarus.home.lan> References: <20110326003818.GT78089@deviant.kiev.zoral.com.ua> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.21 (2010-09-15) Cc: freebsd-fs@freebsd.org Subject: Re: tying down adaX to physical interfaces X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 26 Mar 2011 05:16:47 -0000 On Fri, Mar 25, 2011 at 05:57:10PM -0700, Michael DeMan wrote: > Yeah - this is what I was looking for. I can morph it for mvsch. > > In regards to prior post... > The glabel solution works too, but again gets complicated when a regular staff person just needs to swap out a drive, has the disk drive in hand (which could either not have a glabel on it or maybe has one from somewhere else by accident), and just needs to replace a drive. > > Thanks! > > - mike > > On Mar 25, 2011, at 5:38 PM, Kostik Belousov wrote: > > > On Fri, Mar 25, 2011 at 05:21:50PM -0700, Michael DeMan wrote: > >> Hi All, > >> > >> I seem to recall that there is a way to do this, but can no longer google it. > >> > >> Basically, for NCQ support with SATA devices we are using the 'ada' driver, which of course has SCSI like behavior. > >> > >> With two 8-port disk controllers in the system, we end up in the uncomfortable situation where if the box is rebooted with an 'earlier' drive in the list, the system boots up with different drives mapped to adaX. This becomes more of a problem with multiple drive controllers and not all the ports populated with drives at the start (i.e, add more drives later). > >> > >> In our case, we have two Marvell controllers, and are doing ZFS mirroring with drives off each controller. > >> > >> We end up with mvsch.0 through mvsch.15 showing up (16 controller interfaces), but with only 10 drives right now, we end up with ada0 through ada9. If we lose a drive and reboot, then we end up with the upper ones each being ada(X-1), etc. If we add more drives, the situation becomes even more confusing. > >> > >> Is there a way to force at boot time a mapping from the mvsch interface to the specific ada disk that the operating system sees? Pretty sure I saw something about this last summer but can't seem to find it now. > >> > >> What we would like to have, with 10 disks, is them showing up as: > >> ada0, ada1, ada2, ada3, ada4 > >> ada8, ada9, ada10, ada11, ada12 > >> > >> If we add another pair of disks in the future, we would know where to physically insert them for ada5 and ada13, etc. > > > > I use the following stanza in /boot/device.hints for machine with intel > > on-board ahci and siis in pcie: > > hint.scbus.0.at="ahcich0" > > hint.ada.0.at="scbus0" > > hint.scbus.1.at="ahcich1" > > hint.ada.1.at="scbus1" > > hint.scbus.2.at="ahcich2" > > hint.ada.2.at="scbus2" > > hint.scbus.3.at="ahcich3" > > hint.ada.3.at="scbus3" > > hint.scbus.4.at="ahcich4" > > hint.ada.4.at="scbus4" > > hint.scbus.5.at="siisch0" > > hint.ada.5.at="scbus5" > > hint.scbus.6.at="siisch1" > > hint.ada.6.at="scbus6" > > > > You should get an idea from this. Why would the disk number change on a drive swap? This doesn't happen on my Intel ICH9R controller on *any* Supermicro X7xxx or Supermicro PDxxx board I have access to. The ONLY TIME I have seen this happen is in the following situation: 1) State: physical SATA port 0 <--> hot-swap backplane port 0 <--> disk ada0 physical SATA port 1 <--> hot-swap backplane port 1 <--> disk ada1 physical SATA port 2 <--> hot-swap backplane port 2 <--> empty physical SATA port 3 <--> hot-swap backplane port 3 <--> disk ada2 physical SATA port 4 <--> hot-swap backplane port 4 <--> disk ada3 physical SATA port 5 <--> hot-swap backplane port 5 <--> disk ada4 2) Administrator adds disk to backplane port 2. Disk appears as ada5. 3) Machine gets shut off, rebooted, whatever. 4) Kernel starts. State then becomes: physical SATA port 0 <--> hot-swap backplane port 0 <--> disk ada0 physical SATA port 1 <--> hot-swap backplane port 1 <--> disk ada1 physical SATA port 2 <--> hot-swap backplane port 2 <--> disk ada2 physical SATA port 3 <--> hot-swap backplane port 3 <--> disk ada3 physical SATA port 4 <--> hot-swap backplane port 4 <--> disk ada4 physical SATA port 5 <--> hot-swap backplane port 5 <--> disk ada5 I should note that in my experience (on multiple systems/boards/chassis) hot-swapping a disk *does not* change the disk number. E.g. yanking ada3 results in the SATA channel seeing a disconnection, and an insertion results in the SATA channel seeing an insertion of ada3. Two weeks ago I did this a good 15-16 times while I zeroed out all the old SATA disks I had laying around my flat. Anyway, the way I've addressed the problem described is: leave your empty disks at the "end of the SATA port chain" (I'm using the word chain incorrectly here; I am well aware SATA is 1:1 and not chained). If you *have* to move disks around/empty bays out which were previously filled, you should boot into single-user and fix things there before actually bringing up multi-user mode. I've mentioned this before -- and I'm certain others will point out that said problem was fixed, etc. etc. (done so before, and I've simply forgotten) -- but there are complexities introduced into the fray when using glabel(8) and similar utilities. There have been complaints on the lists about it. I've had a very, very long week and can't be bothered to dig up the evidence; sorry. This is one great thing about ZFS and disk tasting -- it doesn't matter if the disks all "shift up by one" (number-wise), it figures it out and does the Right Thing(tm) with pools as a result. This is why I tend to use ZFS on non-OS disks, that way if the above situation happens, I'm not fighting with the system after a disk replacement. -- | Jeremy Chadwick jdc@parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP 4BD6C0CB |