From owner-freebsd-fs@FreeBSD.ORG Mon Jun 10 02:20:14 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 0F5C263D for ; Mon, 10 Jun 2013 02:20:14 +0000 (UTC) (envelope-from jdc@koitsu.org) Received: from relay5-d.mail.gandi.net (relay5-d.mail.gandi.net [217.70.183.197]) by mx1.freebsd.org (Postfix) with ESMTP id AA69F1622 for ; Mon, 10 Jun 2013 02:20:13 +0000 (UTC) Received: from mfilter14-d.gandi.net (mfilter14-d.gandi.net [217.70.178.142]) by relay5-d.mail.gandi.net (Postfix) with ESMTP id 15EC441C064; Mon, 10 Jun 2013 04:19:57 +0200 (CEST) X-Virus-Scanned: Debian amavisd-new at mfilter14-d.gandi.net Received: from relay5-d.mail.gandi.net ([217.70.183.197]) by mfilter14-d.gandi.net (mfilter14-d.gandi.net [10.0.15.180]) (amavisd-new, port 10024) with ESMTP id A5-+RIutWa4L; Mon, 10 Jun 2013 04:19:55 +0200 (CEST) X-Originating-IP: 76.102.14.35 Received: from jdc.koitsu.org (c-76-102-14-35.hsd1.ca.comcast.net [76.102.14.35]) (Authenticated sender: jdc@koitsu.org) by relay5-d.mail.gandi.net (Postfix) with ESMTPSA id B5BCB41C05C; Mon, 10 Jun 2013 04:19:54 +0200 (CEST) Received: by icarus.home.lan (Postfix, from userid 1000) id D00C773A1C; Sun, 9 Jun 2013 19:19:50 -0700 (PDT) Date: Sun, 9 Jun 2013 19:19:50 -0700 From: Jeremy Chadwick To: "Reed A. Cartwright" Subject: Re: ZFS and Glabel Message-ID: <20130610021950.GA50356@icarus.home.lan> References: <20130609065430.GA28206@icarus.home.lan> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.21 (2010-09-15) Cc: freebsd-fs@freebsd.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 10 Jun 2013 02:20:14 -0000 On Sun, Jun 09, 2013 at 01:25:47PM -0700, Reed A. Cartwright wrote: > Thanks, it makes sense now. > > Would it make sense to have a script that processes the output of "cam > devlist -v" to produce such an example output? > > On Sat, Jun 8, 2013 at 11:54 PM, Jeremy Chadwick wrote: > > On Sat, Jun 08, 2013 at 09:46:18PM -0700, Reed A. Cartwright wrote: > >> I'm looking at my dmesg.boot to figure out what settings I need to > >> wire down my HDDs. I read the cam(4) documentation but I'm not sure I > >> know what I'm doing. Any advice would be helpful. > >> > >> Let's assume that I want to wire everything down to their current > >> positions, what should I put in loader.conf? I'll paste below some of > >> my hardware configuration and lines from dmesg.boot that I think I > >> need to look at. > >> > >> I have 4 LSI cards in the system: mps0, mps1, mps2, mps3. > >> > >> mps0: port 0xd000-0xd0ff mem > >> 0xdff3c000-0xdff3ffff,0xdff40000-0xdff7ffff irq 24 at device 0.0 on > >> pci5 > >> mps1: port 0xc000-0xc0ff mem > >> 0xdfe3c000-0xdfe3ffff,0xdfe40000-0xdfe7ffff irq 44 at device 0.0 on > >> pci4 > >> mps2: port 0xb000-0xb0ff mem > >> 0xdfd3c000-0xdfd3ffff,0xdfd40000-0xdfd7ffff irq 32 at device 0.0 on > >> pci3 > >> mps3: port 0xe000-0xe0ff mem > >> 0xdbf3c000-0xdbf3ffff,0xdbf40000-0xdbf7ffff irq 56 at device 0.0 on > >> pci65 > >> > >> I have drives attached to two of those cards: > >> > >> da0 at mps0 bus 0 scbus0 target 0 lun 0 > >> da1 at mps0 bus 0 scbus0 target 1 lun 0 > >> da2 at mps0 bus 0 scbus0 target 2 lun 0 > >> da3 at mps0 bus 0 scbus0 target 3 lun 0 > >> da4 at mps0 bus 0 scbus0 target 4 lun 0 > >> da5 at mps0 bus 0 scbus0 target 5 lun 0 > >> da6 at mps0 bus 0 scbus0 target 6 lun 0 > >> da7 at mps0 bus 0 scbus0 target 7 lun 0 > > > >> da8 at mps3 bus 0 scbus9 target 0 lun 0 > >> da9 at mps3 bus 0 scbus9 target 1 lun 0 > >> da10 at mps3 bus 0 scbus9 target 2 lun 0 > >> da11 at mps3 bus 0 scbus9 target 3 lun 0 > >> da12 at mps3 bus 0 scbus9 target 4 lun 0 > >> > >> {snip} > > > > As usual, the situation is insane because you have so many controllers > > on the system (more than just mps(4)) -- specifically 11 separate > > controllers or systems using CAM (hence scbus0 to scbus10). > > > > Below is for mps(4). If you want to wire down ahci(4), things are > > a bit different, but you can read this post of mine: > > > > http://lists.freebsd.org/pipermail/freebsd-stable/2013-January/071851.html > > > > Enjoy: > > > > hint.scbus.0.at="mps0" > > hint.scbus.1.at="mps1" > > hint.scbus.2.at="mps2" > > hint.scbus.9.at="mps3" > > hint.da.0.at="scbus0" > > hint.da.1.at="scbus0" > > hint.da.2.at="scbus0" > > hint.da.3.at="scbus0" > > hint.da.4.at="scbus0" > > hint.da.5.at="scbus0" > > hint.da.6.at="scbus0" > > hint.da.7.at="scbus0" > > hint.da.8.at="scbus9" > > hint.da.9.at="scbus9" > > hint.da.10.at="scbus9" > > hint.da.11.at="scbus9" > > hint.da.12.at="scbus9" > > hint.da.13.at="scbus9" > > hint.da.14.at="scbus9" > > hint.da.15.at="scbus9" > > hint.da.16.at="scbus1" > > hint.da.17.at="scbus1" > > hint.da.18.at="scbus1" > > hint.da.19.at="scbus1" > > hint.da.20.at="scbus1" > > hint.da.21.at="scbus1" > > hint.da.22.at="scbus1" > > hint.da.23.at="scbus1" > > hint.da.24.at="scbus2" > > hint.da.25.at="scbus2" > > hint.da.26.at="scbus2" > > hint.da.27.at="scbus2" > > hint.da.28.at="scbus2" > > hint.da.29.at="scbus2" > > hint.da.30.at="scbus2" > > hint.da.31.at="scbus2" > > hint.da.0.target="0" > > hint.da.1.target="1" > > hint.da.2.target="2" > > hint.da.3.target="3" > > hint.da.4.target="4" > > hint.da.5.target="5" > > hint.da.6.target="6" > > hint.da.7.target="7" > > hint.da.8.target="0" > > hint.da.9.target="1" > > hint.da.10.target="2" > > hint.da.11.target="3" > > hint.da.12.target="4" > > hint.da.13.target="5" > > hint.da.14.target="6" > > hint.da.15.target="7" > > hint.da.16.target="0" > > hint.da.17.target="1" > > hint.da.18.target="2" > > hint.da.19.target="3" > > hint.da.20.target="4" > > hint.da.21.target="5" > > hint.da.22.target="6" > > hint.da.23.target="7" > > hint.da.24.target="0" > > hint.da.25.target="1" > > hint.da.26.target="2" > > hint.da.27.target="3" > > hint.da.28.target="4" > > hint.da.29.target="5" > > hint.da.30.target="6" > > hint.da.31.target="7" The script would be ugly and require one-offs per driver. For example, look at your "camcontrol devlist -v" output with regards to mps1 and mps2. There's no indication of what the bus #, target #, or lun # should be because there are no disks on the controller. I made an educated guess based off of mps0/mps3 and previous familiarity (on the lists -- I've never used one of these controllers) with mps(4). Had you shown me "camcontrol devlist -v" output with only 1 controller and 1 disk, I would have had to go purely off of what I've seen in the past. The behaviour could change as well, depending on firmware upgrades or driver changes (many of the storage drivers in FreeBSD in the past 3-4 years have gone through massive changes), or even operational mode (RAID vs. non-RAID), where the target then is always 0 but the lun # increases, or maybe the bus number, or maybe a combination. If someone really wants to take a stab at writing some script that does this, be my guest, but I definitely don't. :-) There's just too many one-offs or assumptions that have to be made which a human mind + experience can do more reliably, IMO. Because remember: the last thing you want to do is modify loader.conf for wiring down and botch it/break it. -- | Jeremy Chadwick jdc@koitsu.org | | UNIX Systems Administrator http://jdc.koitsu.org/ | | Making life hard for others since 1977. PGP 4BD6C0CB |