Date: Mon, 29 Sep 2014 19:24:00 -0500 From: Karl Denninger <karl@denninger.net> To: freebsd-stable@freebsd.org Subject: Re: MPS Message-ID: <5429F820.6040305@denninger.net> In-Reply-To: <CAN6yY1v2MyOMahcTEagjxKQ33q56MmSwHx87_ZqyVYR5RDH8Mw@mail.gmail.com> References: <5429BB41.8080609@denninger.net> <018BC41041EE4DF589EF5D834AD45BAD@multiplay.co.uk> <CAN6yY1v2MyOMahcTEagjxKQ33q56MmSwHx87_ZqyVYR5RDH8Mw@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
[-- Attachment #1 --] On 9/29/2014 7:17 PM, Kevin Oberman wrote: > On Mon, Sep 29, 2014 at 1:25 PM, Steven Hartland <killing@multiplay.co.uk> > wrote: > >> I seem to remember detection being made parallel for speed, as there >> can be lots of drives. The Avago guys may be able to shed more light. >> >> Regards >> Steve >> >> ----- Original Message ----- From: "Karl Denninger" <karl@denninger.net> >> To: <freebsd-stable@freebsd.org> >> Sent: Monday, September 29, 2014 9:04 PM >> Subject: MPS >> >> >> >> Has anyone found a set of configuration settings for the mps driver (LSI >> SAS series cards) that gets the following situation under control? >> >> I have a machine here with one of these cards and a SAS expander. The >> expander has 16 ports, the card has 8 -- one connector attached to the >> expander. So I have 20 potential drives associated with this adapter. >> >> The issue is that the driver makes utterly no sense when it comes to >> assigning drive designations. In theory it should enumerate in series; >> that is, target 0 on the card should be da0, target 1 da1, etc. This >> assumes there are no gaps in the attached drives; if there are then >> things may move around, which is why some people wire drives. >> >> Ok so far, but it doesn't behave that way! >> >> This is the excerpt of what it returns: >> >> root@Dbms-10:/boot # dmesg|grep mps >> mps0: <LSI SAS2008> port 0xc000-0xc0ff mem >> 0xfbb3c000-0xfbb3ffff,0xfbb40000-0xfbb7ffff irq 30 at device 0.0 on pci3 >> mps0: Firmware: 19.00.00.00, Driver: 19.00.00.00-fbsd >> mps0: IOCCapabilities: >> 1285c<ScsiTaskFull,DiagTrace,SnapBuf,EEDP,TransRetry,EventReplay,HostDisc> >> ses0 at mps0 bus 0 scbus0 target 8 lun 0 >> da0 at mps0 bus 0 scbus0 target 0 lun 0 >> da1 at mps0 bus 0 scbus0 target 1 lun 0 >> da2 at mps0 bus 0 scbus0 target 2 lun 0 >> da3 at mps0 bus 0 scbus0 target 3 lun 0 >> *da4 at mps0 bus 0 scbus0 target 6 lun 0** >> **da5 at mps0 bus 0 scbus0 target 7 lun 0* >> >> Those last two are the drives the card identifies as being on targets 0 >> and 1 when looked at in the BIOS. The other four drives (0-3) are on >> the SAS expander -- and there's a two drive "gap" in the targets >> identified as well which has zippo for logic associated with it. >> >> Even more-oddly if I swap the plugs -- that is, plug the two boot disks >> into the OTHER connector (putatively targets 4-7) and put the SAS >> expander on the 0-3 port connector the BIOS *does* show the target shift >> but the machine, when it boots, still places those two drives on targets >> 6 and 7! >> >> There is nothing in /boot/loader.conf or /boot/device.hints that relates >> to these devices at all. >> >> I would like to wire down the various ports so Drive Bay #x corresponds >> to da[x] but how can you when you don't know what order they will come >> up in predicated on either the physical port they're occupying or, for >> that matter, what the card's BIOS shows them as? >> >> I looked in the driver's man page and saw nothing related to this; I get >> around it by labeling the drives and then using "glabel list" to see >> what da[x] number it grabbed if I need to pull a disk while the machine >> is up, but this is a rather-unfriendly way to handle that. >> >> -- >> Karl Denninger >> karl@denninger.net <mailto:karl@denninger.net> >> /The Market Ticker/ >> > Karl, > > This problem seems to crop up with some controllers, especially when port > expanders are used. > > If at all possible, I really recommend labeling them and using the labels > in lieu of device names. I stopped using device names some time ago just > because of this. I do recommend using the labeling for the file system > involved as opposed to glabel, if it has one. Certainly UFS does and I > thought ZFS did, as well, but I am less sure of that as ZFS started to > stabilize about the time I stopped dealing with big systems. I still find > UFS quite adequate for my baby server and laptop. I do use glabel for swap. > > In any case, using labeling will give you stable names to put into things > like your fstab and not make you worry about the actual device number. And, > of course, talking to the manufacturer is also a good idea, espacially when > they don't say "FreeBSD? We don't support that.". > -- > R. Kevin Oberman, Network Engineer, Retired > E-mail: rkoberman@gmail.com > Yeah, that's been my answer along with lettering the drive carriers but it pisses me off. Ah well if that's how it is then that's how it is. :-) Glabel works fine across the board provided you put it on a slice rather than a disk, which I do anyway so as to maintain "nice-nice" with AF (4k) disks along with leaving a small amount of space at the end unallocated, so if by some chance a different brand replacement drive isn't quite the same number of sectors it can be dropped into the pool as a replacement. -- Karl Denninger karl@denninger.net <mailto:karl@denninger.net> /The Market Ticker/ [-- Attachment #2 --] 0 *H 01 0 + 0 *H O0K030 *H 01 0 UUS10UFlorida10U Niceville10U Cuda Systems LLC1 0UCuda Systems LLC CA1/0- *H customer-service@cudasystems.net0 130824190344Z 180823190344Z0[1 0 UUS10UFlorida10UKarl Denninger1!0 *H karl@denninger.net0"0 *H 0 bi՞] MNԿawx?`)'ҴcWgR@BlWh+ u}ApdCF JVй~FOL }EW^bچYp3K& ׂ(R lxڝ.xz?6 &nsJ +1v9 v/( kqĪp[vjcK%fϻe?iq]z lyzFO'ppdX//Lw(3 JI A*S#՟H[f|CGqJKoo y.oEuOw$ /섀$삻J9b|A P~8]D1YI<"""Y^T2iQ2 b yH)] Ƶ0y$ _N 6XqMC 9 XgώjGTP"#nˋ"Bk1 00 U 0 0 `HB0 U 0, `HB OpenSSL Generated Certificate0 U |8 ˴d[20 U #0]Af4 U3x&^"408 `HB+)https://cudasystems.net:11443/revoked.crl0 *H gBwH] j\x`( &gW32"Uf^. ^Iϱ k!DQA g{(w /)\N'[ oRW @C HO>)XrTNɘ!u`xt5(=f\-l3<@C6mnhv##1ŃbH͍_Nq aʷ?rk$^9TIa!kh,D -ct1 0001 0 UUS10UFlorida10U Niceville10U Cuda Systems LLC1 0UCuda Systems LLC CA1/0- *H customer-service@cudasystems.net0 + ;0 *H 1 *H 0 *H 1 140930002400Z0# *H 1uA<~ i# `0l *H 1_0]0 `He*0 `He0 *H 0*H 0 *H @0+0 *H (0 +71001 0 UUS10UFlorida10U Niceville10U Cuda Systems LLC1 0UCuda Systems LLC CA1/0- *H customer-service@cudasystems.net0 *H 101 0 UUS10UFlorida10U Niceville10U Cuda Systems LLC1 0UCuda Systems LLC CA1/0- *H customer-service@cudasystems.net0 *H ʯu + T, *9^ GʶӮ'>Xs]UBjI$ ??g_n&V$kSD_k!pDjۖ穙|6h JQ.;^E]η 9BzE㤎GoՖ}a9@ʼ hx@a"&(SI@ g˸'%`A~H;C1J0LgH+=տI7$\u991hqtWa=ܟQi]UOo,}V4t8 2-Qh z B (#m))q3Qb4I4ͨ^N5@^Rx' ۶twxO (xD!fo7&b0D~,u)\Je"v |Br0TZA lѕ"Ou.ͦ6cQ[6 ЙWsط }
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?5429F820.6040305>