Date: Thu, 8 Apr 2021 02:04:30 -0400 From: Daniel Morante <daniel@morante.net> To: freebsd-arm@freebsd.org Subject: LSI SAS3008 on 13-RC5 and 14-CURRENT Message-ID: <1db2b369-cec6-058a-67ad-f762064b3832@morante.net>
index | next in thread | raw e-mail
[-- Attachment #1 --] I done extensive testing with many different configurations. I apologize in advance for the length of this 😉. I have an arm64 system with an onboard SAS controller (Broadcom® SAS3008) and expander (Broadcom® SAS35x36R). This system also has an onboard SATA controller that is currently unusable in any version of FreeBSD ARM64 (https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=254651). As a result I am limited to using the SAS3008. I've noticed strange behavior on 13-RC3/4/5 and on the latest 14-CURRENT snapshot (2021-04-01).In the controller firmware I can create a logical volume (RAID) or use the individual drives (non-RAID/pass thru). In RAID mode there is a only logical volume and no individual drives. I use the DVD (or disc1) ISO image as the installation media. I load the kernel module (kldload mpr) using a shell and then proceed to install FreeBSD using either ZFS or UFS on root. With 13.0 the drives or logical volumes are detected correctly. On 14.0 no logical volumes are detected. mpr0: <Avago Technologies (LSI) SAS3008> mem 0x42140000-0x4214ffff,0x42100000-0x4213ffff at device 0.0 numa-domain 0 on pci14 mpr0: Firmware: 15.00.00.00, Driver: 23.00.00.00-fbsd mpr0: IOCCapabilities: 6985c<ScsiTaskFull,DiagTrace,SnapBuf,EEDP,TransRetry,IR,MSIXIndex,FastPath,RDPQArray> mpr0: Found device <81<SataDev>,End Device> <6.0Gbps> handle<0x000a> enclosureHandle<0x0002> slot 0 mpr0: At enclosure level 0 and connector name ( ) mpr0: Found device <81<SataDev>,End Device> <6.0Gbps> handle<0x000b> enclosureHandle<0x0002> slot 1 mpr0: At enclosure level 0 and connector name ( ) mpr0: Found device <4411<SmpInit,SspTarg,SepDev>,End Device> <12.0Gbps> handle<0x000c> enclosureHandle<0x0002> slot 16 mpr0: At enclosure level 0 and connector name ( ) This volume only shows up on 13.0 and not on 14.0: da1 at mpr0 bus 0 scbus2 target 0 lun 0 da1: <LSI Logical Volume 3000> Fixed Direct Access SPC-4 SCSI device da1: Serial Number 2692521697153322079 da1: 150.000MB/s transfers da1: Command Queueing enabled da1: 226974MB (464842752 512 byte sectors) In Non-RAID mode individual drives will be shown in both 13.0 and 14.0. I can proceed through the OS installation, but it's painfully slow. Towards the end (at which point I assume it's writing bootcode... before asking to set the root password), the following messages begin to appear in the console: mpr0: IOC Fault 0x40002622, Resetting mpr0: Reinitializing controller mpr0: Firmware: 15.00.00.00, Driver: 23.00.00.00-fbsd mpr0: IOCCapabilities: 6985c<ScsiTaskFull,DiagTrace,SnapBuf,EEDP,TransRetry,IR,MSIXIndex,FastPath,RDPQArray> mpr0: Found device <81<SataDev>,End Device> <6.0Gbps> handle<0x000a> enclosureHandle<0x0002> slot 0 mpr0: At enclosure level 0 and connector name ( ) mpr0: Found device <81<SataDev>,End Device> <6.0Gbps> handle<0x000b> enclosureHandle<0x0002> slot 1 mpr0: At enclosure level 0 and connector name ( ) mpr0: Found device <4411<SmpInit,SspTarg,SepDev>,End Device> <12.0Gbps> handle<0x000c> enclosureHandle<0x0002> slot 16 mpr0: At enclosure level 0 and connector name ( ) mpr0: IOC Fault 0x40002622, Resetting mpr0: Reinitializing controller mpr0: Firmware: 15.00.00.00, Driver: 23.00.00.00-fbsd mpr0: IOCCapabilities: 6985c<ScsiTaskFull,DiagTrace,SnapBuf,EEDP,TransRetry,IR,MSIXIndex,FastPath,RDPQArray> mpr0: Found device <81<SataDev>,End Device> <6.0Gbps> handle<0x000a> enclosureHandle<0x0002> slot 0 mpr0: At enclosure level 0 and connector name ( ) mpr0: Found device <81<SataDev>,End Device> <6.0Gbps> handle<0x000b> enclosureHandle<0x0002> slot 1 mpr0: At enclosure level 0 and connector name ( ) mpr0: Found device <4411<SmpInit,SspTarg,SepDev>,End Device> <12.0Gbps> handle<0x000c> enclosureHandle<0x0002> slot 16 mpr0: At enclosure level 0 and connector name ( ) mpr0: IOC Fault 0x40002622, Resetting mpr0: Reinitializing controller mpr0: Firmware: 15.00.00.00, Driver: 23.00.00.00-fbsd mpr0: IOCCapabilities: 6985c<ScsiTaskFull,DiagTrace,SnapBuf,EEDP,TransRetry,IR,MSIXIndex,FastPath,RDPQArray> mpr0: Found device <81<SataDev>,End Device> <6.0Gbps> handle<0x000a> enclosureHandle<0x0002> slot 0 mpr0: At enclosure level 0 and connector name ( ) mpr0: Found device <81<SataDev>,End Device> <6.0Gbps> handle<0x000b> enclosureHandle<0x0002> slot 1 mpr0: At enclosure level 0 and connector name ( ) mpr0: Found device <4411<SmpInit,SspTarg,SepDev>,End Device> <12.0Gbps> handle<0x000c> enclosureHandle<0x0002> slot 16 mpr0: At enclosure level 0 and connector name ( ) mpr0: IOC Fault 0x40002622, Resetting mpr0: Reinitializing controller mpr0: Firmware: 15.00.00.00, Driver: 23.00.00.00-fbsd mpr0: IOCCapabilities: 6985c<ScsiTaskFull,DiagTrace,SnapBuf,EEDP,TransRetry,IR,MSIXIndex,FastPath,RDPQArray> (da1:mpr0:0:0:0): Invalidating pack g_vfs_done():da1p1[WRITE(offset=512, length=4096)]error = 6 g_vfs_done():da1p1[WRITE(offset=33792, length=4096)]error = 6 da1 at mpr0 bus 0 scbus2 target 0 lun 0 da1: <LSI Logical Volume 3000> s/n 2692521697153322079 detached g_vfs_done():da1p1[WRITE(offset=99840, length=16384)]error = 6 mpr0: Found device <81<SataDev>,End Device> <6.0Gbps> handle<0x000a> enclosureHandle<0x0002> slot 0 mpr0: At enclosure level 0 and connector name ( ) mpr0: Found device <81<SataDev>,End Device> <6.0Gbps> handle<0x000b> enclosureHandle<0x0002> slot 1 mpr0: At enclosure level 0 and connector name ( ) mpr0: Found device <4411<SmpInit,SspTarg,SepDev>,End Device> <12.0Gbps> handle<0x000c> enclosureHandle<0x0002> slot 16 mpr0: At enclosure level 0 and connector name ( ) mpr0: IOC Fault 0x40002622, Resetting mpr0: Reinitializing controller (da1:mpr0:0:0:0): WRITE(10). CDB: 2a 00 00 00 0a cb 00 08 00 00 (da1:mpr0:0:0:0): CAM status: SCSI Bus Reset Sent/Received (da1:mpr0:0:0:0): Error 5, Periph was invalidated g_vfs_done():da1p1[WRITE(offset=1394176, length=1048576)]error = 5 (da1:mpr0:0:0:0): WRITE(6). CDB: 0a 00 01 0b 20 00 (da1:mpr0:0:0:0): CAM status: SCSI Bus Reset Sent/Received (da1:mpr0:0:0:0): Error 5, Periph was invalidated g_vfs_done():da1p1[WRITE(offset=116224, length=16384)]error = 5 fsync: giving up on dirty (error = 5) (da1:mpr0:0:0:0): WRITE(10). CDB: 2a 00 01 d7 0d 08 00 00 08 00 0xffffa0002797b5b8: (da1:mpr0:0:0:0): CAM status: SCSI Bus Reset Sent/Received type VCHR (da1:mpr0:0:0:0): Error 5, Periph was invalidated usecount 1, writecount 0, refcount 6 seqc users 0(da1:mpr0:0:0:0): WRITE(10). CDB: 2a 00 01 52 b0 a8 00 00 08 00 rdev 0xffffa0000af73400 (da1:mpr0:0:0:0): CAM status: SCSI Bus Reset Sent/Received hold count flags () mpr0: Firmware: 15.00.00.00, Driver: 23.00.00.00-fbsd flags ()mpr0: IOCCapabilities: 6985c<ScsiTaskFull,DiagTrace,SnapBuf,EEDP,TransRetry,IR,MSIXIndex,FastPath,RDPQArray> (da1:mpr0:0:0:0): Error 5, Periph was invalidated v_object 0xffffa00027a6f528 ref 0 pages 27 cleanbuf 2 dirtybuf 1 lock type devfs: EXCL by thread 0xffffa0000aac3000 (pid 271, syncer, tid 101634) with exclusive waiters pending dev da1p1 mpr0: Found device <81<SataDev>,End Device> <6.0Gbps> handle<0x000a> enclosureHandle<0x0002> slot 0 mpr0: At enclosure level 0 and connector name ( ) mpr0: Found device <81<SataDev>,End Device> <6.0Gbps> handle<0x000b> enclosureHandle<0x0002> slot 1 mpr0: At enclosure level 0 and connector name ( ) mpr0: Found device <4411<SmpInit,SspTarg,SepDev>,End Device> <12.0Gbps> handle<0x000c> enclosureHandle<0x0002> slot 16 mpr0: At enclosure level 0 and connector name ( ) The above continues indefinitely and the system eventually locks up (no kernel panic). I've tried it countless times and it always ends the same. Regardless of how I configure the controller (RAID 1, 0, or pass thru). To rule out a bad controller I test with Ubuntu 20.10 for arm64. No issues, everything is detected, SAS, onboard SATA, and Ethernet. Not surprising since Gigabyte has Ubuntu on the compatibility list for this system. All hardware is confirmed to be functional. This is were things get strange... I go back to test FreeBSD 13.0 and I no longer have issues with the SAS controller (onboard SATA still not working). It seems that installing/booting Ubuntu did something to 'fix' the problem? Maybe it loaded some type of firmware or places some persistent data in UEFI? (I'm not sure how that works). Regardless I was able to install FreeBSD on to the SAS controller, and I can boot it. Has anyone seen this happen before? Should I file another bug report? 14.0 btw, still can't see logical volumes. Should that be a separate bug report? [-- Attachment #2 --] 0 *H 010 `He 0 *H 00 0 *H 01 0 UUS10U Florida10 U Naples1)0'U The Daniel Morante Company, Inc.10U Pacy World, LLC10U Pacy World Root CA0 180707173531Z 280704173531Z01 0 UUS10U Florida1)0'U The Daniel Morante Company, Inc.10U Pacy World, LLC1#0!U Pacy World Intermediate CA0"0 *H 0 4,?6sK?-A\8z鹘̞y, 'W!m${R x뗹ЃQ rM UM݆j 56%, ҭO!]D GV'VzAc^e^ h't @Q)K8L e{/a-*& ʃ0V5* rn]/"/,GS(ya<Og nbG<*~KRe q1s?B۶Li9 r x^38-@.fpn*ǬG0TPpaMv e"wSAvmY2@;y}h"0LMPGR6IGׂ<D^|}Ӕ gM+Ȉ`($u7IѾavG >w5hpbfs[]yx'N_![4 HENϤDZ\%R|3>9 "SjC3i_`Xd4a3 f0d0 U | r6)A ]Dz)U0 U #0zF;ߢl'+hH=v]0U 0 0U 0 *H ,O(r|AQQMl T Fo۪+X:}|pNH uzmg1jcHWi/o ʢ"'s! 9& Y1yP-Hiu(s"/>B%g[6gdr KiS(x cBKkⅳP&G7dBoB{ hLџ9c2jfWgv{Ȕ5 NQPdnH/J v$4W <SVh Uwv Φ ayMa!}ڲN{}X<Oo KpQLŲI. 19 HEFa( ;.dNNƈZc33. MPNBU<brI^+7@Rl] g>$0?P;Cj_~"!?%IZ֜o@l <" Ҍ!wY qB{5"zZ $~500y 0 *H 01 0 UUS10U Florida1)0'U The Daniel Morante Company, Inc.10U Pacy World, LLC1#0!U Pacy World Intermediate CA0 210329042746Z 310327042746Z01 0 UUS10U Florida10 U Naples1&0$U Personal Identity Certificate10U Home PC10U Daniel Morante1!0 *H daniel@morante.net0"0 *H 0 4Y$im)ۧ䟊?W*: ;BZav{x)e4=AcW :|~ g]ϼ%ۆVx:>45Ofn Y1Ko P4KK n['ߵ*Qg AF0qFb ߗ:wN2%[}a^6z ʆ)dk__( G]!~44.lLJ- |ӿTz ]40 d_(. zdzS:˂H |ZDj"zw,qhJ*":?n.47Z9I9s4ÐCu}Ph\+$Ev8uϾaJlebX\BxKn/Ch<adZoiIb(Vy; L`ZЦlR\Yf"K1̉X%"paD/Sa2I *!K9%;#O%c/]%ʝ0B#lS7^ 00 U 0 0U 0 U c`7ynuR3cO30 U #0| r6)A ]Dz)U0 U 0daniel@morante.net0'U % 0 +++0;+/0-0++0 http://www.pacyworld.com/ca.php0 *H fjV< qL4kR <,ev3.BHwb=+ꣽG9en3_,3%n,, ^(TFِ}an_>!K+@u2a~`d# B~@&NYc6Q*Y*+ xcJ7JeoG0_NL&}' >⍙ R3.&*ִ&?℟'xQ$Br=#'iǜt3=fBQ7Q, ͔ U4o-vWmY? UXlvs;n2ժXa&:쓹(C6KQόD2J12k$V@S; k*1 $ h1}ڂ-s;MPϴ}# wЗG&`|,XTz#~KVU9i n}Xs%=10001 0 UUS10U Florida1)0'U The Daniel Morante Company, Inc.10U Pacy World, LLC1#0!U Pacy World Intermediate CA 0 `He A0 *H 1 *H 0 *H 1 210408060430Z0O *H 1B@ɀL* kdʧqKkâgqҧDJ ¦q #R>GȲ(T t "0l *H 1_0]0 `He*0 `He0 *H 0*H 0 *H @0+0 *H (0 +71001 0 UUS10U Florida1)0'U The Daniel Morante Company, Inc.10U Pacy World, LLC1#0!U Pacy World Intermediate CA 0 *H 101 0 UUS10U Florida1)0'U The Daniel Morante Company, Inc.10U Pacy World, LLC1#0!U Pacy World Intermediate CA 0 *H ' }PNIE[dcҞ4NۧvA"K}T?3É;5;fztƑ`mG0R yXԥ~|CՉjݦd YBur(Mu %̷*K PYWtL euU h $ E39NE|$]~=ԫ} o K=hsچ{ y)hj R974|2 ݧicfp{Q:*S܅ 4džQL-ojj-){,Rɠ[W .7)U*iYi:;2*9w>!MS {ib_> %o ۢB:t8Y?ІRx7y33ITnBBV䯓^aňF3 :YBi.8*~*| aiS"Z7 )f zlHAL+Npjpdd]N 21help
Want to link to this message? Use this
URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?1db2b369-cec6-058a-67ad-f762064b3832>