From nobody Sat Jun 18 08:54:12 2022 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 1026E85B199 for ; Sat, 18 Jun 2022 08:54:18 +0000 (UTC) (envelope-from grembo@freebsd.org) Received: from mail.evolve.de (mail.evolve.de [213.239.217.29]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA512 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mail.evolve.de", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4LQ8pw55T3z4XWS; Sat, 18 Jun 2022 08:54:16 +0000 (UTC) (envelope-from grembo@freebsd.org) Received: by mail.evolve.de (OpenSMTPD) with ESMTP id c45b933d; Sat, 18 Jun 2022 08:54:14 +0000 (UTC) Received: by mail.evolve.de (OpenSMTPD) with ESMTPSA id ae051716 (TLSv1.3:AEAD-AES256-GCM-SHA384:256:NO); Sat, 18 Jun 2022 08:54:14 +0000 (UTC) Content-Type: multipart/alternative; boundary=Apple-Mail-A1E7CE4B-3070-4935-B3D7-D687F871FECC Content-Transfer-Encoding: 7bit List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org Mime-Version: 1.0 (1.0) Subject: Re: SAS/SATA controllers: 8 port that support 8TB Drives From: Michael Gmelin In-Reply-To: <5936c6368df3c7d762d4710d23e7c57c@lerctr.org> Cc: Alexander Motin , Freebsd current Date: Sat, 18 Jun 2022 10:54:12 +0200 Message-Id: References: <5936c6368df3c7d762d4710d23e7c57c@lerctr.org> To: Larry Rosenman X-Mailer: iPhone Mail (19E258) X-Rspamd-Queue-Id: 4LQ8pw55T3z4XWS X-Spamd-Bar: / Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=softfail (mx1.freebsd.org: 213.239.217.29 is neither permitted nor denied by domain of grembo@freebsd.org) smtp.mailfrom=grembo@freebsd.org X-Spamd-Result: default: False [-0.04 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; FREEFALL_USER(0.00)[grembo]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; MV_CASE(0.50)[]; NEURAL_HAM_LONG(-0.36)[-0.359]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; DMARC_NA(0.00)[freebsd.org]; R_SPF_SOFTFAIL(0.00)[~all:c]; NEURAL_SPAM_MEDIUM(0.92)[0.916]; RCVD_COUNT_THREE(0.00)[3]; TO_MATCH_ENVRCPT_SOME(0.00)[]; TO_DN_ALL(0.00)[]; NEURAL_HAM_SHORT(-1.00)[-1.000]; MLMMJ_DEST(0.00)[freebsd-current]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; ASN(0.00)[asn:24940, ipnet:213.239.192.0/18, country:DE]; RCVD_TLS_ALL(0.00)[]; MID_RHS_MATCH_FROM(0.00)[] X-ThisMailContainsUnwantedMimeParts: N --Apple-Mail-A1E7CE4B-3070-4935-B3D7-D687F871FECC Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable > On 18. Jun 2022, at 01:31, Larry Rosenman wrote: > =EF=BB=BFOn 06/17/2022 6:20 pm, Michael Gmelin wrote: >>>> On 18. Jun 2022, at 00:57, Larry Rosenman wrote: >>> =EF=BB=BFOn 06/17/2022 5:48 pm, Michael Gmelin wrote: >>>>> On 18. Jun 2022, at 00:31, Alexander Motin wrote: >>>>> =EF=BB=BF >>>>>> On 17.06.2022 18:24, Alexander Motin wrote: >>>>>>> On 17.06.2022 18:16, Larry Rosenman wrote: >>>>>>> On 06/17/2022 5:08 pm, Alexander Motin wrote: >>>>>>>> On 17.06.2022 11:59, Larry Rosenman wrote: >>>>>>>>> I'm looking to upgrade the controllers in my TrueNAS box to someth= ing that will >>>>>>>>> support 8TB drives because apparently my LSI 2108 controllers do n= ot support 8TB drives. >>>>>>>>> What's the communities recommendation? >>>>>>>>> needs to support SFF connectors for a total of 4 SFF connectors, a= s I have 16 slots. >>>>>>>> We at iX are still using LSI/Broadcom HBAs, just moved from long >>>>>>>> discontinued mps(4) to newer mpr(4). And I don't believe the probl= em >>>>>>>> is directly related to capacity. According to my observations it m= ay >>>>>>>> be Seagate HDDs of/above certain (8TB) generation. We do not use >>>>>>>> Seagate HDDs in our products, so about that instability I only hear= d >>>>>>>> from forums and TrueNAS community user reports. >>>>>>> This is a mfi(4) set of controllers, and a ST80000Nm0045 8TB (CMR) d= rive. >>>>>>> Is this a bad combo? >>>>>>> mfi0: 9973 (708793330s/0x0002/WARN) - PD 00(e0xfc/s3) is not support= ed >>>>>>> (probe0:mfi0:0:0:0): INQUIRY. CDB: 12 00 00 00 24 00 >>>>>>> (probe0:mfi0:0:0:0): CAM status: CCB request completed with an error= >>>>>>> (probe0:mfi0:0:0:0): Retrying command, 3 more tries remain >>>>>>> (probe0:mfi0:0:0:0): INQUIRY. CDB: 12 00 00 00 24 00 >>>>>>> (probe0:mfi0:0:0:0): CAM status: CCB request completed with an error= >>>>>>> (probe0:mfi0:0:0:0): Retrying command, 2 more tries remain >>>>>>> (probe0:mfi0:0:0:0): INQUIRY. CDB: 12 00 00 00 24 00 >>>>>>> (probe0:mfi0:0:0:0): CAM status: CCB request completed with an error= >>>>>>> (probe0:mfi0:0:0:0): Retrying command, 1 more tries remain >>>>>>> (probe0:mfi0:0:0:0): INQUIRY. CDB: 12 00 00 00 24 00 >>>>>>> (probe0:mfi0:0:0:0): CAM status: CCB request completed with an error= >>>>>>> (probe0:mfi0:0:0:0): Retrying command, 0 more tries remain >>>>>>> (probe0:mfi0:0:0:0): INQUIRY. CDB: 12 00 00 00 24 00 >>>>>>> (probe0:mfi0:0:0:0): CAM status: CCB request completed with an error= >>>>>>> (probe0:mfi0:0:0:0): Error 5, Retries exhausted >>>>>>> mfi0 Physical Drives: >>>>>>> 0 ( 932G) UNCONFIGURED GOOD SATA E1:S3 >>>>>> mfi(4) are RAIDs, not HBAs. We do not recommend RAIDs with TrueNAS d= ue to problems with hot-plug, disk identification, etc. and so have limited e= xperience with them. But I know some of LSI RAIDs can be reflashed into equ= ivalent HBAs, so if they share the hardware, I can speculate that they may s= hare some issues. >>>>> I've just noticed "932G" instead of "8000G". It is obviously a bigger= problem than what we heard for HBAs. It looks like a kind of problems that= should not happen to HBAs, since they should not care about disk capacity. >>>> What does `smartctl -a ` report (especially sector sizes)? >>>> -m >>>>> -- >>>>> Alexander Motin >>> It's not even making a mfid* node (it is a 4Kn disk) >> Ok, that=E2=80=99s sad (and explains the wrong size calculation as 4096/5= 12=3D8). >> Is this in HBA mode? (Like Alexander suggested, re-/crossflashing >> using an IT firmware might be an option). What controller / firmware >> image version is it? >> -m >>> -- >>> Larry Rosenman http://www.lerctr.org/~ler >>> Phone: +1 214-642-9640 E-Mail: ler@lerctr.org >>> US Mail: 5708 Sabbia Dr, Round Rock, TX 78665-2106 >=20 >=20 > mfi0@pci0:8:0:0: class=3D0x010400 rev=3D0x05 hdr=3D0x00 vendor=3D0x1000= device=3D0x0079 subvendor=3D0x1734 subdevice=3D0x1176 > vendor =3D 'Broadcom / LSI' > device =3D 'MegaRAID SAS 2108 [Liberator]' > class =3D mass storage > subclass =3D RAID >=20 > mfi1@pci0:3:0:0: class=3D0x010400 rev=3D0x05 hdr=3D0x00 vendor=3D0x1000= device=3D0x0079 subvendor=3D0x1734 subdevice=3D0x1176 > vendor =3D 'Broadcom / LSI' > device =3D 'MegaRAID SAS 2108 [Liberator]' > class =3D mass storage > subclass =3D RAID >=20 > mfi0: port 0xd000-0xd0ff mem 0xfbc9c000-0xfbc9ffff,0xfb= cc0000 > -0xfbcfffff irq 26 at device 0.0 on pci3 > mfi0: Using MSI > mfi0: Megaraid SAS driver Ver 4.23 > mfi0: FW MaxCmds =3D 1008, limiting to 128 > mfip0: on mfi0 > mfi0: 10014 (708822708s/0x0020/info) - Shutdown command received from host= > mfi0: 10015 (boot + 3s/0x0020/info) - Firmware initialization started (PCI= ID 00 > 79/1000/1176/1734) > mfi0: 10016 (boot + 3s/0x0020/info) - Firmware version 2.130.353-2727 > mfi0: 10017 (boot + 6s/0x0020/info) - Package version 12.12.0-0174 > mfi0: 10018 (boot + 6s/0x0020/info) - Board Revision >=20 > mfi1: port 0xc000-0xc0ff mem 0xfba9c000-0xfba9ffff,0xfb= ac0000 > -0xfbafffff irq 16 at device 0.0 on pci8 > mfi1: Using MSI > mfi1: Megaraid SAS driver Ver 4.23 > mfi1: FW MaxCmds =3D 1008, limiting to 128 >=20 Subvendor is Fujitsu Siemens - so I guess this is integrated into a system b= y them. Seems like flashing the 2108 to an IT firmware isn=E2=80=99t an option (base= d on what I found online). You could check if there are firmware updates ava= ilable though. How did you configure the drives in the megaraid utility (ctr= l-h after boot)? Did you create a RAID-0 for each disk? And what capacity is= shown in there? Based on [0], 2108 based controllers don=E2=80=99t support 4kn. IT mode woul= d help (true passthrough), but as written above, I don=E2=80=99t think it=E2= =80=99s an option for this model. -m [0] https://bitdeals.tech/blogs/news/4kn-lsi-compatibility-list >=20 > --=20 > Larry Rosenman http://www.lerctr.org/~ler > Phone: +1 214-642-9640 E-Mail: ler@lerctr.org > US Mail: 5708 Sabbia Dr, Round Rock, TX 78665-2106 --Apple-Mail-A1E7CE4B-3070-4935-B3D7-D687F871FECC Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: quoted-printable

On 18. Jun 2022, at 01:31, Larry Rosenman <ler= @lerctr.org> wrote:

<= div dir=3D"ltr">=EF=BB=BFOn 06/17/2022 6:20 pm, Michael Gmelin wrote:<= /span>
On 18. J= un 2022, at 00:57, Larry Rosenman <ler@lerctr.org> wrote:
= =EF=BB=BFOn 06/17/2022 5:48 pm, Michael Gmelin wrote:
On 18. Jun 2022, at 00= :31, Alexander Motin <mav@freebsd.org> wrote:
<= /blockquote>
=EF=BB= =BF
On 17.06.2022 18:24, Alexa= nder Motin wrote:
On 17.06.2022 18:16, Larry Rosenman wrote:=
On 06/17/2022 5:08 pm, Alexander Motin wrote:
<= blockquote type=3D"cite">On 17.06.2022 11:59, Larry Rosenman wrote:
<= blockquote type=3D"cite">
I'm looking to upgrade the controllers in my TrueNAS box to somethi= ng that will
<= /blockquote>
support 8TB drives because apparently m= y LSI 2108 controllers do not support 8TB drives.
<= /blockquote>
Wha= t's the communities recommendation?
=
needs to support= SFF connectors for a total of 4 SFF connectors, as I have 16 slots.<= br>
We at iX ar= e still using LSI/Broadcom HBAs, just moved from long
discontinued mps(4) to newer mpr(4). &= nbsp;And I don't believe the problem
is directly related to capacity.  According to my o= bservations it may
be Seagate HDDs of/above certain (8TB) generation.  We do not use
=
Seagate HDDs in o= ur products, so about that instability I only heard
<= /blockquote>
from forums and TrueNAS community user r= eports.
This is a mfi(4) set of controllers, a= nd a ST80000Nm0045 8TB (CMR) drive.
=
Is this a bad comb= o?
mfi0: 9973 (708793330s/0x0002/WARN) - PD 00(e0xfc/s= 3) is not supported
(probe0:mfi0:0:0:0): INQUIRY. CDB: 12= 00 00 00 24 00
(probe0:mfi0:0:0:0): CAM status: CCB r= equest completed with an error
(probe0:mfi0:0:0:0): Re= trying command, 3 more tries remain
=
(probe0:mfi0:0:0:0= ): INQUIRY. CDB: 12 00 00 00 24 00
<= blockquote type=3D"cite">
(probe0:mfi0:0:0:0)= : CAM status: CCB request completed with an error
(prob= e0:mfi0:0:0:0): Retrying command, 2 more tries remain
= (probe0:mfi0:0:0:0): INQUIRY. CDB: 12 00 00 00 24 00
=
(= probe0:mfi0:0:0:0): CAM status: CCB request completed with an error
(probe0:mfi0:0:0:0): Retrying command, 1 more tries remain
(probe0:mfi0:0:0:0): INQUIRY. CDB: 12 00 00 00 24 00
(probe0:mfi0:0:0:0): CAM status: CCB request completed with an e= rror
(probe0:mfi0:0:0:0): Retrying command, 0 more tri= es remain
=
(probe0:mfi0:0:0:0): INQUIRY. CDB: 12 00 00 0= 0 24 00
(probe0:mfi0:0:0:0): CAM status: CCB request c= ompleted with an error
(probe0:mfi0:0:0:0): Error 5, R= etries exhausted
mfi0 Physical Drives:
<= blockquote type=3D"cite">
0 (  932G) UNCONFIGURED GOOD <ST8000NM0045-1RL UG07 serial=3D= ZA1AC912> SATA E1:S3
mfi(4) are RAIDs, not HBAs.  We do not recommend RAI= Ds with TrueNAS due to problems with hot-plug, disk identification, etc. and= so have limited experience with them.  But I know some of LSI RAIDs ca= n be reflashed into equivalent HBAs, so if they share the hardware, I can sp= eculate that they may share some issues.
I'= ve just noticed "932G" instead of "8000G".  It is obviously a bigger pr= oblem than what we heard for HBAs.  It looks like a kind of problems th= at should not happen to HBAs, since they should not care about disk capacity= .
What d= oes `smartctl -a <device>` report (especially sector sizes)?
-m
=
--
<= span>Alexander Motin
It's not ev= en making a mfid* node (it is a 4Kn disk)
Ok, that=E2=80=99s sad (and explains the w= rong size calculation as 4096/512=3D8).
Is this in HBA mode? (Like Alexander suggested, re-/cross= flashing
using an IT f= irmware might be an option). What controller / firmware
image version is it?
-m
--
Larry Rosenm= an             &= nbsp;       http://www.lerctr.org/~ler
Phone: +1 214-642-9640       &= nbsp;         E-Mail: ler@lerct= r.org
US Mail: 5708 Sabbia Dr, Round Rock, TX 78665-2106


m= fi0@pci0:8:0:0:    class=3D0x010400 rev=3D0x05 hdr=3D0x00 vendor=3D= 0x1000 device=3D0x0079 subvendor=3D0x1734 subdevice=3D0x1176
   vendor     =3D 'Broadcom / LSI'
   device     =3D 'MegaRAID S= AS 2108 [Liberator]'
   class   &n= bsp;  =3D mass storage
   subclass=   =3D RAID

mfi1@pci0:3:0:0: &nbs= p;  class=3D0x010400 rev=3D0x05 hdr=3D0x00 vendor=3D0x1000 device=3D0x0= 079 subvendor=3D0x1734 subdevice=3D0x1176
   = ;vendor     =3D 'Broadcom / LSI'
 =   device     =3D 'MegaRAID SAS 2108 [Liberator= ]'
   class      =3D= mass storage
   subclass   =3D RA= ID

mfi0: <LSI MegaSAS Gen2> port 0xd0= 00-0xd0ff mem 0xfbc9c000-0xfbc9ffff,0xfbcc0000
-0xfbcfffff i= rq 26 at device 0.0 on pci3
mfi0: Using MSI
= mfi0: Megaraid SAS driver Ver 4.23
mfi0: FW MaxCmds =3D 1008= , limiting to 128
mfip0: <SCSI Passthrough Bus> on mfi= 0
mfi0: 10014 (708822708s/0x0020/info) - Shutdown command re= ceived from host
mfi0: 10015 (boot + 3s/0x0020/info) - Firmw= are initialization started (PCI ID 00
79/1000/1176/1734)
mfi0: 10016 (boot + 3s/0x0020/info) - Firmware version 2.130.35= 3-2727
mfi0: 10017 (boot + 6s/0x0020/info) - Package version= 12.12.0-0174
mfi0: 10018 (boot + 6s/0x0020/info) - Board Re= vision

mfi1: <LSI MegaSAS Gen2> port 0= xc000-0xc0ff mem 0xfba9c000-0xfba9ffff,0xfbac0000
-0xfbaffff= f irq 16 at device 0.0 on pci8
mfi1: Using MSI
mfi1: Megaraid SAS driver Ver 4.23

mfi1: FW MaxCmds =3D 1= 008, limiting to 128


=
Subvendor is Fujitsu Siemens - so I guess this is integrated into= a system by them.

Seems like flashing the 2108 to a= n IT firmware isn=E2=80=99t an option (based on what I found online). You co= uld check if there are firmware updates available though. How did you config= ure the drives in the megaraid utility (ctrl-h after boot)? Did you create a= RAID-0 for each disk? And what capacity is shown in there?

Based on [0], 2108 based controllers don=E2=80=99t support 4kn. IT m= ode would help (true passthrough), but as written above, I don=E2=80=99t thi= nk it=E2=80=99s an option for this model.

-m
<= div dir=3D"ltr">
[0] https://bitdeals.tech/blogs/news/4kn-lsi-comp= atibility-list


--
Larry Rosenman  &= nbsp;            = ;      http://www.lerctr.org/~ler
Phone: +1 214-642-9640         &= nbsp;       E-Mail: ler@lerctr.org=
US Mail: 5708 Sabbia Dr, Round Rock, TX 78665-2106
= --Apple-Mail-A1E7CE4B-3070-4935-B3D7-D687F871FECC--