From nobody Thu Mar 14 18:47:01 2024 X-Original-To: freebsd-hardware@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Twbvx6Nf1z5DbLv for ; Thu, 14 Mar 2024 18:47:09 +0000 (UTC) (envelope-from mike@sentex.net) Received: from smarthost1.sentex.ca (smarthost1.sentex.ca [IPv6:2607:f3e0:0:1::12]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smarthost1.sentex.ca", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Twbvw4V8Yz4NGt for ; Thu, 14 Mar 2024 18:47:08 +0000 (UTC) (envelope-from mike@sentex.net) Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of mike@sentex.net designates 2607:f3e0:0:1::12 as permitted sender) smtp.mailfrom=mike@sentex.net Received: from pyroxene2a.sentex.ca (pyroxene19.sentex.ca [199.212.134.19]) by smarthost1.sentex.ca (8.17.1/8.16.1) with ESMTPS id 42EIl1HG000825 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=FAIL) for ; Thu, 14 Mar 2024 14:47:01 -0400 (EDT) (envelope-from mike@sentex.net) Received: from [IPV6:2607:f3e0:0:4:a1d0:ee73:adac:870f] ([IPv6:2607:f3e0:0:4:a1d0:ee73:adac:870f]) by pyroxene2a.sentex.ca (8.17.1/8.15.2) with ESMTPS id 42EIl0ZH039626 (version=TLSv1.3 cipher=TLS_AES_128_GCM_SHA256 bits=128 verify=NO) for ; Thu, 14 Mar 2024 14:47:00 -0400 (EDT) (envelope-from mike@sentex.net) Message-ID: Date: Thu, 14 Mar 2024 14:47:01 -0400 List-Id: General discussion of FreeBSD hardware List-Archive: https://lists.freebsd.org/archives/freebsd-hardware List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hardware@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Content-Language: en-US To: freebsd-hardware@freebsd.org From: mike tancsa Subject: WD Blue 510 SSD and strange write performance Autocrypt: addr=mike@sentex.net; keydata= xsBNBFywzOMBCACoNFpwi5MeyEREiCeHtbm6pZJI/HnO+wXdCAWtZkS49weOoVyUj5BEXRZP xflV2ib2hflX4nXqhenaNiia4iaZ9ft3I1ebd7GEbGnsWCvAnob5MvDZyStDAuRxPJK1ya/s +6rOvr+eQiXYNVvfBhrCfrtR/esSkitBGxhUkBjOti8QwzD71JVF5YaOjBAs7jZUKyLGj0kW yDg4jUndudWU7G2yc9GwpHJ9aRSUN8e/mWdIogK0v+QBHfv/dsI6zVB7YuxCC9Fx8WPwfhDH VZC4kdYCQWKXrm7yb4TiVdBh5kgvlO9q3js1yYdfR1x8mjK2bH2RSv4bV3zkNmsDCIxjABEB AAHNHW1pa2UgdGFuY3NhIDxtaWtlQHNlbnRleC5uZXQ+wsCOBBMBCAA4FiEEmuvCXT0aY6hs 4SbWeVOEFl5WrMgFAl+pQfkCGwMFCwkIBwIGFQoJCAsCBBYCAwECHgECF4AACgkQeVOEFl5W rMiN6ggAk3H5vk8QnbvGbb4sinxZt/wDetgk0AOR9NRmtTnPaW+sIJEfGBOz47Xih+f7uWJS j+uvc9Ewn2Z7n8z3ZHJlLAByLVLtcNXGoRIGJ27tevfOaNqgJHBPbFOcXCBBFTx4MYMM4iAZ cDT5vsBTSaM36JZFtHZBKkuFEItbA/N8ZQSHKdTYMIA7A3OCLGbJBqloQ8SlW4MkTzKX4u7R yefAYQ0h20x9IqC5Ju8IsYRFacVZconT16KS81IBceO42vXTN0VexbVF2rZIx3v/NT75r6Vw 0FlXVB1lXOHKydRA2NeleS4NEG2vWqy/9Boj0itMfNDlOhkrA/0DcCurMpnpbM7ATQRcsMzk AQgA1Dpo/xWS66MaOJLwA28sKNMwkEk1Yjs+okOXDOu1F+0qvgE8sVmrOOPvvWr4axtKRSG1 t2QUiZ/ZkW/x/+t0nrM39EANV1VncuQZ1ceIiwTJFqGZQ8kb0+BNkwuNVFHRgXm1qzAJweEt RdsCMohB+H7BL5LGCVG5JaU0lqFU9pFP40HxEbyzxjsZgSE8LwkI6wcu0BLv6K6cLm0EiHPO l5G8kgRi38PS7/6s3R8QDsEtbGsYy6O82k3zSLIjuDBwA9GRaeigGppTxzAHVjf5o9KKu4O7 gC2KKVHPegbXS+GK7DU0fjzX57H5bZ6komE5eY4p3oWT/CwVPSGfPs8jOwARAQABwsB2BBgB CAAgFiEEmuvCXT0aY6hs4SbWeVOEFl5WrMgFAl+pQfkCGwwACgkQeVOEFl5WrMiVqwf9GwU8 c6cylknZX8QwlsVudTC8xr/L17JA84wf03k3d4wxP7bqy5AYy7jboZMbgWXngAE/HPQU95NM aukysSnknzoIpC96XZJ0okLBXVS6Y0ylZQ+HrbIhMpuQPoDweoF5F9wKrsHRoDaUK1VR706X rwm4HUzh7Jk+auuMYfuCh0FVlFBEuiJWMLhg/5WCmcRfiuB6F59ZcUQrwLEZeNhF2XJV4KwB Tlg7HCWO/sy1foE5noaMyACjAtAQE9p5kGYaj+DuRhPdWUTsHNuqrhikzIZd2rrcMid+ktb0 NvtvswzMO059z1YGMtGSqQ4srCArju+XHIdTFdiIYbd7+jeehg== Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-Scanned-By: MIMEDefang 2.86 on 64.7.153.18 X-Spamd-Bar: --- X-Spamd-Result: default: False [-3.37 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.98)[-0.985]; R_SPF_ALLOW(-0.20)[+ip6:2607:f3e0::/32]; MIME_GOOD(-0.10)[text/plain]; RCVD_IN_DNSWL_LOW(-0.10)[199.212.134.19:received]; XM_UA_NO_VERSION(0.01)[]; RCPT_COUNT_ONE(0.00)[1]; ASN(0.00)[asn:11647, ipnet:2607:f3e0::/32, country:CA]; MID_RHS_MATCH_FROM(0.00)[]; MIME_TRACE(0.00)[0:+]; FREEFALL_USER(0.00)[mike]; R_DKIM_NA(0.00)[]; MLMMJ_DEST(0.00)[freebsd-hardware@freebsd.org]; DMARC_NA(0.00)[sentex.net]; FROM_EQ_ENVFROM(0.00)[]; FROM_HAS_DN(0.00)[]; ARC_NA(0.00)[]; RCVD_COUNT_TWO(0.00)[2]; TO_MATCH_ENVRCPT_ALL(0.00)[]; TO_DN_NONE(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[freebsd-hardware@freebsd.org]; RCVD_TLS_ALL(0.00)[] X-Rspamd-Queue-Id: 4Twbvw4V8Yz4NGt This might be more of a hardware question than anything, but I noticed the drive is fairly fast on initial writes, but dramatically slows down over time with a consistent write. At bootup time, I can blast out a file (UFS2 mount) #  dd if=/dev/zero of=/mnt/tmp/junk.bin.`date "+%s"` bs=1m status=progress count=4000   3735027712 bytes (3735 MB, 3562 MiB) transferred 7.014s, 533 MB/s # nice and fast.  But subsequent writes really start to tank. #  dd if=/dev/zero of=/mnt/tmp/junk.bin.`date "+%s"` bs=1m status=progress count=4000   4128243712 bytes (4128 MB, 3937 MiB) transferred 46.048s, 90 MB/s # dd if=/dev/zero of=/mnt/tmp/junk.bin.`date "+%s"` bs=1m status=progress count=4000   3992977408 bytes (3993 MB, 3808 MiB) transferred 31.016s, 129 MB/s If I wait for 2min, it seems to be back to normal  # sleep 120 ; dd if=/dev/zero of=/mnt/tmp/junk.bin.`date "+%s"` bs=1m status=progress count=4000   4137680896 bytes (4138 MB, 3946 MiB) transferred 9.025s, 458 MB/s Is there something going on behind the scenes limiting the amount of sustained writes these drives can handle ? Is it just a limitation of the SSD ?  I didnt notice such issues on some consumer Samsungs. The problem initially showed up for me when I was doing a series of zfs send | zfs recv on the same pool (so a lot of reads and writes at the same time) and I would get a bunch of errors on the WD disks.  Replacing them with Samsungs avoids the errors. === START OF INFORMATION SECTION === Device Model:     WD Blue SA510 2.5 1000GB Serial Number:    240406800001 LU WWN Device Id: 5 001b44 8b334e00b Firmware Version: 52046100 User Capacity:    1,000,204,886,016 bytes [1.00 TB] Sector Size:      512 bytes logical/physical Rotation Rate:    Solid State Device Form Factor:      2.5 inches TRIM Command:     Available, deterministic Device is:        Not in smartctl database 7.3/5528 ATA Version is:   ACS-4, ACS-2 T13/2015-D revision 3 SATA Version is:  SATA 3.2, 6.0 Gb/s (current: 6.0 Gb/s) Local Time is:    Thu Mar 14 14:41:58 2024 EDT SMART support is: Available - device has SMART capability. SMART support is: Enabled # camcontrol iden ada1 pass1: ACS-4 ATA SATA 3.x device pass1: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 512bytes) protocol              ACS-4 ATA SATA 3.x device model          WD Blue SA510 2.5 1000GB firmware revision     52046100 serial number         240406800001 WWN                   5001b448b334e00b additional product id cylinders             16383 heads                 16 sectors/track         63 sector size           logical 512, physical 512, offset 0 LBA supported         268435455 sectors LBA48 supported       1953525168 sectors PIO supported         PIO4 DMA supported         WDMA2 UDMA6 media RPM             non-rotating Zoned-Device Commands no Feature                      Support  Enabled   Value Vendor read ahead                     yes      yes write cache                    yes      yes flush cache                    yes      yes Native Command Queuing (NCQ)   yes              32 tags NCQ Priority Information       no NCQ Non-Data Command           no NCQ Streaming                  no Receive & Send FPDMA Queued    no NCQ Autosense                  no SMART                          yes      yes security                       yes      no power management               yes      yes microcode download             yes      yes advanced power management      yes      no      0/0x00 automatic acoustic management  no       no media status notification      no       no power-up in Standby            no       no write-read-verify              no       no unload                         no       no general purpose logging        yes      yes free-fall                      no       no sense data reporting           no       no extended power conditions      no       no device statistics notification no       no Data Set Management (DSM/TRIM) yes DSM - max 512byte blocks       yes              8 DSM - deterministic read       yes              any value Trusted Computing              no encrypts all user data         no Sanitize                       yes              block, Sanitize - commands allowed    yes Sanitize - antifreeze lock     yes Host Protected Area (HPA)      no Accessible Max Address Config  yes      no 1953525168/1953525168 RELENG_14 from today From nobody Thu Mar 14 18:56:48 2024 X-Original-To: freebsd-hardware@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Twc7S3tRyz5Dc3F for ; Thu, 14 Mar 2024 18:57:08 +0000 (UTC) (envelope-from rb@gid.co.uk) Received: from gid2.gid.co.uk (ns0.gid.co.uk [IPv6:2001:470:94de::240]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "gid2.gid.co.uk", Issuer "gid2.gid.co.uk" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Twc7R4C9vz4Pxm for ; Thu, 14 Mar 2024 18:57:07 +0000 (UTC) (envelope-from rb@gid.co.uk) Authentication-Results: mx1.freebsd.org; none Received: from mx0.gid.co.uk (mx0.gid.co.uk [194.32.164.250]) by gid2.gid.co.uk (8.15.2/8.15.2) with ESMTP id 42EIusJE085508; Thu, 14 Mar 2024 18:56:54 GMT (envelope-from rb@gid.co.uk) Received: from smtpclient.apple (moriarty.gid.co.uk [194.32.164.17]) by mx0.gid.co.uk (8.14.2/8.14.2) with ESMTP id 42EIunN4011260; Thu, 14 Mar 2024 18:56:49 GMT (envelope-from rb@gid.co.uk) Content-Type: text/plain; charset=utf-8 List-Id: General discussion of FreeBSD hardware List-Archive: https://lists.freebsd.org/archives/freebsd-hardware List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hardware@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3696.120.41.1.4\)) Subject: Re: WD Blue 510 SSD and strange write performance From: Bob Bishop In-Reply-To: Date: Thu, 14 Mar 2024 18:56:48 +0000 Cc: "freebsd-hardware@freebsd.org" Content-Transfer-Encoding: quoted-printable Message-Id: References: To: mike tancsa X-Mailer: Apple Mail (2.3696.120.41.1.4) X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:6939, ipnet:2001:470::/32, country:US] X-Rspamd-Queue-Id: 4Twc7R4C9vz4Pxm Hi, > On 14 Mar 2024, at 18:47, mike tancsa wrote: >=20 > This might be more of a hardware question than anything, but I noticed = the drive is fairly fast on initial writes, but dramatically slows down = over time with a consistent write. >=20 > At bootup time, I can blast out a file (UFS2 mount) >=20 > # dd if=3D/dev/zero of=3D/mnt/tmp/junk.bin.`date "+%s"` bs=3D1m = status=3Dprogress count=3D4000 > 3735027712 bytes (3735 MB, 3562 MiB) transferred 7.014s, 533 MB/s > # >=20 > nice and fast. But subsequent writes really start to tank. >=20 > # dd if=3D/dev/zero of=3D/mnt/tmp/junk.bin.`date "+%s"` bs=3D1m = status=3Dprogress count=3D4000 > 4128243712 bytes (4128 MB, 3937 MiB) transferred 46.048s, 90 MB/s >=20 > # dd if=3D/dev/zero of=3D/mnt/tmp/junk.bin.`date "+%s"` bs=3D1m = status=3Dprogress count=3D4000 > 3992977408 bytes (3993 MB, 3808 MiB) transferred 31.016s, 129 MB/s >=20 > If I wait for 2min, it seems to be back to normal >=20 > # sleep 120 ; dd if=3D/dev/zero of=3D/mnt/tmp/junk.bin.`date "+%s"` = bs=3D1m status=3Dprogress count=3D4000 > 4137680896 bytes (4138 MB, 3946 MiB) transferred 9.025s, 458 MB/s >=20 > Is there something going on behind the scenes limiting the amount of = sustained writes these drives can handle ? Is it just a limitation of = the SSD ? Probably (I don=E2=80=99t know for sure) these drives have a RAM write = cache and really suck at committing from there to NVM. > I didnt notice such issues on some consumer Samsungs. The problem = initially showed up for me when I was doing a series of zfs send | zfs = recv on the same pool (so a lot of reads and writes at the same time) = and I would get a bunch of errors on the WD disks. Replacing them with = Samsungs avoids the errors. >=20 >=20 > =3D=3D=3D START OF INFORMATION SECTION =3D=3D=3D > Device Model: WD Blue SA510 2.5 1000GB > Serial Number: 240406800001 > LU WWN Device Id: 5 001b44 8b334e00b > Firmware Version: 52046100 > User Capacity: 1,000,204,886,016 bytes [1.00 TB] > Sector Size: 512 bytes logical/physical > Rotation Rate: Solid State Device > Form Factor: 2.5 inches > TRIM Command: Available, deterministic > Device is: Not in smartctl database 7.3/5528 > ATA Version is: ACS-4, ACS-2 T13/2015-D revision 3 > SATA Version is: SATA 3.2, 6.0 Gb/s (current: 6.0 Gb/s) > Local Time is: Thu Mar 14 14:41:58 2024 EDT > SMART support is: Available - device has SMART capability. > SMART support is: Enabled >=20 > # camcontrol iden ada1 > pass1: ACS-4 ATA SATA 3.x device > pass1: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 512bytes) >=20 > protocol ACS-4 ATA SATA 3.x > device model WD Blue SA510 2.5 1000GB > firmware revision 52046100 > serial number 240406800001 > WWN 5001b448b334e00b > additional product id > cylinders 16383 > heads 16 > sectors/track 63 > sector size logical 512, physical 512, offset 0 > LBA supported 268435455 sectors > LBA48 supported 1953525168 sectors > PIO supported PIO4 > DMA supported WDMA2 UDMA6 > media RPM non-rotating > Zoned-Device Commands no >=20 > Feature Support Enabled Value Vendor > read ahead yes yes > write cache yes yes > flush cache yes yes > Native Command Queuing (NCQ) yes 32 tags > NCQ Priority Information no > NCQ Non-Data Command no > NCQ Streaming no > Receive & Send FPDMA Queued no > NCQ Autosense no > SMART yes yes > security yes no > power management yes yes > microcode download yes yes > advanced power management yes no 0/0x00 > automatic acoustic management no no > media status notification no no > power-up in Standby no no > write-read-verify no no > unload no no > general purpose logging yes yes > free-fall no no > sense data reporting no no > extended power conditions no no > device statistics notification no no > Data Set Management (DSM/TRIM) yes > DSM - max 512byte blocks yes 8 > DSM - deterministic read yes any value > Trusted Computing no > encrypts all user data no > Sanitize yes block, > Sanitize - commands allowed yes > Sanitize - antifreeze lock yes > Host Protected Area (HPA) no > Accessible Max Address Config yes no 1953525168/1953525168 >=20 >=20 > RELENG_14 from today >=20 >=20 -- Bob Bishop rb@gid.co.uk From nobody Thu Mar 14 19:36:41 2024 X-Original-To: freebsd-hardware@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Twd175NRhz5Dfwc for ; Thu, 14 Mar 2024 19:36:43 +0000 (UTC) (envelope-from mike@sentex.net) Received: from smarthost1.sentex.ca (smarthost1.sentex.ca [IPv6:2607:f3e0:0:1::12]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smarthost1.sentex.ca", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Twd173Ycgz4Wwn for ; Thu, 14 Mar 2024 19:36:43 +0000 (UTC) (envelope-from mike@sentex.net) Authentication-Results: mx1.freebsd.org; none Received: from pyroxene2a.sentex.ca (pyroxene19.sentex.ca [199.212.134.19]) by smarthost1.sentex.ca (8.17.1/8.16.1) with ESMTPS id 42EJagCL008344 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=FAIL); Thu, 14 Mar 2024 15:36:42 -0400 (EDT) (envelope-from mike@sentex.net) Received: from [IPV6:2607:f3e0:0:4:a1d0:ee73:adac:870f] ([IPv6:2607:f3e0:0:4:a1d0:ee73:adac:870f]) by pyroxene2a.sentex.ca (8.17.1/8.15.2) with ESMTPS id 42EJafBM044086 (version=TLSv1.3 cipher=TLS_AES_128_GCM_SHA256 bits=128 verify=NO); Thu, 14 Mar 2024 15:36:41 -0400 (EDT) (envelope-from mike@sentex.net) Content-Type: multipart/alternative; boundary="------------eT9mUKr5WO7CZYKUB0UtN0Ff" Message-ID: <6504bd49-eca5-4e0a-b2bd-23d29405bb7a@sentex.net> Date: Thu, 14 Mar 2024 15:36:41 -0400 List-Id: General discussion of FreeBSD hardware List-Archive: https://lists.freebsd.org/archives/freebsd-hardware List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hardware@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: WD Blue 510 SSD and strange write performance Content-Language: en-US To: Bob Bishop Cc: "freebsd-hardware@freebsd.org" References: From: mike tancsa Autocrypt: addr=mike@sentex.net; keydata= xsBNBFywzOMBCACoNFpwi5MeyEREiCeHtbm6pZJI/HnO+wXdCAWtZkS49weOoVyUj5BEXRZP xflV2ib2hflX4nXqhenaNiia4iaZ9ft3I1ebd7GEbGnsWCvAnob5MvDZyStDAuRxPJK1ya/s +6rOvr+eQiXYNVvfBhrCfrtR/esSkitBGxhUkBjOti8QwzD71JVF5YaOjBAs7jZUKyLGj0kW yDg4jUndudWU7G2yc9GwpHJ9aRSUN8e/mWdIogK0v+QBHfv/dsI6zVB7YuxCC9Fx8WPwfhDH VZC4kdYCQWKXrm7yb4TiVdBh5kgvlO9q3js1yYdfR1x8mjK2bH2RSv4bV3zkNmsDCIxjABEB AAHNHW1pa2UgdGFuY3NhIDxtaWtlQHNlbnRleC5uZXQ+wsCOBBMBCAA4FiEEmuvCXT0aY6hs 4SbWeVOEFl5WrMgFAl+pQfkCGwMFCwkIBwIGFQoJCAsCBBYCAwECHgECF4AACgkQeVOEFl5W rMiN6ggAk3H5vk8QnbvGbb4sinxZt/wDetgk0AOR9NRmtTnPaW+sIJEfGBOz47Xih+f7uWJS j+uvc9Ewn2Z7n8z3ZHJlLAByLVLtcNXGoRIGJ27tevfOaNqgJHBPbFOcXCBBFTx4MYMM4iAZ cDT5vsBTSaM36JZFtHZBKkuFEItbA/N8ZQSHKdTYMIA7A3OCLGbJBqloQ8SlW4MkTzKX4u7R yefAYQ0h20x9IqC5Ju8IsYRFacVZconT16KS81IBceO42vXTN0VexbVF2rZIx3v/NT75r6Vw 0FlXVB1lXOHKydRA2NeleS4NEG2vWqy/9Boj0itMfNDlOhkrA/0DcCurMpnpbM7ATQRcsMzk AQgA1Dpo/xWS66MaOJLwA28sKNMwkEk1Yjs+okOXDOu1F+0qvgE8sVmrOOPvvWr4axtKRSG1 t2QUiZ/ZkW/x/+t0nrM39EANV1VncuQZ1ceIiwTJFqGZQ8kb0+BNkwuNVFHRgXm1qzAJweEt RdsCMohB+H7BL5LGCVG5JaU0lqFU9pFP40HxEbyzxjsZgSE8LwkI6wcu0BLv6K6cLm0EiHPO l5G8kgRi38PS7/6s3R8QDsEtbGsYy6O82k3zSLIjuDBwA9GRaeigGppTxzAHVjf5o9KKu4O7 gC2KKVHPegbXS+GK7DU0fjzX57H5bZ6komE5eY4p3oWT/CwVPSGfPs8jOwARAQABwsB2BBgB CAAgFiEEmuvCXT0aY6hs4SbWeVOEFl5WrMgFAl+pQfkCGwwACgkQeVOEFl5WrMiVqwf9GwU8 c6cylknZX8QwlsVudTC8xr/L17JA84wf03k3d4wxP7bqy5AYy7jboZMbgWXngAE/HPQU95NM aukysSnknzoIpC96XZJ0okLBXVS6Y0ylZQ+HrbIhMpuQPoDweoF5F9wKrsHRoDaUK1VR706X rwm4HUzh7Jk+auuMYfuCh0FVlFBEuiJWMLhg/5WCmcRfiuB6F59ZcUQrwLEZeNhF2XJV4KwB Tlg7HCWO/sy1foE5noaMyACjAtAQE9p5kGYaj+DuRhPdWUTsHNuqrhikzIZd2rrcMid+ktb0 NvtvswzMO059z1YGMtGSqQ4srCArju+XHIdTFdiIYbd7+jeehg== In-Reply-To: X-Scanned-By: MIMEDefang 2.86 on 64.7.153.18 X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:11647, ipnet:2607:f3e0::/32, country:CA] X-Rspamd-Queue-Id: 4Twd173Ycgz4Wwn This is a multi-part message in MIME format. --------------eT9mUKr5WO7CZYKUB0UtN0Ff Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit On 3/14/2024 2:56 PM, Bob Bishop wrote: > Probably (I don’t know for sure) these drives have a RAM write cache > and really suck at committing from there to NVM. ZFS doesnt seem to like it, or, possibly something else in addition going on.   I will start to get errors like this (da4:mrsas0:0:23:0): READ(10). CDB: 28 00 3c 39 f2 60 00 00 08 00 (da4:mrsas0:0:23:0): CAM status: SCSI Status Error (da4:mrsas0:0:23:0): SCSI status: OK (da5:mrsas0:0:24:0): READ(10). CDB: 28 00 28 0b b5 38 00 00 f0 00 (da5:mrsas0:0:24:0): CAM status: SCSI Status Error (da5:mrsas0:0:24:0): SCSI status: OK when I have a heavy stream of zfs send | zfs recv going on.  Not sure if its possible to work around this or its some other bad interaction, or just a plain old bug in the firmware of these SSDs     ---Mike --------------eT9mUKr5WO7CZYKUB0UtN0Ff Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: 8bit


On 3/14/2024 2:56 PM, Bob Bishop wrote:
Probably (I don’t know for sure) these drives have a RAM write cache and really suck at committing from there to NVM.


ZFS doesnt seem to like it, or, possibly something else in addition going on.   I will start to get errors like this

(da4:mrsas0:0:23:0): READ(10). CDB: 28 00 3c 39 f2 60 00 00 08 00
(da4:mrsas0:0:23:0): CAM status: SCSI Status Error
(da4:mrsas0:0:23:0): SCSI status: OK
(da5:mrsas0:0:24:0): READ(10). CDB: 28 00 28 0b b5 38 00 00 f0 00
(da5:mrsas0:0:24:0): CAM status: SCSI Status Error
(da5:mrsas0:0:24:0): SCSI status: OK

when I have a heavy stream of zfs send | zfs recv going on.  Not sure if its possible to work around this or its some other bad interaction, or just a plain old bug in the firmware of these SSDs

    ---Mike

--------------eT9mUKr5WO7CZYKUB0UtN0Ff-- From nobody Thu Mar 14 19:39:32 2024 X-Original-To: freebsd-hardware@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Twd4Z6WBvz5Dgsj for ; Thu, 14 Mar 2024 19:39:42 +0000 (UTC) (envelope-from freebsd-doc@fjl.co.uk) Received: from bs1.fjl.org.uk (bs1.fjl.org.uk [84.45.41.196]) by mx1.freebsd.org (Postfix) with ESMTP id 4Twd4Y0VR1z4XHY for ; Thu, 14 Mar 2024 19:39:40 +0000 (UTC) (envelope-from freebsd-doc@fjl.co.uk) Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of freebsd-doc@fjl.co.uk designates 84.45.41.196 as permitted sender) smtp.mailfrom=freebsd-doc@fjl.co.uk Received: from [127.0.0.1] ([185.69.144.64]) (authenticated bits=0) by bs1.fjl.org.uk (8.14.4/8.14.4) with ESMTP id 42EJdXeQ066920 for ; Thu, 14 Mar 2024 19:39:33 GMT (envelope-from freebsd-doc@fjl.co.uk) Date: Thu, 14 Mar 2024 19:39:32 +0000 From: Frank Leonhardt To: freebsd-hardware@freebsd.org Subject: Re: WD Blue 510 SSD and strange write performance User-Agent: K-9 Mail for Android In-Reply-To: References: Message-ID: <4C9E3F3A-90B3-43B1-B01F-E157E1107E1E@fjl.co.uk> List-Id: General discussion of FreeBSD hardware List-Archive: https://lists.freebsd.org/archives/freebsd-hardware List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hardware@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spamd-Bar: -- X-Spamd-Result: default: False [-2.50 / 15.00]; NEURAL_HAM_SHORT(-0.98)[-0.977]; NEURAL_HAM_MEDIUM(-0.98)[-0.976]; NEURAL_HAM_LONG(-0.45)[-0.449]; R_SPF_ALLOW(-0.20)[+ip4:84.45.41.196]; ONCE_RECEIVED(0.10)[]; RCVD_NO_TLS_LAST(0.10)[]; MIME_GOOD(-0.10)[text/plain]; ASN(0.00)[asn:25577, ipnet:84.45.0.0/17, country:GB]; RCVD_COUNT_ONE(0.00)[1]; RCPT_COUNT_ONE(0.00)[1]; MIME_TRACE(0.00)[0:+]; FROM_HAS_DN(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; MLMMJ_DEST(0.00)[freebsd-hardware@freebsd.org]; TO_DN_NONE(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; DMARC_NA(0.00)[fjl.co.uk]; PREVIOUSLY_DELIVERED(0.00)[freebsd-hardware@freebsd.org]; ARC_NA(0.00)[] X-Rspamd-Queue-Id: 4Twd4Y0VR1z4XHY Flash storage is complicated=2E I doubt there's a huge cache in them, as as= it would be volatile it'd be a big no-no for synchronous writes=2E The OS = could cache it, of course=2E And if you're using ZFS then all bets are off= =2E ZFS guarantees (for POSIX) that a synchronous write goes to non volatil= e memory before the system call returns=2E However this does not mean it go= es to the final location on the disk=2E ZFS has an intent log of stuff that= needs to be sorted out sometime, but is safe for now=2E So with can write = away, especially with highly compressible blocks, and it'll go fast enough = until it gets to the point it needs to get the FS in order (you run out of = ZIL)=2E I'm speculating about the cause of the effect in your case, but I'm not su= rprised you're getting an effect=2E Look up Flash storage strategies and the workings of the ZIL for more info= rmation=2E From nobody Thu Mar 14 19:48:41 2024 X-Original-To: freebsd-hardware@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4TwdHW3D18z5Dhfq for ; Thu, 14 Mar 2024 19:49:11 +0000 (UTC) (envelope-from freebsd-doc@fjl.co.uk) Received: from bs1.fjl.org.uk (bs1.fjl.org.uk [84.45.41.196]) by mx1.freebsd.org (Postfix) with ESMTP id 4TwdHW1nGqz4Z01 for ; Thu, 14 Mar 2024 19:49:11 +0000 (UTC) (envelope-from freebsd-doc@fjl.co.uk) Authentication-Results: mx1.freebsd.org; none Received: from [127.0.0.1] ([185.69.144.64]) (authenticated bits=0) by bs1.fjl.org.uk (8.14.4/8.14.4) with ESMTP id 42EJn8J3069069; Thu, 14 Mar 2024 19:49:10 GMT (envelope-from freebsd-doc@fjl.co.uk) Date: Thu, 14 Mar 2024 19:48:41 +0000 From: Frank Leonhardt To: freebsd-hardware@freebsd.org, mike tancsa , Bob Bishop Subject: Re: WD Blue 510 SSD and strange write performance User-Agent: K-9 Mail for Android In-Reply-To: <6504bd49-eca5-4e0a-b2bd-23d29405bb7a@sentex.net> References: <6504bd49-eca5-4e0a-b2bd-23d29405bb7a@sentex.net> Message-ID: <72F0A91C-AADC-46F4-B73E-7A47091BEFFD@fjl.co.uk> List-Id: General discussion of FreeBSD hardware List-Archive: https://lists.freebsd.org/archives/freebsd-hardware List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hardware@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:25577, ipnet:84.45.0.0/17, country:GB] X-Rspamd-Queue-Id: 4TwdHW1nGqz4Z01 "CAM status: SCSI Status Error" suggests to me that the drive was just too = busy when asked=2E I'm not saying it's nothing to worry about, but neither = am I saying it is=2E From nobody Thu Mar 14 19:56:13 2024 X-Original-To: freebsd-hardware@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4TwdRk4KXBz5DjCl for ; Thu, 14 Mar 2024 19:56:18 +0000 (UTC) (envelope-from mike@sentex.net) Received: from smarthost1.sentex.ca (smarthost1.sentex.ca [IPv6:2607:f3e0:0:1::12]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smarthost1.sentex.ca", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4TwdRk1G45z4bSC for ; Thu, 14 Mar 2024 19:56:18 +0000 (UTC) (envelope-from mike@sentex.net) Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of mike@sentex.net designates 2607:f3e0:0:1::12 as permitted sender) smtp.mailfrom=mike@sentex.net Received: from pyroxene2a.sentex.ca (pyroxene19.sentex.ca [199.212.134.19]) by smarthost1.sentex.ca (8.17.1/8.16.1) with ESMTPS id 42EJuEwp013173 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=FAIL); Thu, 14 Mar 2024 15:56:14 -0400 (EDT) (envelope-from mike@sentex.net) Received: from [IPV6:2607:f3e0:0:4:a1d0:ee73:adac:870f] ([IPv6:2607:f3e0:0:4:a1d0:ee73:adac:870f]) by pyroxene2a.sentex.ca (8.17.1/8.15.2) with ESMTPS id 42EJuDNi045992 (version=TLSv1.3 cipher=TLS_AES_128_GCM_SHA256 bits=128 verify=NO); Thu, 14 Mar 2024 15:56:13 -0400 (EDT) (envelope-from mike@sentex.net) Message-ID: <69e47494-01aa-4149-a326-91d82dfdc46e@sentex.net> Date: Thu, 14 Mar 2024 15:56:13 -0400 List-Id: General discussion of FreeBSD hardware List-Archive: https://lists.freebsd.org/archives/freebsd-hardware List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hardware@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: WD Blue 510 SSD and strange write performance Content-Language: en-US To: Frank Leonhardt , freebsd-hardware@freebsd.org, Bob Bishop References: <6504bd49-eca5-4e0a-b2bd-23d29405bb7a@sentex.net> <4832DE6A-5C82-4805-99BB-220D4342AE0F@fjl.co.uk> From: mike tancsa Autocrypt: addr=mike@sentex.net; keydata= xsBNBFywzOMBCACoNFpwi5MeyEREiCeHtbm6pZJI/HnO+wXdCAWtZkS49weOoVyUj5BEXRZP xflV2ib2hflX4nXqhenaNiia4iaZ9ft3I1ebd7GEbGnsWCvAnob5MvDZyStDAuRxPJK1ya/s +6rOvr+eQiXYNVvfBhrCfrtR/esSkitBGxhUkBjOti8QwzD71JVF5YaOjBAs7jZUKyLGj0kW yDg4jUndudWU7G2yc9GwpHJ9aRSUN8e/mWdIogK0v+QBHfv/dsI6zVB7YuxCC9Fx8WPwfhDH VZC4kdYCQWKXrm7yb4TiVdBh5kgvlO9q3js1yYdfR1x8mjK2bH2RSv4bV3zkNmsDCIxjABEB AAHNHW1pa2UgdGFuY3NhIDxtaWtlQHNlbnRleC5uZXQ+wsCOBBMBCAA4FiEEmuvCXT0aY6hs 4SbWeVOEFl5WrMgFAl+pQfkCGwMFCwkIBwIGFQoJCAsCBBYCAwECHgECF4AACgkQeVOEFl5W rMiN6ggAk3H5vk8QnbvGbb4sinxZt/wDetgk0AOR9NRmtTnPaW+sIJEfGBOz47Xih+f7uWJS j+uvc9Ewn2Z7n8z3ZHJlLAByLVLtcNXGoRIGJ27tevfOaNqgJHBPbFOcXCBBFTx4MYMM4iAZ cDT5vsBTSaM36JZFtHZBKkuFEItbA/N8ZQSHKdTYMIA7A3OCLGbJBqloQ8SlW4MkTzKX4u7R yefAYQ0h20x9IqC5Ju8IsYRFacVZconT16KS81IBceO42vXTN0VexbVF2rZIx3v/NT75r6Vw 0FlXVB1lXOHKydRA2NeleS4NEG2vWqy/9Boj0itMfNDlOhkrA/0DcCurMpnpbM7ATQRcsMzk AQgA1Dpo/xWS66MaOJLwA28sKNMwkEk1Yjs+okOXDOu1F+0qvgE8sVmrOOPvvWr4axtKRSG1 t2QUiZ/ZkW/x/+t0nrM39EANV1VncuQZ1ceIiwTJFqGZQ8kb0+BNkwuNVFHRgXm1qzAJweEt RdsCMohB+H7BL5LGCVG5JaU0lqFU9pFP40HxEbyzxjsZgSE8LwkI6wcu0BLv6K6cLm0EiHPO l5G8kgRi38PS7/6s3R8QDsEtbGsYy6O82k3zSLIjuDBwA9GRaeigGppTxzAHVjf5o9KKu4O7 gC2KKVHPegbXS+GK7DU0fjzX57H5bZ6komE5eY4p3oWT/CwVPSGfPs8jOwARAQABwsB2BBgB CAAgFiEEmuvCXT0aY6hs4SbWeVOEFl5WrMgFAl+pQfkCGwwACgkQeVOEFl5WrMiVqwf9GwU8 c6cylknZX8QwlsVudTC8xr/L17JA84wf03k3d4wxP7bqy5AYy7jboZMbgWXngAE/HPQU95NM aukysSnknzoIpC96XZJ0okLBXVS6Y0ylZQ+HrbIhMpuQPoDweoF5F9wKrsHRoDaUK1VR706X rwm4HUzh7Jk+auuMYfuCh0FVlFBEuiJWMLhg/5WCmcRfiuB6F59ZcUQrwLEZeNhF2XJV4KwB Tlg7HCWO/sy1foE5noaMyACjAtAQE9p5kGYaj+DuRhPdWUTsHNuqrhikzIZd2rrcMid+ktb0 NvtvswzMO059z1YGMtGSqQ4srCArju+XHIdTFdiIYbd7+jeehg== In-Reply-To: <4832DE6A-5C82-4805-99BB-220D4342AE0F@fjl.co.uk> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-Scanned-By: MIMEDefang 2.86 on 64.7.153.18 X-Spamd-Bar: --- X-Spamd-Result: default: False [-3.39 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-1.000]; R_SPF_ALLOW(-0.20)[+ip6:2607:f3e0::/32]; RCVD_IN_DNSWL_LOW(-0.10)[199.212.134.19:received]; MIME_GOOD(-0.10)[text/plain]; XM_UA_NO_VERSION(0.01)[]; ARC_NA(0.00)[]; TO_DN_SOME(0.00)[]; ASN(0.00)[asn:11647, ipnet:2607:f3e0::/32, country:CA]; MIME_TRACE(0.00)[0:+]; FREEFALL_USER(0.00)[mike]; RCVD_TLS_ALL(0.00)[]; MLMMJ_DEST(0.00)[freebsd-hardware@freebsd.org]; R_DKIM_NA(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; FROM_HAS_DN(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_COUNT_TWO(0.00)[2]; DMARC_NA(0.00)[sentex.net]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCPT_COUNT_THREE(0.00)[3] X-Rspamd-Queue-Id: 4TwdRk1G45z4bSC On 3/14/2024 3:48 PM, Frank Leonhardt wrote: > "CAM status: SCSI Status Error" suggests to me that the drive was just too busy when asked. I'm not saying it's nothing to worry about, but neither am I saying it is. Given enough of them it does cause checksum errors on the test pool unfortunately.  Could a buggy TRIM play a role here too ? I noticed a commit the other day for a Segate SSD that had a broken NCQ TRIM. Could these units suffer from that ? https://cgit.freebsd.org/src/commit/?h=stable/14&id=47fff7407c22c2c4b36b4f9f27ddfa70bb8f3fee Is there a way to turn that off via camcontrol ? Or perhaps instrument some other settings ?  I am not wedded to this hardware, but it would be good to know if they can be made workable without too much effort. kern.cam.da.2.delete_max: 17179607040 kern.cam.da.2.delete_method: ATA_TRIM kern.cam.da.6.delete_max: 17179607040 kern.cam.da.6.delete_method: ATA_TRIM kern.cam.da.0.delete_max: 17179607040 kern.cam.da.0.delete_method: ATA_TRIM kern.cam.da.8.delete_max: 17179607040 kern.cam.da.8.delete_method: ATA_TRIM kern.cam.da.3.delete_max: 17179607040 kern.cam.da.3.delete_method: ATA_TRIM kern.cam.da.7.delete_max: 17179607040 kern.cam.da.7.delete_method: ATA_TRIM kern.cam.da.5.delete_max: 17179607040 kern.cam.da.5.delete_method: ATA_TRIM kern.cam.da.4.delete_max: 17179607040 kern.cam.da.4.delete_method: ATA_TRIM kern.cam.da.1.delete_max: 17179607040 kern.cam.da.1.delete_method: ATA_TRIM kern.cam.ada.0.delete_method: DSM_TRIM  # camcontrol iden da3 pass3: ACS-4 ATA SATA 3.x device pass3: 600.000MB/s transfers, Command Queueing Enabled protocol              ACS-4 ATA SATA 3.x device model          WD Blue SA510 2.5 1000GB firmware revision     52046100 serial number         23261Y800771 WWN                   5001b448bf6521a5 additional product id cylinders             16383 heads                 16 sectors/track         63 sector size           logical 512, physical 512, offset 0 LBA supported         268435455 sectors LBA48 supported       1953525168 sectors PIO supported         PIO4 DMA supported         WDMA2 UDMA6 media RPM             non-rotating Zoned-Device Commands no Feature                      Support  Enabled   Value Vendor read ahead                     yes      yes write cache                    yes      yes flush cache                    yes      yes Native Command Queuing (NCQ)   yes              32 tags NCQ Priority Information       no NCQ Non-Data Command           no NCQ Streaming                  no Receive & Send FPDMA Queued    no NCQ Autosense                  no SMART                          yes      yes security                       yes      no power management               yes      yes microcode download             yes      yes advanced power management      yes      no      0/0x00 automatic acoustic management  no       no media status notification      no       no power-up in Standby            no       no write-read-verify              no       no unload                         no       no general purpose logging        yes      yes free-fall                      no       no sense data reporting           no       no extended power conditions      no       no device statistics notification no       no Data Set Management (DSM/TRIM) yes DSM - max 512byte blocks       yes              8 DSM - deterministic read       yes              any value Trusted Computing              no encrypts all user data         no Sanitize                       yes              block, Sanitize - commands allowed    yes Sanitize - antifreeze lock     yes Host Protected Area (HPA)      no Accessible Max Address Config  yes      no 1953525168/1953525168 From nobody Thu Mar 14 20:26:15 2024 X-Original-To: freebsd-hardware@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Twf6Q1lX1z5DlHp for ; Thu, 14 Mar 2024 20:26:22 +0000 (UTC) (envelope-from mike@sentex.net) Received: from smarthost1.sentex.ca (smarthost1.sentex.ca [IPv6:2607:f3e0:0:1::12]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smarthost1.sentex.ca", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Twf6M5lkpz4fXj for ; Thu, 14 Mar 2024 20:26:19 +0000 (UTC) (envelope-from mike@sentex.net) Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of mike@sentex.net designates 2607:f3e0:0:1::12 as permitted sender) smtp.mailfrom=mike@sentex.net Received: from pyroxene2a.sentex.ca (pyroxene19.sentex.ca [199.212.134.19]) by smarthost1.sentex.ca (8.17.1/8.16.1) with ESMTPS id 42EKQFBC015677 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=FAIL); Thu, 14 Mar 2024 16:26:15 -0400 (EDT) (envelope-from mike@sentex.net) Received: from [IPV6:2607:f3e0:0:4:a1d0:ee73:adac:870f] ([IPv6:2607:f3e0:0:4:a1d0:ee73:adac:870f]) by pyroxene2a.sentex.ca (8.17.1/8.15.2) with ESMTPS id 42EKQEoH048748 (version=TLSv1.3 cipher=TLS_AES_128_GCM_SHA256 bits=128 verify=NO); Thu, 14 Mar 2024 16:26:14 -0400 (EDT) (envelope-from mike@sentex.net) Message-ID: <8f7ec337-16ab-4dd3-a291-e6830f31c51e@sentex.net> Date: Thu, 14 Mar 2024 16:26:15 -0400 List-Id: General discussion of FreeBSD hardware List-Archive: https://lists.freebsd.org/archives/freebsd-hardware List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hardware@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: WD Blue 510 SSD and strange write performance Content-Language: en-US From: mike tancsa To: Frank Leonhardt , freebsd-hardware@freebsd.org, Bob Bishop References: <6504bd49-eca5-4e0a-b2bd-23d29405bb7a@sentex.net> <4832DE6A-5C82-4805-99BB-220D4342AE0F@fjl.co.uk> <69e47494-01aa-4149-a326-91d82dfdc46e@sentex.net> Autocrypt: addr=mike@sentex.net; keydata= xsBNBFywzOMBCACoNFpwi5MeyEREiCeHtbm6pZJI/HnO+wXdCAWtZkS49weOoVyUj5BEXRZP xflV2ib2hflX4nXqhenaNiia4iaZ9ft3I1ebd7GEbGnsWCvAnob5MvDZyStDAuRxPJK1ya/s +6rOvr+eQiXYNVvfBhrCfrtR/esSkitBGxhUkBjOti8QwzD71JVF5YaOjBAs7jZUKyLGj0kW yDg4jUndudWU7G2yc9GwpHJ9aRSUN8e/mWdIogK0v+QBHfv/dsI6zVB7YuxCC9Fx8WPwfhDH VZC4kdYCQWKXrm7yb4TiVdBh5kgvlO9q3js1yYdfR1x8mjK2bH2RSv4bV3zkNmsDCIxjABEB AAHNHW1pa2UgdGFuY3NhIDxtaWtlQHNlbnRleC5uZXQ+wsCOBBMBCAA4FiEEmuvCXT0aY6hs 4SbWeVOEFl5WrMgFAl+pQfkCGwMFCwkIBwIGFQoJCAsCBBYCAwECHgECF4AACgkQeVOEFl5W rMiN6ggAk3H5vk8QnbvGbb4sinxZt/wDetgk0AOR9NRmtTnPaW+sIJEfGBOz47Xih+f7uWJS j+uvc9Ewn2Z7n8z3ZHJlLAByLVLtcNXGoRIGJ27tevfOaNqgJHBPbFOcXCBBFTx4MYMM4iAZ cDT5vsBTSaM36JZFtHZBKkuFEItbA/N8ZQSHKdTYMIA7A3OCLGbJBqloQ8SlW4MkTzKX4u7R yefAYQ0h20x9IqC5Ju8IsYRFacVZconT16KS81IBceO42vXTN0VexbVF2rZIx3v/NT75r6Vw 0FlXVB1lXOHKydRA2NeleS4NEG2vWqy/9Boj0itMfNDlOhkrA/0DcCurMpnpbM7ATQRcsMzk AQgA1Dpo/xWS66MaOJLwA28sKNMwkEk1Yjs+okOXDOu1F+0qvgE8sVmrOOPvvWr4axtKRSG1 t2QUiZ/ZkW/x/+t0nrM39EANV1VncuQZ1ceIiwTJFqGZQ8kb0+BNkwuNVFHRgXm1qzAJweEt RdsCMohB+H7BL5LGCVG5JaU0lqFU9pFP40HxEbyzxjsZgSE8LwkI6wcu0BLv6K6cLm0EiHPO l5G8kgRi38PS7/6s3R8QDsEtbGsYy6O82k3zSLIjuDBwA9GRaeigGppTxzAHVjf5o9KKu4O7 gC2KKVHPegbXS+GK7DU0fjzX57H5bZ6komE5eY4p3oWT/CwVPSGfPs8jOwARAQABwsB2BBgB CAAgFiEEmuvCXT0aY6hs4SbWeVOEFl5WrMgFAl+pQfkCGwwACgkQeVOEFl5WrMiVqwf9GwU8 c6cylknZX8QwlsVudTC8xr/L17JA84wf03k3d4wxP7bqy5AYy7jboZMbgWXngAE/HPQU95NM aukysSnknzoIpC96XZJ0okLBXVS6Y0ylZQ+HrbIhMpuQPoDweoF5F9wKrsHRoDaUK1VR706X rwm4HUzh7Jk+auuMYfuCh0FVlFBEuiJWMLhg/5WCmcRfiuB6F59ZcUQrwLEZeNhF2XJV4KwB Tlg7HCWO/sy1foE5noaMyACjAtAQE9p5kGYaj+DuRhPdWUTsHNuqrhikzIZd2rrcMid+ktb0 NvtvswzMO059z1YGMtGSqQ4srCArju+XHIdTFdiIYbd7+jeehg== In-Reply-To: <69e47494-01aa-4149-a326-91d82dfdc46e@sentex.net> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-Scanned-By: MIMEDefang 2.86 on 64.7.153.18 X-Spamd-Bar: --- X-Spamd-Result: default: False [-3.38 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.99)[-0.994]; R_SPF_ALLOW(-0.20)[+ip6:2607:f3e0::/32]; RCVD_IN_DNSWL_LOW(-0.10)[199.212.134.19:received]; MIME_GOOD(-0.10)[text/plain]; XM_UA_NO_VERSION(0.01)[]; ARC_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:11647, ipnet:2607:f3e0::/32, country:CA]; TO_DN_SOME(0.00)[]; FREEFALL_USER(0.00)[mike]; RCVD_TLS_ALL(0.00)[]; MLMMJ_DEST(0.00)[freebsd-hardware@freebsd.org]; R_DKIM_NA(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; FROM_HAS_DN(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_COUNT_TWO(0.00)[2]; DMARC_NA(0.00)[sentex.net]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCPT_COUNT_THREE(0.00)[3] X-Rspamd-Queue-Id: 4Twf6M5lkpz4fXj On 3/14/2024 3:56 PM, mike tancsa wrote: > On 3/14/2024 3:48 PM, Frank Leonhardt wrote: >> "CAM status: SCSI Status Error" suggests to me that the drive was >> just too busy when asked. I'm not saying it's nothing to worry about, >> but neither am I saying it is. > > Given enough of them it does cause checksum errors on the test pool > unfortunately.  Could a buggy TRIM play a role here too ? I noticed a > commit the other day for a Segate SSD that had a broken NCQ TRIM. > Could these units suffer from that ? > https://cgit.freebsd.org/src/commit/?h=stable/14&id=47fff7407c22c2c4b36b4f9f27ddfa70bb8f3fee > > > Is there a way to turn that off via camcontrol ? Or perhaps instrument > some other settings ?  I am not wedded to this hardware, but it would > be good to know if they can be made workable without too much effort. > On another test box with an MPR controller and same WD drives, a few more messages after the zfs send is about 50% done on a dataset thats about 1TB but compressed to about 260G. Some 29 million files. But with Samsungs, reliably no issue :( (da6:mpr0:0:16:0): WRITE(10). CDB: 2a 00 48 1e 9b 90 00 00 80 00 mpr0: Controller reported scsi ioc terminated tgt 16 SMID 897 loginfo 31110f00 mpr0: Controller reported scsi ioc terminated tgt 16 SMID 1358 loginfo 31110f00 mpr0: Controller reported scsi ioc terminated tgt 16 SMID 1742 loginfo 31110f00 mpr0: Controller reported scsi ioc terminated tgt 16 SMID 1187 loginfo 31110f00 mpr0: Controller reported scsi ioc terminated tgt 16 SMID 1006 loginfo 31110f00 mpr0: Controller reported scsi ioc terminated tgt 16 SMID 758 loginfo 31110f00 (da6:mpr0:0:16:0): CAM status: CCB request completed with an error (da6:mpr0:0:16:0): Retrying command, 3 more tries remain (da6:mpr0:0:16:0): WRITE(10). CDB: 2a 00 48 1e 9c 10 00 00 b8 00 (da6:mpr0:0:16:0): CAM status: CCB request completed with an error (da6:mpr0:0:16:0): Retrying command, 3 more tries remain (da6:mpr0:0:16:0): WRITE(10). CDB: 2a 00 46 93 47 18 00 00 08 00 (da6:mpr0:0:16:0): CAM status: CCB request completed with an error (da6:mpr0:0:16:0): Retrying command, 3 more tries remain (da6:mpr0:0:16:0): READ(10). CDB: 28 00 1c c7 dc 40 00 01 00 00 (da6:mpr0:0:16:0): CAM status: CCB request completed with an error (da6:mpr0:0:16:0): Retrying command, 3 more tries remain (da6:mpr0:0:16:0): READ(10). CDB: 28 00 1c c7 d9 30 00 00 f8 00 (da6:mpr0:0:16:0): CAM status: CCB request completed with an error (da6:mpr0:0:16:0): Retrying command, 3 more tries remain (da6:mpr0:0:16:0): WRITE(10). CDB: 2a 00 46 93 47 10 00 00 08 00 (da6:mpr0:0:16:0): CAM status: CCB request completed with an error (da6:mpr0:0:16:0): Retrying command, 3 more tries remain (da6:mpr0:0:16:0): READ(10). CDB: 28 00 1c c7 d8 30 00 01 00 00 (da6:mpr0:0:16:0): CAM status: CCB request completed with an error (da6:mpr0:0:16:0): Retrying command, 3 more tries remain (da6:mpr0:0:16:0): WRITE(10). CDB: 2a 00 49 55 29 20 00 00 08 00 (da6:mpr0:0:16:0): CAM status: CCB request completed with an error (da6:mpr0:0:16:0): Retrying command, 3 more tries remain (da6:mpr0:0:16:0): WRITE(10). CDB: 2a 00 48 1e 9b 90 00 00 80 00 (da6:mpr0:0:16:0): CAM status: SCSI Status Error (da6:mpr0:0:16:0): SCSI status: Check Condition (da6:mpr0:0:16:0): SCSI sense: UNIT ATTENTION asc:29,0 (Power on, reset, or bus device reset occurred) (da6:mpr0:0:16:0): Retrying command (per sense data) mpr0: Controller reported scsi ioc terminated tgt 13 SMID 1023 loginfo 31110f00 mpr0: Controller reported scsi ioc terminated tgt 13 SMID 297 loginfo 31110f00 (da3:mpr0:0:13:0): WRITE(10). CDB: 2a 00 4c 49 50 18 00 00 a0 00 mpr0: Controller reported scsi ioc terminated tgt 13 SMID 1999 loginfo 31110f00 mpr0: Controller reported scsi ioc terminated tgt 13 SMID 280 loginfo 31110f00 mpr0: Controller reported scsi ioc terminated tgt 13 SMID 1970 loginfo 31110f00 mpr0: Controller reported scsi ioc terminated tgt 13 SMID 859 loginfo 31110f00 mpr0: Controller reported scsi ioc terminated tgt 13 SMID 1652 loginfo 31110f00 mpr0: Controller reported scsi ioc terminated tgt 13 SMID 613 loginfo 31110f00 (da3:mpr0:0:13:0): CAM status: CCB request completed with an error (da3:mpr0:0:13:0): Retrying command, 3 more tries remain (da3:mpr0:0:13:0): WRITE(10). CDB: 2a 00 4c 49 4e a8 00 01 00 00 (da3:mpr0:0:13:0): CAM status: CCB request completed with an error (da3:mpr0:0:13:0): Retrying command, 3 more tries remain (da3:mpr0:0:13:0): WRITE(10). CDB: 2a 00 4c 49 4f a8 00 00 70 00 (da3:mpr0:0:13:0): CAM status: CCB request completed with an error (da3:mpr0:0:13:0): Retrying command, 3 more tries remain (da3:mpr0:0:13:0): WRITE(10). CDB: 2a 00 4c bc 65 30 00 01 08 00 (da3:mpr0:0:13:0): CAM status: CCB request completed with an error (da3:mpr0:0:13:0): Retrying command, 3 more tries remain (da3:mpr0:0:13:0): READ(10). CDB: 28 00 2c 99 68 80 00 01 00 00 (da3:mpr0:0:13:0): CAM status: CCB request completed with an error (da3:mpr0:0:13:0): Retrying command, 3 more tries remain (da3:mpr0:0:13:0): WRITE(10). CDB: 2a 00 46 98 1e b8 00 00 18 00 (da3:mpr0:0:13:0): CAM status: CCB request completed with an error (da3:mpr0:0:13:0): Retrying command, 3 more tries remain (da3:mpr0:0:13:0): READ(10). CDB: 28 00 1e cd a8 28 00 00 18 00 (da3:mpr0:0:13:0): CAM status: CCB request completed with an error (da3:mpr0:0:13:0): Retrying command, 3 more tries remain (da3:mpr0:0:13:0): WRITE(10). CDB: 2a 00 4c bc 64 30 00 01 00 00 (da3:mpr0:0:13:0): CAM status: CCB request completed with an error (da3:mpr0:0:13:0): Retrying command, 3 more tries remain (da3:mpr0:0:13:0): WRITE(10). CDB: 2a 00 4c 49 4e a8 00 01 00 00 (da3:mpr0:0:13:0): CAM status: SCSI Status Error (da3:mpr0:0:13:0): SCSI status: Check Condition (da3:mpr0:0:13:0): SCSI sense: UNIT ATTENTION asc:29,0 (Power on, reset, or bus device reset occurred) (da3:mpr0:0:13:0): Retrying command (per sense data) From nobody Thu Mar 14 20:58:52 2024 X-Original-To: freebsd-hardware@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4TwgN275Yqz5Dbg8 for ; Thu, 14 Mar 2024 21:23:14 +0000 (UTC) (envelope-from freebsd-doc@fjl.co.uk) Received: from bs1.fjl.org.uk (bs1.fjl.org.uk [84.45.41.196]) by mx1.freebsd.org (Postfix) with ESMTP id 4TwgN21zKHz4nKB for ; Thu, 14 Mar 2024 21:23:14 +0000 (UTC) (envelope-from freebsd-doc@fjl.co.uk) Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of freebsd-doc@fjl.co.uk designates 84.45.41.196 as permitted sender) smtp.mailfrom=freebsd-doc@fjl.co.uk Received: from [127.0.0.1] ([185.69.144.64]) (authenticated bits=0) by bs1.fjl.org.uk (8.14.4/8.14.4) with ESMTP id 42ELNCYH090905 for ; Thu, 14 Mar 2024 21:23:13 GMT (envelope-from freebsd-doc@fjl.co.uk) Date: Thu, 14 Mar 2024 20:58:52 +0000 From: Frank Leonhardt To: freebsd-hardware@freebsd.org Subject: Re: WD Blue 510 SSD and strange write performance User-Agent: K-9 Mail for Android In-Reply-To: <69e47494-01aa-4149-a326-91d82dfdc46e@sentex.net> References: <6504bd49-eca5-4e0a-b2bd-23d29405bb7a@sentex.net> <4832DE6A-5C82-4805-99BB-220D4342AE0F@fjl.co.uk> <69e47494-01aa-4149-a326-91d82dfdc46e@sentex.net> Message-ID: List-Id: General discussion of FreeBSD hardware List-Archive: https://lists.freebsd.org/archives/freebsd-hardware List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hardware@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spamd-Bar: -- X-Spamd-Result: default: False [-2.53 / 15.00]; NEURAL_HAM_SHORT(-0.99)[-0.992]; NEURAL_HAM_MEDIUM(-0.98)[-0.976]; NEURAL_HAM_LONG(-0.46)[-0.459]; R_SPF_ALLOW(-0.20)[+ip4:84.45.41.196:c]; ONCE_RECEIVED(0.10)[]; RCVD_NO_TLS_LAST(0.10)[]; MIME_GOOD(-0.10)[text/plain]; ASN(0.00)[asn:25577, ipnet:84.45.0.0/17, country:GB]; RCVD_COUNT_ONE(0.00)[1]; ARC_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; FROM_HAS_DN(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; MLMMJ_DEST(0.00)[freebsd-hardware@freebsd.org]; TO_DN_NONE(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; DMARC_NA(0.00)[fjl.co.uk]; PREVIOUSLY_DELIVERED(0.00)[freebsd-hardware@freebsd.org]; RCPT_COUNT_ONE(0.00)[1] X-Rspamd-Queue-Id: 4TwgN21zKHz4nKB Sorry - not that deeply into modern SSD (never written a driver for one), b= ut based on my understanding your TRIM theory makes sense to me=2E I'd try = turning it off=2E It does seem to be an ongoing source of snafus=2E I did use WD Blue SSDs but I suspect they vary quite a bit=2E I've had rat= her too many early failures=2E I wouldn't use them in production but okay f= or Windoze=2E We all know deep down there's a reason the enterprise SSDs co= st what they do :-) I'll keep thinking From nobody Fri Mar 15 18:17:52 2024 X-Original-To: freebsd-hardware@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4TxCCq0Hztz5Dt0T for ; Fri, 15 Mar 2024 18:17:59 +0000 (UTC) (envelope-from mike@sentex.net) Received: from smarthost1.sentex.ca (smarthost1.sentex.ca [IPv6:2607:f3e0:0:1::12]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smarthost1.sentex.ca", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4TxCCp5L5pz4tPs for ; Fri, 15 Mar 2024 18:17:58 +0000 (UTC) (envelope-from mike@sentex.net) Authentication-Results: mx1.freebsd.org; none Received: from pyroxene2a.sentex.ca (pyroxene19.sentex.ca [199.212.134.19]) by smarthost1.sentex.ca (8.17.1/8.16.1) with ESMTPS id 42FIHra6074343 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=FAIL); Fri, 15 Mar 2024 14:17:53 -0400 (EDT) (envelope-from mike@sentex.net) Received: from [IPV6:2607:f3e0:0:4:c034:f6ba:23ac:a443] ([IPv6:2607:f3e0:0:4:c034:f6ba:23ac:a443]) by pyroxene2a.sentex.ca (8.17.1/8.15.2) with ESMTPS id 42FIHpxQ070340 (version=TLSv1.3 cipher=TLS_AES_128_GCM_SHA256 bits=128 verify=NO); Fri, 15 Mar 2024 14:17:52 -0400 (EDT) (envelope-from mike@sentex.net) Message-ID: Date: Fri, 15 Mar 2024 14:17:52 -0400 List-Id: General discussion of FreeBSD hardware List-Archive: https://lists.freebsd.org/archives/freebsd-hardware List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hardware@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: WD Blue 510 SSD and strange write performance Content-Language: en-US To: Frank Leonhardt , freebsd-hardware@freebsd.org References: <6504bd49-eca5-4e0a-b2bd-23d29405bb7a@sentex.net> <4832DE6A-5C82-4805-99BB-220D4342AE0F@fjl.co.uk> <69e47494-01aa-4149-a326-91d82dfdc46e@sentex.net> From: mike tancsa Autocrypt: addr=mike@sentex.net; keydata= xsBNBFywzOMBCACoNFpwi5MeyEREiCeHtbm6pZJI/HnO+wXdCAWtZkS49weOoVyUj5BEXRZP xflV2ib2hflX4nXqhenaNiia4iaZ9ft3I1ebd7GEbGnsWCvAnob5MvDZyStDAuRxPJK1ya/s +6rOvr+eQiXYNVvfBhrCfrtR/esSkitBGxhUkBjOti8QwzD71JVF5YaOjBAs7jZUKyLGj0kW yDg4jUndudWU7G2yc9GwpHJ9aRSUN8e/mWdIogK0v+QBHfv/dsI6zVB7YuxCC9Fx8WPwfhDH VZC4kdYCQWKXrm7yb4TiVdBh5kgvlO9q3js1yYdfR1x8mjK2bH2RSv4bV3zkNmsDCIxjABEB AAHNHW1pa2UgdGFuY3NhIDxtaWtlQHNlbnRleC5uZXQ+wsCOBBMBCAA4FiEEmuvCXT0aY6hs 4SbWeVOEFl5WrMgFAl+pQfkCGwMFCwkIBwIGFQoJCAsCBBYCAwECHgECF4AACgkQeVOEFl5W rMiN6ggAk3H5vk8QnbvGbb4sinxZt/wDetgk0AOR9NRmtTnPaW+sIJEfGBOz47Xih+f7uWJS j+uvc9Ewn2Z7n8z3ZHJlLAByLVLtcNXGoRIGJ27tevfOaNqgJHBPbFOcXCBBFTx4MYMM4iAZ cDT5vsBTSaM36JZFtHZBKkuFEItbA/N8ZQSHKdTYMIA7A3OCLGbJBqloQ8SlW4MkTzKX4u7R yefAYQ0h20x9IqC5Ju8IsYRFacVZconT16KS81IBceO42vXTN0VexbVF2rZIx3v/NT75r6Vw 0FlXVB1lXOHKydRA2NeleS4NEG2vWqy/9Boj0itMfNDlOhkrA/0DcCurMpnpbM7ATQRcsMzk AQgA1Dpo/xWS66MaOJLwA28sKNMwkEk1Yjs+okOXDOu1F+0qvgE8sVmrOOPvvWr4axtKRSG1 t2QUiZ/ZkW/x/+t0nrM39EANV1VncuQZ1ceIiwTJFqGZQ8kb0+BNkwuNVFHRgXm1qzAJweEt RdsCMohB+H7BL5LGCVG5JaU0lqFU9pFP40HxEbyzxjsZgSE8LwkI6wcu0BLv6K6cLm0EiHPO l5G8kgRi38PS7/6s3R8QDsEtbGsYy6O82k3zSLIjuDBwA9GRaeigGppTxzAHVjf5o9KKu4O7 gC2KKVHPegbXS+GK7DU0fjzX57H5bZ6komE5eY4p3oWT/CwVPSGfPs8jOwARAQABwsB2BBgB CAAgFiEEmuvCXT0aY6hs4SbWeVOEFl5WrMgFAl+pQfkCGwwACgkQeVOEFl5WrMiVqwf9GwU8 c6cylknZX8QwlsVudTC8xr/L17JA84wf03k3d4wxP7bqy5AYy7jboZMbgWXngAE/HPQU95NM aukysSnknzoIpC96XZJ0okLBXVS6Y0ylZQ+HrbIhMpuQPoDweoF5F9wKrsHRoDaUK1VR706X rwm4HUzh7Jk+auuMYfuCh0FVlFBEuiJWMLhg/5WCmcRfiuB6F59ZcUQrwLEZeNhF2XJV4KwB Tlg7HCWO/sy1foE5noaMyACjAtAQE9p5kGYaj+DuRhPdWUTsHNuqrhikzIZd2rrcMid+ktb0 NvtvswzMO059z1YGMtGSqQ4srCArju+XHIdTFdiIYbd7+jeehg== In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-Scanned-By: MIMEDefang 2.86 on 64.7.153.18 X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:11647, ipnet:2607:f3e0::/32, country:CA] X-Rspamd-Queue-Id: 4TxCCp5L5pz4tPs On 3/14/2024 4:58 PM, Frank Leonhardt wrote: > Sorry - not that deeply into modern SSD (never written a driver for one), but based on my understanding your TRIM theory makes sense to me. I'd try turning it off. It does seem to be an ongoing source of snafus. > > I did use WD Blue SSDs but I suspect they vary quite a bit. I've had rather too many early failures. I wouldn't use them in production but okay for Windoze. We all know deep down there's a reason the enterprise SSDs cost what they do :-) > > I'll keep thinking Thanks for the input. I think these drives are just kinda broken :( I noticed we had the 2TB versions of this line, but they seem rather different and I am not able to trigger the errors with them thankfully.   Even stranger, I have a 1TB version of this drive I bought from a while back that has the same firmware, but does NOT have this issue. However, the output of the identifier is slightly different.  Who knows, it could be some component WD uses that *should be* the same but is not and is causing some subtle pathology. I tried turning off NCQ on the controller and it didnt seem to make a difference. Then I turned off autotrim and did a manual trim of the pool, then did the tests and same sorts of errors.  I think I am just gonna cut my losses with these disks :(   Even if I figured out some work around at this point, I would not deploy them into production.  I doubt I will be able to get anywhere with WD. Farewell my 400 bucks :( (da6:mpr0:0:16:0): READ(10). CDB: 28 00 6d e0 ae 28 00 00 08 00 (da6:mpr0:0:16:0): CAM status: CCB request completed with an error (da6:mpr0:0:16:0): Retrying command, 3 more tries remain (da6:mpr0:0:16:0): WRITE(10). CDB: 2a 00 0c cb 3f 00 00 00 e8 00 (da6:mpr0:0:16:0): CAM status: CCB request completed with an error (da6:mpr0:0:16:0): Retrying command, 3 more tries remain (da6:mpr0:0:16:0): READ(10). CDB: 28 00 6d e0 ad 28 00 01 00 00 (da6:mpr0:0:16:0): CAM status: CCB request completed with an error (da6:mpr0:0:16:0): Retrying command, 3 more tries remain (da6:mpr0:0:16:0): READ(10). CDB: 28 00 6d e0 ac 28 00 00 f8 00 (da6:mpr0:0:16:0): CAM status: CCB request completed with an error (da6:mpr0:0:16:0): Retrying command, 3 more tries remain (da6:mpr0:0:16:0): WRITE(10). CDB: 2a 00 40 07 df 88 00 01 00 00 (da6:mpr0:0:16:0): CAM status: CCB request completed with an error (da6:mpr0:0:16:0): Retrying command, 3 more tries remain (da6:mpr0:0:16:0): WRITE(10). CDB: 2a 00 3f 48 72 08 00 01 00 00 (da6:mpr0:0:16:0): CAM status: SCSI Status Error (da6:mpr0:0:16:0): SCSI status: Check Condition (da6:mpr0:0:16:0): SCSI sense: UNIT ATTENTION asc:29,0 (Power on, reset, or bus device reset occurred) (da6:mpr0:0:16:0): Retrying command (per sense data) mpr0: Controller reported scsi ioc terminated tgt 15 SMID 2036 loginfo 31110f00 mpr0: Controller reported scsi ioc terminated tgt 15 SMID 637 loginfo 31110f00 (da5:mpr0:0:15:0): WRITE(10). CDB: 2a 00 41 98 42 00 00 01 00 00 mpr0: Controller reported scsi ioc terminated tgt 15 SMID 1242 loginfo 31110f00 mpr0: Controller reported scsi ioc terminated tgt 15 SMID 979 loginfo 31110f00 mpr0: Controller reported scsi ioc terminated tgt 15 SMID 1243 loginfo 31110f00 mpr0: Controller reported scsi ioc terminated tgt 15 SMID 2091 loginfo 31110f00 mpr0: Controller reported scsi ioc terminated tgt 15 SMID 1612 loginfo 31110f00 mpr0: Controller reported scsi ioc terminated tgt 15 SMID 2093 loginfo 31110f00 mpr0: Controller reported scsi ioc terminated tgt 15 SMID 152 loginfo 31110f00 mpr0: Controller reported scsi ioc terminated tgt 15 SMID 2132 loginfo 31110f00 (da5:mpr0:0:15:0): CAM status: CCB request completed with an error (da5:mpr0:0:15:0): Retrying command, 3 more tries remain (da5:mpr0:0:15:0): WRITE(10). CDB: 2a 00 43 17 dc 88 00 01 00 00 (da5:mpr0:0:15:0): CAM status: CCB request completed with an error (da5:mpr0:0:15:0): Retrying command, 3 more tries remain (da5:mpr0:0:15:0): WRITE(10). CDB: 2a 00 41 98 43 00 00 00 50 00 (da5:mpr0:0:15:0): CAM status: CCB request completed with an error (da5:mpr0:0:15:0): Retrying command, 3 more tries remain (da5:mpr0:0:15:0): WRITE(10). CDB: 2a 00 0c d4 f6 80 00 00 68 00 (da5:mpr0:0:15:0): CAM status: CCB request completed with an error (da5:mpr0:0:15:0): Retrying command, 3 more tries remain (da5:mpr0:0:15:0): WRITE(10). CDB: 2a 00 0c d4 f5 80 00 01 00 00 (da5:mpr0:0:15:0): CAM status: CCB request completed with an error (da5:mpr0:0:15:0): Retrying command, 3 more tries remain (da5:mpr0:0:15:0): READ(10). CDB: 28 00 05 dc 12 28 00 00 f8 00 (da5:mpr0:0:15:0): CAM status: CCB request completed with an error (da5:mpr0:0:15:0): Retrying command, 3 more tries remain (da5:mpr0:0:15:0): READ(10). CDB: 28 00 05 dc 0f b0 00 00 88 00 (da5:mpr0:0:15:0): CAM status: CCB request completed with an error (da5:mpr0:0:15:0): Retrying command, 3 more tries remain (da5:mpr0:0:15:0): WRITE(10). CDB: 2a 00 02 96 7e 80 00 00 10 00 (da5:mpr0:0:15:0): CAM status: CCB request completed with an error (da5:mpr0:0:15:0): Retrying command, 3 more tries remain (da5:mpr0:0:15:0): READ(10). CDB: 28 00 6f 5b 8d 68 00 01 00 00 (da5:mpr0:0:15:0): CAM status: CCB request completed with an error (da5:mpr0:0:15:0): Retrying command, 3 more tries remain (da5:mpr0:0:15:0): WRITE(10). CDB: 2a 00 41 98 42 00 00 01 00 00 (da5:mpr0:0:15:0): CAM status: SCSI Status Error (da5:mpr0:0:15:0): SCSI status: Check Condition (da5:mpr0:0:15:0): SCSI sense: UNIT ATTENTION asc:29,0 (Power on, reset, or bus device reset occurred) (da5:mpr0:0:15:0): Retrying command (per sense data) From nobody Fri Mar 15 19:09:56 2024 X-Original-To: freebsd-hardware@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4TxDNV63zWz5DxfM for ; Fri, 15 Mar 2024 19:10:34 +0000 (UTC) (envelope-from karl@denninger.net) Received: from colo1.denninger.net (colo1.denninger.net [104.236.120.189]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4TxDNT5vbMz41WB for ; Fri, 15 Mar 2024 19:10:33 +0000 (UTC) (envelope-from karl@denninger.net) Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=pass (policy=none) header.from=denninger.net; spf=pass (mx1.freebsd.org: domain of karl@denninger.net designates 104.236.120.189 as permitted sender) smtp.mailfrom=karl@denninger.net Received: from denninger.net (096-033-195-197.res.spectrum.com [96.33.195.197]) by colo1.denninger.net (Postfix) with ESMTP id 077E42110F3 for ; Fri, 15 Mar 2024 15:10:01 -0400 (EDT) Received: from [192.168.10.16] (D6.Denninger.Net [192.168.10.16]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by denninger.net (Postfix) with ESMTPSA id A4B43BB9E for ; Fri, 15 Mar 2024 15:09:56 -0400 (EDT) Message-ID: <0b98d5fc-0bfb-4d13-8e14-79cb3ee873c0@denninger.net> Date: Fri, 15 Mar 2024 15:09:56 -0400 List-Id: General discussion of FreeBSD hardware List-Archive: https://lists.freebsd.org/archives/freebsd-hardware List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hardware@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: WD Blue 510 SSD and strange write performance Content-Language: en-US To: freebsd-hardware@freebsd.org References: <6504bd49-eca5-4e0a-b2bd-23d29405bb7a@sentex.net> <4832DE6A-5C82-4805-99BB-220D4342AE0F@fjl.co.uk> <69e47494-01aa-4149-a326-91d82dfdc46e@sentex.net> From: Karl Denninger In-Reply-To: Content-Type: multipart/signed; protocol="application/pkcs7-signature"; micalg=sha-512; boundary="------------ms020200020503040109090109" X-Spamd-Bar: ----- X-Spamd-Result: default: False [-5.83 / 15.00]; SIGNED_SMIME(-2.00)[]; NEURAL_HAM_LONG(-1.00)[-0.998]; NEURAL_HAM_SHORT(-0.99)[-0.993]; NEURAL_HAM_MEDIUM(-0.94)[-0.945]; DMARC_POLICY_ALLOW(-0.50)[denninger.net,none]; MIME_GOOD(-0.20)[multipart/signed,text/plain]; R_SPF_ALLOW(-0.20)[+mx]; XM_UA_NO_VERSION(0.01)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; MIME_TRACE(0.00)[0:+,1:+,2:~]; RCVD_TLS_LAST(0.00)[]; FREEFALL_USER(0.00)[karl]; ASN(0.00)[asn:14061, ipnet:104.236.64.0/18, country:US]; ARC_NA(0.00)[]; R_DKIM_NA(0.00)[]; MLMMJ_DEST(0.00)[freebsd-hardware@freebsd.org]; FROM_EQ_ENVFROM(0.00)[]; FROM_HAS_DN(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_COUNT_TWO(0.00)[2]; TO_MATCH_ENVRCPT_ALL(0.00)[]; TO_DN_NONE(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[freebsd-hardware@freebsd.org]; HAS_ATTACHMENT(0.00)[] X-Rspamd-Queue-Id: 4TxDNT5vbMz41WB This is a cryptographically signed message in MIME format. --------------ms020200020503040109090109 Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit I've got both the Kingston and Micron versions of these in production use and have seen nothing like this at all; they get hit pretty hard too, including release building (both direct and cross-builds) and similar stuff. On 3/15/2024 2:17 PM, mike tancsa wrote: > On 3/14/2024 4:58 PM, Frank Leonhardt wrote: >> Sorry - not that deeply into modern SSD (never written a driver for >> one), but based on my understanding your TRIM theory makes sense to >> me. I'd try turning it off. It does seem to be an ongoing source of >> snafus. >> >> I did use WD Blue SSDs but I suspect they vary quite a bit. I've had >> rather too many early failures. I wouldn't use them in production but >> okay for Windoze. We all know deep down there's a reason the >> enterprise SSDs cost what they do :-) >> >> I'll keep thinking > > Thanks for the input. I think these drives are just kinda broken :( I > noticed we had the 2TB versions of this line, but they seem rather > different and I am not able to trigger the errors with them > thankfully.   Even stranger, I have a 1TB version of this drive I > bought from a while back that has the same firmware, but does NOT have > this issue. However, the output of the identifier is slightly > different.  Who knows, it could be some component WD uses that *should > be* the same but is not and is causing some subtle pathology. > > > I tried turning off NCQ on the controller and it didnt seem to make a > difference. Then I turned off autotrim and did a manual trim of the > pool, then did the tests and same sorts of errors.  I think I am just > gonna cut my losses with these disks :(   Even if I figured out some > work around at this point, I would not deploy them into production.  I > doubt I will be able to get anywhere with WD. Farewell my 400 bucks :( > > (da6:mpr0:0:16:0): READ(10). CDB: 28 00 6d e0 ae 28 00 00 08 00 > (da6:mpr0:0:16:0): CAM status: CCB request completed with an error > (da6:mpr0:0:16:0): Retrying command, 3 more tries remain > (da6:mpr0:0:16:0): WRITE(10). CDB: 2a 00 0c cb 3f 00 00 00 e8 00 > (da6:mpr0:0:16:0): CAM status: CCB request completed with an error > (da6:mpr0:0:16:0): Retrying command, 3 more tries remain > (da6:mpr0:0:16:0): READ(10). CDB: 28 00 6d e0 ad 28 00 01 00 00 > (da6:mpr0:0:16:0): CAM status: CCB request completed with an error > (da6:mpr0:0:16:0): Retrying command, 3 more tries remain > (da6:mpr0:0:16:0): READ(10). CDB: 28 00 6d e0 ac 28 00 00 f8 00 > (da6:mpr0:0:16:0): CAM status: CCB request completed with an error > (da6:mpr0:0:16:0): Retrying command, 3 more tries remain > (da6:mpr0:0:16:0): WRITE(10). CDB: 2a 00 40 07 df 88 00 01 00 00 > (da6:mpr0:0:16:0): CAM status: CCB request completed with an error > (da6:mpr0:0:16:0): Retrying command, 3 more tries remain > (da6:mpr0:0:16:0): WRITE(10). CDB: 2a 00 3f 48 72 08 00 01 00 00 > (da6:mpr0:0:16:0): CAM status: SCSI Status Error > (da6:mpr0:0:16:0): SCSI status: Check Condition > (da6:mpr0:0:16:0): SCSI sense: UNIT ATTENTION asc:29,0 (Power on, > reset, or bus device reset occurred) > (da6:mpr0:0:16:0): Retrying command (per sense data) > mpr0: Controller reported scsi ioc terminated tgt 15 SMID 2036 loginfo > 31110f00 > mpr0: Controller reported scsi ioc terminated tgt 15 SMID 637 loginfo > 31110f00 > (da5:mpr0:0:15:0): WRITE(10). CDB: 2a 00 41 98 42 00 00 01 00 00 > mpr0: Controller reported scsi ioc terminated tgt 15 SMID 1242 loginfo > 31110f00 > mpr0: Controller reported scsi ioc terminated tgt 15 SMID 979 loginfo > 31110f00 > mpr0: Controller reported scsi ioc terminated tgt 15 SMID 1243 loginfo > 31110f00 > mpr0: Controller reported scsi ioc terminated tgt 15 SMID 2091 loginfo > 31110f00 > mpr0: Controller reported scsi ioc terminated tgt 15 SMID 1612 loginfo > 31110f00 > mpr0: Controller reported scsi ioc terminated tgt 15 SMID 2093 loginfo > 31110f00 > mpr0: Controller reported scsi ioc terminated tgt 15 SMID 152 loginfo > 31110f00 > mpr0: Controller reported scsi ioc terminated tgt 15 SMID 2132 loginfo > 31110f00 > (da5:mpr0:0:15:0): CAM status: CCB request completed with an error > (da5:mpr0:0:15:0): Retrying command, 3 more tries remain > (da5:mpr0:0:15:0): WRITE(10). CDB: 2a 00 43 17 dc 88 00 01 00 00 > (da5:mpr0:0:15:0): CAM status: CCB request completed with an error > (da5:mpr0:0:15:0): Retrying command, 3 more tries remain > (da5:mpr0:0:15:0): WRITE(10). CDB: 2a 00 41 98 43 00 00 00 50 00 > (da5:mpr0:0:15:0): CAM status: CCB request completed with an error > (da5:mpr0:0:15:0): Retrying command, 3 more tries remain > (da5:mpr0:0:15:0): WRITE(10). CDB: 2a 00 0c d4 f6 80 00 00 68 00 > (da5:mpr0:0:15:0): CAM status: CCB request completed with an error > (da5:mpr0:0:15:0): Retrying command, 3 more tries remain > (da5:mpr0:0:15:0): WRITE(10). CDB: 2a 00 0c d4 f5 80 00 01 00 00 > (da5:mpr0:0:15:0): CAM status: CCB request completed with an error > (da5:mpr0:0:15:0): Retrying command, 3 more tries remain > (da5:mpr0:0:15:0): READ(10). CDB: 28 00 05 dc 12 28 00 00 f8 00 > (da5:mpr0:0:15:0): CAM status: CCB request completed with an error > (da5:mpr0:0:15:0): Retrying command, 3 more tries remain > (da5:mpr0:0:15:0): READ(10). CDB: 28 00 05 dc 0f b0 00 00 88 00 > (da5:mpr0:0:15:0): CAM status: CCB request completed with an error > (da5:mpr0:0:15:0): Retrying command, 3 more tries remain > (da5:mpr0:0:15:0): WRITE(10). CDB: 2a 00 02 96 7e 80 00 00 10 00 > (da5:mpr0:0:15:0): CAM status: CCB request completed with an error > (da5:mpr0:0:15:0): Retrying command, 3 more tries remain > (da5:mpr0:0:15:0): READ(10). CDB: 28 00 6f 5b 8d 68 00 01 00 00 > (da5:mpr0:0:15:0): CAM status: CCB request completed with an error > (da5:mpr0:0:15:0): Retrying command, 3 more tries remain > (da5:mpr0:0:15:0): WRITE(10). CDB: 2a 00 41 98 42 00 00 01 00 00 > (da5:mpr0:0:15:0): CAM status: SCSI Status Error > (da5:mpr0:0:15:0): SCSI status: Check Condition > (da5:mpr0:0:15:0): SCSI sense: UNIT ATTENTION asc:29,0 (Power on, > reset, or bus device reset occurred) > (da5:mpr0:0:15:0): Retrying command (per sense data) > > > > -- -- Karl Denninger /The Market-Ticker/ S/MIME Email accepted and preferred --------------ms020200020503040109090109 Content-Type: application/pkcs7-signature; name="smime.p7s" Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="smime.p7s" Content-Description: S/MIME Cryptographic Signature MIAGCSqGSIb3DQEHAqCAMIACAQExDzANBglghkgBZQMEAgMFADCABgkqhkiG9w0BBwEAAKCC DbowggagMIIEiKADAgECAhMA5EiKghDOXrvfxYxjITXYDdhIMA0GCSqGSIb3DQEBCwUAMIGL MQswCQYDVQQGEwJVUzEQMA4GA1UECAwHRmxvcmlkYTESMBAGA1UEBwwJTmljZXZpbGxlMRkw FwYDVQQKDBBDdWRhIFN5c3RlbXMgTExDMRgwFgYDVQQLDA9DdWRhIFN5c3RlbXMgQ0ExITAf BgNVBAMMGEN1ZGEgU3lzdGVtcyBMTEMgMjAxNyBDQTAeFw0xNzA4MTcxNjQyMTdaFw0yNzA4 MTUxNjQyMTdaMHsxCzAJBgNVBAYTAlVTMRAwDgYDVQQIDAdGbG9yaWRhMRkwFwYDVQQKDBBD dWRhIFN5c3RlbXMgTExDMRgwFgYDVQQLDA9DdWRhIFN5c3RlbXMgQ0ExJTAjBgNVBAMMHEN1 ZGEgU3lzdGVtcyBMTEMgMjAxNyBJbnQgQ0EwggIiMA0GCSqGSIb3DQEBAQUAA4ICDwAwggIK AoICAQC1aJotNUI+W4jP7xQDO8L/b4XiF4Rss9O0B+3vMH7Njk85fZ052QhZpMVlpaaO+sCI KqG3oNEbuOHzJB/NDJFnqh7ijBwhdWutdsq23Ux6TvxgakyMPpT6TRNEJzcBVQA0kpby1DVD 0EKSK/FrWWBiFmSxg7qUfmIq/mMzgE6epHktyRM3OGq3dbRdOUgfumWrqHXOrdJz06xE9NzY vc9toqZnd79FUtE/nSZVm1VS3Grq7RKV65onvX3QOW4W1ldEHwggaZxgWGNiR/D4eosAGFxn uYeWlKEC70c99Mp1giWux+7ur6hc2E+AaTGh+fGeijO5q40OGd+dNMgK8Es0nDRw81lRcl24 SWUEky9y8DArgIFlRd6d3ZYwgc1DMTWkTavx3ZpASp5TWih6yI8ACwboTvlUYeooMsPtNa9E 6UQ1nt7VEi5syjxnDltbEFoLYcXBcqhRhFETJe9CdenItAHAtOya3w5+fmC2j/xJz29og1KH YqWHlo3Kswi9G77an+zh6nWkMuHs+03DU8DaOEWzZEav3lVD4u76bKRDTbhh0bMAk4eXriGL h4MUoX3Imfcr6JoyheVrAdHDL/BixbMH1UUspeRuqQMQ5b2T6pabXP0oOB4FqldWiDgJBGRd zWLgCYG8wPGJGYgHibl5rFiI5Ix3FQncipc6SdUzOQIDAQABo4IBCjCCAQYwHQYDVR0OBBYE FF3AXsKnjdPND5+bxVECGKtc047PMIHABgNVHSMEgbgwgbWAFBu1oRhUMNEzjODolDka5k4Q EDBioYGRpIGOMIGLMQswCQYDVQQGEwJVUzEQMA4GA1UECAwHRmxvcmlkYTESMBAGA1UEBwwJ TmljZXZpbGxlMRkwFwYDVQQKDBBDdWRhIFN5c3RlbXMgTExDMRgwFgYDVQQLDA9DdWRhIFN5 c3RlbXMgQ0ExITAfBgNVBAMMGEN1ZGEgU3lzdGVtcyBMTEMgMjAxNyBDQYIJAKxAy1WBo2kY MBIGA1UdEwEB/wQIMAYBAf8CAQAwDgYDVR0PAQH/BAQDAgGGMA0GCSqGSIb3DQEBCwUAA4IC AQCB5686UCBVIT52jO3sz9pKuhxuC2npi8ZvoBwt/IH9piPA15/CGF1XeXUdu2qmhOjHkVLN gO7XB1G8CuluxofOIUce0aZGyB+vZ1ylHXlMeB0R82f5dz3/T7RQso55Y2Vog2Zb7PYTC5B9 oNy3ylsnNLzanYlcW3AAfzZcbxYuAdnuq0Im3EpGm8DoItUcf1pDezugKm/yKtNtY6sDyENj tExZ377cYA3IdIwqn1Mh4OAT/Rmh8au2rZAo0+bMYBy9C11Ex0hQ8zWcvPZBDn4v4RtO8g+K uQZQcJnO09LJNtw94W3d2mj4a7XrsKMnZKvm6W9BJIQ4Nmht4wXAtPQ1xA+QpxPTmsGAU0Cv HmqVC7XC3qxFhaOrD2dsvOAK6Sn3MEpH/YrfYCX7a7cz5zW3DsJQ6o3pYfnnQz+hnwLlz4MK 17NIA0WOdAF9IbtQqarf44+PEyUbKtz1r0KGeGLs+VGdd2FLA0e7yuzxJDYcaBTVwqaHhU2/ Fna/jGU7BhrKHtJbb/XlLeFJ24yvuiYKpYWQSSyZu1R/gvZjHeGb344jGBsZdCDrdxtQQcVA 6OxsMAPSUPMrlg9LWELEEYnVulQJerWxpUecGH92O06wwmPgykkz//UmmgjVSh7ErNvL0lUY UMfunYVO/O5hwhW+P4gviCXzBFeTtDZH259O7TCCBxIwggT6oAMCAQICEgLG8yH4PQFdbd9x Ugmpzl1jXzANBgkqhkiG9w0BAQsFADB7MQswCQYDVQQGEwJVUzEQMA4GA1UECAwHRmxvcmlk YTEZMBcGA1UECgwQQ3VkYSBTeXN0ZW1zIExMQzEYMBYGA1UECwwPQ3VkYSBTeXN0ZW1zIENB MSUwIwYDVQQDDBxDdWRhIFN5c3RlbXMgTExDIDIwMTcgSW50IENBMB4XDTIyMDYyOTE2MTYz NloXDTI3MDYyODE2MTYzNlowOjELMAkGA1UEBhMCVVMxEjAQBgNVBAgMCVRlbm5lc3NlZTEX MBUGA1UEAwwOS2FybCBEZW5uaW5nZXIwggIiMA0GCSqGSIb3DQEBAQUAA4ICDwAwggIKAoIC AQC+HVSyxVtJhy3Ohs+PAGRuO//Dha9A16l5FPATr6wude9zjX5f2lrkRyU8vhCXTZW7WbvW ZKpcZ8r0dtZmiK9uF58Ec6hhvfkxJzbg96WHBw5Fumd5ahZzuCJDtCAWW8R7/KN+zwzQf1+B 3MVLmbaXAFBuKzySKhKMcHbK3/wjUYTgy+3UK6v2SBrowvkUBC+jxNg3Wy12GsTXcUS/8FYI XgVVPgfZZrbJJb5HWOQpvvhILpPCD3xsYJFNKEPltXKWHT7Qtc2HNqikgNwj8oqOb+PeZGMi WapsatKm8mxuOOGOEBhAoTVTwUHlMNTg6QUCJtuWFCK38qOCyk9Haj+86lUU8RG6FkRXWgMb NQm1mWREQhw3axgGLSntjjnznJr5vsvXSYR6c+XKLd5KQZcS6LL8FHYNjqVKHBYM+hDnrTZM qa20JLAF1YagutDiMRURU23iWS7bA9tMcXcqkclTSDtFtxahRifXRI7Epq2GSKuEXe/1Tfb5 CE8QsbCpGsfSwv2tZ/SpqVG08MdRiXxN5tmZiQWo15IyWoeKOXl/hKxA9KPuDHngXX022b1l y+5ZOZbxBAZZMod4y4b4FiRUhRI97r9lCxsP/EPHuuTIZ82BYhrhbtab8HuRo2ofne2TfAWY 2BlA7ExM8XShMd9bRPZrNTokPQPUCWCgCdIATQIDAQABo4IBzzCCAcswPAYIKwYBBQUHAQEE MDAuMCwGCCsGAQUFBzABhiBodHRwOi8vb2NzcC5jdWRhc3lzdGVtcy5uZXQ6ODg4ODAJBgNV HRMEAjAAMBEGCWCGSAGG+EIBAQQEAwIFoDAOBgNVHQ8BAf8EBAMCBeAwHQYDVR0lBBYwFAYI KwYBBQUHAwIGCCsGAQUFBwMEMDMGCWCGSAGG+EIBDQQmFiRPcGVuU1NMIEdlbmVyYXRlZCBD bGllbnQgQ2VydGlmaWNhdGUwHQYDVR0OBBYEFLElmNWeVgsBPe7O8NiBzjvjYnpRMIHKBgNV HSMEgcIwgb+AFF3AXsKnjdPND5+bxVECGKtc047PoYGRpIGOMIGLMQswCQYDVQQGEwJVUzEQ MA4GA1UECAwHRmxvcmlkYTESMBAGA1UEBwwJTmljZXZpbGxlMRkwFwYDVQQKDBBDdWRhIFN5 c3RlbXMgTExDMRgwFgYDVQQLDA9DdWRhIFN5c3RlbXMgQ0ExITAfBgNVBAMMGEN1ZGEgU3lz dGVtcyBMTEMgMjAxNyBDQYITAORIioIQzl6738WMYyE12A3YSDAdBgNVHREEFjAUgRJrYXJs QGRlbm5pbmdlci5uZXQwDQYJKoZIhvcNAQELBQADggIBAKquc7cu0xc8FNtAQauZvocDzWQj 7HG9YvMbWnMi+ckhiA3rdW5NwWg0HBhBho1YlnqV+ntCVE2L8ezohHWm+KAdfXgpraL86Vsn 3ywNlZu/3COMpo2ALuHln8YQtH3Y8ebvzKMdlf2b5WB+7mOFIxXIr4AnNOLKCkq5ZhzC6JW6 Jvw3P0csiGa3UrfatYID5NvPgkaQvEgimEjG3psZqwQTL2Wxohvw783PrDt3wS0XeNhvQ61g 3QJFZKuv+bmGH3YBSPo1t6NUGAr+JozX5lDihB8JGkBt/NwdYec49a08uL0BbPaAJ7NjuIPG 7Y0Ak7PXZT37yx/Zla9PzLMJFgbelOkaatdzbblMZPDEVZ27l4lGMmV83Lm3YP17sdAyS/Wp mav7WmJUkQ9iuIKzSpdc82i9Mfujl1vbBtwtkHNPPtKuulIFM4ZwrPKjlVdLqTSqD8m9yHEi Y0PuAooq63OpJWF6hvMaiIPBWEAVIaDW9uG0MshLl9DnHnMyrJTfuC33Z9mOGMz7dRBjJd5Y W02xAzYnUuEBOpj+LQv5R8XIFMHFXktqEKvQrXeM2RU+PcZqKOBkTktxBLn3NI5VfA15Jk0c 5V5XcOqo3p2hvrwvXrinrb2pEREnoqmfrkXT3zOq5Y6ryRH8u734lGEF0dILXzoV4PM7XFit oTePoEjmMYIFBDCCBQACAQEwgZEwezELMAkGA1UEBhMCVVMxEDAOBgNVBAgMB0Zsb3JpZGEx GTAXBgNVBAoMEEN1ZGEgU3lzdGVtcyBMTEMxGDAWBgNVBAsMD0N1ZGEgU3lzdGVtcyBDQTEl MCMGA1UEAwwcQ3VkYSBTeXN0ZW1zIExMQyAyMDE3IEludCBDQQISAsbzIfg9AV1t33FSCanO XWNfMA0GCWCGSAFlAwQCAwUAoIICQzAYBgkqhkiG9w0BCQMxCwYJKoZIhvcNAQcBMBwGCSqG SIb3DQEJBTEPFw0yNDAzMTUxOTA5NTZaME8GCSqGSIb3DQEJBDFCBECcoDVxDdG1qdJQ2QPn WfT4VRKtmvsrM2WMR27FxNVumIE1lAcaPiy98yixpFZPEHzPSrYvsMNSmsbhHAg22YrDMGwG CSqGSIb3DQEJDzFfMF0wCwYJYIZIAWUDBAEqMAsGCWCGSAFlAwQBAjAKBggqhkiG9w0DBzAO BggqhkiG9w0DAgICAIAwDQYIKoZIhvcNAwICAUAwBwYFKw4DAgcwDQYIKoZIhvcNAwICASgw gaIGCSsGAQQBgjcQBDGBlDCBkTB7MQswCQYDVQQGEwJVUzEQMA4GA1UECAwHRmxvcmlkYTEZ MBcGA1UECgwQQ3VkYSBTeXN0ZW1zIExMQzEYMBYGA1UECwwPQ3VkYSBTeXN0ZW1zIENBMSUw IwYDVQQDDBxDdWRhIFN5c3RlbXMgTExDIDIwMTcgSW50IENBAhICxvMh+D0BXW3fcVIJqc5d Y18wgaQGCyqGSIb3DQEJEAILMYGUoIGRMHsxCzAJBgNVBAYTAlVTMRAwDgYDVQQIDAdGbG9y aWRhMRkwFwYDVQQKDBBDdWRhIFN5c3RlbXMgTExDMRgwFgYDVQQLDA9DdWRhIFN5c3RlbXMg Q0ExJTAjBgNVBAMMHEN1ZGEgU3lzdGVtcyBMTEMgMjAxNyBJbnQgQ0ECEgLG8yH4PQFdbd9x Ugmpzl1jXzANBgkqhkiG9w0BAQEFAASCAgCDKQx+d8w+va/Rs6Atc34Y4yaQmEPBrQ1+H/Yx 227DMicJ47AF+VNCb8LqxyABhYkjv45qHyyydLYLDoN45svC/TBVl07lx7oRuPkiZM/CMTZM XOsLr9pv72sVLcnpGVhWs2G9GK/yZVqV3GTpNtWH3ilxCNTRe5Tty2jsuf1RczMNvrOxKAvN B/upVicr6thgea2D5YRa5cH5V2bqN5Q3H6KgQwvxbbyOQHBYTH4HEfXAV2XpOgHrH1kITDV5 yCOsJtYQCKHBECuzFb3IIHpeHLirDX8Cptm0uKLIDVb4LaVaf0PQSSOAS5MF9U3QLV76nrW6 I3TzrYm1ZjIkW/VrLbm+3iC2VeoZfgwo+S2hpfyIp3HlgP8dLl5mp+KJBUxgxnVhK6pAspZd 3uV0yAH4ZEB3nzgFvl8X61HUqaZtKeSubt8T+Ai675qIRXQFgqB4OSGTI76nDOW4GmCC44TW DOkeTgGEKCmoUnKIJe+H6FplQ6KTnHPGQ10cbzR2S3qOIsMSFvF+Bl8q3zGzTxJgziLvBSEg tQuy1F+Vz/ZFso6lLDqqqasDZID26Fi8kzXlKKs+cRWAMtqkB2nldxfe3VezK90vQCcqG3fy wGoNsbcODVAxnfrojh2FXM8HYosVYYuCjQzGA3RwM7nuYKZSF+iRQh9Q+1Hh2ijShlI9/gAA AAAAAA== --------------ms020200020503040109090109-- From nobody Fri Mar 15 19:20:01 2024 X-Original-To: freebsd-hardware@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4TxDbX24k8z5Dyc5 for ; Fri, 15 Mar 2024 19:20:08 +0000 (UTC) (envelope-from mike@sentex.net) Received: from smarthost1.sentex.ca (smarthost1.sentex.ca [IPv6:2607:f3e0:0:1::12]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smarthost1.sentex.ca", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4TxDbX1Y9Pz42KT for ; Fri, 15 Mar 2024 19:20:08 +0000 (UTC) (envelope-from mike@sentex.net) Authentication-Results: mx1.freebsd.org; none Received: from pyroxene2a.sentex.ca (pyroxene19.sentex.ca [199.212.134.19]) by smarthost1.sentex.ca (8.17.1/8.16.1) with ESMTPS id 42FJK1Ef082053 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=FAIL); Fri, 15 Mar 2024 15:20:01 -0400 (EDT) (envelope-from mike@sentex.net) Received: from [IPV6:2607:f3e0:0:4:c034:f6ba:23ac:a443] ([IPv6:2607:f3e0:0:4:c034:f6ba:23ac:a443]) by pyroxene2a.sentex.ca (8.17.1/8.15.2) with ESMTPS id 42FJK0YC075882 (version=TLSv1.3 cipher=TLS_AES_128_GCM_SHA256 bits=128 verify=NO); Fri, 15 Mar 2024 15:20:00 -0400 (EDT) (envelope-from mike@sentex.net) Message-ID: <9880e18c-a003-487e-84da-c5af9fb0e485@sentex.net> Date: Fri, 15 Mar 2024 15:20:01 -0400 List-Id: General discussion of FreeBSD hardware List-Archive: https://lists.freebsd.org/archives/freebsd-hardware List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hardware@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: WD Blue 510 SSD and strange write performance Content-Language: en-US To: Karl Denninger , freebsd-hardware@freebsd.org References: <6504bd49-eca5-4e0a-b2bd-23d29405bb7a@sentex.net> <4832DE6A-5C82-4805-99BB-220D4342AE0F@fjl.co.uk> <69e47494-01aa-4149-a326-91d82dfdc46e@sentex.net> <0b98d5fc-0bfb-4d13-8e14-79cb3ee873c0@denninger.net> From: mike tancsa Autocrypt: addr=mike@sentex.net; keydata= xsBNBFywzOMBCACoNFpwi5MeyEREiCeHtbm6pZJI/HnO+wXdCAWtZkS49weOoVyUj5BEXRZP xflV2ib2hflX4nXqhenaNiia4iaZ9ft3I1ebd7GEbGnsWCvAnob5MvDZyStDAuRxPJK1ya/s +6rOvr+eQiXYNVvfBhrCfrtR/esSkitBGxhUkBjOti8QwzD71JVF5YaOjBAs7jZUKyLGj0kW yDg4jUndudWU7G2yc9GwpHJ9aRSUN8e/mWdIogK0v+QBHfv/dsI6zVB7YuxCC9Fx8WPwfhDH VZC4kdYCQWKXrm7yb4TiVdBh5kgvlO9q3js1yYdfR1x8mjK2bH2RSv4bV3zkNmsDCIxjABEB AAHNHW1pa2UgdGFuY3NhIDxtaWtlQHNlbnRleC5uZXQ+wsCOBBMBCAA4FiEEmuvCXT0aY6hs 4SbWeVOEFl5WrMgFAl+pQfkCGwMFCwkIBwIGFQoJCAsCBBYCAwECHgECF4AACgkQeVOEFl5W rMiN6ggAk3H5vk8QnbvGbb4sinxZt/wDetgk0AOR9NRmtTnPaW+sIJEfGBOz47Xih+f7uWJS j+uvc9Ewn2Z7n8z3ZHJlLAByLVLtcNXGoRIGJ27tevfOaNqgJHBPbFOcXCBBFTx4MYMM4iAZ cDT5vsBTSaM36JZFtHZBKkuFEItbA/N8ZQSHKdTYMIA7A3OCLGbJBqloQ8SlW4MkTzKX4u7R yefAYQ0h20x9IqC5Ju8IsYRFacVZconT16KS81IBceO42vXTN0VexbVF2rZIx3v/NT75r6Vw 0FlXVB1lXOHKydRA2NeleS4NEG2vWqy/9Boj0itMfNDlOhkrA/0DcCurMpnpbM7ATQRcsMzk AQgA1Dpo/xWS66MaOJLwA28sKNMwkEk1Yjs+okOXDOu1F+0qvgE8sVmrOOPvvWr4axtKRSG1 t2QUiZ/ZkW/x/+t0nrM39EANV1VncuQZ1ceIiwTJFqGZQ8kb0+BNkwuNVFHRgXm1qzAJweEt RdsCMohB+H7BL5LGCVG5JaU0lqFU9pFP40HxEbyzxjsZgSE8LwkI6wcu0BLv6K6cLm0EiHPO l5G8kgRi38PS7/6s3R8QDsEtbGsYy6O82k3zSLIjuDBwA9GRaeigGppTxzAHVjf5o9KKu4O7 gC2KKVHPegbXS+GK7DU0fjzX57H5bZ6komE5eY4p3oWT/CwVPSGfPs8jOwARAQABwsB2BBgB CAAgFiEEmuvCXT0aY6hs4SbWeVOEFl5WrMgFAl+pQfkCGwwACgkQeVOEFl5WrMiVqwf9GwU8 c6cylknZX8QwlsVudTC8xr/L17JA84wf03k3d4wxP7bqy5AYy7jboZMbgWXngAE/HPQU95NM aukysSnknzoIpC96XZJ0okLBXVS6Y0ylZQ+HrbIhMpuQPoDweoF5F9wKrsHRoDaUK1VR706X rwm4HUzh7Jk+auuMYfuCh0FVlFBEuiJWMLhg/5WCmcRfiuB6F59ZcUQrwLEZeNhF2XJV4KwB Tlg7HCWO/sy1foE5noaMyACjAtAQE9p5kGYaj+DuRhPdWUTsHNuqrhikzIZd2rrcMid+ktb0 NvtvswzMO059z1YGMtGSqQ4srCArju+XHIdTFdiIYbd7+jeehg== In-Reply-To: <0b98d5fc-0bfb-4d13-8e14-79cb3ee873c0@denninger.net> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-Scanned-By: MIMEDefang 2.86 on 64.7.153.18 X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:11647, ipnet:2607:f3e0::/32, country:CA] X-Rspamd-Queue-Id: 4TxDbX1Y9Pz42KT On 3/15/2024 3:09 PM, Karl Denninger wrote: > I've got both the Kingston and Micron versions of these in production > use and have seen nothing like this at all; they get hit pretty hard > too, including release building (both direct and cross-builds) and > similar stuff. > What firmware are your devices running ? There are 2 out there I know of.  One one server I have running in prod from last November, it seems fine, but the firmware is older. === START OF INFORMATION SECTION === Device Model:     WD Blue SA510 2.5 1000GB Serial Number:    23020L807259 LU WWN Device Id: 5 001b44 8b2463dcb Firmware Version: 52020100 Device Model:     WD Blue SA510 2.5 1000GB Serial Number:    23261Y804499 LU WWN Device Id: 5 001b44 8bf6bb6c1 Firmware Version: 52046100 From nobody Sun Mar 17 08:32:20 2024 X-Original-To: freebsd-hardware@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4TyB7L70zTz4qLhb for ; Sun, 17 Mar 2024 08:32:30 +0000 (UTC) (envelope-from ml@netfence.it) Received: from soth.netfence.it (mailserver.netfence.it [78.134.96.152]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "mailserver.netfence.it", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4TyB7L07vmz46NR for ; Sun, 17 Mar 2024 08:32:29 +0000 (UTC) (envelope-from ml@netfence.it) Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=pass (policy=none) header.from=netfence.it; spf=pass (mx1.freebsd.org: domain of ml@netfence.it designates 78.134.96.152 as permitted sender) smtp.mailfrom=ml@netfence.it Received: from [10.1.2.18] (alamar.local.netfence.it [10.1.2.18]) (authenticated bits=0) by soth.netfence.it (8.17.2/8.17.2) with ESMTPSA id 42H8WKrl065585 (version=TLSv1.3 cipher=TLS_AES_128_GCM_SHA256 bits=128 verify=NO); Sun, 17 Mar 2024 09:32:20 +0100 (CET) (envelope-from ml@netfence.it) X-Authentication-Warning: soth.netfence.it: Host alamar.local.netfence.it [10.1.2.18] claimed to be [10.1.2.18] Message-ID: <27933f54-2959-4071-b084-d796a7c3ae75@netfence.it> Date: Sun, 17 Mar 2024 09:32:20 +0100 List-Id: General discussion of FreeBSD hardware List-Archive: https://lists.freebsd.org/archives/freebsd-hardware List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hardware@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: WD Blue 510 SSD and strange write performance Content-Language: en-US To: mike tancsa References: <6504bd49-eca5-4e0a-b2bd-23d29405bb7a@sentex.net> <4832DE6A-5C82-4805-99BB-220D4342AE0F@fjl.co.uk> <69e47494-01aa-4149-a326-91d82dfdc46e@sentex.net> From: Andrea Venturoli Cc: freebsd-hardware@freebsd.org In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Spamd-Bar: --- X-Spamd-Result: default: False [-3.75 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.96)[-0.959]; DMARC_POLICY_ALLOW(-0.50)[netfence.it,none]; R_SPF_ALLOW(-0.20)[+ip4:78.134.96.152]; MIME_GOOD(-0.10)[text/plain]; XM_UA_NO_VERSION(0.01)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; ASN(0.00)[asn:35612, ipnet:78.134.0.0/17, country:IT]; RCVD_COUNT_ONE(0.00)[1]; ARC_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; RCPT_COUNT_TWO(0.00)[2]; R_DKIM_NA(0.00)[]; TO_DN_SOME(0.00)[]; FROM_HAS_DN(0.00)[]; HAS_XAW(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; MLMMJ_DEST(0.00)[freebsd-hardware@freebsd.org]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_TLS_ALL(0.00)[] X-Rspamd-Queue-Id: 4TyB7L07vmz46NR On 3/15/24 19:17, mike tancsa wrote: > (da5:mpr0:0:15:0): SCSI sense: UNIT ATTENTION asc:29,0 (Power on, reset, > or bus device reset occurred) Hello. I know I'm probably blaming the wrong component, but is your PSU up to the task? How many drives do you have? Are they power-hungrier than the others you tried (Samsung ???)? Do you have a spare PSU to test/add? Probably this is not the cause... still, before you bit farewell to 400 bucks... bye av. From nobody Sun Mar 17 12:03:44 2024 X-Original-To: freebsd-hardware@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4TyGqD0Vwyz5D024 for ; Sun, 17 Mar 2024 12:03:52 +0000 (UTC) (envelope-from mike@sentex.net) Received: from smarthost1.sentex.ca (smarthost1.sentex.ca [IPv6:2607:f3e0:0:1::12]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smarthost1.sentex.ca", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4TyGqC3CgJz4WfX; Sun, 17 Mar 2024 12:03:51 +0000 (UTC) (envelope-from mike@sentex.net) Authentication-Results: mx1.freebsd.org; none Received: from pyroxene2a.sentex.ca (pyroxene19.sentex.ca [199.212.134.19]) by smarthost1.sentex.ca (8.17.1/8.16.1) with ESMTPS id 42HC3jQs067359 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=FAIL); Sun, 17 Mar 2024 08:03:45 -0400 (EDT) (envelope-from mike@sentex.net) Received: from [IPV6:2607:f3e0:0:4:838:bb8d:b41c:78ce] ([IPv6:2607:f3e0:0:4:838:bb8d:b41c:78ce]) by pyroxene2a.sentex.ca (8.17.1/8.15.2) with ESMTPS id 42HC3hIU004945 (version=TLSv1.3 cipher=TLS_AES_128_GCM_SHA256 bits=128 verify=NO); Sun, 17 Mar 2024 08:03:44 -0400 (EDT) (envelope-from mike@sentex.net) Content-Type: multipart/alternative; boundary="------------CnN0rO33ICebgnjizmMEUWHh" Message-ID: <00cf68fe-73e2-4d28-bb49-6aad7eeaf884@sentex.net> Date: Sun, 17 Mar 2024 08:03:44 -0400 List-Id: General discussion of FreeBSD hardware List-Archive: https://lists.freebsd.org/archives/freebsd-hardware List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hardware@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: WD Blue 510 SSD and strange write performance Content-Language: en-US To: Andrea Venturoli Cc: freebsd-hardware@freebsd.org References: <6504bd49-eca5-4e0a-b2bd-23d29405bb7a@sentex.net> <4832DE6A-5C82-4805-99BB-220D4342AE0F@fjl.co.uk> <69e47494-01aa-4149-a326-91d82dfdc46e@sentex.net> <27933f54-2959-4071-b084-d796a7c3ae75@netfence.it> From: mike tancsa Autocrypt: addr=mike@sentex.net; keydata= xsBNBFywzOMBCACoNFpwi5MeyEREiCeHtbm6pZJI/HnO+wXdCAWtZkS49weOoVyUj5BEXRZP xflV2ib2hflX4nXqhenaNiia4iaZ9ft3I1ebd7GEbGnsWCvAnob5MvDZyStDAuRxPJK1ya/s +6rOvr+eQiXYNVvfBhrCfrtR/esSkitBGxhUkBjOti8QwzD71JVF5YaOjBAs7jZUKyLGj0kW yDg4jUndudWU7G2yc9GwpHJ9aRSUN8e/mWdIogK0v+QBHfv/dsI6zVB7YuxCC9Fx8WPwfhDH VZC4kdYCQWKXrm7yb4TiVdBh5kgvlO9q3js1yYdfR1x8mjK2bH2RSv4bV3zkNmsDCIxjABEB AAHNHW1pa2UgdGFuY3NhIDxtaWtlQHNlbnRleC5uZXQ+wsCOBBMBCAA4FiEEmuvCXT0aY6hs 4SbWeVOEFl5WrMgFAl+pQfkCGwMFCwkIBwIGFQoJCAsCBBYCAwECHgECF4AACgkQeVOEFl5W rMiN6ggAk3H5vk8QnbvGbb4sinxZt/wDetgk0AOR9NRmtTnPaW+sIJEfGBOz47Xih+f7uWJS j+uvc9Ewn2Z7n8z3ZHJlLAByLVLtcNXGoRIGJ27tevfOaNqgJHBPbFOcXCBBFTx4MYMM4iAZ cDT5vsBTSaM36JZFtHZBKkuFEItbA/N8ZQSHKdTYMIA7A3OCLGbJBqloQ8SlW4MkTzKX4u7R yefAYQ0h20x9IqC5Ju8IsYRFacVZconT16KS81IBceO42vXTN0VexbVF2rZIx3v/NT75r6Vw 0FlXVB1lXOHKydRA2NeleS4NEG2vWqy/9Boj0itMfNDlOhkrA/0DcCurMpnpbM7ATQRcsMzk AQgA1Dpo/xWS66MaOJLwA28sKNMwkEk1Yjs+okOXDOu1F+0qvgE8sVmrOOPvvWr4axtKRSG1 t2QUiZ/ZkW/x/+t0nrM39EANV1VncuQZ1ceIiwTJFqGZQ8kb0+BNkwuNVFHRgXm1qzAJweEt RdsCMohB+H7BL5LGCVG5JaU0lqFU9pFP40HxEbyzxjsZgSE8LwkI6wcu0BLv6K6cLm0EiHPO l5G8kgRi38PS7/6s3R8QDsEtbGsYy6O82k3zSLIjuDBwA9GRaeigGppTxzAHVjf5o9KKu4O7 gC2KKVHPegbXS+GK7DU0fjzX57H5bZ6komE5eY4p3oWT/CwVPSGfPs8jOwARAQABwsB2BBgB CAAgFiEEmuvCXT0aY6hs4SbWeVOEFl5WrMgFAl+pQfkCGwwACgkQeVOEFl5WrMiVqwf9GwU8 c6cylknZX8QwlsVudTC8xr/L17JA84wf03k3d4wxP7bqy5AYy7jboZMbgWXngAE/HPQU95NM aukysSnknzoIpC96XZJ0okLBXVS6Y0ylZQ+HrbIhMpuQPoDweoF5F9wKrsHRoDaUK1VR706X rwm4HUzh7Jk+auuMYfuCh0FVlFBEuiJWMLhg/5WCmcRfiuB6F59ZcUQrwLEZeNhF2XJV4KwB Tlg7HCWO/sy1foE5noaMyACjAtAQE9p5kGYaj+DuRhPdWUTsHNuqrhikzIZd2rrcMid+ktb0 NvtvswzMO059z1YGMtGSqQ4srCArju+XHIdTFdiIYbd7+jeehg== In-Reply-To: <27933f54-2959-4071-b084-d796a7c3ae75@netfence.it> X-Scanned-By: MIMEDefang 2.86 on 64.7.153.18 X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:11647, ipnet:2607:f3e0::/32, country:CA] X-Rspamd-Queue-Id: 4TyGqC3CgJz4WfX This is a multi-part message in MIME format. --------------CnN0rO33ICebgnjizmMEUWHh Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit On 3/17/2024 4:32 AM, Andrea Venturoli wrote: > On 3/15/24 19:17, mike tancsa wrote: > >> (da5:mpr0:0:15:0): SCSI sense: UNIT ATTENTION asc:29,0 (Power on, >> reset, or bus device reset occurred) > > Hello. > I know I'm probably blaming the wrong component, but is your PSU up to > the task? > How many drives do you have? Are they power-hungrier than the others > you tried (Samsung ???)? > Do you have a spare PSU to test/add? > > Probably this is not the cause... still, before you bit farewell to > 400 bucks... > hehe, thanks Andrea :)  I too dont want to be out the money. Power supply for sure is a good thing to check. In this case, the main server chassis is sized with a couple of redundant 1000W power supplies that should handle 12 full HDDs. Pretty sure in this case 6 SSDs should not stress it beyond the point. But I had 2 other test boxes on the bench and the one common variable seems to be the WDs. I feel like this is a sunk cost I am pushing myself into, but I did do some more testing.  My co-worker came across this post which was interesting. https://forum.hddguru.com/viewtopic.php?f=10&t=43284 The very last entry says "For WD BLUE SA 510 there are some problems with this type of SSD. This YODA model To fix the SSD if it is still recognized, use the firmware update tools. And then do a secure erase or full wipe of the SSD. After this it will work well. I can give you a link to this utility if it necessary. Also ossible download it from manufacture FTP. If it is not recognized by the computer or is identified as a SSD device, there only one way, use production tools with new firmware to begin the production process by testing the controller and NAND chip and forming a translator. The SSD will be like brand new. " After I did the erase, the tests worked for a good 5 cycles and performance was MUCH smoother and consistent. But then the drives started to fail again.  So I really wonder if TRIM has something to do with it as my test is essentially writing a 250G data set with about 28 million txt files, destroying the dataset and then copying it again. I noticed these 2 commits for other drives. I wonder if the WD is having similar issues. https://cgit.freebsd.org/src/commit/?h=stable/14&id=bf11fee6a5cf97102f87695185cadb63d5a2a7de and https://cgit.freebsd.org/src/commit/?h=stable/14&id=50aa22323424ccea00ef5d8f24e729a480cc77eb I hope you dont mind bcc'ing you Andriy.  I noticed you only added the NCQ quirks for CAM ata and not for CAM scsi. I am running into odd issues with some WD drives and wondering if there is the same root limitation of these WD SA 510 drives like the Samsungs ? However, in my use of the Samsungs I have not been able to trigger these bugs so far.     ---Mike --------------CnN0rO33ICebgnjizmMEUWHh Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: 8bit
On 3/17/2024 4:32 AM, Andrea Venturoli wrote:
On 3/15/24 19:17, mike tancsa wrote:

(da5:mpr0:0:15:0): SCSI sense: UNIT ATTENTION asc:29,0 (Power on, reset, or bus device reset occurred)

Hello.
I know I'm probably blaming the wrong component, but is your PSU up to the task?
How many drives do you have? Are they power-hungrier than the others you tried (Samsung ???)?
Do you have a spare PSU to test/add?

Probably this is not the cause... still, before you bit farewell to 400 bucks...


hehe, thanks Andrea :)  I too dont want to be out the money. Power supply for sure is a good thing to check. In this case, the main server chassis is sized with a couple of redundant 1000W power supplies that should handle 12 full HDDs. Pretty sure in this case 6 SSDs should not stress it beyond the point. But I had 2 other test boxes on the bench and the one common variable seems to be the WDs. 

I feel like this is a sunk cost I am pushing myself into, but I did do some more testing.  My co-worker came across this post which was interesting.

https://forum.hddguru.com/viewtopic.php?f=10&t=43284

The very last entry says

"For WD BLUE SA 510 there are some problems with this type of SSD. This YODA model
To fix the SSD if it is still recognized, use the firmware update tools.
And then do a secure erase or full wipe of the SSD. After this it will work well. I can give you a link to this utility if it necessary. Also ossible download it from manufacture FTP.
If it is not recognized by the computer or is identified as a SSD device, there only one way, use production tools with new firmware to begin the production process by testing the controller and NAND chip and forming a translator. The SSD will be like brand new.
"

After I did the erase, the tests worked for a good 5 cycles and performance was MUCH smoother and consistent. But then the drives started to fail again.  So I really wonder if TRIM has something to do with it as my test is essentially writing a 250G data set with about 28 million txt files, destroying the dataset and then copying it again.

I noticed these 2 commits for other drives. I wonder if the WD is having similar issues. 

https://cgit.freebsd.org/src/commit/?h=stable/14&id=bf11fee6a5cf97102f87695185cadb63d5a2a7de
and
https://cgit.freebsd.org/src/commit/?h=stable/14&id=50aa22323424ccea00ef5d8f24e729a480cc77eb

I hope you dont mind bcc'ing you Andriy.  I noticed you only added the NCQ quirks for CAM ata and not for CAM scsi. I am running into odd issues with some WD drives and wondering if there is the same root limitation of these WD SA 510 drives like the Samsungs ? However, in my use of the Samsungs I have not been able to trigger these bugs so far.

    ---Mike

--------------CnN0rO33ICebgnjizmMEUWHh--