From nobody Thu Mar 14 20:26:15 2024 X-Original-To: freebsd-hardware@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Twf6Q1lX1z5DlHp for ; Thu, 14 Mar 2024 20:26:22 +0000 (UTC) (envelope-from mike@sentex.net) Received: from smarthost1.sentex.ca (smarthost1.sentex.ca [IPv6:2607:f3e0:0:1::12]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smarthost1.sentex.ca", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Twf6M5lkpz4fXj for ; Thu, 14 Mar 2024 20:26:19 +0000 (UTC) (envelope-from mike@sentex.net) Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of mike@sentex.net designates 2607:f3e0:0:1::12 as permitted sender) smtp.mailfrom=mike@sentex.net Received: from pyroxene2a.sentex.ca (pyroxene19.sentex.ca [199.212.134.19]) by smarthost1.sentex.ca (8.17.1/8.16.1) with ESMTPS id 42EKQFBC015677 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=FAIL); Thu, 14 Mar 2024 16:26:15 -0400 (EDT) (envelope-from mike@sentex.net) Received: from [IPV6:2607:f3e0:0:4:a1d0:ee73:adac:870f] ([IPv6:2607:f3e0:0:4:a1d0:ee73:adac:870f]) by pyroxene2a.sentex.ca (8.17.1/8.15.2) with ESMTPS id 42EKQEoH048748 (version=TLSv1.3 cipher=TLS_AES_128_GCM_SHA256 bits=128 verify=NO); Thu, 14 Mar 2024 16:26:14 -0400 (EDT) (envelope-from mike@sentex.net) Message-ID: <8f7ec337-16ab-4dd3-a291-e6830f31c51e@sentex.net> Date: Thu, 14 Mar 2024 16:26:15 -0400 List-Id: General discussion of FreeBSD hardware List-Archive: https://lists.freebsd.org/archives/freebsd-hardware List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hardware@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: WD Blue 510 SSD and strange write performance Content-Language: en-US From: mike tancsa To: Frank Leonhardt , freebsd-hardware@freebsd.org, Bob Bishop References: <6504bd49-eca5-4e0a-b2bd-23d29405bb7a@sentex.net> <4832DE6A-5C82-4805-99BB-220D4342AE0F@fjl.co.uk> <69e47494-01aa-4149-a326-91d82dfdc46e@sentex.net> Autocrypt: addr=mike@sentex.net; keydata= xsBNBFywzOMBCACoNFpwi5MeyEREiCeHtbm6pZJI/HnO+wXdCAWtZkS49weOoVyUj5BEXRZP xflV2ib2hflX4nXqhenaNiia4iaZ9ft3I1ebd7GEbGnsWCvAnob5MvDZyStDAuRxPJK1ya/s +6rOvr+eQiXYNVvfBhrCfrtR/esSkitBGxhUkBjOti8QwzD71JVF5YaOjBAs7jZUKyLGj0kW yDg4jUndudWU7G2yc9GwpHJ9aRSUN8e/mWdIogK0v+QBHfv/dsI6zVB7YuxCC9Fx8WPwfhDH VZC4kdYCQWKXrm7yb4TiVdBh5kgvlO9q3js1yYdfR1x8mjK2bH2RSv4bV3zkNmsDCIxjABEB AAHNHW1pa2UgdGFuY3NhIDxtaWtlQHNlbnRleC5uZXQ+wsCOBBMBCAA4FiEEmuvCXT0aY6hs 4SbWeVOEFl5WrMgFAl+pQfkCGwMFCwkIBwIGFQoJCAsCBBYCAwECHgECF4AACgkQeVOEFl5W rMiN6ggAk3H5vk8QnbvGbb4sinxZt/wDetgk0AOR9NRmtTnPaW+sIJEfGBOz47Xih+f7uWJS j+uvc9Ewn2Z7n8z3ZHJlLAByLVLtcNXGoRIGJ27tevfOaNqgJHBPbFOcXCBBFTx4MYMM4iAZ cDT5vsBTSaM36JZFtHZBKkuFEItbA/N8ZQSHKdTYMIA7A3OCLGbJBqloQ8SlW4MkTzKX4u7R yefAYQ0h20x9IqC5Ju8IsYRFacVZconT16KS81IBceO42vXTN0VexbVF2rZIx3v/NT75r6Vw 0FlXVB1lXOHKydRA2NeleS4NEG2vWqy/9Boj0itMfNDlOhkrA/0DcCurMpnpbM7ATQRcsMzk AQgA1Dpo/xWS66MaOJLwA28sKNMwkEk1Yjs+okOXDOu1F+0qvgE8sVmrOOPvvWr4axtKRSG1 t2QUiZ/ZkW/x/+t0nrM39EANV1VncuQZ1ceIiwTJFqGZQ8kb0+BNkwuNVFHRgXm1qzAJweEt RdsCMohB+H7BL5LGCVG5JaU0lqFU9pFP40HxEbyzxjsZgSE8LwkI6wcu0BLv6K6cLm0EiHPO l5G8kgRi38PS7/6s3R8QDsEtbGsYy6O82k3zSLIjuDBwA9GRaeigGppTxzAHVjf5o9KKu4O7 gC2KKVHPegbXS+GK7DU0fjzX57H5bZ6komE5eY4p3oWT/CwVPSGfPs8jOwARAQABwsB2BBgB CAAgFiEEmuvCXT0aY6hs4SbWeVOEFl5WrMgFAl+pQfkCGwwACgkQeVOEFl5WrMiVqwf9GwU8 c6cylknZX8QwlsVudTC8xr/L17JA84wf03k3d4wxP7bqy5AYy7jboZMbgWXngAE/HPQU95NM aukysSnknzoIpC96XZJ0okLBXVS6Y0ylZQ+HrbIhMpuQPoDweoF5F9wKrsHRoDaUK1VR706X rwm4HUzh7Jk+auuMYfuCh0FVlFBEuiJWMLhg/5WCmcRfiuB6F59ZcUQrwLEZeNhF2XJV4KwB Tlg7HCWO/sy1foE5noaMyACjAtAQE9p5kGYaj+DuRhPdWUTsHNuqrhikzIZd2rrcMid+ktb0 NvtvswzMO059z1YGMtGSqQ4srCArju+XHIdTFdiIYbd7+jeehg== In-Reply-To: <69e47494-01aa-4149-a326-91d82dfdc46e@sentex.net> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-Scanned-By: MIMEDefang 2.86 on 64.7.153.18 X-Spamd-Bar: --- X-Spamd-Result: default: False [-3.38 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.99)[-0.994]; R_SPF_ALLOW(-0.20)[+ip6:2607:f3e0::/32]; RCVD_IN_DNSWL_LOW(-0.10)[199.212.134.19:received]; MIME_GOOD(-0.10)[text/plain]; XM_UA_NO_VERSION(0.01)[]; ARC_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:11647, ipnet:2607:f3e0::/32, country:CA]; TO_DN_SOME(0.00)[]; FREEFALL_USER(0.00)[mike]; RCVD_TLS_ALL(0.00)[]; MLMMJ_DEST(0.00)[freebsd-hardware@freebsd.org]; R_DKIM_NA(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; FROM_HAS_DN(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_COUNT_TWO(0.00)[2]; DMARC_NA(0.00)[sentex.net]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCPT_COUNT_THREE(0.00)[3] X-Rspamd-Queue-Id: 4Twf6M5lkpz4fXj On 3/14/2024 3:56 PM, mike tancsa wrote: > On 3/14/2024 3:48 PM, Frank Leonhardt wrote: >> "CAM status: SCSI Status Error" suggests to me that the drive was >> just too busy when asked. I'm not saying it's nothing to worry about, >> but neither am I saying it is. > > Given enough of them it does cause checksum errors on the test pool > unfortunately.  Could a buggy TRIM play a role here too ? I noticed a > commit the other day for a Segate SSD that had a broken NCQ TRIM. > Could these units suffer from that ? > https://cgit.freebsd.org/src/commit/?h=stable/14&id=47fff7407c22c2c4b36b4f9f27ddfa70bb8f3fee > > > Is there a way to turn that off via camcontrol ? Or perhaps instrument > some other settings ?  I am not wedded to this hardware, but it would > be good to know if they can be made workable without too much effort. > On another test box with an MPR controller and same WD drives, a few more messages after the zfs send is about 50% done on a dataset thats about 1TB but compressed to about 260G. Some 29 million files. But with Samsungs, reliably no issue :( (da6:mpr0:0:16:0): WRITE(10). CDB: 2a 00 48 1e 9b 90 00 00 80 00 mpr0: Controller reported scsi ioc terminated tgt 16 SMID 897 loginfo 31110f00 mpr0: Controller reported scsi ioc terminated tgt 16 SMID 1358 loginfo 31110f00 mpr0: Controller reported scsi ioc terminated tgt 16 SMID 1742 loginfo 31110f00 mpr0: Controller reported scsi ioc terminated tgt 16 SMID 1187 loginfo 31110f00 mpr0: Controller reported scsi ioc terminated tgt 16 SMID 1006 loginfo 31110f00 mpr0: Controller reported scsi ioc terminated tgt 16 SMID 758 loginfo 31110f00 (da6:mpr0:0:16:0): CAM status: CCB request completed with an error (da6:mpr0:0:16:0): Retrying command, 3 more tries remain (da6:mpr0:0:16:0): WRITE(10). CDB: 2a 00 48 1e 9c 10 00 00 b8 00 (da6:mpr0:0:16:0): CAM status: CCB request completed with an error (da6:mpr0:0:16:0): Retrying command, 3 more tries remain (da6:mpr0:0:16:0): WRITE(10). CDB: 2a 00 46 93 47 18 00 00 08 00 (da6:mpr0:0:16:0): CAM status: CCB request completed with an error (da6:mpr0:0:16:0): Retrying command, 3 more tries remain (da6:mpr0:0:16:0): READ(10). CDB: 28 00 1c c7 dc 40 00 01 00 00 (da6:mpr0:0:16:0): CAM status: CCB request completed with an error (da6:mpr0:0:16:0): Retrying command, 3 more tries remain (da6:mpr0:0:16:0): READ(10). CDB: 28 00 1c c7 d9 30 00 00 f8 00 (da6:mpr0:0:16:0): CAM status: CCB request completed with an error (da6:mpr0:0:16:0): Retrying command, 3 more tries remain (da6:mpr0:0:16:0): WRITE(10). CDB: 2a 00 46 93 47 10 00 00 08 00 (da6:mpr0:0:16:0): CAM status: CCB request completed with an error (da6:mpr0:0:16:0): Retrying command, 3 more tries remain (da6:mpr0:0:16:0): READ(10). CDB: 28 00 1c c7 d8 30 00 01 00 00 (da6:mpr0:0:16:0): CAM status: CCB request completed with an error (da6:mpr0:0:16:0): Retrying command, 3 more tries remain (da6:mpr0:0:16:0): WRITE(10). CDB: 2a 00 49 55 29 20 00 00 08 00 (da6:mpr0:0:16:0): CAM status: CCB request completed with an error (da6:mpr0:0:16:0): Retrying command, 3 more tries remain (da6:mpr0:0:16:0): WRITE(10). CDB: 2a 00 48 1e 9b 90 00 00 80 00 (da6:mpr0:0:16:0): CAM status: SCSI Status Error (da6:mpr0:0:16:0): SCSI status: Check Condition (da6:mpr0:0:16:0): SCSI sense: UNIT ATTENTION asc:29,0 (Power on, reset, or bus device reset occurred) (da6:mpr0:0:16:0): Retrying command (per sense data) mpr0: Controller reported scsi ioc terminated tgt 13 SMID 1023 loginfo 31110f00 mpr0: Controller reported scsi ioc terminated tgt 13 SMID 297 loginfo 31110f00 (da3:mpr0:0:13:0): WRITE(10). CDB: 2a 00 4c 49 50 18 00 00 a0 00 mpr0: Controller reported scsi ioc terminated tgt 13 SMID 1999 loginfo 31110f00 mpr0: Controller reported scsi ioc terminated tgt 13 SMID 280 loginfo 31110f00 mpr0: Controller reported scsi ioc terminated tgt 13 SMID 1970 loginfo 31110f00 mpr0: Controller reported scsi ioc terminated tgt 13 SMID 859 loginfo 31110f00 mpr0: Controller reported scsi ioc terminated tgt 13 SMID 1652 loginfo 31110f00 mpr0: Controller reported scsi ioc terminated tgt 13 SMID 613 loginfo 31110f00 (da3:mpr0:0:13:0): CAM status: CCB request completed with an error (da3:mpr0:0:13:0): Retrying command, 3 more tries remain (da3:mpr0:0:13:0): WRITE(10). CDB: 2a 00 4c 49 4e a8 00 01 00 00 (da3:mpr0:0:13:0): CAM status: CCB request completed with an error (da3:mpr0:0:13:0): Retrying command, 3 more tries remain (da3:mpr0:0:13:0): WRITE(10). CDB: 2a 00 4c 49 4f a8 00 00 70 00 (da3:mpr0:0:13:0): CAM status: CCB request completed with an error (da3:mpr0:0:13:0): Retrying command, 3 more tries remain (da3:mpr0:0:13:0): WRITE(10). CDB: 2a 00 4c bc 65 30 00 01 08 00 (da3:mpr0:0:13:0): CAM status: CCB request completed with an error (da3:mpr0:0:13:0): Retrying command, 3 more tries remain (da3:mpr0:0:13:0): READ(10). CDB: 28 00 2c 99 68 80 00 01 00 00 (da3:mpr0:0:13:0): CAM status: CCB request completed with an error (da3:mpr0:0:13:0): Retrying command, 3 more tries remain (da3:mpr0:0:13:0): WRITE(10). CDB: 2a 00 46 98 1e b8 00 00 18 00 (da3:mpr0:0:13:0): CAM status: CCB request completed with an error (da3:mpr0:0:13:0): Retrying command, 3 more tries remain (da3:mpr0:0:13:0): READ(10). CDB: 28 00 1e cd a8 28 00 00 18 00 (da3:mpr0:0:13:0): CAM status: CCB request completed with an error (da3:mpr0:0:13:0): Retrying command, 3 more tries remain (da3:mpr0:0:13:0): WRITE(10). CDB: 2a 00 4c bc 64 30 00 01 00 00 (da3:mpr0:0:13:0): CAM status: CCB request completed with an error (da3:mpr0:0:13:0): Retrying command, 3 more tries remain (da3:mpr0:0:13:0): WRITE(10). CDB: 2a 00 4c 49 4e a8 00 01 00 00 (da3:mpr0:0:13:0): CAM status: SCSI Status Error (da3:mpr0:0:13:0): SCSI status: Check Condition (da3:mpr0:0:13:0): SCSI sense: UNIT ATTENTION asc:29,0 (Power on, reset, or bus device reset occurred) (da3:mpr0:0:13:0): Retrying command (per sense data)