From nobody Thu Jul 13 19:24:39 2023 X-Original-To: freebsd-embedded@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4R24LW6H33z4mlqG for ; Thu, 13 Jul 2023 19:24:51 +0000 (UTC) (envelope-from karl@denninger.net) Received: from colo1.denninger.net (colo1.denninger.net [104.236.120.189]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4R24LT41Lfz3C81 for ; Thu, 13 Jul 2023 19:24:49 +0000 (UTC) (envelope-from karl@denninger.net) Authentication-Results: mx1.freebsd.org; dkim=none; spf=pass (mx1.freebsd.org: domain of karl@denninger.net designates 104.236.120.189 as permitted sender) smtp.mailfrom=karl@denninger.net; dmarc=pass (policy=none) header.from=denninger.net Received: from denninger.net (096-033-195-197.res.spectrum.com [96.33.195.197]) by colo1.denninger.net (Postfix) with ESMTP id 64C1D211089 for ; Thu, 13 Jul 2023 15:24:43 -0400 (EDT) Received: from [192.168.10.25] (D15.Denninger.Net [192.168.10.25]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by denninger.net (Postfix) with ESMTPSA id 06D0A3AAC53 for ; Thu, 13 Jul 2023 15:24:43 -0400 (EDT) Message-ID: <811be6b2-e94a-5143-cd37-f5af4ed7fcda@denninger.net> Date: Thu, 13 Jul 2023 15:24:39 -0400 List-Id: Dedicated and Embedded Systems List-Archive: https://lists.freebsd.org/archives/freebsd-embedded List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-embedded@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:102.0) Gecko/20100101 Thunderbird/102.13.0 Subject: Re: SD card corruption Content-Language: en-US To: freebsd-embedded@freebsd.org References: <709521ba-5719-5f80-10bf-1de05d99d5c1@sentex.net> <028a3f80-4d90-3368-6f51-ab8f77c56154@sentex.net> From: Karl Denninger In-Reply-To: <028a3f80-4d90-3368-6f51-ab8f77c56154@sentex.net> Content-Type: multipart/signed; protocol="application/pkcs7-signature"; micalg=sha-512; boundary="------------ms010406030800050500060209" X-Spamd-Result: default: False [-5.83 / 15.00]; SIGNED_SMIME(-2.00)[]; NEURAL_HAM_LONG(-0.99)[-0.992]; NEURAL_HAM_MEDIUM(-0.99)[-0.988]; NEURAL_HAM_SHORT(-0.95)[-0.950]; DMARC_POLICY_ALLOW(-0.50)[denninger.net,none]; R_SPF_ALLOW(-0.20)[+mx]; MIME_GOOD(-0.20)[multipart/signed,multipart/alternative,text/plain]; MLMMJ_DEST(0.00)[freebsd-embedded@freebsd.org]; RCVD_VIA_SMTP_AUTH(0.00)[]; R_DKIM_NA(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; ASN(0.00)[asn:14061, ipnet:104.236.64.0/18, country:US]; MIME_TRACE(0.00)[0:+,1:+,2:+,3:~,4:~]; RCVD_COUNT_THREE(0.00)[3]; RCPT_COUNT_ONE(0.00)[1]; TO_DN_NONE(0.00)[]; FREEFALL_USER(0.00)[karl]; ARC_NA(0.00)[]; RCVD_TLS_LAST(0.00)[]; FROM_HAS_DN(0.00)[]; HAS_ATTACHMENT(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[freebsd-embedded@freebsd.org]; MID_RHS_MATCH_FROM(0.00)[] X-Rspamd-Queue-Id: 4R24LT41Lfz3C81 X-Spamd-Bar: ----- X-ThisMailContainsUnwantedMimeParts: N This is a cryptographically signed message in MIME format. --------------ms010406030800050500060209 Content-Type: multipart/alternative; boundary="------------swXlFisRyHaczHYiAv9YHTkE" --------------swXlFisRyHaczHYiAv9YHTkE Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit On 7/13/2023 15:12, mike tancsa wrote: > On 7/13/2023 3:02 PM, Bob Bishop wrote: >> >> What’s the environment like? SD cards really don’t like being run hot >> for instance. Are there radios or other electrical noise nearby? > > Its a messy retail environment, but not too hot. The design of the > APUs have excellent passive cooling and I monitor the CPU temp at a > few hundred sites and have a good baseline. There is very little > variation and failures dont seem to correlate with the few hot > outliers. CPU is usually around 50C. > (https://www.pcengines.ch/apucool.htm). The recent fail site the > weekly avg was 49.9C with almost no variation / spikes > > Could exposure to a burst of intense em scramble the SD card ?  I > would think other devices would be impacted if that were the case. > Whats odd is that I am testing one of the returned cards right now. I > wiped it, filled it with 15GB of a few random files and am > continuously checksuming the files and they are fine. I would think > that if the SD card failed, it would continue to fail. Hence, I am > just trying to better understand what causes this. The SD card > hardware layer is all "black box" to me > I have a decent number of these (and this type of config, on pcEngines machines, in the field) and have yet to see this specific sort of failure.   Typically when I see SDs fail they go write-locked AND corrupted -- and are unrecoverable. This sounds like a bug in the on-SD-card firmware that is scrambling the allocation table.  That's nasty and its also likely out of your control.  The only thing I can come up with that might contribute to it would be unstable power. -- Karl Denninger karl@denninger.net /The Market Ticker/ /[S/MIME encrypted email preferred]/ --------------swXlFisRyHaczHYiAv9YHTkE Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: 8bit
On 7/13/2023 15:12, mike tancsa wrote:
On 7/13/2023 3:02 PM, Bob Bishop wrote:

What’s the environment like? SD cards really don’t like being run hot for instance. Are there radios or other electrical noise nearby?

Its a messy retail environment, but not too hot. The design of the APUs have excellent passive cooling and I monitor the CPU temp at a few hundred sites and have a good baseline. There is very little variation and failures dont seem to correlate with the few hot outliers. CPU is usually around 50C. (https://www.pcengines.ch/apucool.htm). The recent fail site the weekly avg was 49.9C with almost no variation / spikes

Could exposure to a burst of intense em scramble the SD card ?  I would think other devices would be impacted if that were the case. Whats odd is that I am testing one of the returned cards right now. I wiped it, filled it with 15GB of a few random files and am continuously checksuming the files and they are fine. I would think that if the SD card failed, it would continue to fail. Hence, I am just trying to better understand what causes this. The SD card hardware layer is all "black box" to me

I have a decent number of these (and this type of config, on pcEngines machines, in the field) and have yet to see this specific sort of failure.   Typically when I see SDs fail they go write-locked AND corrupted -- and are unrecoverable.

This sounds like a bug in the on-SD-card firmware that is scrambling the allocation table.  That's nasty and its also likely out of your control.  The only thing I can come up with that might contribute to it would be unstable power.

--
Karl Denninger
karl@denninger.net
The Market Ticker
[S/MIME encrypted email preferred]
--------------swXlFisRyHaczHYiAv9YHTkE-- --------------ms010406030800050500060209 Content-Type: application/pkcs7-signature; name="smime.p7s" Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="smime.p7s" Content-Description: S/MIME Cryptographic Signature MIAGCSqGSIb3DQEHAqCAMIACAQExDzANBglghkgBZQMEAgMFADCABgkqhkiG9w0BBwEAAKCC DbowggagMIIEiKADAgECAhMA5EiKghDOXrvfxYxjITXYDdhIMA0GCSqGSIb3DQEBCwUAMIGL MQswCQYDVQQGEwJVUzEQMA4GA1UECAwHRmxvcmlkYTESMBAGA1UEBwwJTmljZXZpbGxlMRkw FwYDVQQKDBBDdWRhIFN5c3RlbXMgTExDMRgwFgYDVQQLDA9DdWRhIFN5c3RlbXMgQ0ExITAf BgNVBAMMGEN1ZGEgU3lzdGVtcyBMTEMgMjAxNyBDQTAeFw0xNzA4MTcxNjQyMTdaFw0yNzA4 MTUxNjQyMTdaMHsxCzAJBgNVBAYTAlVTMRAwDgYDVQQIDAdGbG9yaWRhMRkwFwYDVQQKDBBD dWRhIFN5c3RlbXMgTExDMRgwFgYDVQQLDA9DdWRhIFN5c3RlbXMgQ0ExJTAjBgNVBAMMHEN1 ZGEgU3lzdGVtcyBMTEMgMjAxNyBJbnQgQ0EwggIiMA0GCSqGSIb3DQEBAQUAA4ICDwAwggIK AoICAQC1aJotNUI+W4jP7xQDO8L/b4XiF4Rss9O0B+3vMH7Njk85fZ052QhZpMVlpaaO+sCI KqG3oNEbuOHzJB/NDJFnqh7ijBwhdWutdsq23Ux6TvxgakyMPpT6TRNEJzcBVQA0kpby1DVD 0EKSK/FrWWBiFmSxg7qUfmIq/mMzgE6epHktyRM3OGq3dbRdOUgfumWrqHXOrdJz06xE9NzY vc9toqZnd79FUtE/nSZVm1VS3Grq7RKV65onvX3QOW4W1ldEHwggaZxgWGNiR/D4eosAGFxn uYeWlKEC70c99Mp1giWux+7ur6hc2E+AaTGh+fGeijO5q40OGd+dNMgK8Es0nDRw81lRcl24 SWUEky9y8DArgIFlRd6d3ZYwgc1DMTWkTavx3ZpASp5TWih6yI8ACwboTvlUYeooMsPtNa9E 6UQ1nt7VEi5syjxnDltbEFoLYcXBcqhRhFETJe9CdenItAHAtOya3w5+fmC2j/xJz29og1KH YqWHlo3Kswi9G77an+zh6nWkMuHs+03DU8DaOEWzZEav3lVD4u76bKRDTbhh0bMAk4eXriGL h4MUoX3Imfcr6JoyheVrAdHDL/BixbMH1UUspeRuqQMQ5b2T6pabXP0oOB4FqldWiDgJBGRd zWLgCYG8wPGJGYgHibl5rFiI5Ix3FQncipc6SdUzOQIDAQABo4IBCjCCAQYwHQYDVR0OBBYE FF3AXsKnjdPND5+bxVECGKtc047PMIHABgNVHSMEgbgwgbWAFBu1oRhUMNEzjODolDka5k4Q EDBioYGRpIGOMIGLMQswCQYDVQQGEwJVUzEQMA4GA1UECAwHRmxvcmlkYTESMBAGA1UEBwwJ TmljZXZpbGxlMRkwFwYDVQQKDBBDdWRhIFN5c3RlbXMgTExDMRgwFgYDVQQLDA9DdWRhIFN5 c3RlbXMgQ0ExITAfBgNVBAMMGEN1ZGEgU3lzdGVtcyBMTEMgMjAxNyBDQYIJAKxAy1WBo2kY MBIGA1UdEwEB/wQIMAYBAf8CAQAwDgYDVR0PAQH/BAQDAgGGMA0GCSqGSIb3DQEBCwUAA4IC AQCB5686UCBVIT52jO3sz9pKuhxuC2npi8ZvoBwt/IH9piPA15/CGF1XeXUdu2qmhOjHkVLN gO7XB1G8CuluxofOIUce0aZGyB+vZ1ylHXlMeB0R82f5dz3/T7RQso55Y2Vog2Zb7PYTC5B9 oNy3ylsnNLzanYlcW3AAfzZcbxYuAdnuq0Im3EpGm8DoItUcf1pDezugKm/yKtNtY6sDyENj tExZ377cYA3IdIwqn1Mh4OAT/Rmh8au2rZAo0+bMYBy9C11Ex0hQ8zWcvPZBDn4v4RtO8g+K uQZQcJnO09LJNtw94W3d2mj4a7XrsKMnZKvm6W9BJIQ4Nmht4wXAtPQ1xA+QpxPTmsGAU0Cv HmqVC7XC3qxFhaOrD2dsvOAK6Sn3MEpH/YrfYCX7a7cz5zW3DsJQ6o3pYfnnQz+hnwLlz4MK 17NIA0WOdAF9IbtQqarf44+PEyUbKtz1r0KGeGLs+VGdd2FLA0e7yuzxJDYcaBTVwqaHhU2/ Fna/jGU7BhrKHtJbb/XlLeFJ24yvuiYKpYWQSSyZu1R/gvZjHeGb344jGBsZdCDrdxtQQcVA 6OxsMAPSUPMrlg9LWELEEYnVulQJerWxpUecGH92O06wwmPgykkz//UmmgjVSh7ErNvL0lUY UMfunYVO/O5hwhW+P4gviCXzBFeTtDZH259O7TCCBxIwggT6oAMCAQICEgLG8yH4PQFdbd9x Ugmpzl1jXzANBgkqhkiG9w0BAQsFADB7MQswCQYDVQQGEwJVUzEQMA4GA1UECAwHRmxvcmlk YTEZMBcGA1UECgwQQ3VkYSBTeXN0ZW1zIExMQzEYMBYGA1UECwwPQ3VkYSBTeXN0ZW1zIENB MSUwIwYDVQQDDBxDdWRhIFN5c3RlbXMgTExDIDIwMTcgSW50IENBMB4XDTIyMDYyOTE2MTYz NloXDTI3MDYyODE2MTYzNlowOjELMAkGA1UEBhMCVVMxEjAQBgNVBAgMCVRlbm5lc3NlZTEX MBUGA1UEAwwOS2FybCBEZW5uaW5nZXIwggIiMA0GCSqGSIb3DQEBAQUAA4ICDwAwggIKAoIC AQC+HVSyxVtJhy3Ohs+PAGRuO//Dha9A16l5FPATr6wude9zjX5f2lrkRyU8vhCXTZW7WbvW ZKpcZ8r0dtZmiK9uF58Ec6hhvfkxJzbg96WHBw5Fumd5ahZzuCJDtCAWW8R7/KN+zwzQf1+B 3MVLmbaXAFBuKzySKhKMcHbK3/wjUYTgy+3UK6v2SBrowvkUBC+jxNg3Wy12GsTXcUS/8FYI XgVVPgfZZrbJJb5HWOQpvvhILpPCD3xsYJFNKEPltXKWHT7Qtc2HNqikgNwj8oqOb+PeZGMi WapsatKm8mxuOOGOEBhAoTVTwUHlMNTg6QUCJtuWFCK38qOCyk9Haj+86lUU8RG6FkRXWgMb NQm1mWREQhw3axgGLSntjjnznJr5vsvXSYR6c+XKLd5KQZcS6LL8FHYNjqVKHBYM+hDnrTZM qa20JLAF1YagutDiMRURU23iWS7bA9tMcXcqkclTSDtFtxahRifXRI7Epq2GSKuEXe/1Tfb5 CE8QsbCpGsfSwv2tZ/SpqVG08MdRiXxN5tmZiQWo15IyWoeKOXl/hKxA9KPuDHngXX022b1l y+5ZOZbxBAZZMod4y4b4FiRUhRI97r9lCxsP/EPHuuTIZ82BYhrhbtab8HuRo2ofne2TfAWY 2BlA7ExM8XShMd9bRPZrNTokPQPUCWCgCdIATQIDAQABo4IBzzCCAcswPAYIKwYBBQUHAQEE MDAuMCwGCCsGAQUFBzABhiBodHRwOi8vb2NzcC5jdWRhc3lzdGVtcy5uZXQ6ODg4ODAJBgNV HRMEAjAAMBEGCWCGSAGG+EIBAQQEAwIFoDAOBgNVHQ8BAf8EBAMCBeAwHQYDVR0lBBYwFAYI KwYBBQUHAwIGCCsGAQUFBwMEMDMGCWCGSAGG+EIBDQQmFiRPcGVuU1NMIEdlbmVyYXRlZCBD bGllbnQgQ2VydGlmaWNhdGUwHQYDVR0OBBYEFLElmNWeVgsBPe7O8NiBzjvjYnpRMIHKBgNV HSMEgcIwgb+AFF3AXsKnjdPND5+bxVECGKtc047PoYGRpIGOMIGLMQswCQYDVQQGEwJVUzEQ MA4GA1UECAwHRmxvcmlkYTESMBAGA1UEBwwJTmljZXZpbGxlMRkwFwYDVQQKDBBDdWRhIFN5 c3RlbXMgTExDMRgwFgYDVQQLDA9DdWRhIFN5c3RlbXMgQ0ExITAfBgNVBAMMGEN1ZGEgU3lz dGVtcyBMTEMgMjAxNyBDQYITAORIioIQzl6738WMYyE12A3YSDAdBgNVHREEFjAUgRJrYXJs QGRlbm5pbmdlci5uZXQwDQYJKoZIhvcNAQELBQADggIBAKquc7cu0xc8FNtAQauZvocDzWQj 7HG9YvMbWnMi+ckhiA3rdW5NwWg0HBhBho1YlnqV+ntCVE2L8ezohHWm+KAdfXgpraL86Vsn 3ywNlZu/3COMpo2ALuHln8YQtH3Y8ebvzKMdlf2b5WB+7mOFIxXIr4AnNOLKCkq5ZhzC6JW6 Jvw3P0csiGa3UrfatYID5NvPgkaQvEgimEjG3psZqwQTL2Wxohvw783PrDt3wS0XeNhvQ61g 3QJFZKuv+bmGH3YBSPo1t6NUGAr+JozX5lDihB8JGkBt/NwdYec49a08uL0BbPaAJ7NjuIPG 7Y0Ak7PXZT37yx/Zla9PzLMJFgbelOkaatdzbblMZPDEVZ27l4lGMmV83Lm3YP17sdAyS/Wp mav7WmJUkQ9iuIKzSpdc82i9Mfujl1vbBtwtkHNPPtKuulIFM4ZwrPKjlVdLqTSqD8m9yHEi Y0PuAooq63OpJWF6hvMaiIPBWEAVIaDW9uG0MshLl9DnHnMyrJTfuC33Z9mOGMz7dRBjJd5Y W02xAzYnUuEBOpj+LQv5R8XIFMHFXktqEKvQrXeM2RU+PcZqKOBkTktxBLn3NI5VfA15Jk0c 5V5XcOqo3p2hvrwvXrinrb2pEREnoqmfrkXT3zOq5Y6ryRH8u734lGEF0dILXzoV4PM7XFit oTePoEjmMYIFBDCCBQACAQEwgZEwezELMAkGA1UEBhMCVVMxEDAOBgNVBAgMB0Zsb3JpZGEx GTAXBgNVBAoMEEN1ZGEgU3lzdGVtcyBMTEMxGDAWBgNVBAsMD0N1ZGEgU3lzdGVtcyBDQTEl MCMGA1UEAwwcQ3VkYSBTeXN0ZW1zIExMQyAyMDE3IEludCBDQQISAsbzIfg9AV1t33FSCanO XWNfMA0GCWCGSAFlAwQCAwUAoIICQzAYBgkqhkiG9w0BCQMxCwYJKoZIhvcNAQcBMBwGCSqG SIb3DQEJBTEPFw0yMzA3MTMxOTI0NDFaME8GCSqGSIb3DQEJBDFCBEBJwrDeypwWItKDpJCq FM+eeWWgo/MOtN3QnQKZVVkRzxJi3IakulllNKlvGQyj47Sxe7kPqtKHuZzov658pSY0MGwG CSqGSIb3DQEJDzFfMF0wCwYJYIZIAWUDBAEqMAsGCWCGSAFlAwQBAjAKBggqhkiG9w0DBzAO BggqhkiG9w0DAgICAIAwDQYIKoZIhvcNAwICAUAwBwYFKw4DAgcwDQYIKoZIhvcNAwICASgw gaIGCSsGAQQBgjcQBDGBlDCBkTB7MQswCQYDVQQGEwJVUzEQMA4GA1UECAwHRmxvcmlkYTEZ MBcGA1UECgwQQ3VkYSBTeXN0ZW1zIExMQzEYMBYGA1UECwwPQ3VkYSBTeXN0ZW1zIENBMSUw IwYDVQQDDBxDdWRhIFN5c3RlbXMgTExDIDIwMTcgSW50IENBAhICxvMh+D0BXW3fcVIJqc5d Y18wgaQGCyqGSIb3DQEJEAILMYGUoIGRMHsxCzAJBgNVBAYTAlVTMRAwDgYDVQQIDAdGbG9y aWRhMRkwFwYDVQQKDBBDdWRhIFN5c3RlbXMgTExDMRgwFgYDVQQLDA9DdWRhIFN5c3RlbXMg Q0ExJTAjBgNVBAMMHEN1ZGEgU3lzdGVtcyBMTEMgMjAxNyBJbnQgQ0ECEgLG8yH4PQFdbd9x Ugmpzl1jXzANBgkqhkiG9w0BAQEFAASCAgAiLAc2XH1HxJKhTUkDrS5D86GWzBydQDtbp39K ISFzVok2JyKr17p8d2jSErmSM/r0ao2835qbRWcfdsLrhTM3H/fcEjt6nBSrxsySpkICNW/q zsMBE0/hJUU7WJ6h54gvvvh0LWwDUf6bQQtsP32pwLPCLGYyZvVTc9AsHzuykM9J6mPZK0Ox dsWDbmvAUu5o4X+CsiDv+XBOXLTrjA1tlL18vn+l9gn4Yvw18b08SKp1mdm9px+FiL4KJuro bsyIQATnpjLcYRML+K7u3O302ixkXV+kgOoV/BeNv4nvfK5+YXUU84J8qGk9oRUmaZB1B4bS O/dPHZQzj3ixtBZ9Fs4kMNlbTXSsDqsP4RLZ+IQSX+70J3HsfS54TsJgPf/InAKvsoc11M4y yb8bpjnVCISzndxJiCFHKg5pgxf1ooIyDUPevS7zgU8fajL4LRnXCJmGoD9Sq6U75C0ijng4 HivrDMQaDuV9WT89aihEf3ELB/3QO+KLBjar4cDFB8seyRNvZ4mtOndlstWKO4zey/EDWAJu 4wVIvmucoEuCPKvECF5NORb309pEoxlcf8gmEPvBw42pnzYIcPqU3XfZBYCA7ZAqAAQuKj3J SeXU6AOAvLhaLidaNH0d6hQ6O/AfK6BZki5z5gSik8eR0nwIltv7mBml8L5aboPPhBvrPAAA AAAAAA== --------------ms010406030800050500060209--