Date: Mon, 11 Jul 2016 06:30:06 -0500 From: Karl Denninger <karl@denninger.net> To: freebsd-stable@freebsd.org Subject: Re: Not-so stable if you take a CAM error.... Message-ID: <6e9c07e1-12a6-a7cd-f775-6b0fe5a706bc@denninger.net> In-Reply-To: <op.ykfe1fvbkndu52@ronaldradial.radialsg.local> References: <2b0c454b-c1a0-4b5b-e778-bf0939e90ae1@denninger.net> <op.ykfe1fvbkndu52@ronaldradial.radialsg.local>
next in thread | previous in thread | raw e-mail | index | archive | help
This is a cryptographically signed message in MIME format. --------------ms060303020901030402080803 Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: quoted-printable On 7/11/2016 02:57, Ronald Klop wrote: > On Mon, 11 Jul 2016 02:54:38 +0200, Karl Denninger > <karl@denninger.net> wrote: > >> Got a (nasty) surprise this afternoon on my sandbox machine. >> >> I was updating some Raspberry Pi2 machines which involved taking the s= d >> card out, sticking it in an adapter and plugging it into the sandbox, >> then mounting the partition and using rsync. >> >> Unfortunately one of the cards was, unknown to me, bad and returned a >> write error during the update. >> >> The machine panic'd immediately after the CAM write error popped up. >> >> I was quite surprised by this, since (1) the SD card was (of course) >> mounted as a UFS filesystem; it shows up as a CAM device, (2) the >> machine itself is running off a ZFS root on a normal host-adapter and >> thus there is no comingling of the buffer cache and (3) there were no >> images being run from (can't, wrong architecture!) nor any system I/O >> (e.g. pagefile) going to the SD card. >> >> I certainly understand that under some circumstances (maybe even most >> circumstances) taking a hard I/O error to a system device is going to >> hose you and a panic() is arguably "least astonishment" when the price= >> of being wrong might be a corrupted system file or worse (e.g. corrupt= ed >> paged-out RSS, etc.) But I didn't expect a panic out a failed write t= o >> a device that is mounted and being used purely for data. >> >> I don't have a crash dump but can almost-certainly reproduce this if >> it's something that shouldn't happen and thus merits investigation. >> > > Hi, > > I understand you are surprised by this. I don't think it is the way it > should work. > Is there _any_ debugging information for people to use and try to help > you? Like which FreeBSD version are you running? Which FreeBSD version > was used to create the UFS fs? Does it use softupdates (SU) or also > journaling (SU+J)? > Maybe some output of dmesg? Or type of SD-card and reader. Other > people might have similar problems with similar hardware. > > Regards, > Ronald. > FreeBSD 11.0-BETA1 #0 r302489: Sat Jul 9 10:15:24 CDT 2016 =20 karl@NewFS.denninger.net:/usr/obj/usr/src/sys/KSD-SMP and FreeBSD 11.0-BETA1 #0 r302526: Sun Jul 10 10:39:31 CDT 2016 =20 karl@NewFS.denninger.net:/pics/CrossBuild/obj/arm.armv6/pics/CrossBuild/s= rc/sys/RPI2 Both blew up in the same way when stimulated with same I/O error. The filesystem in question does have softupdates enabled (the RPI images have it turned on by default) but no journaling. It's not card/reader dependent no architecture dependent; when it occurred the first time I stuck the card and reader into one of my Pis and attempted to update it there (thinking that perhaps my sandbox machine's USB port was wonky) and it blew up the Pi2 in the exact same way. This isn't (obviously, given both Intel-style and ARM machines being involved) architecture dependent. It's been a good long while since I took an actual hard I/O error that was 'visible' at the OS level (I've had plenty of disks die on ZFS over last few years but no "double failures" on a mirror or similar, and I on my servers I haven't had a UFS-based system for a while. This definitely looks like some sort of regression in the code; I've run FreeBSD for a hell of a long time and have had plenty of instances where disks have failed without having the machine go out from under me. --=20 Karl Denninger karl@denninger.net <mailto:karl@denninger.net> /The Market Ticker/ /[S/MIME encrypted email preferred]/ --------------ms060303020901030402080803 Content-Type: application/pkcs7-signature; name="smime.p7s" Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="smime.p7s" Content-Description: S/MIME Cryptographic Signature MIAGCSqGSIb3DQEHAqCAMIACAQExDzANBglghkgBZQMEAgMFADCABgkqhkiG9w0BBwEAAKCC Bl8wggZbMIIEQ6ADAgECAgEpMA0GCSqGSIb3DQEBCwUAMIGQMQswCQYDVQQGEwJVUzEQMA4G A1UECBMHRmxvcmlkYTESMBAGA1UEBxMJTmljZXZpbGxlMRkwFwYDVQQKExBDdWRhIFN5c3Rl bXMgTExDMRwwGgYDVQQDExNDdWRhIFN5c3RlbXMgTExDIENBMSIwIAYJKoZIhvcNAQkBFhND dWRhIFN5c3RlbXMgTExDIENBMB4XDTE1MDQyMTAyMjE1OVoXDTIwMDQxOTAyMjE1OVowWjEL MAkGA1UEBhMCVVMxEDAOBgNVBAgTB0Zsb3JpZGExGTAXBgNVBAoTEEN1ZGEgU3lzdGVtcyBM TEMxHjAcBgNVBAMTFUthcmwgRGVubmluZ2VyIChPQ1NQKTCCAiIwDQYJKoZIhvcNAQEBBQAD ggIPADCCAgoCggIBALmEWPhAdphrWd4K5VTvE5pxL3blRQPyGF3ApjUjgtavqU1Y8pbI3Byg XDj2/Uz9Si8XVj/kNbKEjkRh5SsNvx3Fc0oQ1uVjyCq7zC/kctF7yLzQbvWnU4grAPZ3IuAp 3/fFxIVaXpxEdKmyZAVDhk9az+IgHH43rdJRIMzxJ5vqQMb+n2EjadVqiGPbtG9aZEImlq7f IYDTnKyToi23PAnkPwwT+q1IkI2DTvf2jzWrhLR5DTX0fUYC0nxlHWbjgpiapyJWtR7K2YQO aevQb/3vN9gSojT2h+cBem7QIj6U69rEYcEDvPyCMXEV9VcXdcmW42LSRsPvZcBHFkWAJqMZ Myiz4kumaP+s+cIDaXitR/szoqDKGSHM4CPAZV9Yh8asvxQL5uDxz5wvLPgS5yS8K/o7zDR5 vNkMCyfYQuR6PAJxVOk5Arqvj9lfP3JSVapwbr01CoWDBkpuJlKfpQIEeC/pcCBKknllbMYq yHBO2TipLyO5Ocd1nhN/nOsO+C+j31lQHfOMRZaPQykXVPWG5BbhWT7ttX4vy5hOW6yJgeT/ o3apynlp1cEavkQRS8uJHoQszF6KIrQMID/JfySWvVQ4ksnfzwB2lRomrdrwnQ4eG/HBS+0l eozwOJNDIBlAP+hLe8A5oWZgooIIK/SulUAsfI6Sgd8dTZTTYmlhAgMBAAGjgfQwgfEwNwYI KwYBBQUHAQEEKzApMCcGCCsGAQUFBzABhhtodHRwOi8vY3VkYXN5c3RlbXMubmV0Ojg4ODgw CQYDVR0TBAIwADARBglghkgBhvhCAQEEBAMCBaAwCwYDVR0PBAQDAgXgMCwGCWCGSAGG+EIB DQQfFh1PcGVuU1NMIEdlbmVyYXRlZCBDZXJ0aWZpY2F0ZTAdBgNVHQ4EFgQUxRyULenJaFwX RtT79aNmIB/u5VkwHwYDVR0jBBgwFoAUJHGbnYV9/N3dvbDKkpQDofrTbTUwHQYDVR0RBBYw FIESa2FybEBkZW5uaW5nZXIubmV0MA0GCSqGSIb3DQEBCwUAA4ICAQBPf3cYtmKowmGIYsm6 eBinJu7QVWvxi1vqnBz3KE+HapqoIZS8/PolB/hwiY0UAE1RsjBJ7yEjihVRwummSBvkoOyf G30uPn4yg4vbJkR9lTz8d21fPshWETa6DBh2jx2Qf13LZpr3Pj2fTtlu6xMYKzg7cSDgd2bO sJGH/rcvva9Spkx5Vfq0RyOrYph9boshRN3D4tbWgBAcX9POdXCVfJONDxhfBuPHsJ6vEmPb An+XL5Yl26XYFPiODQ+Qbk44Ot1kt9s7oS3dVUrh92Qv0G3J3DF+Vt6C15nED+f+bk4gScu+ JHT7RjEmfa18GT8DcT//D1zEke1Ymhb41JH+GyZchDRWtjxsS5OBFMzrju7d264zJUFtX7iJ 3xvpKN7VcZKNtB6dLShj3v/XDsQVQWXmR/1YKWZ93C3LpRs2Y5nYdn6gEOpL/WfQFThtfnat HNc7fNs5vjotaYpBl5H8+VCautKbGOs219uQbhGZLYTv6okuKcY8W+4EJEtK0xB08vqr9Jd0 FS9MGjQE++GWo+5eQxFt6nUENHbVYnsr6bYPQsZH0CRNycgTG9MwY/UIXOf4W034UpR82TBG 1LiMsYfb8ahQJhs3wdf1nzipIjRwoZKT1vGXh/cj3gwSr64GfenURBxaFZA5O1acOZUjPrRT n3ci4McYW/0WVVA3lDGCBRMwggUPAgEBMIGWMIGQMQswCQYDVQQGEwJVUzEQMA4GA1UECBMH RmxvcmlkYTESMBAGA1UEBxMJTmljZXZpbGxlMRkwFwYDVQQKExBDdWRhIFN5c3RlbXMgTExD MRwwGgYDVQQDExNDdWRhIFN5c3RlbXMgTExDIENBMSIwIAYJKoZIhvcNAQkBFhNDdWRhIFN5 c3RlbXMgTExDIENBAgEpMA0GCWCGSAFlAwQCAwUAoIICTTAYBgkqhkiG9w0BCQMxCwYJKoZI hvcNAQcBMBwGCSqGSIb3DQEJBTEPFw0xNjA3MTExMTMwMDZaME8GCSqGSIb3DQEJBDFCBEB0 OgwSeUrXutPC7pzq5DMn/071IKJpRCJRqcR8g94BCNrB+Adc5Ug6JAHsHzMyVcKxT7gBMCHn FRsrQltpmD4LMGwGCSqGSIb3DQEJDzFfMF0wCwYJYIZIAWUDBAEqMAsGCWCGSAFlAwQBAjAK BggqhkiG9w0DBzAOBggqhkiG9w0DAgICAIAwDQYIKoZIhvcNAwICAUAwBwYFKw4DAgcwDQYI KoZIhvcNAwICASgwgacGCSsGAQQBgjcQBDGBmTCBljCBkDELMAkGA1UEBhMCVVMxEDAOBgNV BAgTB0Zsb3JpZGExEjAQBgNVBAcTCU5pY2V2aWxsZTEZMBcGA1UEChMQQ3VkYSBTeXN0ZW1z IExMQzEcMBoGA1UEAxMTQ3VkYSBTeXN0ZW1zIExMQyBDQTEiMCAGCSqGSIb3DQEJARYTQ3Vk YSBTeXN0ZW1zIExMQyBDQQIBKTCBqQYLKoZIhvcNAQkQAgsxgZmggZYwgZAxCzAJBgNVBAYT AlVTMRAwDgYDVQQIEwdGbG9yaWRhMRIwEAYDVQQHEwlOaWNldmlsbGUxGTAXBgNVBAoTEEN1 ZGEgU3lzdGVtcyBMTEMxHDAaBgNVBAMTE0N1ZGEgU3lzdGVtcyBMTEMgQ0ExIjAgBgkqhkiG 9w0BCQEWE0N1ZGEgU3lzdGVtcyBMTEMgQ0ECASkwDQYJKoZIhvcNAQEBBQAEggIAjZQ/vklQ 9EnK2B0wXuNA0wHMhl2lNFpYRjgTHQFDwhpe/lAkgzrS8CAp1bsPBbZAxeFWOt9TM9u/mB5O vFQ4fPERKDg2lB3utERCzzGz0K4fPwVgAbqveglmlGsHvBD8HVcuOsFhyPUeytOVFrhkzRJ/ 9D+nPQN5WnRwLX8+FAdIV8wHdRuQ99QpEcqRzIrcgsdrUOUInVUPNll1LDZJSDc7FR1ideY1 XOqR8/84LG8dvVyVs0NhLEJ9uPfwAgSXKorJkXIMP8vbGsHQOKPVgowAEgd7L9APbVOlfMVP 6tSqJkrAJwW45xo2xXVLmKDJo74dUPhWPcULwJ+60gYutBIRa7fxp6/vdOHPI7kBycuAapJj uazBMpINpTtJNAgJ1noOXfkZpcTGsIHHuJ3bJ63qYGqIBgC3Dh4bzOypBArRAkQ4sdQ9m8Y/ sL6Q9VimF+wgpcDWeU/ItG9XYAx0J2AmnjIfyXI2f6WAQc0vBsdkUNA6pfLerEIiBfWZL1fU 5l6VNhRxx8hhZOvThAX9tJLaJQHGKVAyGE7hN0H/6uxVGbkTiqXZ1Dr6x1kBZcUNmmugc/op qCMayQhMx+2DIGVA+7dbu56rEJsph7HuG6LuTrN+peThFs/gsF7iFDybGkCFKIvwvpQEpxhW 1PmDXvKifdrM9K9jCqCh94u77CMAAAAAAAA= --------------ms060303020901030402080803--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?6e9c07e1-12a6-a7cd-f775-6b0fe5a706bc>