From owner-freebsd-stable@FreeBSD.ORG Thu Nov 16 08:23:59 2006 Return-Path: X-Original-To: stable@FreeBSD.org Delivered-To: freebsd-stable@FreeBSD.ORG Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id BB3F216A416 for ; Thu, 16 Nov 2006 08:23:59 +0000 (UTC) (envelope-from rink@rink.nu) Received: from mx0.rink.nu (thunderstone.rink.nu [80.112.228.34]) by mx1.FreeBSD.org (Postfix) with ESMTP id A6E0243D86 for ; Thu, 16 Nov 2006 08:23:49 +0000 (GMT) (envelope-from rink@rink.nu) Received: from localhost (localhost [127.0.0.1]) by mx0.rink.nu (Postfix) with ESMTP id 488A61704A for ; Thu, 16 Nov 2006 09:24:13 +0100 (CET) X-Virus-Scanned: amavisd-new at rink.nu Received: from mx0.rink.nu ([127.0.0.1]) by localhost (thunderstone.rink.nu [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id mIWK56blDQg5 for ; Thu, 16 Nov 2006 09:24:07 +0100 (CET) Received: by mx0.rink.nu (Postfix, from userid 1000) id 32D9817048; Thu, 16 Nov 2006 09:24:07 +0100 (CET) Date: Thu, 16 Nov 2006 09:24:07 +0100 From: Rink Springer To: stable@FreeBSD.org Message-ID: <20061116082407.GB33390@rink.nu> MIME-Version: 1.0 Content-Type: multipart/signed; protocol="application/x-pkcs7-signature"; micalg=sha1; boundary="WhfpMioaduB5tiZL" Content-Disposition: inline User-Agent: Mutt/1.5.13 (2006-08-11) Cc: Subject: panic(): vinvalbuf: dirty bufs: perhaps a ffs_syncvnode bug? X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 16 Nov 2006 08:23:59 -0000 --WhfpMioaduB5tiZL Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable Hi people, At work, some storage is put on a SUN StorEdge 3510 FibreChannel array; the disks are divided into two volumes, which are mounted using isp(4) controllers. Two seperate machines each mount a single volume (this is, box1 mounts the first volume (/dev/da1), and box2 mounts the second volume (/dev/da2)). Both volumes are formatted using UFS2. Over the night, we reset the shelf in order to activate its new management IP address, causing the /dev/da[12] devices to be temporarily unavailable. This resulted in the following panic on the rather busy mailstorage server (the other server has minor load and was fine): --- (da0:isp0:0:1:0): lost device (da0:isp0:0:1:0): removing device entry (da1:isp0:0:2:0): lost device g_vfs_done():da1s1[WRITE(offset=3D292316823552, length=3D16384)]error =3D 6 g_vfs_done():da1s1[WRITE(offset=3D240287318016, length=3D16384)]error =3D 6 g_vfs_done():da1s1[READ(offset=3D12175362048, length=3D2048)]error =3D 6 g_vfs_done():da1s1[WRITE(offset=3D240287318016, length=3D16384)]error =3D 6 g_vfs_done():da1s1[READ(offset=3D18370689024, length=3D2048)]error =3D 6 g_vfs_done():da1s1[READ(offset=3D25829486592, length=3D512)]error =3D 6 vnode_pager_getpages: I/O read error vm_fault: pager read error, pid 78035 (lmtpd) g_vfs_done():da1s1[WRITE(offset=3D240287318016, length=3D1638(da1:isp0:0:2:0): Invalidating pack 4)]error =3D 6 g_vfs_done():da1s1[READ(offset=3D13768671232, length=3D6144)]error =3D 6 g_vfs_done():da1s1[READ(offset=3D102126977024, length=3D16384)]error =3D 6 g_vfs_done():da1s1[READ(offset=3D13768671232, length=3D6144)]error =3D 6 g_vfs_dpone():da1s1[READ(offset=3D102319669248, length=3D16384)]error =3D 6a nic: vinvalbuf: dirty bufs cpuid =3D 2 Uptime: 54d15h48m38s kernel trap 12 with interrupts disabled Fatal trap 12: page fault while in kernel mode cpuid =3D 0; apic id =3D 00 fault virtual address =3D 0x56 fault code =3D supervisor read, page not present instruction pointer =3D 0x20:0xc0681303 stack pointer =3D 0x28:0xe8d973f0 frame pointer =3D 0x28:0xe8d973f8 code segment =3D base 0x0, limit 0xfffff, type 0x1b =3D DPL 0, pres 1, def32 1, gran 1 processor eflags =3D resume, IOPL =3D 0 current process =3D 78066 (lmtpd) trap number =3D 12 --- When looking at the source code of vinvalbuf(), which calls bufobj_invalbuf(), it seems that this panic is raised after a bufobj still contains dirty data after waiting for it to complete without error. The code can be found at /sys/kern/vfs_subr.c The sync routine called eventually translates to bufsync(), as in /sys/kern/vfs_bio.c, which calls the filesystem's sync routine. It seems as if the return status of vfs_bio_awrite() in ffs_syncvnode() is not checked; all the other parts are checked. I believe this could provoke this panic. As the machine is in production use, it was instantly rebooted by a collegue and thus I have no vmcore, backtrace or anything. I therefore hope the information provided here is adequate. Can someone with more FreeBSD-VFS knowledge please look at this? Thanks! --=20 Rink P.W. Springer - http://rink.nu "It's you isn't it? THE BASTARD OPERATOR FROM HELL!" "In the flesh, on the phone and in your account..." - BOFH #3 --WhfpMioaduB5tiZL Content-Type: application/x-pkcs7-signature Content-Disposition: attachment; filename="smime.p7s" Content-Transfer-Encoding: base64 MIIJawYJKoZIhvcNAQcCoIIJXDCCCVgCAQExCzAJBgUrDgMCGgUAMAsGCSqGSIb3DQEHAaCC BuIwggObMIIDBKADAgECAhAiuN7bs9pg6t3I0n6G5OOTMA0GCSqGSIb3DQEBBQUAMGIxCzAJ BgNVBAYTAlpBMSUwIwYDVQQKExxUaGF3dGUgQ29uc3VsdGluZyAoUHR5KSBMdGQuMSwwKgYD VQQDEyNUaGF3dGUgUGVyc29uYWwgRnJlZW1haWwgSXNzdWluZyBDQTAeFw0wNjExMDgwOTI2 NTNaFw0wNzExMDgwOTI2NTNaMIHSMREwDwYDVQQEEwhTcHJpbmdlcjEaMBgGA1UEKhMRUmlu ayBQZXRlciBXeWNoZXIxIzAhBgNVBAMTGlJpbmsgUGV0ZXIgV3ljaGVyIFNwcmluZ2VyMRsw GQYJKoZIhvcNAQkBFgxtYWlsQHJpbmsubnUxHzAdBgkqhkiG9w0BCQEWEHJpbmtAZnJlZWJz ZC5vcmcxIDAeBgkqhkiG9w0BCQEWEXJpbmtAaWwuZm9udHlzLm5sMRwwGgYJKoZIhvcNAQkB Fg1yaW5rQHN0YWNrLm5sMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAxryGDfel YzzENX7wodkbVY1NALfaiPfNEG10YjD8ZWdK9zkN26Tc878Shbqapq0KYFD8TACGfEhKoMvo qbf0PHAS/gNYr81Arqa9FRPUfzvtDE/cMbhvI+p7ufBITyYnPJp9MUD72iT+DohRR2ISVi3i NAEgDuSbYYNxctnvXqU6O6EPy3mzoFPDoiOQwBfVtFrjxBbND9BUK2bjtUyGt4x8I/Vulzrt qLPTokva+b97DHRgbCA/aLLYIrU6QoqOFJ8GrAbro/FZLYh4m1oJk3FEHVQOKkk7xzIaFmmP QGJRL8m6nrIZFTrQ+X2wmzfLD55K/UiqbekOuMiWbY9EbwIDAQABo10wWzBLBgNVHREERDBC gQxtYWlsQHJpbmsubnWBEHJpbmtAZnJlZWJzZC5vcmeBEXJpbmtAaWwuZm9udHlzLm5sgQ1y aW5rQHN0YWNrLm5sMAwGA1UdEwEB/wQCMAAwDQYJKoZIhvcNAQEFBQADgYEAIfIcieRjePBA wjZqvOdGpyPcNDnK/ubeQSTV5Y4AHWxm1sXhQxB/XrQ3RVdz1qDnBRL1AjkEBAl8e9+am4s6 D6TaSlmJeNXn6ZPJTQecisz3M+AKiMckShM3oAeUi0ktn1yNYR+hz5aQN612XT5OZRYznJVZ kPf1DiA2RVVyz+MwggM/MIICqKADAgECAgENMA0GCSqGSIb3DQEBBQUAMIHRMQswCQYDVQQG EwJaQTEVMBMGA1UECBMMV2VzdGVybiBDYXBlMRIwEAYDVQQHEwlDYXBlIFRvd24xGjAYBgNV BAoTEVRoYXd0ZSBDb25zdWx0aW5nMSgwJgYDVQQLEx9DZXJ0aWZpY2F0aW9uIFNlcnZpY2Vz IERpdmlzaW9uMSQwIgYDVQQDExtUaGF3dGUgUGVyc29uYWwgRnJlZW1haWwgQ0ExKzApBgkq hkiG9w0BCQEWHHBlcnNvbmFsLWZyZWVtYWlsQHRoYXd0ZS5jb20wHhcNMDMwNzE3MDAwMDAw WhcNMTMwNzE2MjM1OTU5WjBiMQswCQYDVQQGEwJaQTElMCMGA1UEChMcVGhhd3RlIENvbnN1 bHRpbmcgKFB0eSkgTHRkLjEsMCoGA1UEAxMjVGhhd3RlIFBlcnNvbmFsIEZyZWVtYWlsIElz c3VpbmcgQ0EwgZ8wDQYJKoZIhvcNAQEBBQADgY0AMIGJAoGBAMSmPFVzVftOucqZWh5owHUE cJ3f6f+jHuy9zfVb8hp2vX8MOmHyv1HOAdTlUAow1wJjWiyJFXCO3cnwK4Vaqj9xVsuvPAsH 5/EfkTYkKhPPK9Xzgnc9A74r/rsYPge/QIACZNenprufZdHFKlSFD0gEf6e20TxhBEAeZBly YLf7AgMBAAGjgZQwgZEwEgYDVR0TAQH/BAgwBgEB/wIBADBDBgNVHR8EPDA6MDigNqA0hjJo dHRwOi8vY3JsLnRoYXd0ZS5jb20vVGhhd3RlUGVyc29uYWxGcmVlbWFpbENBLmNybDALBgNV HQ8EBAMCAQYwKQYDVR0RBCIwIKQeMBwxGjAYBgNVBAMTEVByaXZhdGVMYWJlbDItMTM4MA0G CSqGSIb3DQEBBQUAA4GBAEiM0VCD6gsuzA2jZqxnD3+vrL7CF6FDlpSdf0whuPg2H6otnzYv wPQcUCCTcDz9reFhYsPZOhl+hLGZGwDFGguCdJ4lUJRix9sncVcljd2pnDmOjCBPZV+V2vf3 h9bGCE6u9uo05RAaWzVNd+NWIXiC3CEZNd4ksdMdRv9dX2VPMYICUTCCAk0CAQEwdjBiMQsw CQYDVQQGEwJaQTElMCMGA1UEChMcVGhhd3RlIENvbnN1bHRpbmcgKFB0eSkgTHRkLjEsMCoG A1UEAxMjVGhhd3RlIFBlcnNvbmFsIEZyZWVtYWlsIElzc3VpbmcgQ0ECECK43tuz2mDq3cjS fobk45MwCQYFKw4DAhoFAKCBsTAYBgkqhkiG9w0BCQMxCwYJKoZIhvcNAQcBMBwGCSqGSIb3 DQEJBTEPFw0wNjExMTYwODI0MDdaMCMGCSqGSIb3DQEJBDEWBBRewA9C3M92zy2bRIS43D7O D/28ijBSBgkqhkiG9w0BCQ8xRTBDMAoGCCqGSIb3DQMHMA4GCCqGSIb3DQMCAgIAgDANBggq hkiG9w0DAgIBQDAHBgUrDgMCBzANBggqhkiG9w0DAgIBKDANBgkqhkiG9w0BAQEFAASCAQBH f1APZmbLSbY065tajDFmsa+KQsuhuHTrnB7hN30hAE7+MksX14re4clUN6C8M1idQAVuWUET KQdkW5ogy1Irk1yYoUuOZJBB42yyxMdfcuizuwgKtf8BFkQJfxYwoszyE0mFMydtHf3QrYRt cVyeYwhw/GAO/tmgojm/swM2geurS/j2EDVoEBojemLYy/9z8xKjtQGFnqLUwRarOoAVjO9p EZPv5SpEFW4A95BLTdniWnUdKS9Kw+SzesVpMfv2w1EQdp41lRLf96cNQGGtSVSN6but6bYh Ai2rDzZQ8r3t9KhnJqJiMU6hLiF05JsKA2S9PTaMNi1T3qr3EEkV --WhfpMioaduB5tiZL--