Skip site navigation (1)Skip section navigation (2) 
Date:      Thu, 16 Nov 2006 09:24:07 +0100
From:      Rink Springer <rink@FreeBSD.org>
To:        stable@FreeBSD.org
Subject:   panic(): vinvalbuf: dirty bufs: perhaps a ffs_syncvnode bug?
Message-ID:  <20061116082407.GB33390@rink.nu>
next in thread | raw e-mail | index | archive | help 

[-- Attachment #1 --]
Hi people,

At work, some storage is put on a SUN StorEdge 3510 FibreChannel array;
the disks are divided into two volumes, which are mounted using isp(4)
controllers. Two seperate machines each mount a single volume (this is,
box1 mounts the first volume (/dev/da1), and box2 mounts the second
volume (/dev/da2)). Both volumes are formatted using UFS2.

Over the night, we reset the shelf in order to activate its new
management IP address, causing the /dev/da[12] devices to be temporarily
unavailable. This resulted in the following panic on the rather busy
mailstorage server (the other server has minor load and was fine):

---
(da0:isp0:0:1:0): lost device
(da0:isp0:0:1:0): removing device entry
(da1:isp0:0:2:0): lost device
g_vfs_done():da1s1[WRITE(offset=292316823552, length=16384)]error = 6
g_vfs_done():da1s1[WRITE(offset=240287318016, length=16384)]error = 6
g_vfs_done():da1s1[READ(offset=12175362048, length=2048)]error = 6
g_vfs_done():da1s1[WRITE(offset=240287318016, length=16384)]error = 6
g_vfs_done():da1s1[READ(offset=18370689024, length=2048)]error = 6
g_vfs_done():da1s1[READ(offset=25829486592, length=512)]error = 6
vnode_pager_getpages: I/O read error
vm_fault: pager read error, pid 78035 (lmtpd)
g_vfs_done():da1s1[WRITE(offset=240287318016,
length=1638(da1:isp0:0:2:0): Invalidating pack
4)]error = 6
g_vfs_done():da1s1[READ(offset=13768671232, length=6144)]error = 6
g_vfs_done():da1s1[READ(offset=102126977024, length=16384)]error = 6
g_vfs_done():da1s1[READ(offset=13768671232, length=6144)]error = 6
g_vfs_dpone():da1s1[READ(offset=102319669248, length=16384)]error = 6a
nic: vinvalbuf: dirty bufs
cpuid = 2
Uptime: 54d15h48m38s

kernel trap 12 with interrupts disabled


Fatal trap 12: page fault while in kernel mode
cpuid = 0; apic id = 00
fault virtual address   = 0x56
fault code              = supervisor read, page not present
instruction pointer     = 0x20:0xc0681303
stack pointer           = 0x28:0xe8d973f0
frame pointer           = 0x28:0xe8d973f8
code segment            = base 0x0, limit 0xfffff, type 0x1b
                        = DPL 0, pres 1, def32 1, gran 1
processor eflags        = resume, IOPL = 0
current process         = 78066 (lmtpd)
trap number             = 12
---

When looking at the source code of vinvalbuf(), which calls
bufobj_invalbuf(), it seems that this panic is raised after a bufobj
still contains dirty data after waiting for it to complete without
error. The code can be found at /sys/kern/vfs_subr.c

The sync routine called eventually translates to bufsync(), as in
/sys/kern/vfs_bio.c, which calls the filesystem's sync routine. It seems
as if the return status of vfs_bio_awrite() in ffs_syncvnode() is not
checked; all the other parts are checked. I believe this could provoke
this panic.

As the machine is in production use, it was instantly rebooted by a
collegue and thus I have no vmcore, backtrace or anything. I therefore
hope the information provided here is adequate.

Can someone with more FreeBSD-VFS knowledge please look at this?

Thanks!

-- 
Rink P.W. Springer                                - http://rink.nu
"It's you isn't it? THE BASTARD OPERATOR FROM HELL!"
"In the flesh, on the phone and in your account..."       - BOFH #3

[-- Attachment #2 --]
0	k	*H
	\0	X10	+0	*H
00"۳`~0
	*H
0b10	UZA1%0#U
Thawte Consulting (Pty) Ltd.1,0*U#Thawte Personal Freemail Issuing CA0
061108092653Z
071108092653Z010USpringer10U*Rink Peter Wycher1#0!URink Peter Wycher Springer10	*H
	mail@rink.nu10	*H
	rink@freebsd.org1 0	*H
	rink@il.fontys.nl10	*H
	
rink@stack.nl0"0
	*H
0
Ƽ
c<5~UMڈmtb0egJ9
ۤ
`PL|HJ詷<pX@;O1o#{HO&'<}1@$QGbV-4 aqr^:;ySâ#մZT+fL|#n:ӢK{t`l ?h":BY-xZ	qDT*I;2i@bQ/ɺ:}7JHmȖmDo]0[0KUD0Bmail@rink.nurink@freebsd.orgrink@il.fontys.nl
rink@stack.nl0U00
	*H
!cx@6jF#49A$lfC^7EWs֠9	|{ߚ:JYxM3
$J7I-\aϖ7v]>Ne3Y 6EUr0?0
0
	*H
010	UZA10UWestern Cape10U	Cape Town10U
Thawte Consulting1(0&UCertification Services Division1$0"UThawte Personal Freemail CA1+0)	*H
	personal-freemail@thawte.com0
030717000000Z
130716235959Z0b10	UZA1%0#U
Thawte Consulting (Pty) Ltd.1,0*U#Thawte Personal Freemail Issuing CA00
	*H
0Ħ<UsUNʙZhup[v:aQP
0cZ,p+Z?qV˯<6$*+w=+>@dקe*TH<a@dr`00U00CU<0:08642http://crl.thawte.com/ThawtePersonalFreemailCA.crl0U0)U"0 010UPrivateLabel2-1380
	*H
HP.
fgCL!6-6/P p<ab:~t%Pb'qW%ݩ9 Oe_N4[5MwV!x!5$F]_eO1Q0M0v0b10	UZA1%0#U
Thawte Consulting (Pty) Ltd.1,0*U#Thawte Personal Freemail Issuing CA"۳`~0	+0	*H
	1	*H
0	*H
	1
061116082407Z0#	*H
	1^Bv-D>0R	*H
	1E0C0
*H
0*H
0
*H
@0+0
*H
(0
	*H
GPffI4Z1fBˡt7}!N2K׊T73X@nYA)d[ R+\KdAl_r賻
D	0I3'mЭmq\cp`٠96K5h#zbs:i*DnKMZu)/Jzi1Qv5
@aIT黭!-6Pg&b1N.!t
d=66-SުI

Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20061116082407.GB33390>