Date: Sat, 16 Jun 2007 19:27:06 +0930 From: Benjamin Close <Benjamin.Close@clearchain.com> To: Kris Kennaway <kris@obsecurity.org> Cc: freebsd-current@freebsd.org Subject: Re: Swapfile on ZFS & Deadlock Message-ID: <4673B3F2.3050006@clearchain.com> In-Reply-To: <20070616034006.GA17514@rot13.obsecurity.org> References: <4672945C.3060304@clearchain.com> <20070615182521.GB9619@rot13.obsecurity.org> <46735745.5000401@clearchain.com> <20070616034006.GA17514@rot13.obsecurity.org>
next in thread | previous in thread | raw e-mail | index | archive | help
Kris Kennaway wrote:
> On Sat, Jun 16, 2007 at 12:51:41PM +0930, Benjamin Close wrote:
>
>> Kris Kennaway wrote:
>>
>>> On Fri, Jun 15, 2007 at 11:00:04PM +0930, Benjamin Close wrote:
>>>
>>>
>>>> Hi All,
>>>> Whilst running out of memory compiling Xorg (scanPCI is a killer) I
>>>> discovered a quick way to deadlock the system:
>>>>
>>>> dd if=/dev/zero of=somefileonzfs bs=something count=something
>>>> mdconfig -a -f something
>>>> swapon /dev/md0
>>>>
>>>> Then do something that needs swap.. instant deadlock. The system is
>>>> still responsive but all disk access become hung.
>>>>
>>>> Known issue? If so is there a way we can warn users/prevent users from
>>>> doing it?
>>>>
>>>>
>>> Enable DEBUG_VFS_LOCKS and DEBUG_LOCKS, then break to DDB when the
>>> deadlock occurs and do 'show lockedvnods'.
>>>
>>>
>> Ok, enabled the above and this time got:
>>
>> swap_pager: indefinite wait buffer: bufobj: 0, blkno: 312865, size: 16384
>>
>> just before the deadlock:
>>
>> Show locked vnods returns (hand transcribed)
>>
>> 0xffffff002c3ab5d0: tagz zfs, type VREG
>> usecout 1, writecount 1, refcount 2 mountedhere 0
>> flags()
>> v_object 0xffffff002f047c80 ref 0 pages 0
>> lock type zfs: EXCL (count 1) by thread 0xffffff0030573360 (pid 1188)
>>
>
> What was needed was the backtrace that should have been also displayed
> here.
>
>
oops, made the kernel, forgot to install it, new trace below - looks
like this one is not directly zil related.
>> Pid 1188 is:
>>
>> 1188 0 0 0 SL zfs:(&zi 0xffffff000208fd58 [md0]
>>
>
> Also a trace of the current backtrace of this. However based on the
> single byte of information 'i' I predict that this is due to zil (ZFS
> intent logging). I have turned this off on my machines because of
> deadlock conditions during low memory, which is also going to be true
> on your system. Try adding to /boot/loader.conf
>
> vfs.zfs.zil_disable=1
>
>
So this is a known issue then.
>> called doadump and though it went through the motions, savecore didn't
>> find anything saved.
>> Not sure what you need debugging wise, let me know.
>>
>> System is Intel Core 2 duo, running in SMP amd64, updated Friday 15th June.
>> The box has 1G physical ram, 1G dedicated swap partition, but needs an
>> extra 300M to compile xf86ScanPCI.c (freaky!).
>>
>
> Or just use -O0 on this file.
>
In the end I did use -O0 but figured the report would be more
worthwhile, it's certainly a issue for when zfs becomes non experimental.
Here's the latest trace:
0xffffff0023daa20: tag zfs, type VREG
usecount 1, writecount 1, refcount 2, mounted here0
flags()
vobject 0xffffff002df1cbb8 ref 0 pages 0
lock type zfs: EXCL (count 1) by thread 0xffffff002ed5b000 (pid 1033)
#0 0xfffffff8045f62b at _lockmgr+0x4cb
#1 at ... VOP_LOCK1_APV+0x79
#2 at ... _vn_lock+0x94
#3 at ... mdstart_vnode+0x15f
#4 at ... md_kthread+0x111
#5 at ... fork_exit
#6 at ... fork_trampoline
1033 0 0 0 SL vmwait 0xffffffff80a573b8 [md0]
Trace 1033
sched_switch()+0x15e
mi_switch+0x231
sleepq_switch+0xc7
sleepq_wait+0x44
_sleep+0x327
kmem_malloc+0x2a2
uma_large_malloc+0x4a
malloc+0x12d
arc_get_data_buf+0x36e
arc_buf_alloc+0xe4
arc_read+0fa
dbuf_read+0x43a
dmu_tx_check_ioerr+09a
dmu_tx_count_write+0x178
dmu_tx_hold_write+0x4a
zfs_freebsd_write+0x399
VOP_WRITE_APV+0xe5
mdstart_vnod()+0x1a5
md_kthread+0x111
fork_exit
fork_trampoline
Looks like zfs is wanting to write but in order to do it needs memory so
asks md for it. md needs to page something out in order to be able to
provide it. Since md is zfs file backed - deadlock.
Got a dump this time but the stack appears corrupt .... hmm, haven't got
a valid dump for a while now, wonder if something else is up.
mdstart_vnode+0x15f:
564 vn_lock(vp,KL_EXCLUSIVE | LK_RETY, td );
Cheers,
Benjamin
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?4673B3F2.3050006>