From owner-freebsd-current@FreeBSD.ORG Sun May 31 01:51:32 2009 Return-Path: Delivered-To: current@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 15D18106566C for ; Sun, 31 May 2009 01:51:32 +0000 (UTC) (envelope-from ler@lerctr.org) Received: from thebighonker.lerctr.org (thebighonker.lerctr.org [192.147.25.65]) by mx1.freebsd.org (Postfix) with ESMTP id DA4188FC19 for ; Sun, 31 May 2009 01:51:31 +0000 (UTC) (envelope-from ler@lerctr.org) DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=lerami; d=lerctr.org; h=Received:Received:Message-ID:In-Reply-To:References:Date:Subject:From:To:Cc:User-Agent:MIME-Version:Content-Type:Content-Transfer-Encoding:X-Priority:Importance:X-Spam-Score:X-LERCTR-Spam-Score:X-Spam-Report:X-LERCTR-Spam-Report:DomainKey-Status; b=GIPRM4ZHIHcSYVMss5TnUp4/UTBOnaP9Sr4hZbYy8aLI5xM1g/vVPALe48nXS2nNszohHqdYO5D+Easl8wt+2hu7GMMr2HYfBRKUd/omY/BkC9kOAYcP0lAbi/mdN720VzPrD+DhliH9gJqcGky4Ocl2sK6jp/d+2bUJGWS+WFQ=; Received: from localhost.lerctr.org ([127.0.0.1]:61441 helo=webmail.lerctr.org) by thebighonker.lerctr.org with esmtp (Exim 4.69 (FreeBSD)) (envelope-from ) id 1MAaCv-000AP5-Pb; Sat, 30 May 2009 20:51:31 -0500 Received: from 76.205.169.61 (SquirrelMail authenticated user ler) by webmail.lerctr.org with HTTP; Sat, 30 May 2009 20:51:29 -0500 Message-ID: In-Reply-To: References: Date: Sat, 30 May 2009 20:51:29 -0500 From: "Larry Rosenman" To: "Randy Bush" User-Agent: SquirrelMail/1.4.19 MIME-Version: 1.0 Content-Type: text/plain;charset=iso-8859-1 Content-Transfer-Encoding: 8bit X-Priority: 3 (Normal) Importance: Normal X-Spam-Score: -3.1 (---) X-LERCTR-Spam-Score: -3.1 (---) X-Spam-Report: SpamScore (-3.1/5.0) ALL_TRUSTED=-1.8, BAYES_00=-2.599, FM_MULTI_ODD2=1.1, TW_CG=0.077, TW_PL=0.077, TW_YD=0.077 X-LERCTR-Spam-Report: SpamScore (-3.1/5.0) ALL_TRUSTED=-1.8, BAYES_00=-2.599, FM_MULTI_ODD2=1.1, TW_CG=0.077, TW_PL=0.077, TW_YD=0.077 DomainKey-Status: no signature Cc: current Subject: Re: kern/134011: [hang] swap_pager_getswapspace(4): failed X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 31 May 2009 01:51:32 -0000 On Sat, May 30, 2009 8:46 pm, Randy Bush wrote: >> this time it actually said something interesting on console! >> >> for some values of 'interesting' :) >> >> swap_pager_getswapspace(3): failed >> swap_pager_getswapspace(3): failed >> swap_pager_getswapspace(3): failed >> swap_pager_getswapspace(3): failed >> swap_pager_getswapsp >> a >> lcatale t(ra1p 612): : pafgae ifaullte wdhi >> e sin wkearnepl m_odpe >> acpugied =r 1_; agpiec itd s= w01a >> afasulpt vairtcuael (ad3dre)ss := 0x0f >> odfalulet dc >> e s =w saupeprv_isopr awgrieter _dagtae, ptagse nowta >> ppressepnat >> 0inest(ru3ct)i:on pfoinateir l= e0dx2 >> :0sxfwfaffpff_ffp80a47gc25e6 >> rst_acgk eptoinsterw a p s =p 0ax2c8:0exf(3ff)ff:f80 7f9fd1a680i >> poieadme >> ntserw a p _ =p 0xag28e:0rxf_fffgff8e07t9fsd16we0 >> 0xdep ssegpmenta c= beas(e 30x0), :lim itf 0xaffifflf,e >> tydpe >> 1bs >> w =a DpPL _0, ppraesg 1,e lorng_ 1g, deef3t2 s0,w >> garapn s1 >> ledaocceseso(r 3ef)la:gs =f ianteirrulpt eendab >> , rseswumae, pIOP_L p= a0 >> gcuerrren_t pgroecests s = w789a (psysslpogad) >> ctreap( n9umb)er: = f12 >> epainilc:e pdag >> fsauwlta >> pcp_uipd =a 1g >> eUptime: 9h50m49s >> Physical memory: 4083 MB >> Dumping 1958 MB: >> >> > > a bit better in last night's syslog, possibly during backup > > randy > > > May 30 00:40:14 work0 kernel: lock order reversal: > May 30 00:40:14 work0 kernel: 1st 0xffffff0057d019d0 ufs (ufs) @ > /usr/src/sys/ufs/ffs/ffs_snapshot.c:423 > May 30 00:40:14 work0 kernel: 2nd 0xffffff8052c01aa0 bufwait (bufwait) @ > /usr/src/sys/kern/vfs_bio.c:2556 > May 30 00:40:14 work0 kernel: 3rd 0xffffff0004b8d098 ufs (ufs) @ > /usr/src/sys/ufs/ffs/ffs_snapshot.c:544 > May 30 00:40:16 work0 kernel: lock order reversal: > May 30 00:40:16 work0 kernel: 1st 0xffffff8052c01aa0 bufwait (bufwait) @ > /usr/src/sys/kern/vfs_bio.c:2556 > May 30 00:40:16 work0 kernel: 2nd 0xffffff00d35c7d30 snaplk (snaplk) @ > /usr/src/sys/ufs/ffs/ffs_snapshot.c:793 > May 30 00:50:14 work0 kernel: lock order reversal: > May 30 00:50:14 work0 kernel: 1st 0xffffff00d35c7d30 snaplk (snaplk) @ > /usr/src/sys/kern/vfs_vnops.c:297 > May 30 00:50:14 work0 kernel: 2nd 0xffffff0057d019d > May 30 00:50:14 work0 kernel: 0 ufs (ufs) @ /u > May 30 00:50:14 work0 kernel: s > May 30 00:50:15 work0 kernel: r/src/sys/ > May 30 00:50:15 work0 kernel: ufs/ffs/ > May 30 00:50:15 work0 kernel: ffs_snap > May 30 00:50:15 work0 kernel: shot.c: > May 30 00:50:15 work0 kernel: 1587 > May 30 01:45:21 work0 kernel: > May 30 01:45:21 work0 kernel: > May 30 01:45:21 work0 kernel: Fatal trap 12: page fault while in kernel > mode > May 30 01:45:21 work0 kernel: cpuid = 0; apic id = 00 > May 30 01:45:21 work0 kernel: fault virtual address = 0x0 > May 30 01:45:21 work0 kernel: fault code = supervisor write data, page > not present > May 30 01:45:21 work0 kernel: instruction pointer = > 0x20:0xffffffff8047c256 > May 30 01:45:21 work0 kernel: sta > May 30 01:45:21 work0 kernel: c > May 30 01:45:21 work0 kernel: k pointer = 0x28:0xffffff807a057680 > May 30 01:45:21 work0 kernel: frame pointer = > 0x28:0xffffff807a0576e0 > May 30 01:45:21 work0 kernel: code segment = base 0x0, limit 0xfffff, > type 0x1b > May 30 01:45:21 work0 kernel: = DPL 0, pres 1, long 1, def32 0, gran > May 30 01:45:21 work0 kernel: 1 > May 30 01:45:21 work0 kernel: processor eflags = interrup > May 30 01:45:21 work0 kernel: t enabled, resume, > May 30 01:45:21 work0 kernel: IOPL = 0 > May 30 01:45:21 work0 kernel: current process > May 30 01:45:21 work0 kernel: = 9181 (nfcapd) > May 30 02:10:04 work0 syslogd: kernel boot file is /boot/kernel/kernel See Kip's replies in my ZFS Crash thread. I suspect you have compression turned on in ZFS, and there is an unbounded allocation of memory in the ZFS (de)compression code. Kip is working on a patch to put some bounds on it. I've changed my tunables to not allow the ARC to grab ALL of physmem (vfs.zfs.arc_max) and testing a full backup now. Waiting, patiently, for code patch from Kip. -- Larry Rosenman http://www.lerctr.org/~ler Phone: +1 512-248-2683 E-Mail: ler@lerctr.org US Mail: 430 Valona Loop, Round Rock, TX 78681-3893