From owner-freebsd-current@FreeBSD.ORG Wed Nov 5 16:47:17 2014 Return-Path: Delivered-To: freebsd-current@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 8BB764A1 for ; Wed, 5 Nov 2014 16:47:17 +0000 (UTC) Received: from citadel.icyb.net.ua (citadel.icyb.net.ua [212.40.38.140]) by mx1.freebsd.org (Postfix) with ESMTP id C270B7AC for ; Wed, 5 Nov 2014 16:47:16 +0000 (UTC) Received: from porto.starpoint.kiev.ua (porto-e.starpoint.kiev.ua [212.40.38.100]) by citadel.icyb.net.ua (8.8.8p3/ICyb-2.3exp) with ESMTP id SAA19801; Wed, 05 Nov 2014 18:49:09 +0200 (EET) (envelope-from avg@FreeBSD.org) Received: from localhost ([127.0.0.1]) by porto.starpoint.kiev.ua with esmtp (Exim 4.34 (FreeBSD)) id 1Xm3jm-000ABy-BZ; Wed, 05 Nov 2014 18:47:14 +0200 Message-ID: <545A5440.2070403@FreeBSD.org> Date: Wed, 05 Nov 2014 18:45:52 +0200 From: Andriy Gapon User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:31.0) Gecko/20100101 Thunderbird/31.1.2 MIME-Version: 1.0 To: "James R. Van Artsdalen" Subject: Re: r273165. ZFS ARC: possible memory leak to Inact References: <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<1415098949.596412362.8vxee7kf@frv41.fwdcdn.com>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> <545A19CA.5000203@jrv.org> <545A1AE8.5090205@FreeBSD.org> <545A5128.8000504@jrv.org> In-Reply-To: <545A5128.8000504@jrv.org> Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 7bit Cc: freebsd-current@FreeBSD.org, "James R. Van Artsdalen" X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 05 Nov 2014 16:47:17 -0000 On 05/11/2014 18:32, James R. Van Artsdalen wrote: > On 11/5/2014 6:41 AM, Andriy Gapon wrote: >> If something hangs (appears to hang) and it's ZFS related, then >> https://wiki.freebsd.org/AvgZfsDeadlockDebug >> > > I don't think the"zpool history" hang is in ZFS or storage layer code: > it seems be stalled in kernel malloc(). See PID 12105 (zpool > history) below. The wiki page has this underscored line :-) when reporting a problem please always include full information about thread stacks, don't cherry pick; the output can be large, upload it somewhere and post a link > SUPERTEX:/root# uname -a > FreeBSD SUPERTEX.housenet.jrv 10.1-PRERELEASE FreeBSD 10.1-PRERELEASE #3 > r273476M: Wed Oct 22 15:05:29 CDT 2014 > root@SUPERTEX.housenet.jrv:/usr/obj/usr/src/sys/GENERIC amd64 > SUPERTEX:/root# procstat -kk -a > kk > SUPERTEX:/root# grep zio_wait kk > SUPERTEX:/root# grep zio_done kk > SUPERTEX:/root# grep zio_int kk > SUPERTEX:/root# grep zfs_ kk > 0 100475 kernel zfs_vn_rele_task mi_switch+0xe1 > sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a > fork_trampoline+0xe > 0 101018 kernel zfs_vn_rele_task mi_switch+0xe1 > sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a > fork_trampoline+0xe > 0 101522 kernel zfs_vn_rele_task mi_switch+0xe1 > sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a > fork_trampoline+0xe > 12105 101599 zpool - mi_switch+0xe1 > sleepq_wait+0x3a _cv_wait+0x16d vmem_xalloc+0x568 vmem_alloc+0x3d > kmem_malloc+0x33 uma_large_malloc+0x49 malloc+0x43 > zfs_ioc_pool_get_history+0xbd zfsdev_ioctl+0x5ca devfs_ioctl_f+0x139 > kern_ioctl+0x255 sys_ioctl+0x13c amd64_syscall+0x351 Xfast_syscall+0xfb > SUPERTEX:/root# grep dmu_ kk > SUPERTEX:/root# grep arc_ kk > 3 100125 zfskern arc_reclaim_thre mi_switch+0xe1 > sleepq_timedwait+0x3a _cv_timedwait_sbt+0x18b arc_reclaim_thread+0x288 > fork_exit+0x9a fork_trampoline+0xe > 3 100126 zfskern l2arc_feed_threa mi_switch+0xe1 > sleepq_timedwait+0x3a _cv_timedwait_sbt+0x18b l2arc_feed_thread+0x19f > fork_exit+0x9a fork_trampoline+0xe > SUPERTEX:/root# grep buf_ kk > 20 100139 bufdaemon - mi_switch+0xe1 > sleepq_timedwait+0x3a _sleep+0x26e buf_daemon+0x78 fork_exit+0x9a > fork_trampoline+0xe > SUPERTEX:/root# ps lp 12105 > UID PID PPID CPU PRI NI VSZ RSS MWCHAN STAT TT TIME COMMAND > 0 12105 12104 0 20 0 105692 35984 kmem are D - 0:03.76 zpool > history BI > SUPERTEX:/root# > > -- Andriy Gapon