From owner-freebsd-fs@freebsd.org Tue Dec 1 01:24:33 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 5DD5CA3D12E for ; Tue, 1 Dec 2015 01:24:33 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4A063139A for ; Tue, 1 Dec 2015 01:24:33 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id tB11OXSA015956 for ; Tue, 1 Dec 2015 01:24:33 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 194513] zfs recv hangs in state kmem arena Date: Tue, 01 Dec 2015 01:24:30 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 10.0-STABLE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: mgoroff@quorum.net X-Bugzilla-Status: Closed X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-fs@FreeBSD.org X-Bugzilla-Target-Milestone: --- X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: cc Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 7bit X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 01 Dec 2015 01:24:33 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=194513 mgoroff@quorum.net changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |mgoroff@quorum.net --- Comment #13 from mgoroff@quorum.net --- We are encountering what appears to be the same issue on a large production server running 10.2-RELEASE. The machine has 128G of RAM and vfs.zfs.arc_max is set to 100G. I have a zfs recv currently stuck waiting on kmem arena. When this problem occurs, the ARC starts rapidly shrinking down to vfs.zfs.arc_min (13G in our case) but the process will remain blocked for a long period of time (hours) even though top shows free memory at 104G. ps -l for the process shows: UID PID PPID CPU PRI NI VSZ RSS MWCHAN STAT TT TIME COMMAND 0 37892 37814 0 20 0 42248 3428 kmem are D 1 0:04.16 zfs recv -u -d ezdata2 while procstat -kk -p 37892 shows: PID TID COMM TDNAME KSTACK 37892 101149 zfs - mi_switch+0xe1 sleepq_wait+0x3a _cv_wait+0x16d vmem_xalloc+0x568 vmem_alloc+0x3d kmem_malloc+0x33 uma_large_malloc+0x49 malloc+0x43 dmu_recv_stream+0x114 zfs_ioc_recv+0x955 zfsdev_ioctl+0x5ca devfs_ioctl_f+0x139 kern_ioctl+0x255 sys_ioctl+0x140 amd64_syscall+0x357 Xfast_syscall+0xfb -- You are receiving this mail because: You are the assignee for the bug.