From owner-freebsd-fs@FreeBSD.ORG Tue Jan 13 00:26:02 2015 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 3B66330D for ; Tue, 13 Jan 2015 00:26:02 +0000 (UTC) Received: from mail.ijs.si (mail.ijs.si [IPv6:2001:1470:ff80::25]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id E10FA68F for ; Tue, 13 Jan 2015 00:26:01 +0000 (UTC) Received: from amavis-proxy-ori.ijs.si (localhost [IPv6:::1]) by mail.ijs.si (Postfix) with ESMTP id 3kLsyZ3q4RzZ9 for ; Tue, 13 Jan 2015 01:25:58 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ijs.si; h= user-agent:message-id:references:in-reply-to:organization :subject:subject:from:from:date:date:content-transfer-encoding :content-type:content-type:mime-version:received:received :received:received; s=jakla4; t=1421108755; x=1423700756; bh=WNe CHLrklcBwrs6UEY7xKY7Xfergi+Y/hJHFb6eJjNw=; b=UTM0TSh07WLKy6C0yFt eWi7+ConQNJpE6pNaoFPEvqa3jHIjUp1z0jfe3tmkudvo10zVOz9jZ4FOPTcuVAi PXpdb0eAyo9STt4BQajJSErey6sHVT97ndGdhi4jUxVT+X3m9psSnLTA87DpQWQb uhpXrGmOA7yUvT9lPVqclGhE= X-Virus-Scanned: amavisd-new at ijs.si Received: from mail.ijs.si ([IPv6:::1]) by amavis-proxy-ori.ijs.si (mail.ijs.si [IPv6:::1]) (amavisd-new, port 10012) with ESMTP id 0pmzIuIFyxIK for ; Tue, 13 Jan 2015 01:25:55 +0100 (CET) Received: from mildred.ijs.si (mailbox.ijs.si [IPv6:2001:1470:ff80::143:1]) by mail.ijs.si (Postfix) with ESMTP for ; Tue, 13 Jan 2015 01:25:55 +0100 (CET) Received: from neli.ijs.si (neli.ijs.si [IPv6:2001:1470:ff80:88:21c:c0ff:feb1:8c91]) by mildred.ijs.si (Postfix) with ESMTP id 3kLsyW0wHmzVd for ; Tue, 13 Jan 2015 01:25:55 +0100 (CET) Received: from sleepy.ijs.si ([2001:1470:ff80:e001::1:1]) by neli.ijs.si with HTTP (HTTP/1.1 POST); Tue, 13 Jan 2015 01:25:55 +0100 MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII; format=flowed Content-Transfer-Encoding: 7bit Date: Tue, 13 Jan 2015 01:25:55 +0100 From: Mark Martinec To: freebsd-fs@freebsd.org Subject: Re: memory exhaustion on 10.1 AMD64 ZFS storage system Organization: J. Stefan Institute In-Reply-To: <20150112211148.15c1fe3c@kerio.tuxis.nl> References: <20150112211148.15c1fe3c@kerio.tuxis.nl> Message-ID: <520c2cb181266d1f28d00dbf7929d591@mailbox.ijs.si> X-Sender: Mark.Martinec+freebsd@ijs.si User-Agent: Roundcube Webmail/1.0.4 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 13 Jan 2015 00:26:02 -0000 Joseph Mingrone wrote: > We've had this storage system running 9.x without problems. After > upgrading to 10.1 we've seen "out of swap space" messages in the logs. > Dec 13 04:29:12 storage2 kernel: pid 723 (rpc.statd), uid 0, was > killed: > out of swap space [...] > What's the best way to determine if this is a ZFS problem? I've read > in > the 10.1 release notes that vfs.zfs.zio.use_uma has been re-enabled. > Has this caused anyone problems with 10.1? Aggressiveness of ARC in 10.0 hurt us pretty badly when switching from 9.2 to 10.0. The ARC / UMA greediness for memory was causing excessive swapping out of still active processes, while keeping ARC luxuriously bathing in memory. The situation in 10.1 may have been slightly improved, although it seems the fix still has not been committed: Bug 187594 - [zfs] [patch] ZFS ARC behavior problem and fix https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=187594 I'd really like to see a solution committed in time for 10.2. Mark Schouten wrote: > Do you have compression enabled and L2ARC? There's this bug that leaks > memory via L2ARC, causing all memory to run out, pushing out the ARC > and (probably) causes crashes and lots of swap usage.. > > I think this is about that bug: > https://github.com/freebsd/freebsd/commit/b98f85d480b770e34d5e08c66dbc668bd5548bdc That too (unrelated to ARC / UMA greediness for memory). Mark