From owner-freebsd-fs@FreeBSD.ORG  Wed Sep  4 06:03:29 2013
Return-Path: <owner-freebsd-fs@FreeBSD.ORG>
Delivered-To: freebsd-fs@FreeBSD.org
Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115])
 (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits))
 (No client certificate requested)
 by hub.freebsd.org (Postfix) with ESMTP id 883E3827
 for <freebsd-fs@FreeBSD.org>; Wed,  4 Sep 2013 06:03:29 +0000 (UTC)
 (envelope-from avg@FreeBSD.org)
Received: from citadel.icyb.net.ua (citadel.icyb.net.ua [212.40.38.140])
 by mx1.freebsd.org (Postfix) with ESMTP id DA394230E
 for <freebsd-fs@FreeBSD.org>; Wed,  4 Sep 2013 06:03:28 +0000 (UTC)
Received: from porto.starpoint.kiev.ua (porto-e.starpoint.kiev.ua
 [212.40.38.100])
 by citadel.icyb.net.ua (8.8.8p3/ICyb-2.3exp) with ESMTP id JAA20707;
 Wed, 04 Sep 2013 09:03:21 +0300 (EEST)
 (envelope-from avg@FreeBSD.org)
Received: from localhost ([127.0.0.1])
 by porto.starpoint.kiev.ua with esmtp (Exim 4.34 (FreeBSD))
 id 1VH6BV-0006BT-62; Wed, 04 Sep 2013 09:03:21 +0300
Message-ID: <5226CCEF.7090002@FreeBSD.org>
Date: Wed, 04 Sep 2013 09:02:23 +0300
From: Andriy Gapon <avg@FreeBSD.org>
User-Agent: Mozilla/5.0 (X11; FreeBSD amd64;
 rv:17.0) Gecko/20130810 Thunderbird/17.0.8
MIME-Version: 1.0
To: Grant Gray <grant@gray.id.au>
Subject: Re: ZFS livelock / deadlock on pure SSD pool
References: <522599A9.9070107@grantgray.id.au> <5225AB77.9020208@FreeBSD.org>
 <5225BB8C.5050802@gray.id.au>
In-Reply-To: <5225BB8C.5050802@gray.id.au>
X-Enigmail-Version: 1.5.1
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit
Cc: freebsd-fs@FreeBSD.org, Grant Gray <grant@grantgray.id.au>
X-BeenThere: freebsd-fs@freebsd.org
X-Mailman-Version: 2.1.14
Precedence: list
List-Id: Filesystems <freebsd-fs.freebsd.org>
List-Unsubscribe: <http://lists.freebsd.org/mailman/options/freebsd-fs>,
 <mailto:freebsd-fs-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-fs>
List-Post: <mailto:freebsd-fs@freebsd.org>
List-Help: <mailto:freebsd-fs-request@freebsd.org?subject=help>
List-Subscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-fs>,
 <mailto:freebsd-fs-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Wed, 04 Sep 2013 06:03:29 -0000

on 03/09/2013 13:35 Grant Gray said the following:
> On 3/09/2013 7:27 PM, Andriy Gapon wrote:
>> arc_lowmem+0x38 kmem_malloc+0xb0

> Thanks for the feedback. Do you think it may be triggered when the ARC is
> evicting pages because it is full, or a genuine low-memory case? The system has
> 32GB of RAM, of which the ARC is typically about 24G (I think).

Given the kmem_malloc -> arc_lowmem call chain it was a KVA shortage.  Probably
because of KVA fragmentation.
Setting KVA size to a value larger than your physical memory size (1.5x or 2x)
may work around this problem.  The cost of the workaround is that some memory
will be used for the additional page table pages.

Some recent changes in head are supposed to help with the KVA fragmentation
problem in general.
-- 
Andriy Gapon