From owner-freebsd-fs@FreeBSD.ORG Thu Jan 21 22:21:37 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 33FBD106568B for ; Thu, 21 Jan 2010 22:21:37 +0000 (UTC) (envelope-from artemb@gmail.com) Received: from mail-iw0-f171.google.com (mail-iw0-f171.google.com [209.85.223.171]) by mx1.freebsd.org (Postfix) with ESMTP id C9FE58FC14 for ; Thu, 21 Jan 2010 22:21:36 +0000 (UTC) Received: by iwn1 with SMTP id 1so462106iwn.28 for ; Thu, 21 Jan 2010 14:21:36 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:sender:received:in-reply-to :references:date:x-google-sender-auth:message-id:subject:from:to:cc :content-type; bh=FN1/oI683hZiYEsRodLCu4ZS1+4HIjRo2uYMdzy57E4=; b=OYVZ93sU9qEW9vaXnKUzPo2N50JqorSMD9Em96VDB7mRf7L5qnTHgqSorZafIY3zhl 03kgvX0v4xV7MF4Xi8fBiA2NKhny+16jt82HCKcBOAdF6c4Ko0rBWeOZqFDA/Q6vqOwD pxgLmyT0nSoXR9zu3osdq4VqwFgAQHpWSSe6Y= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type; b=Q7ALshnkrQN8Bj1TOqFpoSNMQzdk0AYm0EosduYPphO1peN/Qvk2UX2fYiEFDCuTHo ZUSolXv/mX+mxQxRRfVfhYzgQdpbIMs2QbKumEo9al6Z1Xs9kpnJtHEPwc6vaG7TbcCO eay1zHvpOpjHs9AjhV53zOKStcjEIrM53JR7s= MIME-Version: 1.0 Sender: artemb@gmail.com Received: by 10.231.147.210 with SMTP id m18mr3365368ibv.48.1264112494583; Thu, 21 Jan 2010 14:21:34 -0800 (PST) In-Reply-To: <4B58BD2D.30803@rcn.com> References: <4B58976E.1020402@polands.org> <4B58A069.8000802@egr.msu.edu> <4B58BD2D.30803@rcn.com> Date: Thu, 21 Jan 2010 14:21:34 -0800 X-Google-Sender-Auth: 53b91750d8fa3c0a Message-ID: From: Artem Belevich To: Gary Corcoran Content-Type: text/plain; charset=ISO-8859-1 Cc: freebsd-fs@freebsd.org Subject: Re: Repeatable ZFS "kmem map too small" panic on 8.0-STABLE X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 21 Jan 2010 22:21:37 -0000 On Thu, Jan 21, 2010 at 12:46 PM, Gary Corcoran wrote: > Adam McDougall wrote: >> >> Put this in /boot/loader.conf: >> vm.kmem_size="20G" >> >> It is intentionally higher than your amount of ram. > > Would you mind explaining... > 1) why this fixes the kmem_map too small problem ? Because it explicitly makes kmem_map larger. > 2) why it should be larger than the amount of RAM, and by how much ? ZFS needs access to a lot of memory for ARC and it allocates/frees memory fairly randomly. That raises two issues. First issue is that kernel is by default fairly conservative about its memory needs. vm.kmem_size which limits address space for in-kernel memory allocations is by default set to a fairly low value which works reasonably well in most of the cases. However, for ZFS it needs to be bumped up allow large amounts of memory to be allocated by ZFS. Second problem is memory fragmentation. If you set vm.kmem_size == physical memory size, over time you may end up with a situation when there is plenty of physical memory available, but there is no single contiguous block of address space to map that memory into. FreeBSD allocator is pretty good about avoiding fragmentation but you still do need more address space than the amount of memory that could potentially be allocated. I'd say that vm.kmem_size should be few multiples of amounts of memory that you're planning to allocate. Just my $0.02 --Artem