From owner-freebsd-fs@FreeBSD.ORG Tue Oct 15 15:17:54 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id D5A243D4 for ; Tue, 15 Oct 2013 15:17:54 +0000 (UTC) (envelope-from jdavidlists@gmail.com) Received: from mail-ie0-x233.google.com (mail-ie0-x233.google.com [IPv6:2607:f8b0:4001:c03::233]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id AAC352E1E for ; Tue, 15 Oct 2013 15:17:54 +0000 (UTC) Received: by mail-ie0-f179.google.com with SMTP id aq17so13904959iec.10 for ; Tue, 15 Oct 2013 08:17:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:date:message-id:subject:from:to:content-type; bh=g/d0jBD9Iy/9zK8A3u/mr4VN9CdFKXXYvZnC+JhMXVQ=; b=lUutqgtplQENOo0U/FChwUfRWB+F0SM1jx8pgSuQ90wdnU6w+Vhn+PQ7yIcXGaQ15R vZqsq9jk2vKaf/MZX3GEeG3/N0jnlqjiv1OBJoXc+m0Lv1QvwoovzEt1qAgap97ba111 2jBL4j5V4El+OAHyripSmb2u8U7fmi83hNhAKn9MdIQRl9LDqL5oNkK9/VnHir0jMuIV uryEiTvL37zemCBESehMTFVNTZS14dmfYFPbMZEmGWGtEN1uJwxiz8KMWwk6YQ690M3S Ghd7M4RHA1tFiWpjgHXC/cqyCjoRoBId6iGAKV+hrZn/iIZLZ/io5PeXzsHFam4BQGGS H2MA== MIME-Version: 1.0 X-Received: by 10.50.117.40 with SMTP id kb8mr17508871igb.60.1381850274078; Tue, 15 Oct 2013 08:17:54 -0700 (PDT) Sender: jdavidlists@gmail.com Received: by 10.43.180.131 with HTTP; Tue, 15 Oct 2013 08:17:53 -0700 (PDT) Date: Tue, 15 Oct 2013 11:17:53 -0400 X-Google-Sender-Auth: VF1eXMvrHxHFju05Y25BDcSgUs8 Message-ID: Subject: 9.2 + ZFS + i386 = panic: pmap_enter: attempted pmap_enter on 4MB page From: J David To: "freebsd-fs@freebsd.org" Content-Type: text/plain; charset=ISO-8859-1 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 15 Oct 2013 15:17:55 -0000 What are the necessary loader.conf / kernel config invocations to make ZFS stable on 9.2-RELEASE i386 node with 3GB RAM? This machine was rock solid under 8.4, but since upgrading to 9.2 it has been a disaster. It crashes every few hours with "panic: pmap_enter: attempted pmap_enter on 4MB page." Here are a couple of stack traces: panic: pmap_enter: attempted pmap_enter on 4MB page cpuid = 0 KDB: stack backtrace: #0 0xc0afc092 at kdb_backtrace+0x52 #1 0xc0ac249c at panic+0x1bc #2 0xc0f1825d at pmap_enter+0x63d #3 0xc0d34ee5 at vm_fault_hold+0x1c45 #4 0xc0d33262 at vm_fault+0x82 #5 0xc0f1f5d6 at trap_pfault+0x186 #6 0xc0f1ed4b at trap+0x51b #7 0xc0f0887c at calltrap+0x6 #8 0xc16770b6 at zio_execute+0x116 #9 0xc15d9460 at taskq_run_safe+0x10 #10 0xc0b08f26 at taskqueue_run_locked+0xe6 #11 0xc0b097f7 at taskqueue_thread_loop+0xb7 #12 0xc0a90a73 at fork_exit+0xa3 #13 0xc0f08924 at fork_trampoline+0x8 panic: pmap_enter: attempted pmap_enter on 4MB page cpuid = 1 KDB: stack backtrace: #0 0xc0afc092 at kdb_backtrace+0x52 #1 0xc0ac249c at panic+0x1bc #2 0xc0f1825d at pmap_enter+0x63d #3 0xc0d34ee5 at vm_fault_hold+0x1c45 #4 0xc0d33262 at vm_fault+0x82 #5 0xc0f1f5d6 at trap_pfault+0x186 #6 0xc0f1ed4b at trap+0x51b #7 0xc0f0887c at calltrap+0x6 #8 0xc1679bdf at zio_dva_allocate+0x9f #9 0xc16770b6 at zio_execute+0x116 #10 0xc15d9460 at taskq_run_safe+0x10 #11 0xc0b08f26 at taskqueue_run_locked+0xe6 #12 0xc0b097f7 at taskqueue_thread_loop+0xb7 #13 0xc0a90a73 at fork_exit+0xa3 #14 0xc0f08924 at fork_trampoline+0x8 For the last crash, top was running and this is what was on the screen when it died: CPU: 0.6% user, 0.0% nice, 2.2% system, 0.2% interrupt, 97.1% idle Mem: 442M Active, 91M Inact, 79M Wired, 3764K Cache, 4960K Buf, 2380M Free ARC: 40M Total, 8520K MFU, 30M MRU, 304K Anon, 494K Header, 1062K Other Swap: 4096M Total, 4096M Free Write failed: Broken pipe It doesn't seem to matter what KVA_PAGES, vm.kmem_size, vfs.zfs.arc_max or vfs.zfs.vdev.cache.size is set to, and the ZFS tuning guides in the wiki (albeit appearing to be dating to the 7.x era) provides guidelines for tuning down to 768M. So 3GB should be enough for a machine that is 99% idle. (Particularly given that it dies with >2GB free.) This seems to be specific to i386, this problem hasn't cropped up on any amd64 nodes. No compression, deduplication, snapshots or anything like that is in use. Thanks for any suggestions!