From owner-freebsd-stable@freebsd.org Mon Nov 19 01:24:44 2018 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id CE3BE112E6A9 for ; Mon, 19 Nov 2018 01:24:44 +0000 (UTC) (envelope-from wollman@hergotha.csail.mit.edu) Received: from hergotha.csail.mit.edu (tunnel82308-pt.tunnel.tserv4.nyc4.ipv6.he.net [IPv6:2001:470:1f06:ccb::2]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 66DBA7A300 for ; Mon, 19 Nov 2018 01:24:41 +0000 (UTC) (envelope-from wollman@hergotha.csail.mit.edu) Received: from hergotha.csail.mit.edu (localhost [127.0.0.1]) by hergotha.csail.mit.edu (8.15.2/8.15.2) with ESMTP id wAJ1Oe90097833 for ; Sun, 18 Nov 2018 20:24:40 -0500 (EST) (envelope-from wollman@hergotha.csail.mit.edu) Received: (from wollman@localhost) by hergotha.csail.mit.edu (8.15.2/8.14.4/Submit) id wAJ1Oe4B097832; Sun, 18 Nov 2018 20:24:40 -0500 (EST) (envelope-from wollman) MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Message-ID: <23538.4310.710700.401331@hergotha.csail.mit.edu> Date: Sun, 18 Nov 2018 20:24:38 -0500 From: Garrett Wollman To: freebsd-stable@freebsd.org Subject: Trap 12 in vm_page_alloc_after() X-Mailer: VM 8.2.0b under 26.1 (amd64-portbld-freebsd11.2) X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.6.2 (hergotha.csail.mit.edu [127.0.0.1]); Sun, 18 Nov 2018 20:24:40 -0500 (EST) X-Spam-Status: No, score=-1.0 required=5.0 tests=ALL_TRUSTED, HEADER_FROM_DIFFERENT_DOMAINS autolearn=disabled version=3.4.1 X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on hergotha.csail.mit.edu X-Rspamd-Queue-Id: 66DBA7A300 X-Spamd-Result: default: False [-3.18 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-0.98)[-0.983,0]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MIME_GOOD(-0.10)[text/plain]; TO_DN_NONE(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[freebsd-stable@freebsd.org]; RCPT_COUNT_ONE(0.00)[1]; RCVD_COUNT_THREE(0.00)[3]; IP_SCORE(-1.65)[ipnet: 2001:470::/32(-4.55), asn: 6939(-3.58), country: US(-0.09)]; R_SPF_PERMFAIL(0.00)[]; MX_GOOD(-0.01)[hergotha.bimajority.org]; NEURAL_HAM_SHORT(-0.84)[-0.839,0]; FORGED_SENDER(0.30)[wollman@bimajority.org,wollman@hergotha.csail.mit.edu]; R_DKIM_NA(0.00)[]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:6939, ipnet:2001:470::/32, country:US]; FROM_NEQ_ENVFROM(0.00)[wollman@bimajority.org,wollman@hergotha.csail.mit.edu]; DMARC_POLICY_SOFTFAIL(0.10)[bimajority.org : No valid SPF, No valid DKIM, none] X-Rspamd-Server: mx1.freebsd.org X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 19 Nov 2018 01:24:45 -0000 Has anyone seen this before? It's on a busy NFS server, but hasn't been observed on any of our other NFS servers. ------------------------------------------------------------------------ Fatal trap 12: page fault while in kernel mode cpuid = 35; apic id = 35 fault virtual address = 0x5a fault code = supervisor read data, page not present instruction pointer = 0x20:0xffffffff809a903d stack pointer = 0x28:0xfffffe17eb8d0710 frame pointer = 0x28:0xfffffe17eb8d0750 code segment = base 0x0, limit 0xfffff, type 0x1b = DPL 0, pres 1, long 1, def32 0, gran 1 processor eflags = interrupt enabled, resume, IOPL = 0 current process = 878 (nfsd: service) trap number = 12 panic: page fault cpuid = 35 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe17eb8d03c0 vpanic() at vpanic+0x177/frame 0xfffffe17eb8d0420 panic() at panic+0x43/frame 0xfffffe17eb8d0480 trap_fatal() at trap_fatal+0x35f/frame 0xfffffe17eb8d04d0 trap_pfault() at trap_pfault+0x49/frame 0xfffffe17eb8d0530 trap() at trap+0x2c7/frame 0xfffffe17eb8d0640 calltrap() at calltrap+0x8/frame 0xfffffe17eb8d0640 --- trap 0xc, rip = 0xffffffff809a903d, rsp = 0xfffffe17eb8d0710, rbp = 0xfffffe17eb8d0750 --- vm_page_alloc_after() at vm_page_alloc_after+0x15d/frame 0xfffffe17eb8d0750 kmem_back() at kmem_back+0xf2/frame 0xfffffe17eb8d07c0 kmem_malloc() at kmem_malloc+0x60/frame 0xfffffe17eb8d07f0 keg_alloc_slab() at keg_alloc_slab+0xe2/frame 0xfffffe17eb8d0860 keg_fetch_slab() at keg_fetch_slab+0x14e/frame 0xfffffe17eb8d08b0 zone_fetch_slab() at zone_fetch_slab+0x64/frame 0xfffffe17eb8d08e0 zone_import() at zone_import+0x3f/frame 0xfffffe17eb8d0930 uma_zalloc_arg() at uma_zalloc_arg+0x3d9/frame 0xfffffe17eb8d09a0 zil_alloc_lwb() at zil_alloc_lwb+0x9c/frame 0xfffffe17eb8d09e0 zil_lwb_write_issue() at zil_lwb_write_issue+0x2f8/frame 0xfffffe17eb8d0a40 zil_commit_impl() at zil_commit_impl+0x95f/frame 0xfffffe17eb8d0b80 zfs_freebsd_fsync() at zfs_freebsd_fsync+0xa7/frame 0xfffffe17eb8d0bb0 VOP_FSYNC_APV() at VOP_FSYNC_APV+0x82/frame 0xfffffe17eb8d0be0 nfsvno_fsync() at nfsvno_fsync+0xe0/frame 0xfffffe17eb8d0c50 nfsrvd_commit() at nfsrvd_commit+0xe8/frame 0xfffffe17eb8d0e20 nfsrvd_dorpc() at nfsrvd_dorpc+0x621/frame 0xfffffe17eb8d0ff0 nfssvc_program() at nfssvc_program+0x557/frame 0xfffffe17eb8d11a0 svc_run_internal() at svc_run_internal+0xe09/frame 0xfffffe17eb8d12e0 svc_thread_start() at svc_thread_start+0xb/frame 0xfffffe17eb8d12f0 fork_exit() at fork_exit+0x83/frame 0xfffffe17eb8d1330 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe17eb8d1330 --- trap 0xc, rip = 0x80087101a, rsp = 0x7fffffffe688, rbp = 0x7fffffffe930 --- ------------------------------------------------------------------------ At this point the system was frozen: it did not attempt to reboot automatically and was not in the debugger. I had to do a remote reset via the BMC. The kernel is 11.2 r336644 (so no errata applied), but none of the SAs and ENs release so far look like they touch this region of code. -GAWollman