From owner-freebsd-stable@freebsd.org Mon Nov 26 04:35:38 2018 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 24157114CF8A for ; Mon, 26 Nov 2018 04:35:38 +0000 (UTC) (envelope-from wollman@hergotha.csail.mit.edu) Received: from hergotha.csail.mit.edu (tunnel82308-pt.tunnel.tserv4.nyc4.ipv6.he.net [IPv6:2001:470:1f06:ccb::2]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id AA9B0858DF for ; Mon, 26 Nov 2018 04:35:35 +0000 (UTC) (envelope-from wollman@hergotha.csail.mit.edu) Received: from hergotha.csail.mit.edu (localhost [127.0.0.1]) by hergotha.csail.mit.edu (8.15.2/8.15.2) with ESMTP id wAQ4ZUrP045184; Sun, 25 Nov 2018 23:35:32 -0500 (EST) (envelope-from wollman@hergotha.csail.mit.edu) Received: (from wollman@localhost) by hergotha.csail.mit.edu (8.15.2/8.14.4/Submit) id wAQ4ZUWP045183; Sun, 25 Nov 2018 23:35:30 -0500 (EST) (envelope-from wollman) MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Message-ID: <23547.30738.149260.454185@hergotha.csail.mit.edu> Date: Sun, 25 Nov 2018 23:35:30 -0500 From: Garrett Wollman To: Konstantin Belousov Cc: freebsd-stable@freebsd.org Subject: Re: Trap 12 in vm_page_alloc_after() In-Reply-To: <20181119050944.GW2378@kib.kiev.ua> References: <23538.4310.710700.401331@hergotha.csail.mit.edu> <20181119050944.GW2378@kib.kiev.ua> X-Mailer: VM 8.2.0b under 26.1 (amd64-portbld-freebsd11.2) X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.6.2 (hergotha.csail.mit.edu [127.0.0.1]); Sun, 25 Nov 2018 23:35:32 -0500 (EST) X-Spam-Status: No, score=-1.0 required=5.0 tests=ALL_TRUSTED, HEADER_FROM_DIFFERENT_DOMAINS autolearn=disabled version=3.4.1 X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on hergotha.csail.mit.edu X-Rspamd-Queue-Id: AA9B0858DF X-Spamd-Result: default: False [-4.28 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; NEURAL_HAM_SHORT(-0.96)[-0.964,0]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; IP_SCORE(-1.60)[ipnet: 2001:470::/32(-4.47), asn: 6939(-3.46), country: US(-0.09)]; RCVD_COUNT_THREE(0.00)[3]; TO_MATCH_ENVRCPT_SOME(0.00)[]; R_SPF_PERMFAIL(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; MX_GOOD(-0.01)[hergotha.bimajority.org]; FORGED_SENDER(0.30)[wollman@bimajority.org,wollman@hergotha.csail.mit.edu]; FREEMAIL_TO(0.00)[gmail.com]; R_DKIM_NA(0.00)[]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:6939, ipnet:2001:470::/32, country:US]; FROM_NEQ_ENVFROM(0.00)[wollman@bimajority.org,wollman@hergotha.csail.mit.edu]; DMARC_POLICY_SOFTFAIL(0.10)[bimajority.org : No valid SPF, No valid DKIM, none] X-Rspamd-Server: mx1.freebsd.org X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 26 Nov 2018 04:35:38 -0000 < said: > On Sun, Nov 18, 2018 at 08:24:38PM -0500, Garrett Wollman wrote: >> Has anyone seen this before? It's on a busy NFS server, but hasn't >> been observed on any of our other NFS servers. >> >> ------------------------------------------------------------------------ >> Fatal trap 12: page fault while in kernel mode >> --- trap 0xc, rip = 0xffffffff809a903d, rsp = 0xfffffe17eb8d0710, rbp = 0xfffffe17eb8d0750 --- >> vm_page_alloc_after() at vm_page_alloc_after+0x15d/frame 0xfffffe17eb8d0750 > What is the line number for vm_page_alloc_after+0x15d ? > Do you have NUMA enabled on 11 ? If gdb is to be believed, the trap is at line 1687: /* * At this point we had better have found a good page. */ KASSERT(m != NULL, ("missing page")); free_count = vm_phys_freecnt_adj(m, -1); >>>>>> if ((m->flags & PG_ZERO) != 0) vm_page_zero_count--; mtx_unlock(&vm_page_queue_free_mtx); vm_page_alloc_check(m); The faulting instruction is: 0xffffffff809a903d : testb $0x8,0x5a(%r14) There are no options matching /numa/i in the configuration. (This is a non-debugging configuration so the KASSERT is inoperative, I assume.) I have about a dozen other servers with the same kernel and they're not crashing, but obviously they all have different loads and sets of active clients. -GAWollman