From owner-freebsd-stable@freebsd.org Wed Oct 14 13:53:12 2015 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id AC5FDA13BD9 for ; Wed, 14 Oct 2015 13:53:12 +0000 (UTC) (envelope-from frank@zzattack.org) Received: from mail-wi0-x231.google.com (mail-wi0-x231.google.com [IPv6:2a00:1450:400c:c05::231]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4740F13DB for ; Wed, 14 Oct 2015 13:53:11 +0000 (UTC) (envelope-from frank@zzattack.org) Received: by wieq12 with SMTP id q12so84503614wie.1 for ; Wed, 14 Oct 2015 06:53:10 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=zzattack.org; s=google; h=to:from:subject:message-id:date:user-agent:mime-version :content-type:content-transfer-encoding; bh=bbbwAGqdrM0EXyWkRYL/z/f4sUFT8jAX5kfiEnErOPM=; b=k1GEOer0WUtL1T9Cqdvr2oIi1OsGOQCQ9Sq/oHTMErmdTeo8yrObSxYWzUfwE5YRCI 7cwwtiRYBpBW7JQvSMiBXqydHxv4VzSJSUbM2lNmJwwn0Ad3Bjs184iyrmHw8niq3ASC 60zs9olJuHTVYZTzSUuZpG1ipml0+yUuZ9ucw= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:to:from:subject:message-id:date:user-agent :mime-version:content-type:content-transfer-encoding; bh=bbbwAGqdrM0EXyWkRYL/z/f4sUFT8jAX5kfiEnErOPM=; b=UKwzNGZZr0oGt8cNG6Tlkv62pWrZpKKlgoL5aICWNL9dXTMPI4C1l3fwDoQa4X1a45 F7UvC0h8smYl3EpbjjrkSVAvCI/lNiad460QhNi7QvbGoDFqTb3EhbDtEr92FVJyCB0t uLHA01UiN/YrLdUiI01x5iwngDDKUziOt2ZYFO/nQLSUn+5pVbLM+GmBVD4Xh+p97lXh lSJhNJ+q5INNOT/cM8cAX3NAYJrio/VLw0fboFNI7XcIi8P6g5LyyBXoVNDdC8VNPzhN Y7nzlYH4KIwlxvBk67I0klU6HVE1BsM3RHoGiBVQVFoqflw3h0+ZNoRXms9dP+84gzMI /taQ== X-Gm-Message-State: ALoCoQlYBmBlQuASLdvq4DLHu8NsPc7tcpD1owzSkwP5SS8blI5+8V+pjR0Df+Yefc0x/Buyz0uT X-Received: by 10.180.9.74 with SMTP id x10mr4472656wia.61.1444830765799; Wed, 14 Oct 2015 06:52:45 -0700 (PDT) Received: from [10.31.45.20] (38-106-201-31.ftth.glasoperator.nl. [31.201.106.38]) by smtp.googlemail.com with ESMTPSA id x7sm19406781wia.10.2015.10.14.06.52.44 for (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 14 Oct 2015 06:52:45 -0700 (PDT) To: freebsd-stable@freebsd.org From: Frank Razenberg Subject: 10.2-STABLE amd64 panic: page fault while in kernel mode Message-ID: <561E5E2F.90404@zzattack.org> Date: Wed, 14 Oct 2015 15:52:47 +0200 User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64; rv:38.0) Gecko/20100101 Thunderbird/38.2.0 MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 14 Oct 2015 13:53:12 -0000 After upgrading from 9.2 to 10.1 I first started noticing panics. They occurred roughly weekly and since this storage machine isn't frequently used I didn't look into it much further. After updating for 10.2-STABLE the panics have gone from weekly to daily. The machine has 32GB of non-registered ECC DDR3-1066 RAM. There's also a 10-disk raidz2 pool. I've ran memtest86+ for 72 hours straight with no errors. Crash dumps all feature the following: Fatal trap 12: page fault while in kernel mode cpuid = 2; apic id = 12 fault virtual address = 0x1d1c0bec0 fault code = supervisor read data, page not present instruction pointer = 0x20:0xffffffff804fda65 stack pointer = 0x28:0xfffffe0698f21870 frame pointer = 0x28:0xfffffe0698f218d0 code segment = base 0x0, limit 0xfffff, type 0x1b = DPL 0, pres 1, long 1, def32 0, gran 1 processor eflags = interrupt enabled, resume, IOPL = 0 current process = 6106 (pickup) trap number = 12 panic: page fault cpuid = 2 (kgdb) bt #0 doadump (textdump=) at pcpu.h:219 #1 0xffffffff8053ce32 in kern_reboot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:455 #2 0xffffffff8053d215 in vpanic (fmt=, ap=) at /usr/src/sys/kern/kern_shutdown.c:762 #3 0xffffffff8053d0a3 in panic (fmt=0x0) at /usr/src/sys/kern/kern_shutdown.c:691 #4 0xffffffff807755db in trap_fatal (frame=, eva=) at /usr/src/sys/amd64/amd64/trap.c:851 #5 0xffffffff807758dd in trap_pfault (frame=0xfffffe0698dbc7c0, usermode=) at /usr/src/sys/amd64/amd64/trap.c:674 #6 0xffffffff80774f7a in trap (frame=0xfffffe0698dbc7c0) at /usr/src/sys/amd64/amd64/trap.c:440 #7 0xffffffff8075b0f2 in calltrap () at /usr/src/sys/amd64/amd64/exception.S:236 #8 0xffffffff804fda65 in kqueue_close (fp=0xfffff803e4967190, td=0xfffff80014b094a0) at /usr/src/sys/kern/kern_event.c:1750 #9 0xffffffff804f25f9 in _fdrop (fp=0xfffff803e4967190, td=0xfffff802b5d2a000) at file.h:343 #10 0xffffffff804f4e9e in closef (fp=, td=) at /usr/src/sys/kern/kern_descrip.c:2338 #11 0xffffffff804f4ab9 in fdescfree (td=0xfffff80014b094a0) at /usr/src/sys/kern/kern_descrip.c:2106 #12 0xffffffff805013a9 in exit1 (td=0xfffff80014b094a0, rv=) at /usr/src/sys/kern/kern_exit.c:369 #13 0xffffffff80500e3e in sys_sys_exit (td=0xfffffe000782e060, uap=) at /usr/src/sys/kern/kern_exit.c:179 #14 0xffffffff80775efd in amd64_syscall (td=0xfffff80014b094a0, traced=0) at subr_syscall.c:134 #15 0xffffffff8075b3db in Xfast_syscall () at /usr/src/sys/amd64/amd64/exception.S:396 #16 0x000000080120335a in ?? () Most of the dumps list 'pickup' as current process. All of them have 'kqueue_close' in the backtrace. I'm not sure what the next step in diagnosing the issue is. Any pointers would be greatly appreciated. -Frank