From owner-freebsd-stable@FreeBSD.ORG Sun Dec 23 22:49:09 2012 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 61980372; Sun, 23 Dec 2012 22:49:09 +0000 (UTC) (envelope-from takeda@takeda.tk) Received: from chinatsu.takeda.tk (mail.takeda.tk [74.0.89.210]) by mx1.freebsd.org (Postfix) with ESMTP id 1ED238FC13; Sun, 23 Dec 2012 22:49:08 +0000 (UTC) Received: from localhost.takeda.tk (takeda-ws2.lan [10.0.0.123]) (authenticated bits=0) by chinatsu.takeda.tk (8.14.5/8.14.5) with ESMTP id qBNMXa63010389 (version=TLSv1/SSLv3 cipher=AES256-SHA bits=256 verify=NO); Sun, 23 Dec 2012 14:33:37 -0800 (PST) (envelope-from takeda@takeda.tk) Date: Sun, 23 Dec 2012 14:23:08 -0800 From: Derek Kulinski X-Priority: 3 (Normal) Message-ID: <1824023197.20121223142308@takeda.tk> To: freebsd-fs@freebsd.org, freebsd-stable@freebsd.org Subject: FreeBSD 9.1-RELEASE crashes almost daily; backtraces always list zfs routines MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-Virus-Scanned: clamav-milter 0.97.6 at chinatsu.takeda.tk X-Virus-Status: Clean X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 23 Dec 2012 22:49:09 -0000 Please help, I reported this issue on http://www.freebsd.org/cgi/query-pr.cgi?pr=kern/174372 but the crashes are unbearable since they happen regularly at night, most of the time when periodic.daily is called (3am) but there are exceptions. It seems like it can be triggered by any heavy disk activity. In many of the crash dumps the current process is find command, but of course that's does not always is the cause. When calling scrub, it appears to pass successfully. Smartmon tools is not reporting any disk errors. I tested memory using memtest86 about a month ago and it passed tests successfully. I never had this type of issue on 9.0, and not much changed in my kernel config besides installing WiFi card. System: FreeBSD chinatsu.takeda.tk 9.1-RELEASE FreeBSD 9.1-RELEASE #2 r244482: Wed Dec 19 23:28:15 PST 2012 root@chinatsu.takeda.tk:/usr/obj/usr/src/sys/CHINATSU amd64 It's compiled from releng/9.1 branch. Thank you for any help, Derek Example crash messages: Today: panic: general protection fault cpuid = 1 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a kdb_backtrace() at kdb_backtrace+0x37 panic() at panic+0x1cd trap_fatal() at trap_fatal+0x285 trap() at trap+0x23e calltrap() at calltrap+0x8 --- trap 0x9, rip = 0xffffffff8101f5ac, rsp = 0xffffff8230ac18c0, rbp = 0xffffff8230ac18d0 --- list_prev() at list_prev+0xc arc_evict() at arc_evict+0x194 arc_adjust() at arc_adjust+0x1a1 arc_reclaim_thread() at arc_reclaim_thread+0x1a6 fork_exit() at fork_exit+0x11c fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff8230ac1bb0, rbp = 0 --- Uptime: 20h44m51s Dumping 1157 out of 8072 MB:..2%..12%..21%..31%..41%..52%..61%..71%..81%..92% Yesterday: Fatal trap 12: page fault while in kernel mode cpuid = 0; apic id = 00 fault virtual address = 0xffffffffffffffe8 fault code = supervisor read data, page not present instruction pointer = 0x20:0xffffffff8102d1e7 stack pointer = 0x28:0xffffff82315ec190 frame pointer = 0x28:0xffffff82315ec280 code segment = base 0x0, limit 0xfffff, type 0x1b = DPL 0, pres 1, long 1, def32 0, gran 1 processor eflags = interrupt enabled, resume, IOPL = 0 current process = 34744 (find) trap number = 12 panic: page fault cpuid = 0 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a kdb_backtrace() at kdb_backtrace+0x37 panic() at panic+0x1cd trap_fatal() at trap_fatal+0x285 trap_pfault() at trap_pfault+0x216 trap() at trap+0x363 calltrap() at calltrap+0x8 --- trap 0xc, rip = 0xffffffff8102d1e7, rsp = 0xffffff82315ec190, rbp = 0xffffff82315ec280 --- arc_evict() at arc_evict+0x1e7 arc_get_data_buf() at arc_get_data_buf+0x1d5 arc_read_nolock() at arc_read_nolock+0x1ec arc_read() at arc_read+0x93 dbuf_read() at dbuf_read+0x452 dmu_buf_hold() at dmu_buf_hold+0xe0 zap_lockdir() at zap_lockdir+0x58 zap_cursor_retrieve() at zap_cursor_retrieve+0x19b zfs_freebsd_readdir() at zfs_freebsd_readdir+0x2ee VOP_READDIR_APV() at VOP_READDIR_APV+0x4a kern_getdirentries() at kern_getdirentries+0x225 sys_getdirentries() at sys_getdirentries+0x23 amd64_syscall() at amd64_syscall+0x5d8 Xfast_syscall() at Xfast_syscall+0xf7 --- syscall (196, FreeBSD ELF64, sys_getdirentries), rip = 0x80089b29c, rsp = 0x7fffffffd8a8, rbp = 0x1 --- Uptime: 2d3h49m3s Dumping 3701 out of 8072 MB:..1%..11%..21%..31%..41%..51%..61%..71%..81%..91% -- Best regards, Derek mailto:takeda@takeda.tk