From owner-freebsd-fs@FreeBSD.ORG Mon Dec 24 11:48:09 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 8070C38C for ; Mon, 24 Dec 2012 11:48:09 +0000 (UTC) (envelope-from joh.hendriks@gmail.com) Received: from mail-bk0-f47.google.com (mail-bk0-f47.google.com [209.85.214.47]) by mx1.freebsd.org (Postfix) with ESMTP id 019068FC0A for ; Mon, 24 Dec 2012 11:48:08 +0000 (UTC) Received: by mail-bk0-f47.google.com with SMTP id j4so3275605bkw.20 for ; Mon, 24 Dec 2012 03:48:02 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=x-received:message-id:date:from:user-agent:mime-version:to:subject :references:in-reply-to:content-type:content-transfer-encoding; bh=T7wuagg/WZeAD/GTPmOTPVinISbDdFYXRfCFu9KxhX4=; b=PtHgg9RS0XspRvcX7kO25e71N4v230CJ7OkhCfK6YVoPcSooo0fuyuRSlK8IuiyipY oIaQFWevpExOQQYWDgeSUr+r014ocOjT/bH7D2Xc8bMWC6T3GNvfmcxR9DVAEyRbLWSD jUK6Il2eYFe8Vv084TdNnZqtpWprAoWBhrgTqyHLniOIjO7ee0nW7GSKvSHfIuGGMfeS 2o+hs8FRqgsWXQTWkK3hgJyx9tNiyBYt2wSUu3uUFaKsUDSpEahLueCwuYz/7SKxZyEj gkBviVMUe53+dsf6EMCzn51FCWhDY/+tlV6SEISGMKZ7EWutstpKgYPEdW/IAx9v6YMu qzPw== X-Received: by 10.204.13.28 with SMTP id z28mr10289462bkz.113.1356349681706; Mon, 24 Dec 2012 03:48:01 -0800 (PST) Received: from [192.168.50.105] (double-l.xs4all.nl. [80.126.205.144]) by mx.google.com with ESMTPS id u3sm14923613bkw.9.2012.12.24.03.47.59 (version=SSLv3 cipher=OTHER); Mon, 24 Dec 2012 03:48:00 -0800 (PST) Message-ID: <50D840ED.20507@gmail.com> Date: Mon, 24 Dec 2012 12:47:57 +0100 From: Johan Hendriks User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:17.0) Gecko/17.0 Thunderbird/17.0 MIME-Version: 1.0 To: Derek Kulinski , freebsd-fs@freebsd.org Subject: Re: FreeBSD 9.1-RELEASE crashes almost daily; backtraces always list zfs routines References: <1824023197.20121223142308@takeda.tk> In-Reply-To: <1824023197.20121223142308@takeda.tk> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 24 Dec 2012 11:48:09 -0000 Derek Kulinski schreef: > Please help, I reported this issue on > http://www.freebsd.org/cgi/query-pr.cgi?pr=kern/174372 but the crashes > are unbearable since they happen regularly at night, most of the time > when periodic.daily is called (3am) but there are exceptions. It seems > like it can be triggered by any heavy disk activity. In many of the > crash dumps the current process is find command, but of course that's > does not always is the cause. > > When calling scrub, it appears to pass successfully. Smartmon tools is > not reporting any disk errors. I tested memory using memtest86 about a > month ago and it passed tests successfully. > > I never had this type of issue on 9.0, and not much changed in my > kernel config besides installing WiFi card. > > System: > FreeBSD chinatsu.takeda.tk 9.1-RELEASE FreeBSD 9.1-RELEASE #2 r244482: Wed Dec 19 23:28:15 PST 2012 root@chinatsu.takeda.tk:/usr/obj/usr/src/sys/CHINATSU amd64 > It's compiled from releng/9.1 branch. > > Thank you for any help, > Derek > > > Example crash messages: > > Today: > panic: general protection fault > cpuid = 1 > KDB: stack backtrace: > db_trace_self_wrapper() at db_trace_self_wrapper+0x2a > kdb_backtrace() at kdb_backtrace+0x37 > panic() at panic+0x1cd > trap_fatal() at trap_fatal+0x285 > trap() at trap+0x23e > calltrap() at calltrap+0x8 > --- trap 0x9, rip = 0xffffffff8101f5ac, rsp = 0xffffff8230ac18c0, rbp = 0xffffff8230ac18d0 --- > list_prev() at list_prev+0xc > arc_evict() at arc_evict+0x194 > arc_adjust() at arc_adjust+0x1a1 > arc_reclaim_thread() at arc_reclaim_thread+0x1a6 > fork_exit() at fork_exit+0x11c > fork_trampoline() at fork_trampoline+0xe > --- trap 0, rip = 0, rsp = 0xffffff8230ac1bb0, rbp = 0 --- > Uptime: 20h44m51s > Dumping 1157 out of 8072 MB:..2%..12%..21%..31%..41%..52%..61%..71%..81%..92% > > > Yesterday: > Fatal trap 12: page fault while in kernel mode > cpuid = 0; apic id = 00 > fault virtual address = 0xffffffffffffffe8 > fault code = supervisor read data, page not present > instruction pointer = 0x20:0xffffffff8102d1e7 > stack pointer = 0x28:0xffffff82315ec190 > frame pointer = 0x28:0xffffff82315ec280 > code segment = base 0x0, limit 0xfffff, type 0x1b > = DPL 0, pres 1, long 1, def32 0, gran 1 > processor eflags = interrupt enabled, resume, IOPL = 0 > current process = 34744 (find) > trap number = 12 > panic: page fault > cpuid = 0 > KDB: stack backtrace: > db_trace_self_wrapper() at db_trace_self_wrapper+0x2a > kdb_backtrace() at kdb_backtrace+0x37 > panic() at panic+0x1cd > trap_fatal() at trap_fatal+0x285 > trap_pfault() at trap_pfault+0x216 > trap() at trap+0x363 > calltrap() at calltrap+0x8 > --- trap 0xc, rip = 0xffffffff8102d1e7, rsp = 0xffffff82315ec190, rbp = 0xffffff82315ec280 --- > arc_evict() at arc_evict+0x1e7 > arc_get_data_buf() at arc_get_data_buf+0x1d5 > arc_read_nolock() at arc_read_nolock+0x1ec > arc_read() at arc_read+0x93 > dbuf_read() at dbuf_read+0x452 > dmu_buf_hold() at dmu_buf_hold+0xe0 > zap_lockdir() at zap_lockdir+0x58 > zap_cursor_retrieve() at zap_cursor_retrieve+0x19b > zfs_freebsd_readdir() at zfs_freebsd_readdir+0x2ee > VOP_READDIR_APV() at VOP_READDIR_APV+0x4a > kern_getdirentries() at kern_getdirentries+0x225 > sys_getdirentries() at sys_getdirentries+0x23 > amd64_syscall() at amd64_syscall+0x5d8 > Xfast_syscall() at Xfast_syscall+0xf7 > --- syscall (196, FreeBSD ELF64, sys_getdirentries), rip = 0x80089b29c, rsp = 0x7fffffffd8a8, rbp = 0x1 --- > Uptime: 2d3h49m3s > Dumping 3701 out of 8072 MB:..1%..11%..21%..31%..41%..51%..61%..71%..81%..91% > > > I can not help you with the crash itself. The problem with ZFS and the daily run is the check for certain files changes and permissions. If you have a lot of snapshots, find will be looking for those file through all of your snapshots. To disable this you can look at the following thread. http://forums.freebsd.org/archive/index.php/t-29994.html or the following. http://forums.freebsd.org/archive/index.php/t-31846.html regards Johan Hendriks