From owner-freebsd-amd64@FreeBSD.ORG Wed Jan 4 15:20:05 2012 Return-Path: Delivered-To: freebsd-amd64@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id AE335106566C for ; Wed, 4 Jan 2012 15:20:05 +0000 (UTC) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 8A8548FC0A for ; Wed, 4 Jan 2012 15:20:05 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.5/8.14.5) with ESMTP id q04FK53J063039 for ; Wed, 4 Jan 2012 15:20:05 GMT (envelope-from gnats@freefall.freebsd.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.5/8.14.5/Submit) id q04FK5Zd063038; Wed, 4 Jan 2012 15:20:05 GMT (envelope-from gnats) Resent-Date: Wed, 4 Jan 2012 15:20:05 GMT Resent-Message-Id: <201201041520.q04FK5Zd063038@freefall.freebsd.org> Resent-From: FreeBSD-gnats-submit@FreeBSD.org (GNATS Filer) Resent-To: freebsd-amd64@FreeBSD.org Resent-Reply-To: FreeBSD-gnats-submit@FreeBSD.org, Zaphod Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 1BEC4106566C for ; Wed, 4 Jan 2012 15:12:27 +0000 (UTC) (envelope-from nobody@FreeBSD.org) Received: from red.freebsd.org (red.freebsd.org [IPv6:2001:4f8:fff6::22]) by mx1.freebsd.org (Postfix) with ESMTP id E84A28FC15 for ; Wed, 4 Jan 2012 15:12:26 +0000 (UTC) Received: from red.freebsd.org (localhost [127.0.0.1]) by red.freebsd.org (8.14.4/8.14.4) with ESMTP id q04FCQ3E088868 for ; Wed, 4 Jan 2012 15:12:26 GMT (envelope-from nobody@red.freebsd.org) Received: (from nobody@localhost) by red.freebsd.org (8.14.4/8.14.4/Submit) id q04FCQPH088867; Wed, 4 Jan 2012 15:12:26 GMT (envelope-from nobody) Message-Id: <201201041512.q04FCQPH088867@red.freebsd.org> Date: Wed, 4 Jan 2012 15:12:26 GMT From: Zaphod To: freebsd-gnats-submit@FreeBSD.org X-Send-Pr-Version: www-3.1 X-Mailman-Approved-At: Wed, 04 Jan 2012 16:26:57 +0000 Cc: Subject: amd64/163815: HDD timeout on ZFS + SB7x0 SATA Controller [AHCI] X-BeenThere: freebsd-amd64@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to the AMD64 platform List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 04 Jan 2012 15:20:05 -0000 >Number: 163815 >Category: amd64 >Synopsis: HDD timeout on ZFS + SB7x0 SATA Controller [AHCI] >Confidential: no >Severity: critical >Priority: high >Responsible: freebsd-amd64 >State: open >Quarter: >Keywords: >Date-Required: >Class: sw-bug >Submitter-Id: current-users >Arrival-Date: Wed Jan 04 15:20:05 UTC 2012 >Closed-Date: >Last-Modified: >Originator: Zaphod >Release: 9.0 >Organization: NA >Environment: FreeBSD 9.0-PRERELEASE FreeBSD 9.0-PRERELEASE #0 r228984: Fri Dec 30 12:57:09 EET 2011 amd64 >Description: Problem first showed its self during port builds (heavy HDD usage): -------------------------------- swap_pager: indefinite wait buffer: bufobj: 0, blkno: 32262, size: 4096 swap_pager: indefinite wait buffer: bufobj: 0, blkno: 66056, size: 4096 swap_pager: indefinite wait buffer: bufobj: 0, blkno: 82746, size: 8192 swap_pager: indefinite wait buffer: bufobj: 0, blkno: 44091, size: 4096 ahcich0: Timeout on slot 29 port 0 ahcich0: is 00000000 cs 000000ff ss e00000ff rs e00000ff tfd c0 serr 00000000 cmd 0004e017 ahcich0: AHCI reset... ahcich0: SATA connect time=100us status=00000123 ahcich0: AHCI reset: device found (ada0:ahcich0:0:0:0): Command timed out (ada0:ahcich0:0:0:0): Retrying command ------------------------------- Now, after latest update to /usr/src, buildworld breaks with "seg.fault 11" message, but actually due to swap_pager timeout. Break is near clang/lib/ARCMigrate/TransAutoreleasePool.cpp (but where is not so relevant). Also, CPU usage is not very heavy before system freeze. Hardware & Setup Info: - controller: ahci0@pci0:0:17:0: class=0x010601 card=0x43911002 chip=0x43911002 rev=0x00 hdr=0x00 device= 'SB7x0/SB8x0/SB9x0 SATA Controller [AHCI mode]' on 'RS780 Host Bridge'. Board: Biostar A780L - HDD is SAMSUNG HD322HJ, 320GB, ATA-8-ACS revision 3b, all FS on ZFS. - CPU: K8 [Athlon64/Opteron - mem/swap: RAM 1 GB / swap 2 GB (not zfs). Usage during buildworld: max RAM 65% / max swap 58% Previously built full-debug enabled kernel shows some errors as: kernel: lock order reversal: kernel: 1st 0xfffffe0010598248 filedesc structure (filedesc structure) @ /asp/src/sys/kern/kern_descrip.c:1197 kernel: 2nd 0xfffffe001052ccf0 zfs (zfs) @ /asp/src/sys/kern/vfs_subr.c:4245 kernel: KDB: stack backtrace: kernel: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a kernel: kdb_backtrace() at kdb_backtrace+0x37 kernel: _witness_debugger() at _witness_debugger+0x65 kernel: witness_checkorder() at witness_checkorder+0x833 kernel: __lockmgr_args() at __lockmgr_args+0xd9d kernel: vop_stdlock() at vop_stdlock+0x39 kernel: VOP_LOCK1_APV() at VOP_LOCK1_APV+0x9b kernel: _vn_lock() at _vn_lock+0x68 kernel: knlist_remove_kq() at knlist_remove_kq+0xfc kernel: knote_fdclose() at knote_fdclose+0x177 kernel: kern_close() at kern_close+0xe8 kernel: amd64_syscall() at amd64_syscall+0x27b kernel: Xfast_syscall() at Xfast_syscall+0xf7 kernel: --- syscall (6, FreeBSD ELF64, sys_close), rip = 0x8015abcdc, rsp = 0x7fffffffd868, rbp = 0x801807b20 --- Another one: kernel: lock order reversal: kernel: 1st 0xfffffe0018e3a448 filedesc structure (filedesc structure) @ /asp/src/sys/kern/kern_descrip.c:1197 kernel: 2nd 0xfffffe0004533cf0 devfs (devfs) @ /asp/src/sys/kern/vfs_subr.c:4245 kernel: KDB: stack backtrace: kernel: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a kernel: kdb_backtrace() at kdb_backtrace+0x37 kernel: _witness_debugger() at _witness_debugger+0x65 kernel: witness_checkorder() at witness_checkorder+0x833 kernel: __lockmgr_args() at __lockmgr_args+0xd9d kernel: vop_stdlock() at vop_stdlock+0x39 kernel: VOP_LOCK1_APV() at VOP_LOCK1_APV+0x9b kernel: _vn_lock() at _vn_lock+0x68 kernel: knlist_remove_kq() at knlist_remove_kq+0xfc kernel: knote_fdclose() at knote_fdclose+0x177 kernel: kern_close() at kern_close+0xe8 kernel: amd64_syscall() at amd64_syscall+0x27b kernel: Xfast_syscall() at Xfast_syscall+0xf7 kernel: --- syscall (6, FreeBSD ELF64, sys_close), rip = 0x8015abcdc, rsp = 0x7fffffffd498, rbp = 0x80180e230 --- kernel: lock order reversal: kernel: 1st 0xfffffe0018e3a448 filedesc structure (filedesc structure) @ /asp/src/sys/kern/kern_descrip.c:1197 kernel: 2nd 0xfffffe000b4f4a78 pseudofs (pseudofs) @ /asp/src/sys/kern/vfs_subr.c:4245 kernel: KDB: stack backtrace: kernel: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a kernel: kdb_backtrace() at kdb_backtrace+0x37 kernel: _witness_debugger() at _witness_debugger+0x65 kernel: witness_checkorder() at witness_checkorder+0x833 kernel: __lockmgr_args() at __lockmgr_args+0xd9d kernel: vop_stdlock() at vop_stdlock+0x39 kernel: VOP_LOCK1_APV() at VOP_LOCK1_APV+0x9b kernel: _vn_lock() at _vn_lock+0x68 kernel: knlist_remove_kq() at knlist_remove_kq+0xfc kernel: knote_fdclose() at knote_fdclose+0x177 kernel: kern_close() at kern_close+0xe8 kernel: amd64_syscall() at amd64_syscall+0x27b kernel: Xfast_syscall() at Xfast_syscall+0xf7 kernel: --- syscall (6, FreeBSD ELF64, sys_close), rip = 0x8015abcdc, rsp = 0x7fffffffd498, rbp = 0x80180e230 --- More details posted in forum: http://forums.freebsd.org/showthread.php?t=27452 >How-To-Repeat: >Fix: NA >Release-Note: >Audit-Trail: >Unformatted: