From owner-freebsd-stable@FreeBSD.ORG Mon Apr 30 16:31:01 2012 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 11E6D1065675 for ; Mon, 30 Apr 2012 16:31:01 +0000 (UTC) (envelope-from fjwcash@gmail.com) Received: from mail-qc0-f182.google.com (mail-qc0-f182.google.com [209.85.216.182]) by mx1.freebsd.org (Postfix) with ESMTP id C42218FC14 for ; Mon, 30 Apr 2012 16:31:00 +0000 (UTC) Received: by qcsg15 with SMTP id g15so1853219qcs.13 for ; Mon, 30 Apr 2012 09:31:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:date:message-id:subject:from:to:cc:content-type; bh=RIwBQ3gic0+Uxk3lRld0xmabOd592BD17mXAb0X7Giw=; b=UwzkoRB2WPUE+OcqxZ9lDoMAdppOrp8q7VDyKkaj7ZTPE7vs9ieARmgTenazdx+w4m Dh61zAs4g7WfDpW2qYOwqicziyMPUT450KGeooBo0qdLje+QR1UHO0eqVAEzxzmiMMht 2ZBK96IDtI7VhPKssH4pcRsJ3WChMGubpOU3d6xi2pdtNv3YX9AoE4sz0VW5v4yEzoAm /NRi1YEdP2GU0agdgtzWIOreT4fnHW/YJv9bqKl+pHDW1FBL7wYD3DGXYjezcddBxWWS D6aMeO4C3kkuTcla8OdaQRaZKKXaa+nz9hXki1+D8aCcewGIQxeZQuEwocFAQ5lYZ8Gn nIKg== MIME-Version: 1.0 Received: by 10.224.115.208 with SMTP id j16mr18168406qaq.84.1335803460314; Mon, 30 Apr 2012 09:31:00 -0700 (PDT) Received: by 10.229.91.138 with HTTP; Mon, 30 Apr 2012 09:31:00 -0700 (PDT) Date: Mon, 30 Apr 2012 09:31:00 -0700 Message-ID: From: Freddie Cash To: FreeBSD Stable Content-Type: text/plain; charset=UTF-8 Cc: bzeeb+freebsd+lor@zabbadoz.net Subject: Process for getting data to report LoRs X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 30 Apr 2012 16:31:01 -0000 Is it possible to get the backtrace for a LoR from any of the system logs or anything like that, after the fact? Especially after rebooting into a non-debug kernel? I compiled a custom kernel for our ZFS boxes that have been locking up on me lately, adding INVARIANTS and WITNESS. But, to be safe, I booted the debug kernel using nextboot on Friday. The boxes locked up over the weekend, and we restarted, reverting them back to the non-debug kernel. Going through /var/log/messages, I see a couple LoRs that aren't listed on http://ipv4.sources.zabbadoz.net/freebsd/lor.html However, as I'm not running the debug kernel anymore, I can't go through sysctl to grab the backtrace for it. Is there any other way to get the info? Is there anyway to configure the system to log the backtrace to a file? Here's the info from /var/log/messages, which I'm guessing is not enough to track down the cause of the LoR: lock order reversal: 1st 0xfffffe0019415098 zfs (zfs) @ /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vfsops.c:1704 2nd 0xfffffe00191669f8 ufs (ufs) @ /usr/src/sys/kern/vfs_syscalls.c:1665 lock order reversal: 1st 0xfffffe00194d0eb8 rtentry (rtentry) @ /usr/src/sys/net/route.c:374 2nd 0xfffffe000adb2bc0 if_afdata (if_afdata) @ /usr/src/sys/netinet6/scope6.c:417 System info: FreeBSD betadrive.sd73.bc.ca 9.0-STABLE FreeBSD 9.0-STABLE #0 r234466: Fri Apr 20 10:57:30 PDT 2012 root@betadrive.sd73.bc.ca:/usr/obj/usr/src/sys/ZFSHOST90 amd64 -- Freddie Cash fjwcash@gmail.com