From owner-freebsd-hackers@FreeBSD.ORG Thu Aug 27 20:14:40 2009 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 9AA37106568E for ; Thu, 27 Aug 2009 20:14:40 +0000 (UTC) (envelope-from linda.messerschmidt@gmail.com) Received: from qw-out-2122.google.com (qw-out-2122.google.com [74.125.92.25]) by mx1.freebsd.org (Postfix) with ESMTP id 5578F8FC3D for ; Thu, 27 Aug 2009 20:14:40 +0000 (UTC) Received: by qw-out-2122.google.com with SMTP id 3so311842qwe.7 for ; Thu, 27 Aug 2009 13:14:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:in-reply-to:references :date:message-id:subject:from:to:content-type; bh=MOG30b9mAkpXtPXFXxkGxwMda0hctps7yuICHuU0QT4=; b=APeTYGJuPL52cvyCQ3iqVo5UrF32151okVvMROC3FQQxzWPUZX/GlkRWCmLAENmIAc qzwgQ2ksWs4hMBP+rW8YftO50VUUrbTy+oNsMnZhQrGK+YOae5uGntv19YdxEbLdh+HG jhzLhOnGVlOxlkOKTHgylX1ecMhfCGV8UBtbE= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; b=tIxHVpXHzZtP3pt1AaZGjUdjC8sV7l02/wopkuqCoX/FzzUtU3cP1wUFGy5F3mhcmN OA++xQCw/K+5s1trgieG4cqNAIHtz+PQ0zBkg5MVhG77WDsL11KrFNUT1UYnhhA1VSXR 4TrkcAP63ogBJszuxk5ScaqtjXAA8938g8CG8= MIME-Version: 1.0 Received: by 10.229.93.4 with SMTP id t4mr206829qcm.93.1251404079506; Thu, 27 Aug 2009 13:14:39 -0700 (PDT) In-Reply-To: <200908261642.59419.jhb@freebsd.org> References: <237c27100908261203g7e771400o2d9603220d1f1e0b@mail.gmail.com> <200908261642.59419.jhb@freebsd.org> Date: Thu, 27 Aug 2009 16:14:39 -0400 Message-ID: <237c27100908271314v28e6c710s8a278064333d1d20@mail.gmail.com> From: Linda Messerschmidt To: freebsd-hackers@freebsd.org Content-Type: text/plain; charset=ISO-8859-1 Subject: Re: Intermittent system hangs on 7.2-RELEASE-p1 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 27 Aug 2009 20:14:40 -0000 On Wed, Aug 26, 2009 at 4:42 PM, John Baldwin wrote: > One thing to note is that ktrace only logs voluntary context switches (i.e. > call to tsleep or waiting on a condition variable). It specifically does not > log preemptions or blocking on a mutex, I was not aware, thanks. > so in theory if your machine was > livelocked temporarily that might explain this. How would we determine that? We are now able to reproduce this on a test machine, even after slipping in a 7.2-STABLE kernel with KTR enabled. So we have a lot more options now. Unfortunately, I don't really "get" KTR yet. It looks like it has relevant info, but I was unable to correlate its huge timestamps (e.g. 6795522404430562) to ktrace output times (e.g. 1251387606.225544) showing problem areas. What's my best bet from here?