From owner-freebsd-stable@FreeBSD.ORG Tue May 5 03:41:37 2009 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id B3696106568C; Tue, 5 May 2009 03:41:37 +0000 (UTC) (envelope-from pluknet@gmail.com) Received: from mail-fx0-f162.google.com (mail-fx0-f162.google.com [209.85.220.162]) by mx1.freebsd.org (Postfix) with ESMTP id E8E288FC27; Tue, 5 May 2009 03:41:36 +0000 (UTC) (envelope-from pluknet@gmail.com) Received: by fxm6 with SMTP id 6so4217631fxm.43 for ; Mon, 04 May 2009 20:41:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:in-reply-to:references :date:message-id:subject:from:to:cc:content-type :content-transfer-encoding; bh=EKCcalFeVAxJX9h187YvxRIBA0ubyNIIHIQq6X+bnAA=; b=VStTyXLzUcvJN2EGJeaC+I7HLpyUpeBc5NPzyikn7QC1QpDJBqsm6rjnUfdr2A1NDK BtB8/ndi2N9eFaObbL8NvY6hHMvvAMKSuMXgdkwn9TeGSCFdFpT9ntRphKret0m4sPO6 MBuaZaPLghHVnp8FBfGN0eGK02UpVFGtBw6TI= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type:content-transfer-encoding; b=NAOiOgeRdhFrq5Lwn7bg+/P5XE8eClPEK6KWM0oTLUtgizhGEEmw4iib6sE4XE5+rS 4kliGY8OIEqsCU8mrHJcSyNq1heflrxaLYor9OsJF+RfN3tkBARRiRdRzyWz3KpHz2mD zeKvGItLLFGHHBHwQMPpx6r2whmDHRH59QM7s= MIME-Version: 1.0 Received: by 10.103.175.9 with SMTP id c9mr4100382mup.3.1241494895966; Mon, 04 May 2009 20:41:35 -0700 (PDT) In-Reply-To: <200905010949.45927.jhb@freebsd.org> References: <200905010949.45927.jhb@freebsd.org> Date: Tue, 5 May 2009 07:41:35 +0400 Message-ID: From: pluknet To: John Baldwin Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Cc: freebsd-stable@freebsd.org Subject: Re: lock up in 6.2 (procs massively stuck in Giant) X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 05 May 2009 03:41:38 -0000 2009/5/1 John Baldwin : > On Thursday 30 April 2009 2:36:34 am pluknet wrote: >> Hi folks. >> >> Today I got a new locking issue. >> This is the first time I got it, and it's merely reproduced. >> >> The box has lost both remote connection and local access. >> No SIGINFO output on the local console even. >> Jumping in ddb> shows the next: >> >> 1) first, this is a 8-way web server. No processes on runqueue except one > httpd >> (i.e. ps shows R in its state): > > You need to find who owns Giant and what that thread is doing. You can try > using 'show lock Giant' as well as 'show lockchain 11568'. > Hi, John! Just reproduced now on another box. Hmm.. stack of the process owing Giant looks garbled. db> show lock Giant class: sleep mutex name: Giant flags: {DEF, RECURSE} state: {OWNED, CONTESTED} owner: 0xd0d79320 (tid 102754, pid 34594, "httpd") db> show lockchain 34594 thread 102754 (pid 34594, httpd) running on CPU 7 db> show lockchain 102754 thread 102754 (pid 34594, httpd) running on CPU 7 db> bt 102754 Tracing pid 34594 tid 102754 td 0xd0d79320 sched_switch(2,2,1,f1a3fb3c,c08a55d9,...) at sched_switch+0x143 MAXCPU(e5895590,62e85356,e8000110,ffffa3d5,ffb988e8,...) at 0xf7 *** error reading from address f8658d94 *** What can I do else? db> show allpcpu Current CPU: 1 cpuid = 0 curthread = 0xcd963960: pid 95678 "httpd" curpcb = 0xf0593d90 fpcurthread = none idlethread = 0xc7cfeaf0: pid 17 "idle: cpu0" APIC ID = 0 currentldt = 0x50 cpuid = 1 curthread = 0xc7dfaaf0: pid 40 "swi0: sio" curpcb = 0xe82ebd90 fpcurthread = none idlethread = 0xc7cfe000: pid 16 "idle: cpu1" APIC ID = 1 currentldt = 0x50 cpuid = 2 curthread = 0xca8aa640: pid 31167 "httpd" curpcb = 0xf1279d90 fpcurthread = none idlethread = 0xc7cfde10: pid 15 "idle: cpu2" APIC ID = 2 currentldt = 0x50 cpuid = 3 curthread = 0xc8b62e10: pid 17221 "httpd" curpcb = 0xee951d90 fpcurthread = none idlethread = 0xc7cfdc80: pid 14 "idle: cpu3" APIC ID = 3 currentldt = 0x50 cpuid = 4 curthread = 0xca1f2c80: pid 39078 "httpd" curpcb = 0xeec17d90 fpcurthread = none idlethread = 0xc7cfdaf0: pid 13 "idle: cpu4" APIC ID = 4 currentldt = 0x50 cpuid = 5 curthread = 0xcd423af0: pid 74960 "httpd" curpcb = 0xf03f2d90 fpcurthread = none idlethread = 0xc7cfd960: pid 12 "idle: cpu5" APIC ID = 5 currentldt = 0x50 cpuid = 6 curthread = 0xcaa89c80: pid 23426 "httpd" curpcb = 0xef1aed90 fpcurthread = none idlethread = 0xc7cfd7d0: pid 11 "idle: cpu6" APIC ID = 6 currentldt = 0x50 cpuid = 7 curthread = 0xd0d79320: pid 34594 "httpd" curpcb = 0xf1a3fd90 fpcurthread = none idlethread = 0xc7cfd640: pid 10 "idle: cpu7" APIC ID = 7 currentldt = 0x50 -- wbr, pluknet