From owner-freebsd-hackers@freebsd.org Thu Mar 31 21:23:49 2016 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id F29ADAE38D0; Thu, 31 Mar 2016 21:23:48 +0000 (UTC) (envelope-from mail@eax.me) Received: from relay15.nicmail.ru (relay15.nicmail.ru [195.208.5.161]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id A6EC61873; Thu, 31 Mar 2016 21:23:47 +0000 (UTC) (envelope-from mail@eax.me) Received: from [109.70.25.187] (port=48124 helo=portege) by f19.mail.nic.ru with esmtp (Exim 5.55) (envelope-from ) id 1alk41-000AjB-CT; Fri, 01 Apr 2016 00:23:37 +0300 Received: from [188.123.231.37] (account mail@eax.me HELO portege) by proxy08.mail.nic.ru (Exim 5.55) with id 1alk41-0007vx-QO; Fri, 01 Apr 2016 00:23:37 +0300 Date: Fri, 1 Apr 2016 00:23:37 +0300 From: Aleksander Alekseev To: Mark Johnston Cc: Adrian Chadd , Hans Petter Selasky , Andriy Voskoboinyk , freebsd-dtrace@freebsd.org, "freebsd-hackers@freebsd.org" Subject: Re: I need a little help in fixing `exclusive sleep mutex urtwn0_com_lock` in CURRENT Message-ID: <20160401002337.36700d9b@portege> In-Reply-To: <20160331180333.GA25235@wkstn-mjohnston.west.isilon.com> References: <20160330123048.3361a9e4@fujitsu> <56FBBC62.6040905@selasky.org> <20160331204256.5cb1fdaf@portege> <20160331180333.GA25235@wkstn-mjohnston.west.isilon.com> MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 31 Mar 2016 21:23:49 -0000 > > By any chance is it possible to trace mtx_lock / mtx_unlock calls > > using DTrace? I see number of probes in `dtrace -l` which look like > > something I need. Unfortunately they are named like knlist_mtx_lock > > or do_lock_umutex so I can't figure out whether these are probes I > > looking for or not. > > Yes. You can use the lockstat provider to trace lock events: > "dtrace -l -P lockstat". Predicates can be used to isolate events > related to a specific lock. For example: > > # dtrace -n > 'lockstat:::adaptive-acquire /args[0]->lock_object.lo_name == > "so_snd"/{stack();}' > > will print a stack every time a socket send buffer lock is acquired. > In general, the lock name is specified as an initialization parameter. > In your case it'll be "urtwn0_com_lock". > > lockstat(1) is a command-line program that's good at aggregating data > collected from lockstat probes; for the type of debugging you're doing > it's probably not very useful. > > Note that DTrace probably isn't very helpful here if the panic occurs > immediately after the event you're interested in occurs, since > dtrace(1) won't have time to retrieve the trace record and print it. Thanks, Mark. Here is a funny thing. I thought that by writing debug.witness.watch=0 to /etc/sysctl.conf and rebooting I will give DTrace some time to report who acquired a lock and didn't released it. But in this case kernel crashed differently (I checked twice): http://pastebin.com/raw/f08Fd70B And here is saved DTrace log for lock "urtwn0_com_lock": http://pastebin.com/raw/0X9zgeiY If I'm not wrong everything is OK. No missing mtx_unlock this time. Not sure what to do next so far. And it's pretty late here in UTC+3 already so I will think about this tomorrow. Any advices would be appreciated. -- Best regards, Aleksander Alekseev http://eax.me/