Date: Thu, 31 Mar 2016 15:59:02 -0700 From: Mark Johnston <markj@FreeBSD.org> To: Aleksander Alekseev <mail@eax.me> Cc: Adrian Chadd <adrian.chadd@gmail.com>, Hans Petter Selasky <hps@selasky.org>, Andriy Voskoboinyk <s3erios@gmail.com>, freebsd-dtrace@freebsd.org, "freebsd-hackers@freebsd.org" <freebsd-hackers@freebsd.org> Subject: Re: I need a little help in fixing `exclusive sleep mutex urtwn0_com_lock` in CURRENT Message-ID: <20160331225902.GA32570@wkstn-mjohnston.west.isilon.com> In-Reply-To: <20160401002337.36700d9b@portege> References: <20160330123048.3361a9e4@fujitsu> <op.ye4x9fosiew4ia@localhost> <56FBBC62.6040905@selasky.org> <CAJ-Vmon=s-uKMhzgRm--DttXJ8_x_cAKnpALu4UeLOdQQ9nXpw@mail.gmail.com> <20160331204256.5cb1fdaf@portege> <20160331180333.GA25235@wkstn-mjohnston.west.isilon.com> <20160401002337.36700d9b@portege>
next in thread | previous in thread | raw e-mail | index | archive | help
On Fri, Apr 01, 2016 at 12:23:37AM +0300, Aleksander Alekseev wrote:
> > > By any chance is it possible to trace mtx_lock / mtx_unlock calls
> > > using DTrace? I see number of probes in `dtrace -l` which look like
> > > something I need. Unfortunately they are named like knlist_mtx_lock
> > > or do_lock_umutex so I can't figure out whether these are probes I
> > > looking for or not.
> >
> > Yes. You can use the lockstat provider to trace lock events:
> > "dtrace -l -P lockstat". Predicates can be used to isolate events
> > related to a specific lock. For example:
> >
> > # dtrace -n
> > 'lockstat:::adaptive-acquire /args[0]->lock_object.lo_name ==
> > "so_snd"/{stack();}'
> >
> > will print a stack every time a socket send buffer lock is acquired.
> > In general, the lock name is specified as an initialization parameter.
> > In your case it'll be "urtwn0_com_lock".
> >
> > lockstat(1) is a command-line program that's good at aggregating data
> > collected from lockstat probes; for the type of debugging you're doing
> > it's probably not very useful.
> >
> > Note that DTrace probably isn't very helpful here if the panic occurs
> > immediately after the event you're interested in occurs, since
> > dtrace(1) won't have time to retrieve the trace record and print it.
>
> Thanks, Mark.
>
> Here is a funny thing. I thought that by writing debug.witness.watch=0
> to /etc/sysctl.conf and rebooting I will give DTrace some time to
> report who acquired a lock and didn't released it. But in this case
> kernel crashed differently (I checked twice):
>
> http://pastebin.com/raw/f08Fd70B
>
> And here is saved DTrace log for lock "urtwn0_com_lock":
>
> http://pastebin.com/raw/0X9zgeiY
>
> If I'm not wrong everything is OK. No missing mtx_unlock this time.
Looking at your original post, the problem isn't related to a missing
unlock. The attempt to lock the ieee80211com is triggering a fault,
presumably because it was freed while the lock was dropped. So DTrace
isn't going to help much here.
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20160331225902.GA32570>