Date: Fri, 1 Apr 2016 00:23:37 +0300 From: Aleksander Alekseev <mail@eax.me> To: Mark Johnston <markj@FreeBSD.org> Cc: Adrian Chadd <adrian.chadd@gmail.com>, Hans Petter Selasky <hps@selasky.org>, Andriy Voskoboinyk <s3erios@gmail.com>, freebsd-dtrace@freebsd.org, "freebsd-hackers@freebsd.org" <freebsd-hackers@freebsd.org> Subject: Re: I need a little help in fixing `exclusive sleep mutex urtwn0_com_lock` in CURRENT Message-ID: <20160401002337.36700d9b@portege> In-Reply-To: <20160331180333.GA25235@wkstn-mjohnston.west.isilon.com> References: <20160330123048.3361a9e4@fujitsu> <op.ye4x9fosiew4ia@localhost> <56FBBC62.6040905@selasky.org> <CAJ-Vmon=s-uKMhzgRm--DttXJ8_x_cAKnpALu4UeLOdQQ9nXpw@mail.gmail.com> <20160331204256.5cb1fdaf@portege> <20160331180333.GA25235@wkstn-mjohnston.west.isilon.com>
next in thread | previous in thread | raw e-mail | index | archive | help
> > By any chance is it possible to trace mtx_lock / mtx_unlock calls
> > using DTrace? I see number of probes in `dtrace -l` which look like
> > something I need. Unfortunately they are named like knlist_mtx_lock
> > or do_lock_umutex so I can't figure out whether these are probes I
> > looking for or not.
>
> Yes. You can use the lockstat provider to trace lock events:
> "dtrace -l -P lockstat". Predicates can be used to isolate events
> related to a specific lock. For example:
>
> # dtrace -n
> 'lockstat:::adaptive-acquire /args[0]->lock_object.lo_name ==
> "so_snd"/{stack();}'
>
> will print a stack every time a socket send buffer lock is acquired.
> In general, the lock name is specified as an initialization parameter.
> In your case it'll be "urtwn0_com_lock".
>
> lockstat(1) is a command-line program that's good at aggregating data
> collected from lockstat probes; for the type of debugging you're doing
> it's probably not very useful.
>
> Note that DTrace probably isn't very helpful here if the panic occurs
> immediately after the event you're interested in occurs, since
> dtrace(1) won't have time to retrieve the trace record and print it.
Thanks, Mark.
Here is a funny thing. I thought that by writing debug.witness.watch=0
to /etc/sysctl.conf and rebooting I will give DTrace some time to
report who acquired a lock and didn't released it. But in this case
kernel crashed differently (I checked twice):
http://pastebin.com/raw/f08Fd70B
And here is saved DTrace log for lock "urtwn0_com_lock":
http://pastebin.com/raw/0X9zgeiY
If I'm not wrong everything is OK. No missing mtx_unlock this time.
Not sure what to do next so far. And it's pretty late here in UTC+3
already so I will think about this tomorrow. Any advices would be
appreciated.
--
Best regards,
Aleksander Alekseev
http://eax.me/
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20160401002337.36700d9b>