From owner-freebsd-hackers@freebsd.org Thu Mar 31 18:00:30 2016 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 26FE5AE3FF6; Thu, 31 Mar 2016 18:00:30 +0000 (UTC) (envelope-from markjdb@gmail.com) Received: from mail-pa0-x229.google.com (mail-pa0-x229.google.com [IPv6:2607:f8b0:400e:c03::229]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id E9FCD18AC; Thu, 31 Mar 2016 18:00:29 +0000 (UTC) (envelope-from markjdb@gmail.com) Received: by mail-pa0-x229.google.com with SMTP id fe3so71266638pab.1; Thu, 31 Mar 2016 11:00:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=sender:date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to:user-agent; bh=8FZTPvShhj+3LK7LHN9fNMQK6zadYVqSOepvY9OlhkE=; b=0Rvk+i5fTf4txXNnjnlLNg+0edxdsk7YiAhFwnpH8zZYY3omCqFi/B+5CLSK5awtOS buRH1zawhPg/IaH8JMC8XSaefxp7BNkWyIDmUNthT3HuLynAtRWsJ+U/0sXSemArAdc1 63ZBIo0CVwxI53r9MiN8YIr40IbOHmJO2BCBVWx1V5eZ3MiySCaMgdzq45wVV+5yM7zC wbcZLFHesj4b2fpqtM8Xz4Mevww2pcHHbowan9yynXZv5ed/MRi1vnPH6qihdCqKOZ/L 1QAkpRHY0H0qii2Y7lHtYQZbgznQ3qFO2vuGPRKw4JkipBpiU+CWPq+YRip44kxIeW8G khmg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:sender:date:from:to:cc:subject:message-id :references:mime-version:content-disposition:in-reply-to:user-agent; bh=8FZTPvShhj+3LK7LHN9fNMQK6zadYVqSOepvY9OlhkE=; b=Wd2qO8hiuoAZbD5YrCT+39/Dwhm+yTxnw0n1yw+oTaKpsnJ9JOzQMXF8gyljwhgi8c T9rU2Hf0UK1UJUiCkqdb2IaVoWjFeg8/MyLYrEna+XcCJ1cXKXi9U12QGU04FR5o2qq/ 5ZRZ84Gwe8+cdYe1utXjTNah3y/kTvbEaWkmBWbZL2EjWFAwggtbesjk5690/oZShVC7 5T37UI6PBd4+j8/VL2G69O6MBRrXey/76DgyHygj8HBohhusNqyXNKdjDvXb42mO1QIG GmnkJe+tVPdWuHSoSIlnTpnUeMFBd/SF7IOmp//Zka3bvdzOcdRCFEKpdwIpSh3eC8is TOig== X-Gm-Message-State: AD7BkJJ/LS4fY+t/pfEJ1Svk5LSE3XCB1wYzViTqNG6TxaH3ps+I9vB9Noff+FEPcKwAUQ== X-Received: by 10.67.30.163 with SMTP id kf3mr24023412pad.45.1459447229402; Thu, 31 Mar 2016 11:00:29 -0700 (PDT) Received: from wkstn-mjohnston.west.isilon.com (c-67-182-131-225.hsd1.wa.comcast.net. [67.182.131.225]) by smtp.gmail.com with ESMTPSA id e12sm14976613pat.19.2016.03.31.11.00.25 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 31 Mar 2016 11:00:28 -0700 (PDT) Sender: Mark Johnston Date: Thu, 31 Mar 2016 11:03:33 -0700 From: Mark Johnston To: Aleksander Alekseev Cc: Adrian Chadd , Hans Petter Selasky , "freebsd-hackers@freebsd.org" , Andriy Voskoboinyk , freebsd-dtrace@freebsd.org Subject: Re: I need a little help in fixing `exclusive sleep mutex urtwn0_com_lock` in CURRENT Message-ID: <20160331180333.GA25235@wkstn-mjohnston.west.isilon.com> References: <20160330123048.3361a9e4@fujitsu> <56FBBC62.6040905@selasky.org> <20160331204256.5cb1fdaf@portege> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20160331204256.5cb1fdaf@portege> User-Agent: Mutt/1.5.24 (2015-08-30) X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 31 Mar 2016 18:00:30 -0000 On Thu, Mar 31, 2016 at 08:42:56PM +0300, Aleksander Alekseev wrote: > > So as I understand some code acquired a mutex and didn't release it. > > And naturally it has something to do with USB. Now how can I figure > > out what code acquired this lock? Best approach I can think of > > currently is to find all places where this lock is accessed and add a > > debug logging. But perhaps there is a better way? > > By any chance is it possible to trace mtx_lock / mtx_unlock calls using > DTrace? I see number of probes in `dtrace -l` which look like > something I need. Unfortunately they are named like knlist_mtx_lock or > do_lock_umutex so I can't figure out whether these are probes I looking > for or not. Yes. You can use the lockstat provider to trace lock events: "dtrace -l -P lockstat". Predicates can be used to isolate events related to a specific lock. For example: # dtrace -n 'lockstat:::adaptive-acquire /args[0]->lock_object.lo_name == "so_snd"/{stack();}' will print a stack every time a socket send buffer lock is acquired. In general, the lock name is specified as an initialization parameter. In your case it'll be "urtwn0_com_lock". lockstat(1) is a command-line program that's good at aggregating data collected from lockstat probes; for the type of debugging you're doing it's probably not very useful. Note that DTrace probably isn't very helpful here if the panic occurs immediately after the event you're interested in occurs, since dtrace(1) won't have time to retrieve the trace record and print it.