From owner-svn-src-all@freebsd.org Thu Jan 21 01:25:55 2016 Return-Path: Delivered-To: svn-src-all@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 9817FA8A2F9; Thu, 21 Jan 2016 01:25:55 +0000 (UTC) (envelope-from markjdb@gmail.com) Received: from mail-pf0-x229.google.com (mail-pf0-x229.google.com [IPv6:2607:f8b0:400e:c00::229]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 6622A1A44; Thu, 21 Jan 2016 01:25:55 +0000 (UTC) (envelope-from markjdb@gmail.com) Received: by mail-pf0-x229.google.com with SMTP id e65so13986387pfe.0; Wed, 20 Jan 2016 17:25:55 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=sender:date:from:to:cc:subject:message-id:references:mime-version :content-type:content-disposition:in-reply-to:user-agent; bh=IAUPjbJNLErpjGWbJEeYLNxlxnmfzUZuT4rd/Q1sAM4=; b=Ly7C2RbU7E1L1AnxEHhx57vBfkNJOye7GTB0MkZkIEuXeGmTP7MEr2NfPQJgGuY6z4 qUfwbg4IXBH3BaY3ryCk6QSxAiGU4hZmGuPRNPTzqocbsDbZPoeFSmicUlOnT4Vg1yeU uMfaes9uStuP1L/hLAJjryWZBrrD/bAuHBdMpqcSMslQ0XUL5ul7NFYrefWBzHy2M3mt NxHMdVH1PJhPmZDTkrrF+pZzIKzJUTMWJ+A+MKWB9JT0xf7FtR+veZWiVxmHIepGW/b+ s45fcib9FIdzu6q+1cQ0NxKtG+9dbV2lO5oJ3/wIfqMqz3LS1fIV+VtXjx0+Ltu61oZY /1RQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:sender:date:from:to:cc:subject:message-id :references:mime-version:content-type:content-disposition :in-reply-to:user-agent; bh=IAUPjbJNLErpjGWbJEeYLNxlxnmfzUZuT4rd/Q1sAM4=; b=XyzkMd+1egCTjHsceTXyqCWkuM/l9MP/AeTNplQKHUtFq0ZRL3VxhJfsu0cbTyhI/4 elpDgdSQAweuZXvZl2oFylaxK2mqUSR4+QeCsKm9BwcgYpfCsMTwHAoYvO/ZpHVDkWzs i5M6pa6xIyKOA/PpXEOAk4G4PkmxnmJtr6oJgvSyZsvkgO2Y2Y/dA3HQqyynhwOiSc2O l7UKyMACoQXDnV3CKEJf0vAxmR0KQNmdGmH7MaZYprhKeXr1r3v28AX9HsSrZ3mIFKvZ /s7mIqGIrjZxBt8Yv9pd0DqbgmWCT4R8OUU2Z7aXrLbNDfK4o9LZV68WdSlIZDQ5wJ8O Dnng== X-Gm-Message-State: ALoCoQllxKGGYeWss/T9kJ78J7tEXxNY45kSwCwt5vXBw1z2OvWC3Kfvh1NDYIXiXv0tVbh9n28JoWBsGz5jh0+RZgGZft+omQ== X-Received: by 10.98.14.29 with SMTP id w29mr56365579pfi.160.1453339555072; Wed, 20 Jan 2016 17:25:55 -0800 (PST) Received: from wkstn-mjohnston.west.isilon.com (c-67-182-131-225.hsd1.wa.comcast.net. [67.182.131.225]) by smtp.gmail.com with ESMTPSA id 76sm45773695pfl.36.2016.01.20.17.25.54 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 20 Jan 2016 17:25:54 -0800 (PST) Sender: Mark Johnston Date: Wed, 20 Jan 2016 17:28:25 -0800 From: Mark Johnston To: Ravi Pokala Cc: Adrian Chadd , Andriy Voskoboinyk , "src-committers@freebsd.org" , "svn-src-all@freebsd.org" , "svn-src-head@freebsd.org" Subject: Re: svn commit: r294471 - head/sys/dev/usb/wlan Message-ID: <20160121012825.GA9303@wkstn-mjohnston.west.isilon.com> References: <201601202327.u0KNR2Hh066219@repo.freebsd.org> <4FBE9133-AB30-4D30-A298-1742952700C5@panasas.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <4FBE9133-AB30-4D30-A298-1742952700C5@panasas.com> User-Agent: Mutt/1.5.24 (2015-08-30) X-BeenThere: svn-src-all@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: "SVN commit messages for the entire src tree \(except for " user" and " projects" \)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 21 Jan 2016 01:25:55 -0000 On Wed, Jan 20, 2016 at 04:15:54PM -0800, Ravi Pokala wrote: > -----Original Message----- > > > From: on behalf of Adrian Chadd > Date: 2016-01-20, Wednesday at 16:02 > To: Andriy Voskoboinyk > Cc: "src-committers@freebsd.org" , "svn-src-all@freebsd.org" , "svn-src-head@freebsd.org" > Subject: Re: svn commit: r294471 - head/sys/dev/usb/wlan > > >Hi, > > > >So yeah, I think it's time we just bit the bullet and wrote a > >generic-ish debug/ktr framework for drivers to use so drivers and > >infrastructure doesn't keep spinning its own damned debugging stuff. > > > >(I know people keep saying "dtrace", but ...) > > > >Let's have a think about it. > > One issue I have with DTrace (and KTR, though I'm even less familiar with it than I am with DTrace) is that you only get a stream of what's happening right now - there's no way to see *what just happened*. For example, I wrote an ATA command tracing infrastructure for Panasas which keeps the last several thousand commands in a trace buffer, and a tool that extracts the buffer and saves it to a file. What you describe sounds like it could be accomplished with KTR. All KTR does is maintain a ring of buffers, each containing a static format string with optional arguments, as well as some metadata (CPU, thread, timecounter value). One feature that would be nice would be a way to associate KTR rings with arbitrary objects rather than having a single global ring (or per-CPU rings). For example, KTR_BUF lets one trace operations on filesystem buffers, but there's no good way to find all trace records corresponding to a given buf. At the moment one has to always include a pointer to the buf, and then scan *all* KTR entries for the pointer of interest while hoping that the relevant entries haven't already been overwritten. With per-object rings, one can in effect see a "history" of the object, whether it's a buf or a driver softc or whatever. Isilon has done ad-hoc implementations of this for bufs and mbufs, and they're quite handy for debugging. With BUF_TRACKING enabled in our kernel config, each buf contains a const char *b_records[32], and one adds buf_track(bp, __func__); or so to various functions to record an entry in the buf when the function is invoked. This is really similar to what KTR does already, and I've needed to further hack up our buf tracking to include metadata (thread IDs) that KTR already provides. And, KTR records can be retrieved from vmcores by ktrdump -M. > We can then feed that file into various analysis tools, which can show us exactly what we sent to the drive, and exactly what the drive sent back to us. (The nice thing is, the buffer is also in the vmcore, so we can easily extract it after a panic, and it's identical to what you get from the live system, so the same analysis tools can be used in both instances.) This has come in very handy when we want to see, for example, what new and stupid thing a drive did that caused an error, slowdown or panic.