From owner-freebsd-net@FreeBSD.ORG  Thu Aug 22 00:40:53 2013
Return-Path: <owner-freebsd-net@FreeBSD.ORG>
Delivered-To: freebsd-net@freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org
 [IPv6:2001:1900:2254:206a::19:1])
 (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits))
 (No client certificate requested)
 by hub.freebsd.org (Postfix) with ESMTP id CB8401AB;
 Thu, 22 Aug 2013 00:40:53 +0000 (UTC)
 (envelope-from markjdb@gmail.com)
Received: from mail-qc0-x232.google.com (mail-qc0-x232.google.com
 [IPv6:2607:f8b0:400d:c01::232])
 (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits))
 (No client certificate requested)
 by mx1.freebsd.org (Postfix) with ESMTPS id 7A15A2A13;
 Thu, 22 Aug 2013 00:40:53 +0000 (UTC)
Received: by mail-qc0-f178.google.com with SMTP id s1so649959qcw.37
 for <multiple recipients>; Wed, 21 Aug 2013 17:40:52 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113;
 h=sender:date:from:to:cc:subject:message-id:references:mime-version
 :content-type:content-disposition:in-reply-to:user-agent;
 bh=kTIvIZ51MixjWaQ/QoOqOAnakrsvo5xoAsbMOBi7r6k=;
 b=k5V47JUBbsoFmdonpf2vib70rMXbmN0YaUvbHi3vPwsdUtvYlWJFbQ8VrvGH+LYkQ9
 cYb+gGxk7F+UUpbYxwufYKzqMBgcr/vDyqc2l4b00SUXEUtTS15AK3/cXxDDzAbgYAjm
 4QFSlTNeczqJa0hf9dto/0XCh+Leu6FwMfGPyV1UmXwQMSykse3my2VCdv9OXsLIEzBj
 ovj+d36Uvv681C3EiiVLIT4ok5Ez+np9QxW1n8QJrxR7eEMNIo4lPU8RcOa6yy7G02pc
 A+uqw4jkK+UCeh56m0r7SRW9kbGnmduczMgjk8mSKtNc+djDAyoIVkI1wyNG3JtwHavG
 tPAQ==
X-Received: by 10.224.3.197 with SMTP id 5mr14289860qao.25.1377132052667;
 Wed, 21 Aug 2013 17:40:52 -0700 (PDT)
Received: from raichu (24-212-218-13.cable.teksavvy.com. [24.212.218.13])
 by mx.google.com with ESMTPSA id m10sm14648160qae.12.1969.12.31.16.00.00
 (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128);
 Wed, 21 Aug 2013 17:40:51 -0700 (PDT)
Sender: Mark Johnston <markjdb@gmail.com>
Date: Wed, 21 Aug 2013 20:40:29 -0400
From: Mark Johnston <markj@freebsd.org>
To: Brendan Gregg <brendan.gregg@joyent.com>
Subject: Re: DTrace network providers
Message-ID: <20130822004029.GA49869@raichu>
References: <20130821045926.GA17196@raichu>
 <CA+XzFFhw9beGi8=7heCY07dExvno=oMBYgzUrpht2LYXSbutqg@mail.gmail.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <CA+XzFFhw9beGi8=7heCY07dExvno=oMBYgzUrpht2LYXSbutqg@mail.gmail.com>
User-Agent: Mutt/1.5.21 (2010-09-15)
Cc: freebsd-net@freebsd.org, freebsd-dtrace@freebsd.org
X-BeenThere: freebsd-net@freebsd.org
X-Mailman-Version: 2.1.14
Precedence: list
List-Id: Networking and TCP/IP with FreeBSD <freebsd-net.freebsd.org>
List-Unsubscribe: <http://lists.freebsd.org/mailman/options/freebsd-net>,
 <mailto:freebsd-net-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-net>
List-Post: <mailto:freebsd-net@freebsd.org>
List-Help: <mailto:freebsd-net-request@freebsd.org?subject=help>
List-Subscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-net>,
 <mailto:freebsd-net-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Thu, 22 Aug 2013 00:40:54 -0000

On Tue, Aug 20, 2013 at 11:44:56PM -0700, Brendan Gregg wrote:
> G'Day,
> 
> On Tue, Aug 20, 2013 at 10:00 PM, Mark Johnston <markj@freebsd.org> wrote:
> 
> > Hello!
> >
> > I've ported the ip, tcp and udp DTrace providers to FreeBSD, following
> > the Solaris documentation here:
> >
> > https://wikis.oracle.com/display/DTrace/ip+Provider
> > https://wikis.oracle.com/display/DTrace/tcp+Provider
> > https://wikis.oracle.com/display/DTrace/udp+Provider
> >
> >
> Fantastic!
> 
> 
> > My implementation of these providers makes use of dynamic translators,
> > for which FreeBSD support was added in r254468; this patch won't compile
> > with earlier revisions. The use of dynamic translators means that
> > existing DTrace scripts which use these providers will just work when run
> > on FreeBSD - no modifications needed. In particular, all of the examples
> > in the links above will work properly on FreeBSD with my diff.
> >
> > I've collected a bunch of example scripts for these providers and placed
> > them here:
> >
> > http://people.freebsd.org/~markj/dtrace/network-providers/
> >
> >
> They look familiar. :-) Which is a good sign for this port, as that's
> what's supposed to happen: the same scripts run on any OS.

Yep - by "collect" I mostly meant "copied from the DTrace book." :)
They were very useful for testing. At first all the examples had to be
changed to use the xlate operator on arguments, but now everything just
works.

> 
> To run one you just need to execute "dtrace -s <script>".
> >
> 
> Or chmod 755 the ones with interpreter lines.
> 
> 
> > In general these providers make it trivial to monitor or count packets
> > and bytes on a per-host/port/protocol/interface basis. One can also do
> > neat things like watch TCP connection state transitions in real time
> > (tcpstate.d) and measure connection latency. All of the probes
> > correspond to logical events in their respective protocol
> > implementations; all of the providers have send and receive probes, and
> > the tcp provider has a few more.
> >
> > I didn't have to make any major changes to add support for these
> > providers, but I've made a few small tweaks:
> >
> > 1. Add a little helper function for TCP state changes instead of
> >    sprinkling a bunch of SDT_PROBE calls throughout the code.
> >
> 
> Makes sense.
> 
> 
> > 2. Set the IPv6 payload size in TCP and UDP before calling ip6_output().
> >    This is done for the send probes so that the ipinfo_t argument
> >    can be used to get the payload size. It's not quite correct since it
> >    doesn't include the length of IPv6 option headers (they aren't known
> >    yet), but I think that's ok for the purposes of these providers.
> >
> 
> Excluding the IP header length is correct for ipinfo_t->ip_plength, which
> is just supposed to be the payload length.
> 
> I see the patch makes an assumption about IPv4 header size being 20 bytes,
> instead of picking it from the header (IHL), like:
> 
>         ip_plength = ntohs(I->ipha_length) -
>             ((I->ipha_version_and_hdr_length & 0xf) << 2);

Ah, thanks for pointing this out! I've fixed this in the revision here:

http://people.freebsd.org/~markj/patches/network-providers/network-providers-2.diff

Now that field is obtained with:

ip_plength = p == NULL ? 0 :
    ((struct ip *)p)->ip_v == 4 ?
    ntohs(((struct ip *)p)->ip_len) - (((struct ip *)p)->ip_hl << 2) :
    ntohs(((struct ip6_hdr *)p)->ip6_ctlun.ip6_un1.ip6_un1_plen);

> 
> 3. Set the IPv4 header version in udp_output(). This is already done for
> >    IPv6.
> >
> > I'm hoping that none of this (or the rest of the diff) is controversial,
> > but I wanted to make sure, so any review would be really appreciated.
> >
> > The patch is here:
> >
> >
> > http://people.freebsd.org/~markj/patches/network-providers/network-providers-1.diff
> >
> > It depends on r254468. To use it, just recompile the kernel (assuming
> > that KDTRACE_HOOKS is enabled) and copy
> > $SRC/cddl/lib/libdtrace/{ip,tcp,udp}.d (added by the patch) to
> > /usr/lib/dtrace. Then the example scripts I linked above should just
> > work.
> >
> 
> Great work Mark!
> 
> Here's some (from many) one-liners that this should make possible (more in
> the DTrace book):
> 
> Who is connecting to what:
> dtrace -n 'tcp:::accept-established { @[args[3]->tcps_raddr,
> args[3]->tcps_lport] = count(); }'
> 
> Who isn't connecting to what:
> dtrace -n 'tcp:::accept-refused { @[args[2]->ip_daddr, args[4]->tcp_sport]
> = count(); }'
> 
> What am I connecting to?
> dtrace -n 'tcp:::connect-established { @[args[3]->tcps_raddr ,
> args[3]->tcps_rport] = count(); }'
> 
> IP payload bytes for TCP send, size distribution by destination address:
> dtrace -n 'tcp:::send { @[args[2]->ip_daddr] =
> quantize(args[2]->ip_plength); }'
> 
> Many of these are very low overhead ways to understand network behavior -
> lower than sniffing every packet. Plus you can pull out kernel state that
> is never seen on the wire.
> 
> Brendan
> 
> -- 
> Brendan Gregg, Joyent                      http://dtrace.org/blogs/brendan