From owner-freebsd-net@freebsd.org Sat Jul 23 19:21:59 2016 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id ABBB3BA2518 for ; Sat, 23 Jul 2016 19:21:59 +0000 (UTC) (envelope-from hiren@strugglingcoder.info) Received: from mail.strugglingcoder.info (strugglingcoder.info [104.236.146.68]) by mx1.freebsd.org (Postfix) with ESMTP id 9A05A1E00 for ; Sat, 23 Jul 2016 19:21:59 +0000 (UTC) (envelope-from hiren@strugglingcoder.info) Received: from localhost (unknown [10.1.1.3]) (Authenticated sender: hiren@strugglingcoder.info) by mail.strugglingcoder.info (Postfix) with ESMTPA id 501D517B37; Sat, 23 Jul 2016 12:21:53 -0700 (PDT) Date: Sat, 23 Jul 2016 12:21:53 -0700 From: hiren panchasara To: Kevin Oberman Cc: Chris Dunbar , freebsd-net Subject: Re: Slow performance with Intel X540-T2 10Gb NIC Message-ID: <20160723192153.GB94850@strugglingcoder.info> References: <1244557023.708807.1469061382192.JavaMail.zimbra@dunbar.net> <1441424852.712842.1469134420198.JavaMail.zimbra@dunbar.net> <183608784.713013.1469136611853.JavaMail.zimbra@dunbar.net> <144391790.714645.1469195320062.JavaMail.zimbra@dunbar.net> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="0eh6TmSyL6TZE2Uz" Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.23 (2014-03-12) X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 23 Jul 2016 19:21:59 -0000 --0eh6TmSyL6TZE2Uz Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On 07/22/16 at 11:23P, Kevin Oberman wrote: [skip] >=20 > This sort of problem can be very tricky to diagnose. I'd like to suggest > that one of the tool you use should be SIFTR. It does kernel level > collection of network statistics and is a loadable module. By default it i > IPv4 only. It will have to be re-built with "CFLAGS+=3D-DSIFTR_IPV6" > uncommented in /sys/modules/siftr/Makefile for IPv6 support. It starts, > stops, amd manages collection under the control of 4 sysctls. >=20 > I have found it invaluable for analysis of netork performance issues, but > seems to not be widely known. Another such tool which I (personally) find more powerful and less known is dtrace in this context. For example, in output direction, right when tcp is about to send a packet to ip, there is a dtrace trace point which you can use to get a ton of useful information: # dtrace -n 'tcp:::send / args[2]->ip_saddr =3D=3D "192.168.0.1" / {printf = ("%8u", args[3]->tcps_mss)}' This can let me see MSS for that out going packet. You can see all of tcp control block data this way. The mapping is in /usr/lib/dtrace/tcp.d file. You can always add whatever you want in there and use it. https://github.com/brendangregg/DTrace-book-scripts/blob/master/Chap6/tcpio= =2Ed is one such awesome script that you can modify to match your needs and look at whatever bidirectional traffic in detail. With dtrace predicates, you can filter out what you want to log which is not possible yet with siftr. This makes running siftr a little annoying on a busy box as it logs all the things and you have to do post processing to get what you want to see. Just my 2 rupees. Cheers, Hiren --0eh6TmSyL6TZE2Uz Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQF8BAABCgBmBQJXk8PNXxSAAAAAAC4AKGlzc3Vlci1mcHJAbm90YXRpb25zLm9w ZW5wZ3AuZmlmdGhob3JzZW1hbi5uZXRBNEUyMEZBMUQ4Nzg4RjNGMTdFNjZGMDI4 QjkyNTBFMTU2M0VERkU1AAoJEIuSUOFWPt/lNzIIAKTiYKKySQkVvqZcR08OF/Ei 4f+x+CCiUbCCyG5VK410AYiVw8MB1gHJbPiKj7qNq+34Tf2BIJo0tIJZ8+nEnRbV EsmEFaYv8hTmwldQm60TPKMHyeJOpuY4L1s+bJ8MEmshJCC5tgruHglMmM1gXCgB +v/zmGa2LaVloiLD1JAXBqpkHH2/ue/Or8GgL0vUTham0APqZBXfRSTaUF6GKPo/ ibrXSvLOJwuChFDMzBptuiwdo1fviwthQtnH13d+n8jxKb7uEAiddyh7tOqhh3qS yS/We8RPRjAdWFV17dMEpvTmcorspMn6y5aYtV9E389mdYCqmFQlCM8heacaFBM= =cfTd -----END PGP SIGNATURE----- --0eh6TmSyL6TZE2Uz--