Date: Sat, 23 Jul 2016 12:21:53 -0700 From: hiren panchasara <hiren@strugglingcoder.info> To: Kevin Oberman <rkoberman@gmail.com> Cc: Chris Dunbar <chris@dunbar.net>, freebsd-net <freebsd-net@freebsd.org> Subject: Re: Slow performance with Intel X540-T2 10Gb NIC Message-ID: <20160723192153.GB94850@strugglingcoder.info> In-Reply-To: <CAN6yY1tp29nUc2=18jEYkhgfMpQ0Rg=dB_OtzB0ZpSrXRgN_bQ@mail.gmail.com> References: <1244557023.708807.1469061382192.JavaMail.zimbra@dunbar.net> <CA%2Bb0zg-mXiDZzKmcomfLNxKbpb_R1F50k=vo%2B32sxQwLkRNGvg@mail.gmail.com> <1441424852.712842.1469134420198.JavaMail.zimbra@dunbar.net> <183608784.713013.1469136611853.JavaMail.zimbra@dunbar.net> <CAEW%2BogZwHKhXAb2ra2eq14UteUwzZLyEpPiLEywzaen0nCppbw@mail.gmail.com> <144391790.714645.1469195320062.JavaMail.zimbra@dunbar.net> <CAN6yY1tp29nUc2=18jEYkhgfMpQ0Rg=dB_OtzB0ZpSrXRgN_bQ@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
--0eh6TmSyL6TZE2Uz
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable
On 07/22/16 at 11:23P, Kevin Oberman wrote:
[skip]
>=20
> This sort of problem can be very tricky to diagnose. I'd like to suggest
> that one of the tool you use should be SIFTR. It does kernel level
> collection of network statistics and is a loadable module. By default it i
> IPv4 only. It will have to be re-built with "CFLAGS+=3D-DSIFTR_IPV6"
> uncommented in /sys/modules/siftr/Makefile for IPv6 support. It starts,
> stops, amd manages collection under the control of 4 sysctls.
>=20
> I have found it invaluable for analysis of netork performance issues, but
> seems to not be widely known.
Another such tool which I (personally) find more powerful and less known
is dtrace in this context.
For example, in output direction, right when tcp is about to send a
packet to ip, there is a dtrace trace point which you can use to get a
ton of useful information:
# dtrace -n 'tcp:::send / args[2]->ip_saddr =3D=3D "192.168.0.1" / {printf =
("%8u", args[3]->tcps_mss)}'
This can let me see MSS for that out going packet. You can see all of
tcp control block data this way. The mapping is in /usr/lib/dtrace/tcp.d
file. You can always add whatever you want in there and use it.
https://github.com/brendangregg/DTrace-book-scripts/blob/master/Chap6/tcpio=
=2Ed
is one such awesome script that you can modify to match your needs and
look at whatever bidirectional traffic in detail.
With dtrace predicates, you can filter out what you want to log which is
not possible yet with siftr. This makes running siftr a little annoying
on a busy box as it logs all the things and you have to do post
processing to get what you want to see.
Just my 2 rupees.
Cheers,
Hiren
--0eh6TmSyL6TZE2Uz
Content-Type: application/pgp-signature
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2
iQF8BAABCgBmBQJXk8PNXxSAAAAAAC4AKGlzc3Vlci1mcHJAbm90YXRpb25zLm9w
ZW5wZ3AuZmlmdGhob3JzZW1hbi5uZXRBNEUyMEZBMUQ4Nzg4RjNGMTdFNjZGMDI4
QjkyNTBFMTU2M0VERkU1AAoJEIuSUOFWPt/lNzIIAKTiYKKySQkVvqZcR08OF/Ei
4f+x+CCiUbCCyG5VK410AYiVw8MB1gHJbPiKj7qNq+34Tf2BIJo0tIJZ8+nEnRbV
EsmEFaYv8hTmwldQm60TPKMHyeJOpuY4L1s+bJ8MEmshJCC5tgruHglMmM1gXCgB
+v/zmGa2LaVloiLD1JAXBqpkHH2/ue/Or8GgL0vUTham0APqZBXfRSTaUF6GKPo/
ibrXSvLOJwuChFDMzBptuiwdo1fviwthQtnH13d+n8jxKb7uEAiddyh7tOqhh3qS
yS/We8RPRjAdWFV17dMEpvTmcorspMn6y5aYtV9E389mdYCqmFQlCM8heacaFBM=
=cfTd
-----END PGP SIGNATURE-----
--0eh6TmSyL6TZE2Uz--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20160723192153.GB94850>