From owner-freebsd-questions@FreeBSD.ORG Tue Nov 22 22:01:58 2011 Return-Path: Delivered-To: freebsd-questions@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 484D31065672 for ; Tue, 22 Nov 2011 22:01:58 +0000 (UTC) (envelope-from m.seaman@infracaninophile.co.uk) Received: from smtp.infracaninophile.co.uk (smtp6.infracaninophile.co.uk [IPv6:2001:8b0:151:1:3cd3:cd67:fafa:3d78]) by mx1.freebsd.org (Postfix) with ESMTP id 9D5AE8FC1F for ; Tue, 22 Nov 2011 22:01:57 +0000 (UTC) Received: from seedling.black-earth.co.uk (seedling.black-earth.co.uk [81.187.76.163]) (authenticated bits=0) by smtp.infracaninophile.co.uk (8.14.5/8.14.5) with ESMTP id pAMM1rox081563 (version=TLSv1/SSLv3 cipher=DHE-RSA-CAMELLIA256-SHA bits=256 verify=NO) for ; Tue, 22 Nov 2011 22:01:53 GMT (envelope-from m.seaman@infracaninophile.co.uk) X-DKIM: OpenDKIM Filter v2.4.1 smtp.infracaninophile.co.uk pAMM1rox081563 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=infracaninophile.co.uk; s=201001-infracaninophile; t=1321999313; bh=KUmFJWm1JeUeg74rKTLXx254txjQs8AH9uP7CILGSJg=; h=Message-ID:Date:From:MIME-Version:To:Subject:References: In-Reply-To:Content-Type:Cc; b=AKKWWOVNYpJrjGYixtNnX98AvFeWUZ5p7gv9b6AA0efVRgDGcU9wLetbwkaLbFsLV M74bZjU2cNw1idCd/rhCwUbMq05LqIbc4uP/LORH/k9C4H/GkMQxCgmlfk9I0CceOr ZqVeHr7IDL6AIWsOdcoJFfkzpNYrs3C4V1IH8njc= Message-ID: <4ECC1BC9.7060007@infracaninophile.co.uk> Date: Tue, 22 Nov 2011 22:01:45 +0000 From: Matthew Seaman User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:8.0) Gecko/20111105 Thunderbird/8.0 MIME-Version: 1.0 To: freebsd-questions@freebsd.org References: In-Reply-To: X-Enigmail-Version: 1.3.3 OpenPGP: id=60AE908C Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="------------enig88A62592967565C69995833A" X-Virus-Scanned: clamav-milter 0.97.3 at lucid-nonsense.infracaninophile.co.uk X-Virus-Status: Clean X-Spam-Status: No, score=-0.1 required=5.0 tests=BAYES_20,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,SPF_FAIL autolearn=no version=3.3.2 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on lucid-nonsense.infracaninophile.co.uk Subject: Re: Diagnosing packet loss X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 22 Nov 2011 22:01:58 -0000 This is an OpenPGP/MIME signed message (RFC 2440 and 3156) --------------enig88A62592967565C69995833A Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable On 22/11/2011 20:33, Kees Jan Koster wrote: > I am stuck with a machine that shows serious packet loss (about 1% of > all traffic is dropped). I tried the obvious (new network cable, > different switch port, different ethernet interface on the machine), > but the problems remain. >=20 > Another machine that sits in the same rack and is hooked up to the > same switch shows no such packet loss issues. The problematic machine > is a dual Opteron with FreeBSD 8.2-STABLE from Thu Aug 11 14:05:47 > CEST 2011. >=20 > The machine is lightly loaded. A MySQL slave is running, but the > machine is not serving queries. Plus a Munin server process. >=20 > I am at a loss where to start diagnosing this. Can you advise me > where to look? Are there network buffers that may be overflowing? You say "lightly loaded," but how much does that actually equate to in kb/s or Mb/s? I'd call anything less than about 1Mb/s on a GB ethernet link pretty light, but other people have different ideas. Check for duplex mismatch -- normally everything just works allowing the NIC and the switch to autonegotiate, but every so often some bright spark gets the idea that wiring down the speed setting is a good idea. Trouble is you have to set *both* ends of the ethernet link to the same settings -- if one end is trying to auto and the other is fixed, you'll end up with the auto end defaulting to 100baseTX half-duplex and performance will suck, and suck increasingly hard as network load goes up. Amazing how often that 'set both ends the same' thing leads to grief= =2E Another hideously embarrassing error would be to spend ages debugging before finding out you had a duplicate IP number on your network. Can you definitely rule that out? A third networking problem that also has the potential to make you the butt of a few jokes is if your network cables are kinked, crushed, over stretched or simply cable-tied too tightly. Anything like that can cause signal leakage between the pairs of conductors in the cable which can be enough to disrupt packet transmission. Simply snipping through a too-tight cable tie can have a magical-seeming effect. What sort on NICs are there on your machine? It's well known that re(4) interfaces simply cannot keep up with the throughput of a good server NIC like em(4) or bge(4). [But re(4)'s are cheap and good enough for most home systems...] If you can try swapping in a reasonably good NIC card -- beg, borrow or steal from another machine just for a few hours to use for testing -- and see if that cures the problem. Other considerations: are you doing anything beyond just plain ethernet networking? Any VLANS? What about ipsec or other tunnelled/encapsulated traffic? Are you using RSTP or lagg to make your networking resilient to failures? If the answer to any of these is "yes" -- does temporarily disabling that feature and doing it simple and stupid help with the packet loss? Do you get the same sort of packet loss if you take the switch away and just run a cable direct between two machines. (Nb. If your NICs don't support MSIx you'll need a crossover cable.) On another host on your net, can you use wireshark to capture and examine the traffic from your failing machine? For best results, either wire the two machines directly together or configure the switch port your wireshark box is connected to as a /monitor/ port so it sees all the traffic coming out of your problem box. Does your NIC have hardware checksumming? If so, does disabling that help with the error rate? (see ifconfig(8) and the man page for your NIC in section 4 for how.) There have been a number of instances of buggy checksumming causing problems in the past. Nb. with hardware checksumming, the checksum field is calculated and inserted in packets very late; after any way of examining the packets as they leave your machine has ceased to be possible. Makes it look like the checksums are all wrong if you sample the traffic on the originating machine. This is why you need to use another, external machine to watch for this sort of error. Cheers, Matthew --=20 Dr Matthew J Seaman MA, D.Phil. 7 Priory Courtyard Flat 3 PGP: http://www.infracaninophile.co.uk/pgpkey Ramsgate JID: matthew@infracaninophile.co.uk Kent, CT11 9PW --------------enig88A62592967565C69995833A Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG/MacGPG2 v2.0.16 (Darwin) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk7MG9EACgkQ8Mjk52CukIypbACdEMm/uOze6c/66UfuJmFG+uLh kb8AnRQ9IGvHaP53pQl5dmyG1rH8NL2B =KbEg -----END PGP SIGNATURE----- --------------enig88A62592967565C69995833A--