From owner-freebsd-stable@FreeBSD.ORG Wed Nov 5 11:49:36 2014 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 3039DF7E for ; Wed, 5 Nov 2014 11:49:36 +0000 (UTC) Received: from smtp.infracaninophile.co.uk (smtp6.infracaninophile.co.uk [IPv6:2001:8b0:151:1:3cd3:cd67:fafa:3d78]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "smtp.infracaninophile.co.uk", Issuer "ca.infracaninophile.co.uk" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id C8B8EBDA for ; Wed, 5 Nov 2014 11:49:35 +0000 (UTC) Received: from ox-dell39.ox.adestra.com (no-reverse-dns.metronet-uk.com [85.199.232.226] (may be forged)) (authenticated bits=0) by smtp.infracaninophile.co.uk (8.14.9/8.14.9) with ESMTP id sA5BnLxD007487 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES128-SHA bits=128 verify=NO) for ; Wed, 5 Nov 2014 11:49:30 GMT (envelope-from matthew@freebsd.org) Authentication-Results: smtp.infracaninophile.co.uk; dmarc=none header.from=freebsd.org DKIM-Filter: OpenDKIM Filter v2.9.2 smtp.infracaninophile.co.uk sA5BnLxD007487 Authentication-Results: smtp.infracaninophile.co.uk/sA5BnLxD007487; dkim=none reason="no signature"; dkim-adsp=none; dkim-atps=neutral X-Authentication-Warning: lucid-nonsense.infracaninophile.co.uk: Host no-reverse-dns.metronet-uk.com [85.199.232.226] (may be forged) claimed to be ox-dell39.ox.adestra.com Message-ID: <545A0EB4.4090404@freebsd.org> Date: Wed, 05 Nov 2014 11:49:08 +0000 From: Matthew Seaman User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:31.0) Gecko/20100101 Thunderbird/31.2.0 MIME-Version: 1.0 To: freebsd-stable@freebsd.org Subject: Varnish proxy goes catatonic under heavy load Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="UfFkmOIO9brN0XPF6E1IRRTIJXEqhTMuH" X-Virus-Scanned: clamav-milter 0.98.4 at lucid-nonsense.infracaninophile.co.uk X-Virus-Status: Clean X-Spam-Status: No, score=-0.9 required=5.0 tests=AWL,BAYES_00,RDNS_NONE, SPF_SOFTFAIL autolearn=no autolearn_force=no version=3.4.0 X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on lucid-nonsense.infracaninophile.co.uk X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 05 Nov 2014 11:49:36 -0000 This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --UfFkmOIO9brN0XPF6E1IRRTIJXEqhTMuH Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Dear all, We had an unfortunate set of circumstances which resulted in several million people all trying to download about 1.5MB worth of images from our servers over the course of a few hours. Or, at least, it would have been a few hours, except that our three varnish proxies just crumbled under the load within 10 minutes. Now, that's bad enough, but we could have just about coped if the proxies stopped serving requests for a few minutes. What actually happened was that all three servers went catatonic on the network *and stayed that way*: even when we shunted the traffic away from one, we still couldn't access it via ssh or any network protocol. And it stayed like that for sufficiently long time that we had no recourse other than to get the servers rebooted. Can anyone explain what was happening here? Not having the servers recover accessibility for an extended period even after the excess traffic was stopped is unacceptable. We're also struggling to recreate the effect in the lab: any clues about how to do so, and any suggestions about how to prevent the 'going catatonic' response would be greatly appreciated. Servers are amd64 running FreeBSD 9.1 or 9.2 and Varnish 3.0.5. Cheers, Matthew --UfFkmOIO9brN0XPF6E1IRRTIJXEqhTMuH Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQJ8BAEBCgBmBQJUWg7AXxSAAAAAAC4AKGlzc3Vlci1mcHJAbm90YXRpb25zLm9w ZW5wZ3AuZmlmdGhob3JzZW1hbi5uZXQxOUYxNTRFQ0JGMTEyRTUwNTQ0RTNGMzAw MDUxM0YxMEUwQTlFNEU3AAoJEABRPxDgqeTnT6MP/iQ8borxlNTLRBbLouBz61FH YrBaK/EqaeTp3xSyx7QpVi4tGpfWgnliw27eXiifIX+GEFY5Pzige+uYtRdwdVHl L8iLDM/yx7mJVmqCv2d3BnU7T+RvPcN0BIsPRnIbbK6+dzxkJRELp4f2el1xHZFl ha3k1GxYHcpo3+4aRbJ4Q5J9BaM3uvhAY2kEbs/061L+1ov+J750t7qDALsNBUBR a7Wc79UndpRZhHQKErXlSZK/NiF4VwhuQfRaFISq1firiVPb5Rmhh+sVGj8kVNLo 2tTh/EnuHsazxi96KrJ2O2cnrv0fJy4ShQ3kuYvTeXV7KYIMIlG8PnyBpP/JkBJz VbQQIHBPg/VwMbCgOKTH7jiUqIcTjWUgKnnOlm3FBAID9WhGSSIRyWMvTg+Ft2bW FnlVFtdXrXpJTPC+M/4U72/oLDfAq5ZvPLkLGui5jMzmI43Jdu7eCqEB9WnjyjgB WIIRJW6vBuYKMoAYcdx42bh+qWQNB+QIcoQQ4RGsAfAr96RooZSUFU0VAS29qDKG 7DeTwc0bbWhAi0qlKCS8gDuxl13O/KMhnQyDJhVMM+tuns/vKDzuJmBzEtbszJOg pnVGKK+fBK1aWjkcuIlBmiI2kvUn7tDmA+9nSOQGYrbJAlC/UmQiWy7JuBVp5z4o L5Sr8OuhoYOYUW+A0Ryd =iwPP -----END PGP SIGNATURE----- --UfFkmOIO9brN0XPF6E1IRRTIJXEqhTMuH--