Date: Wed, 05 Nov 2014 11:49:08 +0000 From: Matthew Seaman <matthew@freebsd.org> To: freebsd-stable@freebsd.org Subject: Varnish proxy goes catatonic under heavy load Message-ID: <545A0EB4.4090404@freebsd.org>
next in thread | raw e-mail | index | archive | help
This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --UfFkmOIO9brN0XPF6E1IRRTIJXEqhTMuH Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Dear all, We had an unfortunate set of circumstances which resulted in several million people all trying to download about 1.5MB worth of images from our servers over the course of a few hours. Or, at least, it would have been a few hours, except that our three varnish proxies just crumbled under the load within 10 minutes. Now, that's bad enough, but we could have just about coped if the proxies stopped serving requests for a few minutes. What actually happened was that all three servers went catatonic on the network *and stayed that way*: even when we shunted the traffic away from one, we still couldn't access it via ssh or any network protocol. And it stayed like that for sufficiently long time that we had no recourse other than to get the servers rebooted. Can anyone explain what was happening here? Not having the servers recover accessibility for an extended period even after the excess traffic was stopped is unacceptable. We're also struggling to recreate the effect in the lab: any clues about how to do so, and any suggestions about how to prevent the 'going catatonic' response would be greatly appreciated. Servers are amd64 running FreeBSD 9.1 or 9.2 and Varnish 3.0.5. Cheers, Matthew --UfFkmOIO9brN0XPF6E1IRRTIJXEqhTMuH Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQJ8BAEBCgBmBQJUWg7AXxSAAAAAAC4AKGlzc3Vlci1mcHJAbm90YXRpb25zLm9w ZW5wZ3AuZmlmdGhob3JzZW1hbi5uZXQxOUYxNTRFQ0JGMTEyRTUwNTQ0RTNGMzAw MDUxM0YxMEUwQTlFNEU3AAoJEABRPxDgqeTnT6MP/iQ8borxlNTLRBbLouBz61FH YrBaK/EqaeTp3xSyx7QpVi4tGpfWgnliw27eXiifIX+GEFY5Pzige+uYtRdwdVHl L8iLDM/yx7mJVmqCv2d3BnU7T+RvPcN0BIsPRnIbbK6+dzxkJRELp4f2el1xHZFl ha3k1GxYHcpo3+4aRbJ4Q5J9BaM3uvhAY2kEbs/061L+1ov+J750t7qDALsNBUBR a7Wc79UndpRZhHQKErXlSZK/NiF4VwhuQfRaFISq1firiVPb5Rmhh+sVGj8kVNLo 2tTh/EnuHsazxi96KrJ2O2cnrv0fJy4ShQ3kuYvTeXV7KYIMIlG8PnyBpP/JkBJz VbQQIHBPg/VwMbCgOKTH7jiUqIcTjWUgKnnOlm3FBAID9WhGSSIRyWMvTg+Ft2bW FnlVFtdXrXpJTPC+M/4U72/oLDfAq5ZvPLkLGui5jMzmI43Jdu7eCqEB9WnjyjgB WIIRJW6vBuYKMoAYcdx42bh+qWQNB+QIcoQQ4RGsAfAr96RooZSUFU0VAS29qDKG 7DeTwc0bbWhAi0qlKCS8gDuxl13O/KMhnQyDJhVMM+tuns/vKDzuJmBzEtbszJOg pnVGKK+fBK1aWjkcuIlBmiI2kvUn7tDmA+9nSOQGYrbJAlC/UmQiWy7JuBVp5z4o L5Sr8OuhoYOYUW+A0Ryd =iwPP -----END PGP SIGNATURE----- --UfFkmOIO9brN0XPF6E1IRRTIJXEqhTMuH--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?545A0EB4.4090404>