Date: Thu, 06 Nov 2014 09:44:48 +0000 From: Matthew Seaman <matthew@freebsd.org> To: Konstantin Belousov <kostikbel@gmail.com> Cc: freebsd-stable@freebsd.org, Steven Hartland <killing@multiplay.co.uk> Subject: Re: Varnish proxy goes catatonic under heavy load Message-ID: <545B4310.7000403@freebsd.org> In-Reply-To: <20141106083153.GK53947@kib.kiev.ua> References: <545A0EB4.4090404@freebsd.org> <545A117B.4080606@multiplay.co.uk> <545B1F2A.5010203@FreeBSD.org> <20141106083153.GK53947@kib.kiev.ua>
next in thread | previous in thread | raw e-mail | index | archive | help
This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --VvAqFpo4W0tEH1bcbUbxO1cCWI634SnAJ Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable On 11/06/14 08:31, Konstantin Belousov wrote: > I do not remember exact point in the stable/9 lifetime when the > debug.vn_io_fault_enable was merged. If it is present in your system, > frob its value to 1 and see. I highly suspect that if varnish is in > 'mmap' mode (whatever it is called), and you use UFS, it may help. Seems it is not present in 9.1-RELEASE, but it is in 9.2-RELEASE. I've toggled that sysctl on the one machine running 9.2-RELEASE, but I doubt we're going to get any useful results from it before we start upgrading -- the traffic flood that triggered this was exceptional, and an isolated incident. > I am suggesting this before upgrading to 10 only because I want to > know whether the vn_io_fault code helps in this situation. There > are rumors that it does, but I never seen the confirmation. Hmmm.... well, our theory about this is that we see the effect when the total traffic is sufficiently high that we're hitting the network capacity, and dropping some packets. (The actual traffic load on an individual server was big, but nothing like saturating the network. It's the total that was maxing out our uplink to the Internet.) We simulated the effect by sticking a test box on a 10Mb/s connection and threw a lot of requests for a largeish (1MB) file at it. The packet loss seems to be important -- presumably it's clogging up the available mbufs with old packets that haven't received an ACK yet, so have to be held onto in case they need to be resent. Cheers, Matthew --VvAqFpo4W0tEH1bcbUbxO1cCWI634SnAJ Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQJ8BAEBCgBmBQJUW0MXXxSAAAAAAC4AKGlzc3Vlci1mcHJAbm90YXRpb25zLm9w ZW5wZ3AuZmlmdGhob3JzZW1hbi5uZXQxOUYxNTRFQ0JGMTEyRTUwNTQ0RTNGMzAw MDUxM0YxMEUwQTlFNEU3AAoJEABRPxDgqeTnnrEP/2xfDpzp2fJ1X9kN1hCf2IbL 6+eHA0zDsGkCbUdha6RbyIbj+cB9+PT+7wribyzP6FyVvyA24/osQ5iUL6m2Cqnu OTrdf8/yVnLITjRGZFqSjAjens5eS/TuHk1NiIcTGnIs8RPxTgpeZconicB7uau0 7QmatroQpnOfwXT4x8pcLT8ZOI+9Px0Ng1wAD85TR8e7FjzAezsnvQtLgxNWKSVT rEmRQKP03DeXkyGpbWU2tz/jqCtVufoI/UkATfhDqbDTx5FSleww9kst9h+1tbv8 Ncjr6rIakJ/zan96FNJvT9++SElVYAxJS+1VGJmLFBgpgSKKjYO7x7Q6DUCOJK7I qY180n/M1R8uhRtS5daS4Ji0ZnChUrEmru/vgOC2d70bHGHZuQ3E72gaiNqLJPPn o0zvEVT6uJk1uWwvIIUdcGv/B5DyY/tDNPVH2bRaB1/RRMarkiOmEU6ibmLsykO/ ocI0qPA2hZEqfj1KqPK8+WweRE78tH9blkoK3uYbn0ZuHN6dlHJ36yTJC/S4o2z6 zHUlHLiZ1LCpq3JAt0IDlJwazzVtUokPu7Qv47F7aGZv6sV+oos4P27U7z7N7U/u 5KlCvWxmUNBQDbSLbQjYn1XjCAWFqUX8/3MVBh9Oz8dMuihRyeyKMuRdnLxdHAOp hzvG/7i7dIX/Nyc14g8B =ORlW -----END PGP SIGNATURE----- --VvAqFpo4W0tEH1bcbUbxO1cCWI634SnAJ--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?545B4310.7000403>