From owner-freebsd-stable@FreeBSD.ORG  Thu Nov  6 10:00:58 2014
Return-Path: <owner-freebsd-stable@FreeBSD.ORG>
Delivered-To: freebsd-stable@freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115])
 (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits))
 (No client certificate requested)
 by hub.freebsd.org (Postfix) with ESMTPS id C6DBDAF3;
 Thu,  6 Nov 2014 10:00:58 +0000 (UTC)
Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1])
 (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits))
 (Client did not present a certificate)
 by mx1.freebsd.org (Postfix) with ESMTPS id 3B3E42E0;
 Thu,  6 Nov 2014 10:00:58 +0000 (UTC)
Received: from tom.home (kostik@localhost [127.0.0.1])
 by kib.kiev.ua (8.14.9/8.14.9) with ESMTP id sA6A0nIZ063523
 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO);
 Thu, 6 Nov 2014 12:00:49 +0200 (EET)
 (envelope-from kostikbel@gmail.com)
DKIM-Filter: OpenDKIM Filter v2.9.2 kib.kiev.ua sA6A0nIZ063523
Received: (from kostik@localhost)
 by tom.home (8.14.9/8.14.9/Submit) id sA6A0nFM063521;
 Thu, 6 Nov 2014 12:00:49 +0200 (EET)
 (envelope-from kostikbel@gmail.com)
X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com
 using -f
Date: Thu, 6 Nov 2014 12:00:49 +0200
From: Konstantin Belousov <kostikbel@gmail.com>
To: Matthew Seaman <matthew@freebsd.org>
Subject: Re: Varnish proxy goes catatonic under heavy load
Message-ID: <20141106100049.GO53947@kib.kiev.ua>
References: <545A0EB4.4090404@freebsd.org> <545A117B.4080606@multiplay.co.uk>
 <545B1F2A.5010203@FreeBSD.org> <20141106083153.GK53947@kib.kiev.ua>
 <545B4310.7000403@freebsd.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <545B4310.7000403@freebsd.org>
User-Agent: Mutt/1.5.23 (2014-03-12)
X-Spam-Status: No, score=-2.0 required=5.0 tests=ALL_TRUSTED,BAYES_00,
 DKIM_ADSP_CUSTOM_MED,FREEMAIL_FROM,NML_ADSP_CUSTOM_MED autolearn=no
 autolearn_force=no version=3.4.0
X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on tom.home
Cc: freebsd-stable@freebsd.org, Steven Hartland <killing@multiplay.co.uk>
X-BeenThere: freebsd-stable@freebsd.org
X-Mailman-Version: 2.1.18-1
Precedence: list
List-Id: Production branch of FreeBSD source code <freebsd-stable.freebsd.org>
List-Unsubscribe: <http://lists.freebsd.org/mailman/options/freebsd-stable>,
 <mailto:freebsd-stable-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-stable/>
List-Post: <mailto:freebsd-stable@freebsd.org>
List-Help: <mailto:freebsd-stable-request@freebsd.org?subject=help>
List-Subscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-stable>,
 <mailto:freebsd-stable-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Thu, 06 Nov 2014 10:00:58 -0000

On Thu, Nov 06, 2014 at 09:44:48AM +0000, Matthew Seaman wrote:
> Hmmm.... well, our theory about this is that we see the effect when the
> total traffic is sufficiently high that we're hitting the network
> capacity, and dropping some packets.  (The actual traffic load on an
> individual server was big, but nothing like saturating the network. It's
> the total that was maxing out our uplink to the Internet.)
Gathering information according to
https://www.freebsd.org/doc/en_US.ISO8859-1/books/developers-handbook/kerneldebug-deadlocks.html
would eliminate most doubts.

So, do you use UFS ?

> 
> We simulated the effect by sticking a test box on a 10Mb/s connection
> and threw a lot of requests for a largeish (1MB) file at it.  The packet
> loss seems to be important -- presumably it's clogging up the available
> mbufs with old packets that haven't received an ACK yet, so have to be
> held onto in case they need to be resent.