From owner-freebsd-stable@FreeBSD.ORG  Tue Oct 13 13:42:35 2009
Return-Path: <owner-freebsd-stable@FreeBSD.ORG>
Delivered-To: freebsd-stable@freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34])
	by hub.freebsd.org (Postfix) with ESMTP id 0CF6B1065672;
	Tue, 13 Oct 2009 13:42:35 +0000 (UTC)
	(envelope-from rwatson@freebsd.org)
Received: from cyrus.watson.org (cyrus.watson.org [65.122.17.42])
	by mx1.freebsd.org (Postfix) with ESMTP id D7FA68FC21;
	Tue, 13 Oct 2009 13:42:34 +0000 (UTC)
Received: from lemongrass.sec.cl.cam.ac.uk (lemongrass.sec.cl.cam.ac.uk
	[128.232.18.47])
	by cyrus.watson.org (Postfix) with ESMTPSA id 4A21646B03;
	Tue, 13 Oct 2009 09:42:34 -0400 (EDT)
Mime-Version: 1.0 (Apple Message framework v1076)
Content-Type: text/plain; charset=us-ascii; format=flowed; delsp=yes
From: "Robert N. M. Watson" <rwatson@freebsd.org>
In-Reply-To: <9bbcef730910130633w150571a0k461fb4e67a51fb1d@mail.gmail.com>
Date: Tue, 13 Oct 2009 14:42:32 +0100
Content-Transfer-Encoding: 7bit
Message-Id: <B5BA3ACC-BC06-4042-8434-0D9395A0F478@freebsd.org>
References: <E316139E-FFCF-432F-8DCE-62B120C38E55@exscape.org>
	<CC16B639-7A75-4016-A8A8-5C59E9CD5E95@exscape.org>
	<hb1qs0$qjd$1@ger.gmane.org>
	<alpine.BSF.2.00.0910131406340.26071@fledge.watson.org>
	<9bbcef730910130633w150571a0k461fb4e67a51fb1d@mail.gmail.com>
To: Ivan Voras <ivoras@freebsd.org>
X-Mailer: Apple Mail (2.1076)
Cc: freebsd-stable@freebsd.org
Subject: Re: Extreme console latency during disk IO (8.0-RC1,
	previous  releases also affected according to others)
X-BeenThere: freebsd-stable@freebsd.org
X-Mailman-Version: 2.1.5
Precedence: list
List-Id: Production branch of FreeBSD source code <freebsd-stable.freebsd.org>
List-Unsubscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-stable>, 
	<mailto:freebsd-stable-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-stable>
List-Post: <mailto:freebsd-stable@freebsd.org>
List-Help: <mailto:freebsd-stable-request@freebsd.org?subject=help>
List-Subscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-stable>,
	<mailto:freebsd-stable-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Tue, 13 Oct 2009 13:42:35 -0000


On 13 Oct 2009, at 14:33, Ivan Voras wrote:

>> If (1) is highly variable during I/O, it's almost certainly a  
>> property of
>> the VM technology you're using, and there's nought to be done about  
>> it in
>> the guest OS.
>
> Here's an example of a ping session with 0.1s resolution during a few
> seconds-stall in ssh:
>
> 64 bytes from 161.53.72.188: icmp_seq=1576 ttl=64 time=0.383 ms
> 64 bytes from 161.53.72.188: icmp_seq=1577 ttl=64 time=0.405 ms
> 64 bytes from 161.53.72.188: icmp_seq=1578 ttl=64 time=0.360 ms
>
> 64 bytes from 161.53.72.188: icmp_seq=2304 ttl=64 time=4.194 ms
> 64 bytes from 161.53.72.188: icmp_seq=2305 ttl=64 time=0.454 ms
> 64 bytes from 161.53.72.188: icmp_seq=2306 ttl=64 time=0.376 ms
>
> note huge packet loss. It looks like it's VM fault or something like  
> it.

It sounds like the VM is failing to execute the guest during certain  
types of I/O. A bit of scheduler tracing in the host OS probably  
wouldn't go amiss to confirm that the VM really is suspending the  
guest at about the same time ICMP latency goes up. However, given the  
above I think I you can reasonable assume that the 4ms jump you're  
seeing there is due to global host OS/VM scheduling, and not FreeBSD  
scheduling.

Robert