From owner-freebsd-stable@FreeBSD.ORG Tue Oct 13 13:34:18 2009 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id A20FE1065782; Tue, 13 Oct 2009 13:34:18 +0000 (UTC) (envelope-from ivoras@gmail.com) Received: from mail-ew0-f218.google.com (mail-ew0-f218.google.com [209.85.219.218]) by mx1.freebsd.org (Postfix) with ESMTP id 008D88FC12; Tue, 13 Oct 2009 13:34:17 +0000 (UTC) Received: by ewy18 with SMTP id 18so3765547ewy.43 for ; Tue, 13 Oct 2009 06:34:17 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:sender:received:in-reply-to :references:from:date:x-google-sender-auth:message-id:subject:to:cc :content-type:content-transfer-encoding; bh=pWSIOzQHiOjp2hVuFMedxqzOv92eNQu4XENbBZG8t5I=; b=qDCv5jFE5TdMGfnsTOPlkWWtDT4UJmCRonRTsl2xro6RPhpfh+seAOaNSyr+MD3iGm zJt815oqi3GPqd1ozdq9o43pbBM2AOV7VK98UBbPNGK7q93WjU4bkmUUfIXYwD2zmHpQ umFvC58c/0wl6CgdmsIblhA8vq9Yrrg3ZcxHU= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:sender:in-reply-to:references:from:date :x-google-sender-auth:message-id:subject:to:cc:content-type :content-transfer-encoding; b=ZJl+NSGdQekvJyfKYUtBsCOI1QPIbPKmDKLNlPgszXZHt1/vS8xXSQddFKvv0S/dTo fd5CCUGMxt2rdxFhme4+KzGKA/d+SixPoAcY3emw7McnQEPmzBYrMY8Mf6bm9Gb6WqFy 5KmozmbljllFdANpVNO/p6rikt57TPPCM8MnQ= MIME-Version: 1.0 Sender: ivoras@gmail.com Received: by 10.216.93.2 with SMTP id k2mr2362870wef.210.1255440856126; Tue, 13 Oct 2009 06:34:16 -0700 (PDT) In-Reply-To: References: From: Ivan Voras Date: Tue, 13 Oct 2009 15:33:56 +0200 X-Google-Sender-Auth: 7f53eec9f23384d2 Message-ID: <9bbcef730910130633w150571a0k461fb4e67a51fb1d@mail.gmail.com> To: Robert Watson Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Cc: freebsd-stable@freebsd.org Subject: Re: Extreme console latency during disk IO (8.0-RC1, previous releases also affected according to others) X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 13 Oct 2009 13:34:18 -0000 2009/10/13 Robert Watson : > > On Tue, 13 Oct 2009, Ivan Voras wrote: > >> Thomas Backman wrote: >>> >>> I'm copying this over from the freebsd-performance list, as I'm looking >>> for a few more opinions - not on the problems *I* am having, but rather= to >>> check whether the problem is universal or not, and if not, find a possi= ble >>> common factor. In other words: I want to hear about your experiences, *= good >>> or bad*! >>> >>> Here's the original thread (not from the beginning, though): >>> http://lists.freebsd.org/pipermail/freebsd-performance/2009-October/003= 843.html >>> >>> Long story short, my version: when the disk is stressed hard enough, >>> console IO becomes COMPLETELY unbearable. 10+ seconds to switch between >>> windows in screen(1), running (or even typing) simple commands, etc. Th= is >>> happens both via SSH and the serial console. >> >> Hmm, this looks familiar - I've noticed it before on the physical (VGA) >> console and I notice it all the time under VMWare. It sort of looks like >> disk IO really blocks network IO in this case - I use the VMs over ssh. > > Real hardware and virtual hardware have vastly different performance > properties, so I'd be careful not to assume that the issue described by t= he > original reporter and the issue you're experiencing are the same. =C2=A0I= n our > kernel, low level network protocols will essentially always take preceden= ce > over disk I/O activity. =C2=A0So on face value "disk IO really blocks net= work IO" > is highly unlikely. Yes, I agree for both reasons and that is why I wasn't complaining until encountering this thread. > There are two much more likely possibilities: (1) poor VM implementation > causes the virtual CPU to be suspended behind synchronous host OS I/O or = (2) > the network stack is running fine but the interactive user application is > getting I/O or locks scheduled behind a bulk process. > > A useful diagnostic here is to compare the behavior of three kinds of > network latency tests: > > (1) ping from the host OS to the guest OS > (2) netperf TCP_RR from the host OS to the guest OS > (3) ssh interactive latency > > If (1) is highly variable during I/O, it's almost certainly a property of > the VM technology you're using, and there's nought to be done about it in > the guest OS. Here's an example of a ping session with 0.1s resolution during a few seconds-stall in ssh: 64 bytes from 161.53.72.188: icmp_seq=3D1576 ttl=3D64 time=3D0.383 ms 64 bytes from 161.53.72.188: icmp_seq=3D1577 ttl=3D64 time=3D0.405 ms 64 bytes from 161.53.72.188: icmp_seq=3D1578 ttl=3D64 time=3D0.360 ms 64 bytes from 161.53.72.188: icmp_seq=3D2304 ttl=3D64 time=3D4.194 ms 64 bytes from 161.53.72.188: icmp_seq=3D2305 ttl=3D64 time=3D0.454 ms 64 bytes from 161.53.72.188: icmp_seq=3D2306 ttl=3D64 time=3D0.376 ms note huge packet loss. It looks like it's VM fault or something like it.