From owner-freebsd-current@FreeBSD.ORG Tue Apr 20 12:29:15 2010 Return-Path: Delivered-To: freebsd-current@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 2C74E1065675 for ; Tue, 20 Apr 2010 12:29:15 +0000 (UTC) (envelope-from asmrookie@gmail.com) Received: from mail-bw0-f216.google.com (mail-bw0-f216.google.com [209.85.218.216]) by mx1.freebsd.org (Postfix) with ESMTP id AD9298FC12 for ; Tue, 20 Apr 2010 12:29:14 +0000 (UTC) Received: by bwz8 with SMTP id 8so5482721bwz.3 for ; Tue, 20 Apr 2010 05:29:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:sender:received:in-reply-to :references:date:x-google-sender-auth:received:message-id:subject :from:to:cc:content-type:content-transfer-encoding; bh=XWqOf85/bpN6HAJ2L0ZJlY/sj9ogi2zyiVvrw7dUmb0=; b=pNAT5TcCiCzuBlg8wue8gTTGvnj5K8qW/WacmREEc5rp/j+xLYcYL8Govp8N75eEX8 uUFSvZmNCITGeYMo1i/lqCHqBgbCLy0ned4KPT1wo5XGkSWUxcX2T1dD+w8J27O3plQt KTdg2+oNbET3OPxydg+aXTY+5H3b1jONMbv80= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type :content-transfer-encoding; b=c+K6mClgtNvXIGi10sEVYMScVR95dE+Ke1DUcU2Z3L7yrspjVWwKSMypRF1LvtGmiF LlgyLXVBq5UJki8Lsen4mQ+5UrPQgNlz151Q6+ji713eCY/DX02KK692jGd9tuejXhxZ BgsaduZ5TTJPPrN3W/sROwG+72+kzRtbrCH6w= MIME-Version: 1.0 Sender: asmrookie@gmail.com Received: by 10.239.164.140 with HTTP; Tue, 20 Apr 2010 05:29:13 -0700 (PDT) In-Reply-To: <4BCD5049.8030408@gmail.com> References: <4BCD5049.8030408@gmail.com> Date: Tue, 20 Apr 2010 14:29:13 +0200 X-Google-Sender-Auth: 61cbf56919355b31 Received: by 10.239.193.80 with SMTP id h16mr572565hbi.151.1271766553292; Tue, 20 Apr 2010 05:29:13 -0700 (PDT) Message-ID: From: Attilio Rao To: David Ehrmann Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Cc: freebsd-current@freebsd.org Subject: Re: Strange disk problem X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 20 Apr 2010 12:29:15 -0000 2010/4/20 David Ehrmann : > Initially, I noticed a problem where reading a file on this machine seeme= d > to stop--something like a video would just stop playing. =C2=A0At first, = I > thought it was the machine, but a new motherboard, CPU, and RAM later, th= e > problem persists. =C2=A0The network card uses a different chipset, too. > > The files are on zfs, but scrubs are fine, and zpool status lists no erro= rs > of any kind. =C2=A0Trying to reproduce the problem, I set up a script tha= t > reading a random 1M block every 60 seconds off the drive backing zfs. > =C2=A0That's when I noticed something: one disk seems to be causing the p= roblems. > =C2=A0I logged the dd times, and some of them were huge--more than a minu= te. =C2=A0The > times on the other disk in the mirrored vdev were low. > > I've only seen the problem when I have a vm's disk image hosted on the > machine. =C2=A0That said, the network interface is configured at 100mbps,= so > there's no reason for that to saturate the disk's throughput. =C2=A0Top r= eports > that almost 20% of the CPU is going towards interrupts. =C2=A0I can read = a file > off the zfs pool at over 50MB/s, so that shouldn't be a problem. =C2=A0On= e thing > I'm wondering is why the disk read doesn't timeout quickly? =C2=A0At leas= t that > way zfs could try to use the other drive in the mirrored vdev. > > Any ideas? =C2=A0One thing I should try is switching the drive, see if th= e > problem follows the disk or stays with the lowest /dev/adX device. =C2=A0= I'm > using geli, but the read problems happen with both /dev/adX AND > /dev/adX.eli., so I don't think that's it. =C2=A0I've seen the problem wi= th > Samba, NFS, and dd. David, do you think you are willing to re-create the problem and do a PMC analysis on it? (If you need any guidance let me know, I will be happy to give it). Attilio --=20 Peace can only be achieved by understanding - A. Einstein