From owner-freebsd-hackers@FreeBSD.ORG Tue Apr 17 19:16:09 2012 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 56F55106566B; Tue, 17 Apr 2012 19:16:09 +0000 (UTC) (envelope-from gljennjohn@googlemail.com) Received: from mail-ey0-f182.google.com (mail-ey0-f182.google.com [209.85.215.182]) by mx1.freebsd.org (Postfix) with ESMTP id 913588FC1E; Tue, 17 Apr 2012 19:16:08 +0000 (UTC) Received: by eaaf13 with SMTP id f13so1844878eaa.13 for ; Tue, 17 Apr 2012 12:16:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlemail.com; s=20120113; h=date:from:to:cc:subject:message-id:in-reply-to:references:reply-to :x-mailer:mime-version:content-type:content-transfer-encoding; bh=Q1VuFzBSyYa1uZE5lARsbtqKW0ZghxjkX2QlEe5NLgI=; b=TZ7qB9V9F0+f0muI7n1jqwxRIzVhfFD3TA0YDbc6YpQR5tTYhhPtX3L5QFAqsHya93 7vqzeakmjjQPsa1z+6Xt81EQew/T+d5VyC2qQQUKa/LBCcQ887WHEl08W7bPDtH1O9y1 /SAgdefg44o3hnwp3O7BOeYfFuPc0hIDPuM+dPswdIzlUCuno0nNtEYaZu4TemmH3de2 Y8ecn+OVy7kjDSpemlsUCldjNZx4vvX7vp2sfCLf8XkwxXFtf23snoszTQJAKG3yzgcB c7p25xIRfZfgfBBQXAxJjPGe9cpZ5rJ94uD7x1oAUCv0kJdsJo1t0b1sa6c2dtyndrEk Gygw== Received: by 10.213.110.7 with SMTP id l7mr1275406ebp.7.1334690161570; Tue, 17 Apr 2012 12:16:01 -0700 (PDT) Received: from ernst.jennejohn.org (p578E3B06.dip.t-dialin.net. [87.142.59.6]) by mx.google.com with ESMTPS id d54sm106543825eei.9.2012.04.17.12.15.59 (version=TLSv1/SSLv3 cipher=OTHER); Tue, 17 Apr 2012 12:16:00 -0700 (PDT) Date: Tue, 17 Apr 2012 21:15:58 +0200 From: Gary Jennejohn To: Adrian Chadd Message-ID: <20120417211558.4793b705@ernst.jennejohn.org> In-Reply-To: References: <20120403193124.46ad9de9@ernst.jennejohn.org> <20120411192153.5672b62c@ernst.jennejohn.org> X-Mailer: Claws Mail 3.8.0 (GTK+ 2.24.6; amd64-portbld-freebsd10.0) Mime-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 8bit Cc: freebsd-hackers , Jerry Toung Subject: Re: CAM disk I/O starvation X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: gljennjohn@googlemail.com List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 17 Apr 2012 19:16:09 -0000 On Mon, 16 Apr 2012 14:39:12 -0700 Adrian Chadd wrote: > On 11 April 2012 10:21, Gary Jennejohn wrote: > > > Just for the archive my bad disk performance seems to have been fixed in > > HEAD by svn commit r234074.  Seems that all interrupts were being handled > > by a single CPU/core (I have 6), which resulted in abysmal interrupt > > handling when mutltiple disks were busy. > > > > Since this commit my disk preformance is back to normal and long lags > > are a thing of the past. > > Hi, > > This is kind of worrying. You only have a few disks, a single core > SHOULD be able to handle all the interrupts for those disks whilst > leaving plenty of cycles to spare to drive the rest of your system. > And you have 5 other cores. > > Would you be willing to help out diagnose exactly why that particular > behaviour is causing you so much trouble? It almost sounds like > something in the IO path is blocking for far too long, not allowing > the rest of the system to move forward. That's very worrying for an > interrupt handler. :) > Yes, I agree completely. My first thought was that disk I/O scheduling had somehow been pessimized. But then I thought - wait a minute, I have disk caches enabled and command queuing is enabled for all of them, so that shouldn't really have any noticeable impact. So I was at a loss to explain why disk performance had suddenly gotten so bad. I'd be willing to spend some time on diagnosing it, but I have to come up with a scenario which would reliably reproduce the problem. AFAICR it generally happened when I was running csup/svn because my CVS repositoy is on one disk and /usr/{ports,src} are on a different one. I still have the old problem kernel around, but it's probably not instrumented for any meaningful diagnoses. -- Gary Jennejohn