From owner-freebsd-stable@FreeBSD.ORG Tue May 29 19:40:48 2012 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 69E171065672 for ; Tue, 29 May 2012 19:40:48 +0000 (UTC) (envelope-from fjwcash@gmail.com) Received: from mail-qa0-f47.google.com (mail-qa0-f47.google.com [209.85.216.47]) by mx1.freebsd.org (Postfix) with ESMTP id 2352C8FC18 for ; Tue, 29 May 2012 19:40:48 +0000 (UTC) Received: by qabg1 with SMTP id g1so1721651qab.13 for ; Tue, 29 May 2012 12:40:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type:content-transfer-encoding; bh=1lrUGnlXnBEyWiocHKM7v1SoDNkAGqzrtkwcBuNs4e0=; b=rWnSWxF/9ulCjUMOu0k8hANmcLvgplRDYAxGukq9ieHCc7ccbRnY6GMluVHYa0xO+F 1J921vzMCLxB915Dmd+EJqdJlDekvqZz66kl5sq/yrNN62WnP+TQPsAmjcGw/P7heiTW cCANm6WrhZEC7fntV6mo+Q0ZOTcEJ3FsfA/+jznBu1vEZIE16HLWXPuf6lfx6cwg5vhq S/1TLMHhKEI0aeoOtYv1/XAI09H61ziVOqndkgTHLtiIGN37Xuh6ZyoXebqSqfe9aQZZ 8H0PA1eMRVSeYv6ubXAaUqB+QIrejMqlnJdcHhBOjdgXd0WTCRKE4H0CJmwQKJ2Nxg0Y zVHA== MIME-Version: 1.0 Received: by 10.224.181.83 with SMTP id bx19mr890745qab.79.1338320447433; Tue, 29 May 2012 12:40:47 -0700 (PDT) Received: by 10.229.20.148 with HTTP; Tue, 29 May 2012 12:40:47 -0700 (PDT) In-Reply-To: References: Date: Tue, 29 May 2012 12:40:47 -0700 Message-ID: From: Freddie Cash To: Kees Jan Koster Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Cc: freebsd-stable@freebsd.org Subject: Re: FreeBSD 9.0 hangs on heavy I/O X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 29 May 2012 19:40:48 -0000 On Tue, May 29, 2012 at 12:34 PM, Freddie Cash wrote: > On Tue, May 29, 2012 at 12:26 PM, Kees Jan Koster wr= ote: >> I seem to have a problem where really heavy disk I/O is drowning my mach= ine. I see hangs in the shell where I am logged on using ssh. Network conne= ctions get dropped for no apparent reason and some HTTP requests are served= really slowly. Profiling the app code shows that the hangs are in complete= ly random places. Operations that are no more than a few lines of code apar= t suddenly take seconds to complete. >> >> In my search I seem to find that my machine is quite slow on the disk. I= find that rather odd, given that the device in question is an SSD drive an= d it is a good bit faster than the WD drive that used to carry the data set= that is accessed heavily. This drive is doing 1.5 times the throughput, bu= t the hangs have not gone away. >> >> To clarify, the data set used to live on ada2 (see the devlist below) wh= ich is a spinning disk. When I experienced intermittent hangs I plugged in = an SSD drive (ada3 on the devlist) and moved the data there. This improved = the MB's per second that are being written (it is mostly-write data) but ha= s not changed the hangs. If anything, they got worse since. >> >> Using gstat I notice that I/O service time is quite high. From the gstat= below you can see that it takes just over 2s to servr the requests. The L(= q) seems to never drop far below 100 and %busy hovers around 100% all day l= ong. Can someone please help me troubleshoot that further? What can I do to= make the underlying problem visible? >> >> I should mention all data is referenced through cross-mountpoint symlink= s, would that make a difference? Should I use canonical paths in the code i= nstead? >> >> All file systems are mounted "noatime, soft-updates". > > You may want to play around with gshed, the GEOM Scheduler. > > Matt Dillon did a bunch of tests comparing FreeBSD+UFS to > DragonflyBSD+HAMMER and found that FreeBSD starves read threads in > order to satisfy write threads (or the other way around?). =C2=A0But, > adding gsched into the mix helped things immensely, allowing mixed > reads/writes to better shares disk I/O resources. > > I'll see if I can dig up a link to his testing e-mail messages. Here's the post, part of a thread on benchmarking RAID controllers: http://leaf.dragonflybsd.org/mailarchive/kernel/2011-07/msg00034.html --=20 Freddie Cash fjwcash@gmail.com