From owner-freebsd-stable@FreeBSD.ORG Tue May 29 19:26:37 2012 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 1A712106564A for ; Tue, 29 May 2012 19:26:37 +0000 (UTC) (envelope-from kjkoster@gmail.com) Received: from mail-we0-f182.google.com (mail-we0-f182.google.com [74.125.82.182]) by mx1.freebsd.org (Postfix) with ESMTP id 9B05D8FC12 for ; Tue, 29 May 2012 19:26:36 +0000 (UTC) Received: by werg1 with SMTP id g1so3697682wer.13 for ; Tue, 29 May 2012 12:26:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=from:content-type:content-transfer-encoding:subject:date:message-id :to:mime-version:x-mailer; bh=sdhgGeYx2wcoLFp3jwFt2q4lA5F46WausAstEpmAZAU=; b=zrrumq0h9VpMOk61hR/0YJYOxbVm/cZfvs3II/ExsuL5z8Pzsxb5RxX0cwhg0t1xKG svlxpvZDag5YPGm6Ew3UNkllVkhJxYXkPTPGvJomq8iv4gwM4Iv7p1ew3mmyfkd3uMHg PHk8c60gwmEmrqrihPMadVfn1oezQaZbcTeQYQee7HZGUCqMw3BvNtsCOdu1cyza5QEA ZHBfclRtteXsu0B14whveU0rbuFTsJeVH6rJVK6wZ/nkamPNHqeLkIF40G1C+M4W62fc tYj6TanpnakfL3F0KuabH7CqSXte7vIWIvZ7yZNk5MomXwrlXKmjx/f7yD7/9WB+Tv3F WwBQ== Received: by 10.216.142.167 with SMTP id i39mr7997741wej.94.1338319595367; Tue, 29 May 2012 12:26:35 -0700 (PDT) Received: from kees-jan-kosters-macbook-air.fritz.box (kjkoster.org. [83.163.197.206]) by mx.google.com with ESMTPS id ei4sm47839204wid.5.2012.05.29.12.26.34 (version=TLSv1/SSLv3 cipher=OTHER); Tue, 29 May 2012 12:26:34 -0700 (PDT) From: Kees Jan Koster Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable Date: Tue, 29 May 2012 21:26:32 +0200 Message-Id: To: freebsd-stable@freebsd.org Mime-Version: 1.0 (Apple Message framework v1278) X-Mailer: Apple Mail (2.1278) Subject: FreeBSD 9.0 hangs on heavy I/O X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 29 May 2012 19:26:37 -0000 Dear All, I seem to have a problem where really heavy disk I/O is drowning my = machine. I see hangs in the shell where I am logged on using ssh. = Network connections get dropped for no apparent reason and some HTTP = requests are served really slowly. Profiling the app code shows that the = hangs are in completely random places. Operations that are no more than = a few lines of code apart suddenly take seconds to complete. In my search I seem to find that my machine is quite slow on the disk. I = find that rather odd, given that the device in question is an SSD drive = and it is a good bit faster than the WD drive that used to carry the = data set that is accessed heavily. This drive is doing 1.5 times the = throughput, but the hangs have not gone away. To clarify, the data set used to live on ada2 (see the devlist below) = which is a spinning disk. When I experienced intermittent hangs I = plugged in an SSD drive (ada3 on the devlist) and moved the data there. = This improved the MB's per second that are being written (it is = mostly-write data) but has not changed the hangs. If anything, they got = worse since. Using gstat I notice that I/O service time is quite high. =46rom the = gstat below you can see that it takes just over 2s to servr the = requests. The L(q) seems to never drop far below 100 and %busy hovers = around 100% all day long. Can someone please help me troubleshoot that = further? What can I do to make the underlying problem visible? I should mention all data is referenced through cross-mountpoint = symlinks, would that make a difference? Should I use canonical paths in = the code instead? All file systems are mounted "noatime, soft-updates". Details: # uname -a=20 FreeBSD cumin.java-monitor.com 9.0-STABLE FreeBSD 9.0-STABLE #0: Mon Mar = 26 14:30:19 UTC 2012 = kjkoster@cumin.java-monitor.com:/usr/obj/usr/src/sys/CUMIN amd64 # gstat -f 'ada[0-3]$' -b dT: 1.001s w: 1.000s filter: ada[0-3]$ L(q) ops/s r/s kBps ms/r w/s kBps ms/w %busy Name 0 0 0 0 0.0 0 0 0.0 0.0 ada0 0 0 0 0 0.0 0 0 0.0 0.0 ada1 0 0 0 0 0.0 0 0 0.0 0.0 ada2 103 273 0 0 0.0 273 34630 2062 121.9 ada3 # camcontrol devlist at scbus1 target 0 lun 0 (pass0,ada0) at scbus2 target 0 lun 0 (pass1,ada1) at scbus3 target 0 lun 0 (pass2,ada2) at scbus4 target 0 lun 0 (pass3,ada3) at scbus7 target 0 lun 0 (pass4,cd0) at scbus8 target 0 lun 0 (pass5,cd1) # _ -- Kees Jan http://java-monitor.com/ kjkoster@kjkoster.org +31651838192 The secret of success lies in the stability of the goal. -- Benjamin = Disraeli