From owner-freebsd-performance@FreeBSD.ORG Sat Aug 8 09:02:49 2009 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id B48BB106564A for ; Sat, 8 Aug 2009 09:02:49 +0000 (UTC) (envelope-from grarpamp@gmail.com) Received: from ey-out-2122.google.com (ey-out-2122.google.com [74.125.78.26]) by mx1.freebsd.org (Postfix) with ESMTP id 3A8E48FC1A for ; Sat, 8 Aug 2009 09:02:48 +0000 (UTC) Received: by ey-out-2122.google.com with SMTP id 9so523531eyd.7 for ; Sat, 08 Aug 2009 02:02:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:date:message-id:subject :from:to:content-type:content-transfer-encoding; bh=XtPunWYkcOzmEVObxOH0CxMZx0Wg1s6dYpc1+lIsiC8=; b=fIIGdXZrg9oq0ZLcLVwX2sPcQiiuj6EU0WY05IocPz4LqQRQxWxDVLM5hWFu3gnvsf 1jZRn2QgNEfxdfPEO+N+WZG4VhO3ez9dIUoTbrkpEnu9XD2xDRKwLfqC2EvPrxZ3vh7W 8qNS4DmsnkxiXMYrEE27GFbonJ2PvxUm3Mvm8= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:date:message-id:subject:from:to:content-type :content-transfer-encoding; b=UziNRb/xZ+SFztZfyyBgF3ve0P4es8ZZu2ehwQdpTAZI8UWA2KyYcTInm/u/drC46M X7ejqqqMnSlW6eScesemEMd2FLcRGjmjU+7gc5xk3ayly3qusAGTXoJHdTp0DZzrFS7E mAwTWcjiK86BM8n0n9fD4E6r1hs9znpySVEzA= MIME-Version: 1.0 Received: by 10.210.114.15 with SMTP id m15mr2581371ebc.0.1249722168175; Sat, 08 Aug 2009 02:02:48 -0700 (PDT) Date: Sat, 8 Aug 2009 05:02:48 -0400 Message-ID: From: grarpamp To: freebsd-performance@freebsd.org Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit X-Mailman-Approved-At: Sat, 08 Aug 2009 12:43:56 +0000 Subject: RELENG_7 heavy disk = system crawls X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 08 Aug 2009 09:02:49 -0000 > Are you *sure* that the cpu is your bottleneck? Well, I've got a gig free in /usr/local which is ufs2+softdeps. So I just dd if=dev/zero bs=1m of=zero there. Disk was 100% busy, cpu was 10-15% system, 10% user, about 16MiB/sec. Of course since that is my system spindle, and it was busied out by dd, I had to wait 10 sec or so for vi to load to write this note during it :) But the rest of the interface was responsive. Though I've never run RELENG_4 on this box, I'd venture it would feel similar in this case. I can dd if=/dev/ad[n].eli of=/dev/null bs=1m and use 75% system all in geli, 27% disk busy, 20MiB/sec. Interface was slower but reasonable. Now do a dd if=file of=/dev/null bs=1m where file is on zfs on the geli spindle above and the system melts down. Half cpu in geli, half spread over about 8 spa_zio's, 4MiB/sec, all system time, disk 100% busy. I'm not sure yet how to isolate cpu from i/o under my geli+zfs setup. I think they're mated together. Curiously, I've got about 104 spa_zio procs all in tq->tq DL state. Only about 20 or so have more than a little system time on them. But about 9 zfs mounts so that may be ok, don't know. Don't get me wrong, FreeBSD has been my primary os since RELENG_2_2. And it's been great, still is. I recommend and use it all the time. It's just that this workload has really put the screws to things and I don't see a way out. I'd like to deploy geli+zfs everywhere but if I can't login remotely because some user has it busied out on something I've no knobs to control, umm, yeah :) I'm curious to see what others running geli_aes128+zfs_sha256 are seeing in this regard. And I'd love to see what a fast dual or more core amd64 system would be like under this workload. As to your i/o thing, I think back in RELENG_4 that if all the spindles were on the same pata controller/interrupt, monopolistic loads could occur. Seemed to be a hardware thing, not BSD. IE: At the moment I've got a half dozen spindles and filesystems spread out under RELENG_4 all happily doing find | xargs sha1 at once, no problems. That hardware is set for update to RELENG_7 or RELENG_8 in a few weeks. > we need a way to nice i/o up/down That would be handy for sure. User spindles, system spindles, storage, net, keyboard, etc. # systime spread 11 root 1 171 ki31 0K 8K RUN 46.5H 88.18% idle: cpu0 3215 root 1 -8 - 0K 8K geli:w 737:30 1.76% g_eli[0] ad6 607 root 1 -8 - 0K 8K geli:w 158:12 0.00% g_eli[0] ad4 3235 root 1 -16 - 0K 24K tq->tq 69:41 0.00% spa_zio 3229 root 1 -16 - 0K 24K tq->tq 69:40 0.00% spa_zio 3228 root 1 -16 - 0K 24K tq->tq 69:39 0.00% spa_zio 3234 root 1 -16 - 0K 24K tq->tq 69:39 0.00% spa_zio 3233 root 1 -16 - 0K 24K tq->tq 69:39 0.00% spa_zio 3232 root 1 -16 - 0K 24K tq->tq 69:39 0.00% spa_zio 3231 root 1 -16 - 0K 24K tq->tq 69:38 0.00% spa_zio 3230 root 1 -16 - 0K 24K tq->tq 69:37 0.00% spa_zio 1135 user 1 44 0 169M 152M select 56:02 0.00% XFree86 954 root 1 -16 - 0K 24K tq->tq 17:10 0.00% spa_zio 958 root 1 -16 - 0K 24K tq->tq 17:10 0.00% spa_zio 956 root 1 -16 - 0K 24K tq->tq 17:09 0.00% spa_zio 953 root 1 -16 - 0K 24K tq->tq 17:09 0.00% spa_zio 957 root 1 -16 - 0K 24K tq->tq 17:09 0.00% spa_zio 952 root 1 -16 - 0K 24K tq->tq 17:09 0.00% spa_zio 951 root 1 -16 - 0K 24K tq->tq 17:09 0.00% spa_zio 955 root 1 -16 - 0K 24K tq->tq 17:09 0.00% spa_zio 613 root 1 -8 - 0K 8K geli:w 16:12 0.00% g_eli[0] ad7 3 root 1 -8 - 0K 8K - 15:05 0.00% g_up