From owner-freebsd-current@FreeBSD.ORG Wed Nov 11 21:26:17 2009 Return-Path: Delivered-To: freebsd-current@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 2F0F51065693; Wed, 11 Nov 2009 21:26:17 +0000 (UTC) (envelope-from sfourman@gmail.com) Received: from mail-pz0-f185.google.com (mail-pz0-f185.google.com [209.85.222.185]) by mx1.freebsd.org (Postfix) with ESMTP id EECD08FC1C; Wed, 11 Nov 2009 21:26:16 +0000 (UTC) Received: by pzk15 with SMTP id 15so1111191pzk.3 for ; Wed, 11 Nov 2009 13:26:16 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:in-reply-to:references :date:message-id:subject:from:to:cc:content-type :content-transfer-encoding; bh=ro8ed8yvQCMFzLgf9446Omsj+rHJKHhKhYqh//DtLoQ=; b=pY4KOeBszjUKBDGjMCipnVlTdxAiQuGzz+xPNq2qc3WTyChhIIS+XAb3NFilKVN0Hh wBy/mQEZiN4VSBqDjMyVG0+KI8OFug+/ozUMwOWwzwx7DgVrZbMfi7ZxJ5znpUxHLAqZ iENLQx+SrcqMeeAojDKAyKXfyDM6cv0caSfDM= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type:content-transfer-encoding; b=Nx9oVNhrrHbgoXoeAxyW8JXNp3JKmzK5NCPhN8tlGLlyakdB5CwvPU7flyQ3OBhfRz VOBU9TEo+2DH+54pGapLFQKzwysTXjTaMHEnMhs3or4iQVB6JUXnDcCVrnDB4TF+ExTR ofScF1Q2MuemLyh4e1ZEJrwDjl0CO6yXzTbQ8= MIME-Version: 1.0 Received: by 10.142.55.20 with SMTP id d20mr210928wfa.37.1257974776434; Wed, 11 Nov 2009 13:26:16 -0800 (PST) In-Reply-To: <20091111204903.GI89052@dan.emsphone.com> References: <11167f520911111050j36dd94far667c81e6f5c18e69@mail.gmail.com> <20091111204903.GI89052@dan.emsphone.com> Date: Wed, 11 Nov 2009 15:26:16 -0600 Message-ID: <11167f520911111326v13bb442bt36e853afbecdf834@mail.gmail.com> From: "Sam Fourman Jr." To: Dan Nelson Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Cc: freebsd-current@freebsd.org, Ivan Voras Subject: Re: Help ZFS FreeBSD 8.0 RC2 Write performance issue X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 11 Nov 2009 21:26:17 -0000 On Wed, Nov 11, 2009 at 2:49 PM, Dan Nelson wrote= : > In the last episode (Nov 11), Ivan Voras said: >> Sam Fourman Jr. wrote: >> > I am running FreeBSD 8.0RC2 and I dont understand why my ZFS/NFS is >> > acting weird on writes. =A0I get ~150mbit writes idk if this is good o= r >> > not? =A0but it paused for a few seconds every once and awhile. >> >> You didn't give any "iostat" statistics - I suspect that if you >> correlate ifstat and iostat output that you will see that network >> "pauses" happen during spikes in IO. You should check for this and post >> your results. > > Yes, iostat would be useful here. =A0"iostat -zxC 2" will give you per-di= sk > stats plus CPU usage every 2 seconds (CPU may be a factor if you have > compression enabled). > > On a Solaris box I admin, setting zfs_write_limit_override helped stutter= ing > while doing heavy writes. =A0It's not exported on FreeBSD, but it should = be > easy to add it as a RW sysctl; it lives in dsl_pool.c and can be tweaked = at > runtime. =A0Start big and tune it down so each write burst takes under a > second; it looks like you're writing solid for around 6-8 seconds now. = =A0The > number will vary depending on your disk speed and how much ARC you have. here are some iostats for you. I do not believe I have compression enabled am I mistaken? isn'y SATA2 300MB/s? and I am doing ~6MB/s per disk? I built this machine with 4GB of memory because I thought ZFS would like it= . now maybe a re(4) interface isnt the best choice. if that is the problem here I can change it. We spent ~$800 on a hardware RAID card thinking that it would help performa= nce Why is it that with sftp we do not see the pauses in Network transfer? # iostat -zxC 1 (with NFS and ZFS) extended device statistics cpu device r/s w/s kr/s kw/s wait svc_t %b us ni sy in id ad4 0.1 0.4 2.0 6.1 0 3.1 0 0 0 1 1 98 da0 2.2 7.2 127.0 246.7 0 120.3 6 da1 2.0 7.5 117.3 246.2 0 120.4 6 da2 2.2 7.3 127.1 246.7 0 119.8 6 da3 2.0 7.6 115.5 246.2 0 119.8 5 da4 2.2 7.3 126.1 246.7 0 119.9 5 da5 2.0 7.5 115.6 246.3 0 119.8 5 extended device statistics cpu device r/s w/s kr/s kw/s wait svc_t %b us ni sy in id 0 0 0 0 99 extended device statistics cpu device r/s w/s kr/s kw/s wait svc_t %b us ni sy in id 0 0 0 0 100 extended device statistics cpu device r/s w/s kr/s kw/s wait svc_t %b us ni sy in id 0 0 0 0 100 extended device statistics cpu device r/s w/s kr/s kw/s wait svc_t %b us ni sy in id 0 0 0 0 100 extended device statistics cpu device r/s w/s kr/s kw/s wait svc_t %b us ni sy in id 0 0 0 0 100 extended device statistics cpu device r/s w/s kr/s kw/s wait svc_t %b us ni sy in id 0 0 0 0 100 extended device statistics cpu device r/s w/s kr/s kw/s wait svc_t %b us ni sy in id 0 0 0 0 100 extended device statistics cpu device r/s w/s kr/s kw/s wait svc_t %b us ni sy in id 0 0 0 0 100 extended device statistics cpu device r/s w/s kr/s kw/s wait svc_t %b us ni sy in id 0 0 0 0 100 extended device statistics cpu device r/s w/s kr/s kw/s wait svc_t %b us ni sy in id 0 0 0 0 100 extended device statistics cpu device r/s w/s kr/s kw/s wait svc_t %b us ni sy in id 0 0 0 0 100 extended device statistics cpu device r/s w/s kr/s kw/s wait svc_t %b us ni sy in id ad4 0.0 11.0 0.0 175.6 0 1.4 0 2 0 3 0 95 da0 0.0 47.9 0.0 94.3 0 1.0 1 da1 0.0 54.9 0.0 84.8 0 1.4 2 da2 0.0 44.9 0.0 92.8 0 1.0 1 da3 0.0 48.9 0.0 83.3 0 1.4 2 da4 0.0 47.9 0.0 95.3 0 1.0 1 da5 0.0 53.9 0.0 87.8 0 1.2 2 extended device statistics cpu device r/s w/s kr/s kw/s wait svc_t %b us ni sy in id 0 0 0 0 99 extended device statistics cpu device r/s w/s kr/s kw/s wait svc_t %b us ni sy in id 0 0 0 0 100 extended device statistics cpu device r/s w/s kr/s kw/s wait svc_t %b us ni sy in id 0 0 0 0 100 extended device statistics cpu device r/s w/s kr/s kw/s wait svc_t %b us ni sy in id 0 0 0 0 100 extended device statistics cpu device r/s w/s kr/s kw/s wait svc_t %b us ni sy in id ad4 0.0 28.9 0.0 462.8 0 3.5 1 0 0 0 0 99 extended device statistics cpu device r/s w/s kr/s kw/s wait svc_t %b us ni sy in id da0 131.7 0.0 8392.0 0.0 35 73.6 60 0 0 4 3 93 da1 118.8 0.0 7561.7 0.0 35 74.9 62 da2 127.7 0.0 8136.5 0.0 35 77.7 63 da3 103.8 0.0 6603.7 0.0 29 137.6 79 da4 106.8 0.0 6795.8 0.0 33 112.9 74 da5 123.7 0.0 7919.4 0.0 35 126.9 83 extended device statistics cpu device r/s w/s kr/s kw/s wait svc_t %b us ni sy in id da0 51.9 89.9 3322.5 1487.5 4 77.4 38 0 0 6 5 89 da1 55.9 109.8 3578.1 1469.6 4 76.4 49 da2 58.9 88.9 3769.8 1604.8 1 75.3 46 da3 69.9 105.8 4472.6 1652.8 0 117.0 82 da4 66.9 85.9 4280.9 1490.0 3 112.4 70 da5 66.9 108.8 4280.9 1469.6 3 105.3 69 extended device statistics cpu device r/s w/s kr/s kw/s wait svc_t %b us ni sy in id da0 0.0 107.7 0.0 5069.1 10 53.0 80 0 0 9 8 83 da1 0.0 107.7 0.0 5069.1 10 54.4 81 da2 0.0 103.7 0.0 4953.9 10 50.4 73 da3 0.0 102.7 0.0 4890.1 10 51.7 74 da4 0.0 108.7 0.0 5299.5 6 50.0 74 da5 0.0 108.7 0.0 5299.5 6 51.6 76 extended device statistics cpu device r/s w/s kr/s kw/s wait svc_t %b us ni sy in id da0 0.0 106.7 0.0 5196.4 4 50.5 76 0 0 8 8 85 da1 0.0 106.7 0.0 5196.4 4 52.1 78 da2 0.0 103.7 0.0 5196.4 4 49.3 73 da3 0.0 103.7 0.0 5196.4 4 52.1 75 da4 0.0 103.7 0.0 4991.5 3 51.7 78 da5 0.0 103.7 0.0 4991.5 5 52.8 80 extended device statistics cpu device r/s w/s kr/s kw/s wait svc_t %b us ni sy in id da0 0.0 114.8 0.0 5404.5 0 47.5 81 0 0 9 7 84 da1 0.0 111.8 0.0 5250.9 3 49.4 78 da2 0.0 113.8 0.0 5404.5 0 48.3 73 da3 0.0 113.8 0.0 5404.5 0 50.1 75 da4 0.0 111.8 0.0 5379.1 0 48.0 78 da5 0.0 111.8 0.0 5379.1 0 49.1 81 extended device statistics cpu device r/s w/s kr/s kw/s wait svc_t %b us ni sy in id da0 0.0 102.7 0.0 4890.1 13 50.2 70 0 0 8 9 83 da1 0.0 105.7 0.0 5043.7 13 53.0 77 da2 0.0 105.7 0.0 5120.5 9 47.7 73 da3 0.0 105.7 0.0 5120.5 9 50.5 75 da4 0.0 107.7 0.0 5120.1 9 49.7 77 da5 0.0 107.7 0.0 5120.1 9 51.6 79 extended device statistics cpu device r/s w/s kr/s kw/s wait svc_t %b us ni sy in id da0 0.0 109.8 0.0 5353.4 0 52.6 81 0 0 7 7 85 da1 0.0 109.8 0.0 5353.4 0 53.9 81 da2 0.0 103.8 0.0 5122.3 0 53.0 78 da3 0.0 103.8 0.0 5122.3 0 54.5 78 da4 0.0 105.8 0.0 5122.8 0 52.8 80 da5 0.0 105.8 0.0 5122.8 0 54.7 81 extended device statistics cpu device r/s w/s kr/s kw/s wait svc_t %b us ni sy in id da0 0.0 107.8 0.0 5150.4 8 50.0 77 0 0 9 8 83 da1 0.0 107.8 0.0 5150.4 8 52.3 79 da2 0.0 107.8 0.0 5176.4 8 49.2 77 da3 0.0 107.8 0.0 5176.4 8 51.5 78 da4 0.0 108.8 0.0 5175.9 8 49.9 77 da5 0.0 108.8 0.0 5175.9 8 52.7 77 extended device statistics cpu device r/s w/s kr/s kw/s wait svc_t %b us ni sy in id da0 0.0 107.7 0.0 5299.6 0 53.5 78 0 0 7 7 86 da1 0.0 107.7 0.0 5299.6 0 56.1 78 da2 0.0 105.7 0.0 5146.0 2 54.0 82 da3 0.0 103.7 0.0 5043.3 4 54.1 77 da4 0.0 107.7 0.0 5248.7 1 54.1 78 da5 0.0 107.7 0.0 5248.7 1 56.2 78 extended device statistics cpu device r/s w/s kr/s kw/s wait svc_t %b us ni sy in id da0 0.0 106.8 0.0 5124.5 8 48.7 77 0 0 9 9 82 da1 0.0 102.8 0.0 4893.9 12 52.0 76 da2 0.0 105.8 0.0 5022.2 13 51.7 75 da3 0.0 107.8 0.0 5125.0 13 54.9 81 da4 0.0 107.8 0.0 5150.5 8 50.2 78 da5 0.0 107.8 0.0 5150.5 8 50.7 79 extended device statistics cpu device r/s w/s kr/s kw/s wait svc_t %b us ni sy in id da0 0.0 105.7 0.0 5171.9 1 53.0 78 0 0 9 7 84 da1 0.0 109.7 0.0 5402.3 1 54.8 81 da2 0.0 107.7 0.0 5274.1 3 51.5 81 da3 0.0 105.7 0.0 5171.4 5 52.0 77 da4 0.0 105.7 0.0 5146.0 1 53.8 76 da5 0.0 105.7 0.0 5146.0 1 55.1 78 extended device statistics cpu device r/s w/s kr/s kw/s wait svc_t %b us ni sy in id da0 0.0 107.8 0.0 5124.6 9 49.8 79 0 0 9 11 80 da1 0.0 105.8 0.0 4996.8 11 52.0 78 da2 0.0 105.8 0.0 5022.3 13 51.3 73 da3 0.0 107.8 0.0 5125.1 13 54.5 79 da4 0.0 107.8 0.0 5150.6 9 49.4 76 da5 0.0 107.8 0.0 5150.6 9 51.3 77 extended device statistics cpu device r/s w/s kr/s kw/s wait svc_t %b us ni sy in id da0 0.0 105.8 0.0 5148.1 0 53.2 79 0 0 6 7 87 da1 0.0 107.8 0.0 5275.8 0 54.7 79 da2 0.0 108.8 0.0 5378.6 0 52.2 81 da3 0.0 108.8 0.0 5378.6 0 54.1 81 da4 0.0 106.8 0.0 5148.1 0 53.9 82 da5 0.0 106.8 0.0 5148.1 0 55.6 82 extended device statistics cpu device r/s w/s kr/s kw/s wait svc_t %b us ni sy in id ad4 0.0 4.0 0.0 12.0 0 0.5 0 0 0 16 6 77 da0 0.0 108.8 0.0 4988.2 64 69.9 65 da1 0.0 109.8 0.0 5091.5 64 91.7 73 da2 0.0 107.8 0.0 5014.1 64 48.3 57 da3 0.0 86.8 0.0 3733.3 43 71.4 73 da4 0.0 109.8 0.0 5088.0 64 70.6 71 da5 0.0 104.8 0.0 4578.0 62 59.0 61 extended device statistics cpu device r/s w/s kr/s kw/s wait svc_t %b us ni sy in id da0 0.0 98.8 0.0 5724.0 69 490.3 109 0 0 2 1 97 da1 0.0 86.8 0.0 5007.5 69 562.7 102 da2 0.0 125.7 0.0 7222.9 69 465.5 114 da3 0.0 92.8 0.0 5303.4 70 538.2 100 da4 0.0 113.8 0.0 6557.3 70 486.6 104 da5 0.0 110.8 0.0 6339.2 69 468.6 114 extended device statistics cpu device r/s w/s kr/s kw/s wait svc_t %b us ni sy in id da0 0.0 104.8 0.0 6032.5 68 706.3 100 0 0 2 0 98 da1 0.0 103.8 0.0 5993.6 62 654.4 99 da2 0.0 104.8 0.0 6057.4 67 657.3 100 da3 0.0 104.8 0.0 6057.4 62 654.0 100 da4 0.0 104.8 0.0 6057.4 67 644.2 100 da5 0.0 103.8 0.0 5993.6 61 683.7 99 extended device statistics cpu device r/s w/s kr/s kw/s wait svc_t %b us ni sy in id da0 0.0 104.8 0.0 6057.3 67 624.9 100 0 0 1 1 98 da1 0.0 104.8 0.0 6057.3 63 531.2 100 da2 0.0 104.8 0.0 6057.3 68 625.4 100 da3 0.0 104.8 0.0 6032.3 63 551.6 100 da4 0.0 104.8 0.0 6057.3 68 625.7 100 da5 0.0 105.8 0.0 6121.1 61 510.1 101 extended device statistics cpu device r/s w/s kr/s kw/s wait svc_t %b us ni sy in id da0 0.0 106.8 0.0 6160.2 68 616.8 100 0 0 2 1 97 da1 0.0 106.8 0.0 6160.2 62 522.8 100 da2 0.0 106.8 0.0 6160.2 67 617.1 100 da3 0.0 106.8 0.0 6185.1 62 522.8 100 da4 0.0 106.8 0.0 6160.2 67 617.4 100 da5 0.0 106.8 0.0 6160.2 62 503.4 100 extended device statistics cpu device r/s w/s kr/s kw/s wait svc_t %b us ni sy in id da0 0.0 107.7 0.0 6221.3 68 607.9 100 0 0 0 2 98 da1 0.0 108.7 0.0 6285.1 63 515.2 100 da2 0.0 108.7 0.0 6285.1 68 608.3 101 da3 0.0 108.7 0.0 6260.2 63 515.3 100 da4 0.0 107.7 0.0 6221.3 67 608.6 100 da5 0.0 107.7 0.0 6221.3 62 496.0 100 extended device statistics cpu device r/s w/s kr/s kw/s wait svc_t %b us ni sy in id da0 0.0 109.8 0.0 6354.5 68 599.2 101 0 0 1 1 98 da1 0.0 109.8 0.0 6329.5 63 507.8 100 da2 0.0 108.8 0.0 6290.6 65 599.4 99 da3 0.0 108.8 0.0 6290.6 62 507.9 99 da4 0.0 109.8 0.0 6354.5 66 599.7 100 da5 0.0 109.8 0.0 6329.5 62 489.0 100 extended device statistics cpu device r/s w/s kr/s kw/s wait svc_t %b us ni sy in id da0 0.0 112.8 0.0 6518.3 31 586.1 100 0 0 1 1 98 da1 0.0 112.8 0.0 6518.3 53 480.6 100 da2 0.0 112.8 0.0 6493.4 5 573.1 100 da3 0.0 112.8 0.0 6174.5 63 677.5 100 da4 0.0 112.8 0.0 6493.4 15 576.6 100 da5 0.0 111.8 0.0 5690.1 42 846.4 100 extended device statistics cpu device r/s w/s kr/s kw/s wait svc_t %b us ni sy in id da0 0.0 159.6 0.0 5581.4 1 208.5 86 0 0 7 7 86 da1 0.0 188.5 0.0 6181.4 1 346.6 87 da2 0.0 131.7 0.0 4023.3 2 152.0 76 da3 0.0 191.5 0.0 7381.9 5 244.5 86 da4 0.0 140.6 0.0 4442.3 5 177.2 80 da5 0.0 164.6 0.0 5976.9 5 149.8 84 extended device statistics cpu device r/s w/s kr/s kw/s wait svc_t %b us ni sy in id da0 0.0 103.8 0.0 4945.3 13 50.8 78 0 0 12 7 81 da1 0.0 103.8 0.0 4945.3 13 52.6 79 da2 0.0 105.8 0.0 4983.7 13 52.0 77 da3 0.0 108.8 0.0 5150.5 13 54.2 80 da4 0.0 107.8 0.0 5150.5 13 52.5 79 da5 0.0 107.8 0.0 5150.5 13 54.9 79 extended device statistics cpu device r/s w/s kr/s kw/s wait svc_t %b us ni sy in id da0 0.0 106.8 0.0 5250.5 4 51.8 80 0 0 7 7 86 da1 0.0 106.8 0.0 5250.5 4 53.5 79 da2 0.0 107.8 0.0 5225.1 5 52.8 81 da3 0.0 107.8 0.0 5225.1 5 54.9 82 da4 0.0 102.8 0.0 5020.5 8 51.7 74 da5 0.0 102.8 0.0 5020.5 8 53.2 74 extended device statistics cpu device r/s w/s kr/s kw/s wait svc_t %b us ni sy in id da0 0.0 102.7 0.0 5069.2 0 53.3 77 0 0 7 7 85 da1 0.0 102.7 0.0 5069.2 0 56.2 80 da2 0.0 104.7 0.0 5094.7 0 54.2 80 da3 0.0 104.7 0.0 5094.7 0 55.7 80 da4 0.0 108.7 0.0 5299.6 0 55.0 83 da5 0.0 108.7 0.0 5299.6 0 57.2 83 extended device statistics cpu device r/s w/s kr/s kw/s wait svc_t %b us ni sy in id da0 0.0 109.8 0.0 5304.2 5 48.8 77 0 0 7 8 85 da1 0.0 109.8 0.0 5304.2 5 50.9 79 da2 0.0 105.8 0.0 5048.1 9 49.2 75 da3 0.0 105.8 0.0 5048.1 9 50.7 77 da4 0.0 106.8 0.0 5073.6 9 49.7 74 da5 0.0 106.8 0.0 5073.6 9 50.4 74 extended device statistics cpu device r/s w/s kr/s kw/s wait svc_t %b us ni sy in id da0 0.0 105.7 0.0 5095.2 1 55.4 80 0 0 10 8 83 da1 0.0 105.7 0.0 5095.2 1 57.4 80 da2 0.0 108.7 0.0 5376.5 1 54.4 81 da3 0.0 108.7 0.0 5376.5 1 56.5 82 da4 0.0 104.7 0.0 5120.1 5 53.7 79 da5 0.0 108.7 0.0 5350.5 1 56.6 86 extended device statistics cpu device r/s w/s kr/s kw/s wait svc_t %b us ni sy in id ad4 0.0 13.0 0.0 207.7 0 1.8 0 0 0 10 10 80 da0 0.0 107.8 0.0 5124.5 6 49.7 78 da1 0.0 107.8 0.0 5124.5 6 51.6 79 da2 0.0 104.8 0.0 4970.7 8 49.4 73 da3 0.0 102.8 0.0 4868.4 10 52.2 70 da4 0.0 110.8 0.0 5330.1 6 50.6 80 da5 0.0 106.8 0.0 5099.5 6 51.1 75 extended device statistics cpu device r/s w/s kr/s kw/s wait svc_t %b us ni sy in id ad4 0.0 1.0 0.0 2.0 0 0.2 0 0 0 8 10 82 da0 0.0 102.7 0.0 5069.0 0 51.9 78 da1 0.0 102.7 0.0 5069.0 0 53.1 78 da2 0.0 105.7 0.0 5197.2 0 52.0 77 da3 0.0 107.7 0.0 5299.4 0 53.1 81 da4 0.0 105.7 0.0 5069.0 0 54.6 79 da5 0.0 105.7 0.0 5069.0 0 55.1 81 extended device statistics cpu device r/s w/s kr/s kw/s wait svc_t %b us ni sy in id da0 0.0 105.8 0.0 5099.8 7 49.0 74 0 0 9 7 84 da1 0.0 105.8 0.0 5099.8 7 49.6 75 da2 0.0 110.8 0.0 5329.9 3 49.2 80 da3 0.0 110.8 0.0 5329.9 3 51.2 80 da4 0.0 109.8 0.0 5240.5 4 50.6 78 da5 0.0 106.8 0.0 5073.8 7 51.5 77 extended device statistics cpu device r/s w/s kr/s kw/s wait svc_t %b us ni sy in id da0 0.0 106.8 0.0 5198.3 0 53.9 80 0 0 6 7 87 da1 0.0 106.8 0.0 5198.3 0 56.4 82 da2 0.0 101.8 0.0 4968.3 0 53.5 77 da3 0.0 101.8 0.0 4968.3 0 54.4 80 da4 0.0 103.8 0.0 5057.6 0 53.3 75 da5 0.0 106.8 0.0 5224.3 0 55.6 77 extended device statistics cpu device r/s w/s kr/s kw/s wait svc_t %b us ni sy in id da0 0.0 106.8 0.0 5098.2 7 48.8 73 0 0 7 9 84 da1 0.0 106.8 0.0 5098.2 7 50.7 76 da2 0.0 110.8 0.0 5328.2 3 49.2 78 da3 0.0 110.8 0.0 5328.2 3 51.2 80 da4 0.0 108.8 0.0 5238.9 4 49.8 75 da5 0.0 105.8 0.0 5072.2 7 50.7 73 extended device statistics cpu device r/s w/s kr/s kw/s wait svc_t %b us ni sy in id da0 0.0 137.7 0.0 7561.1 70 223.8 94 0 0 13 2 85 da1 0.0 116.8 0.0 6087.7 70 241.2 94 da2 0.0 92.8 0.0 4976.1 70 297.9 91 da3 0.0 111.8 0.0 5906.1 70 277.4 72 da4 0.0 80.8 0.0 4320.9 70 279.3 88 da5 0.0 106.8 0.0 5625.7 70 316.1 91 extended device statistics cpu device r/s w/s kr/s kw/s wait svc_t %b us ni sy in id da0 0.0 111.7 0.0 6451.7 70 643.3 100 0 0 1 1 98 da1 0.0 111.7 0.0 6451.7 70 643.6 100 da2 0.0 111.7 0.0 6451.7 70 595.3 100 da3 0.0 104.7 0.0 6054.7 69 641.7 118 da4 0.0 112.7 0.0 6515.5 69 620.6 101 da5 0.0 103.7 0.0 5990.8 70 602.5 100 extended device statistics cpu device r/s w/s kr/s kw/s wait svc_t %b us ni sy in id da0 0.0 108.8 0.0 6287.9 69 635.6 100 0 0 2 2 97 da1 0.0 109.8 0.0 6326.8 70 636.0 100 da2 0.0 109.8 0.0 6326.8 70 635.7 100 da3 0.0 109.8 0.0 6351.7 69 635.9 101 da4 0.0 108.8 0.0 6262.9 70 635.4 100 da5 0.0 109.8 0.0 6326.8 70 635.3 100 extended device statistics cpu device r/s w/s kr/s kw/s wait svc_t %b us ni sy in id ad4 0.0 8.0 0.0 127.7 0 1.1 0 0 0 3 0 96 da0 0.0 109.8 0.0 6326.8 65 616.5 100 da1 0.0 108.8 0.0 6287.9 62 630.3 100 da2 0.0 108.8 0.0 6287.9 66 635.8 99 da3 0.0 109.8 0.0 6326.8 60 634.5 100 da4 0.0 109.8 0.0 6351.7 66 635.5 100 da5 0.0 108.8 0.0 6287.9 61 635.4 99 extended device statistics cpu device r/s w/s kr/s kw/s wait svc_t %b us ni sy in id da0 0.0 109.8 0.0 6351.7 65 562.5 100 0 0 4 1 95 da1 0.0 109.8 0.0 6351.7 62 507.9 100 da2 0.0 109.8 0.0 6351.7 66 586.2 100 da3 0.0 108.8 0.0 6287.9 61 471.3 99 da4 0.0 108.8 0.0 6287.9 65 576.8 99 da5 0.0 109.8 0.0 6351.7 61 503.6 100 extended device statistics cpu device r/s w/s kr/s kw/s wait svc_t %b us ni sy in id da0 0.0 108.8 0.0 6263.0 66 562.1 99 0 0 2 0 97 da1 0.0 108.8 0.0 6263.0 63 507.6 99 da2 0.0 109.8 0.0 6326.8 66 580.3 100 da3 0.0 109.8 0.0 6326.8 61 471.1 100 da4 0.0 109.8 0.0 6326.8 65 561.7 100 da5 0.0 109.8 0.0 6326.8 61 488.7 100 extended device statistics cpu device r/s w/s kr/s kw/s wait svc_t %b us ni sy in id da0 0.0 129.8 0.0 4390.7 0 380.0 70 0 0 1 4 95 da1 0.0 154.7 0.0 5853.3 0 440.6 91 da2 0.0 166.7 0.0 6683.4 0 410.9 96 da3 0.0 169.7 0.0 6133.9 0 439.4 95 da4 0.0 176.7 0.0 7427.7 0 424.0 98 da5 0.0 170.7 0.0 6736.9 0 449.1 97 extended device statistics cpu device r/s w/s kr/s kw/s wait svc_t %b us ni sy in id da0 0.0 111.8 0.0 5148.2 8 44.7 75 0 0 7 7 85 da1 0.0 111.8 0.0 5148.2 8 47.3 75 da2 0.0 109.8 0.0 5148.2 8 45.2 75 da3 0.0 109.8 0.0 5148.2 8 47.2 76 da4 0.0 107.8 0.0 4917.7 13 46.1 77 da5 0.0 107.8 0.0 4917.7 13 48.2 76 extended device statistics cpu device r/s w/s kr/s kw/s wait svc_t %b us ni sy in id da0 0.0 105.7 0.0 5196.9 0 52.8 80 0 0 8 9 83 da1 0.0 105.7 0.0 5196.9 0 55.7 83 da2 0.0 106.7 0.0 5171.9 1 53.8 78 da3 0.0 106.7 0.0 5171.9 1 55.9 78 da4 0.0 109.7 0.0 5427.8 0 52.2 81 da5 0.0 109.7 0.0 5427.8 0 54.8 82 extended device statistics cpu device r/s w/s kr/s kw/s wait svc_t %b us ni sy in id da0 0.0 106.8 0.0 5227.4 4 47.3 75 0 0 12 8 80 da1 0.0 106.8 0.0 5227.4 4 48.9 75 da2 0.0 107.8 0.0 4996.7 9 49.3 80 da3 0.0 107.8 0.0 4996.7 9 52.0 81 da4 0.0 104.8 0.0 4996.7 8 47.2 65 da5 0.0 104.8 0.0 4996.7 8 50.0 68 extended device statistics cpu device r/s w/s kr/s kw/s wait svc_t %b us ni sy in id da0 0.0 102.7 0.0 4966.8 0 53.2 76 0 0 8 9 83 da1 0.0 102.7 0.0 4966.8 0 56.1 77 da2 0.0 105.7 0.0 5222.6 0 52.4 76 da3 0.0 105.7 0.0 5222.6 0 55.4 78 da4 0.0 106.7 0.0 5196.7 0 53.6 81 da5 0.0 106.7 0.0 5196.7 0 56.1 81 extended device statistics cpu device r/s w/s kr/s kw/s wait svc_t %b us ni sy in id da0 0.0 111.8 0.0 5279.0 6 50.9 82 0 0 6 10 84 da1 0.0 113.8 0.0 5406.7 4 52.7 82 da2 0.0 106.8 0.0 5150.7 9 49.9 76 da3 0.0 106.8 0.0 5150.7 9 51.4 77 da4 0.0 106.8 0.0 5176.6 8 49.1 76 da5 0.0 106.8 0.0 5176.6 8 50.2 78 extended device statistics cpu device r/s w/s kr/s kw/s wait svc_t %b us ni sy in id da0 0.0 103.7 0.0 5146.0 0 53.0 77 0 0 7 9 84 da1 0.0 101.7 0.0 5018.3 0 55.6 79 da2 0.0 108.7 0.0 5273.7 0 54.7 82 da3 0.0 108.7 0.0 5273.7 0 55.2 82 da4 0.0 107.7 0.0 5248.2 0 54.3 80 da5 0.0 107.7 0.0 5248.2 0 56.9 80 extended device statistics cpu device r/s w/s kr/s kw/s wait svc_t %b us ni sy in id da0 0.0 106.8 0.0 5073.6 9 49.1 74 0 0 10 9 82 da1 0.0 106.8 0.0 5073.6 9 51.1 74 da2 0.0 108.8 0.0 5304.2 5 48.1 76 da3 0.0 108.8 0.0 5304.2 5 50.3 78 da4 0.0 106.8 0.0 5048.1 9 49.3 78 da5 0.0 106.8 0.0 5048.1 9 51.2 78 extended device statistics cpu device r/s w/s kr/s kw/s wait svc_t %b us ni sy in id da0 0.0 107.8 0.0 5122.3 11 50.3 79 0 0 8 8 83 da1 0.0 107.8 0.0 5122.3 11 52.7 83 da2 0.0 103.8 0.0 4892.3 11 50.4 71 da3 0.0 103.8 0.0 4892.3 11 51.7 71 da4 0.0 109.8 0.0 5378.8 6 48.5 74 da5 0.0 109.8 0.0 5378.8 6 50.0 77 extended device statistics cpu device r/s w/s kr/s kw/s wait svc_t %b us ni sy in id da0 0.0 108.5 0.0 5188.6 4 51.0 78 0 0 7 9 84 da1 0.0 108.5 0.0 5188.6 4 53.0 80 da2 0.0 106.6 0.0 5188.6 4 51.6 72 da3 0.0 106.6 0.0 5188.6 4 52.6 73 da4 0.0 99.6 0.0 4958.1 4 50.5 76 da5 0.0 99.6 0.0 4958.1 4 52.4 77 ^C # Here is a quick sample with sftp and ZFS (keep in mind ssh has encryption NFS does not) # ifstat -b -i re0 re0 Kbps in Kbps out 260931.9 8590.13 205778.7 6825.03 285794.0 9565.88 242095.0 8016.92 238060.4 7924.14 262288.7 8660.99 260094.6 8660.25 268791.7 8869.08 303607.5 10010.25 263964.4 8736.56 261743.6 8659.45 246390.6 8191.83 300105.3 10305.06 255019.9 8291.85 283355.3 9390.99 277390.9 9198.74 279070.0 9218.81 278103.5 9339.65 258649.8 8492.39 251343.6 8320.49 283036.9 9346.05 283302.4 9277.87 278886.2 9384.52 300472.5 9772.91 292509.8 9669.22 299081.4 9816.50 313643.0 10299.40 289208.9 9747.82 313738.8 10228.35 207670.6 6837.77 248762.1 8224.13 194959.1 6457.88 265009.2 9495.61 249181.9 8228.59 272940.6 8945.51 226037.8 7485.62 273701.2 9048.08 236890.4 7834.88 # iostat -zxC 1 extended device statistics cpu device r/s w/s kr/s kw/s wait svc_t %b us ni sy in id ad4 0.1 0.4 1.8 6.1 0 3.0 0 1 0 1 1 97 da0 2.0 13.4 117.8 551.1 16 203.8 10 da1 1.9 13.7 108.8 550.6 16 200.5 10 da2 2.1 13.4 117.8 551.1 20 203.0 10 da3 1.9 13.8 107.1 550.6 20 200.1 9 da4 2.0 13.4 116.8 551.1 16 203.4 9 da5 1.9 13.8 107.3 550.6 16 199.8 9 extended device statistics cpu device r/s w/s kr/s kw/s wait svc_t %b us ni sy in id da0 0.0 108.8 0.0 6289.0 69 315.8 101 21 0 18 13 48 da1 0.0 107.8 0.0 6225.1 70 321.8 102 da2 0.0 108.8 0.0 6289.0 69 331.0 100 da3 0.0 111.8 0.0 6455.7 70 322.7 101 da4 0.0 104.8 0.0 6058.4 69 326.7 101 da5 0.0 108.8 0.0 6289.0 69 327.1 101 extended device statistics cpu device r/s w/s kr/s kw/s wait svc_t %b us ni sy in id da0 0.0 109.8 0.0 6326.8 69 633.3 100 15 0 8 9 68 da1 0.0 109.8 0.0 6326.8 70 627.8 100 da2 0.0 109.8 0.0 6351.8 69 624.3 100 da3 0.0 108.8 0.0 6287.9 69 629.0 99 da4 0.0 109.8 0.0 6351.8 69 640.0 100 da5 0.0 109.8 0.0 6351.8 69 625.0 100 extended device statistics cpu device r/s w/s kr/s kw/s wait svc_t %b us ni sy in id da0 0.0 109.8 0.0 6351.9 44 634.9 100 3 0 3 3 91 da1 0.0 108.8 0.0 6288.0 46 634.5 99 da2 0.0 109.8 0.0 6326.9 41 634.8 100 da3 0.0 109.8 0.0 6351.9 39 634.6 100 da4 0.0 109.8 0.0 6326.9 48 635.2 100 da5 0.0 108.8 0.0 6263.0 45 635.2 99 extended device statistics cpu device r/s w/s kr/s kw/s wait svc_t %b us ni sy in id da0 0.0 111.8 0.0 6163.1 68 356.9 98 9 0 18 6 67 da1 0.0 127.7 0.0 7109.6 68 354.4 98 da2 0.0 106.8 0.0 5829.3 68 399.2 91 da3 0.0 104.8 0.0 5701.1 67 349.2 84 da4 0.0 113.8 0.0 6291.3 68 377.4 96 da5 0.0 110.8 0.0 6124.2 68 353.0 79 extended device statistics cpu device r/s w/s kr/s kw/s wait svc_t %b us ni sy in id da0 0.0 129.8 0.0 6805.8 70 519.5 100 19 0 14 19 49 da1 0.0 119.8 0.0 5371.1 70 502.2 100 da2 0.0 93.8 0.0 5408.1 58 607.6 104 da3 0.0 139.8 0.0 6706.4 70 457.6 111 da4 0.0 132.8 0.0 7582.5 60 476.8 102 da5 0.0 130.8 0.0 6156.3 69 511.3 117 extended device statistics cpu device r/s w/s kr/s kw/s wait svc_t %b us ni sy in id da0 0.0 109.8 0.0 6326.8 70 535.4 100 17 0 12 12 59 da1 0.0 109.8 0.0 6351.8 70 523.0 100 da2 0.0 109.8 0.0 6351.8 58 396.3 100 da3 0.0 109.8 0.0 6326.8 70 521.1 100 da4 0.0 109.8 0.0 6351.8 60 431.1 100 da5 0.0 109.8 0.0 6326.8 69 515.6 101 extended device statistics cpu device r/s w/s kr/s kw/s wait svc_t %b us ni sy in id da0 0.0 109.8 0.0 6351.7 70 632.7 100 18 0 11 15 56 da1 0.0 109.8 0.0 6326.8 70 632.7 100 da2 0.0 109.8 0.0 6326.8 58 414.1 100 da3 0.0 110.8 0.0 6415.6 69 632.4 101 da4 0.0 109.8 0.0 6326.8 60 468.3 100 da5 0.0 109.8 0.0 6351.7 69 632.5 100 extended device statistics cpu device r/s w/s kr/s kw/s wait svc_t %b us ni sy in id da0 0.0 104.8 0.0 5996.4 0 631.6 93 22 0 8 14 57 da1 0.0 123.7 0.0 6598.7 0 616.9 99 da2 0.0 135.7 0.0 7264.3 5 668.5 101 da3 0.0 115.8 0.0 6149.6 0 623.4 97 da4 0.0 99.8 0.0 5323.8 0 746.0 89 da5 0.0 125.7 0.0 6735.4 0 615.4 99 extended device statistics cpu device r/s w/s kr/s kw/s wait svc_t %b us ni sy in id da0 0.0 126.7 0.0 4006.1 70 115.8 43 17 0 18 12 54 da1 0.0 126.7 0.0 3634.9 70 115.3 51 da2 0.0 109.8 0.0 3090.1 70 100.1 52 da3 0.0 109.8 0.0 2708.9 70 107.4 48 da4 0.0 115.8 0.0 3163.9 70 92.1 52 da5 0.0 118.8 0.0 3196.3 70 111.5 52 extended device statistics cpu device r/s w/s kr/s kw/s wait svc_t %b us ni sy in id da0 0.0 103.8 0.0 5993.5 67 592.8 108 20 0 12 15 53 da1 0.0 114.8 0.0 6646.1 62 547.2 100 da2 0.0 117.8 0.0 6812.8 67 554.3 100 da3 0.0 108.8 0.0 6287.9 64 566.6 103 da4 0.0 108.8 0.0 6287.9 66 595.2 100 da5 0.0 104.8 0.0 6057.4 62 597.1 100 extended device statistics cpu device r/s w/s kr/s kw/s wait svc_t %b us ni sy in id da0 0.0 109.8 0.0 6326.6 67 597.7 100 23 0 10 15 52 da1 0.0 109.8 0.0 6326.6 62 546.1 100 da2 0.0 109.8 0.0 6326.6 67 601.6 100 da3 0.0 110.8 0.0 6390.5 62 563.9 101 da4 0.0 109.8 0.0 6351.5 66 603.0 100 da5 0.0 109.8 0.0 6326.6 62 566.4 100 extended device statistics cpu device r/s w/s kr/s kw/s wait svc_t %b us ni sy in id ad4 0.0 1.0 0.0 6.0 0 0.4 0 18 0 10 13 59 da0 0.0 109.7 0.0 6349.1 67 578.3 100 da1 0.0 109.7 0.0 6349.1 62 506.0 100 da2 0.0 109.7 0.0 6349.1 67 578.2 100 da3 0.0 109.7 0.0 6324.2 62 486.9 100 da4 0.0 109.7 0.0 6324.2 66 578.2 100 da5 0.0 109.7 0.0 6349.1 61 505.5 100 extended device statistics cpu device r/s w/s kr/s kw/s wait svc_t %b us ni sy in id da0 0.0 110.8 0.0 6367.5 37 577.3 101 18 0 10 12 60 da1 0.0 109.8 0.0 6303.6 40 505.1 100 da2 0.0 109.8 0.0 6303.6 44 577.4 100 da3 0.0 109.8 0.0 6303.6 60 486.3 100 da4 0.0 109.8 0.0 6303.6 47 577.3 100 da5 0.0 109.8 0.0 6316.6 58 500.2 100 extended device statistics cpu device r/s w/s kr/s kw/s wait svc_t %b us ni sy in id da0 0.0 145.7 0.0 5188.2 70 280.0 85 19 0 16 10 55 da1 0.0 146.7 0.0 4827.9 70 360.9 89 da2 0.0 154.7 0.0 6099.2 70 300.0 83 da3 0.0 173.6 0.0 6462.0 70 371.0 84 da4 0.0 163.7 0.0 6266.4 70 297.1 91 da5 0.0 176.6 0.0 6327.8 70 352.1 89 extended device statistics cpu device r/s w/s kr/s kw/s wait svc_t %b us ni sy in id da0 0.0 122.7 0.0 7082.2 69 507.1 102 25 0 11 13 51 da1 0.0 117.8 0.0 6812.8 70 514.6 99 da2 0.0 104.8 0.0 6057.3 69 589.2 108 da3 0.0 102.8 0.0 5929.6 69 591.3 109 da4 0.0 107.8 0.0 6224.0 70 561.0 101 da5 0.0 105.8 0.0 6096.3 70 566.3 104 extended device statistics cpu device r/s w/s kr/s kw/s wait svc_t %b us ni sy in id da0 0.0 109.7 0.0 6348.9 69 632.2 100 20 0 12 11 57 da1 0.0 109.7 0.0 6324.0 70 633.1 100 da2 0.0 109.7 0.0 6324.0 69 632.3 100 da3 0.0 109.7 0.0 6348.9 69 633.1 100 da4 0.0 109.7 0.0 6348.9 70 633.1 100 da5 0.0 109.7 0.0 6348.9 70 632.3 100 ^C #