Skip site navigation (1)Skip section navigation (2)
Date:      Mon, 14 Jan 2013 10:40:10 +0100
From:      Nicolas Rachinsky <fbsd-mas-0@ml.turing-complete.org>
To:        Artem Belevich <art@freebsd.org>
Cc:        freebsd-fs <freebsd-fs@freebsd.org>
Subject:   Re: slowdown of zfs (tx->tx)
Message-ID:  <20130114094010.GA75529@mid.pc5.i.0x5.de>
In-Reply-To: <CAFqOu6gWpMsWN0pTBiv10WfwyGWMfO9GzMLWTtcVxHixr-_i3Q@mail.gmail.com>
References:  <20130108174225.GA17260@mid.pc5.i.0x5.de> <CAFqOu6jgA8RWV5d%2BrOBk8D=3Vu3yWSnDkAi1cFJ0esj4OpBy2Q@mail.gmail.com> <20130109162613.GA34276@mid.pc5.i.0x5.de> <CAFqOu6jrng=v8eVyhqV-PBqJM_dYy%2BU7X4%2B=ahBeoxvK4mxcSA@mail.gmail.com> <20130110193949.GA10023@mid.pc5.i.0x5.de> <20130111073417.GA95100@mid.pc5.i.0x5.de> <CAFqOu6gWpMsWN0pTBiv10WfwyGWMfO9GzMLWTtcVxHixr-_i3Q@mail.gmail.com>

next in thread | previous in thread | raw e-mail | index | archive | help

--6TrnltStXW4iwmi0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline

* Artem Belevich <art@freebsd.org> [2013-01-11 12:39 -0800]:
> On Thu, Jan 10, 2013 at 11:34 PM, Nicolas Rachinsky
> <fbsd-mas-0@ml.turing-complete.org> wrote:
> > * Nicolas Rachinsky <fbsd-mas-0@ml.turing-complete.org> [2013-01-10 20:39 +0100]:
> >> after replacing one of the controllers, all problems seem to have
> >> disappeared. Thank you very much for your advice!
> >
> > Now the problem is back.
> >
> > After changing the controller, there were no more timeouts logged.
> >
> > No UDMA_CRC_Error_Count changed.
> >
> 
> Is there anything special about ada8? It does seem to have noticeably
> higher service time compared to other disks.

Nothing I know of. The disks are Samsung HD103UJ and HD103SI, multiple
of each type.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x000f   100   100   051    Pre-fail  Always       -       0
  3 Spin_Up_Time            0x0007   073   073   011    Pre-fail  Always       -       8890
  4 Start_Stop_Count        0x0032   100   100   000    Old_age   Always       -       32
  5 Reallocated_Sector_Ct   0x0033   094   094   010    Pre-fail  Always       -       166
  7 Seek_Error_Rate         0x000f   100   100   051    Pre-fail  Always       -       0
  8 Seek_Time_Performance   0x0025   100   100   015    Pre-fail  Offline      -       10872
  9 Power_On_Hours          0x0032   099   099   000    Old_age   Always       -       5688
 10 Spin_Retry_Count        0x0033   100   100   051    Pre-fail  Always       -       0
 11 Calibration_Retry_Count 0x0012   100   100   000    Old_age   Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       31
 13 Read_Soft_Error_Rate    0x000e   100   100   000    Old_age   Always       -       0
183 Runtime_Bad_Block       0x0032   100   100   000    Old_age   Always       -       0
184 End-to-End_Error        0x0033   100   100   000    Pre-fail  Always       -       0
187 Reported_Uncorrect      0x0032   100   100   000    Old_age   Always       -       0
188 Command_Timeout         0x0032   100   100   000    Old_age   Always       -       0
190 Airflow_Temperature_Cel 0x0022   078   069   000    Old_age   Always       -       22 (Min/Max 21/25)
194 Temperature_Celsius     0x0022   077   067   000    Old_age   Always       -       23 (Min/Max 21/26)
195 Hardware_ECC_Recovered  0x001a   100   100   000    Old_age   Always       -       1259614646
196 Reallocated_Event_Count 0x0032   096   096   000    Old_age   Always       -       166
197 Current_Pending_Sector  0x0012   100   100   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0030   100   100   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x003e   100   100   000    Old_age   Always       -       0
200 Multi_Zone_Error_Rate   0x000a   100   099   000    Old_age   Always       -       5
201 Soft_Read_Error_Rate    0x000a   100   100   000    Old_age   Always       -       0

Reallocated_Sector_Ct did not increase during the last days.


> Cound you do gstat with 1-second interval. Some of the 5-second
> samples show that ada8 is the bottleneck -- it has its request queue
> full (L(q)=10) when all other drives were done with their jobs. And
> that's a 5-sec average. Its write service time also seems to be a lot
> higher than for other drives.

Attached.  I have replace ada8 by ada9, which is a Western Digital
Caviar Black.

Now ada0 and ada4 seem to be the bottleneck.

But I don't understand the intervalls without any disk activity.

> Does the drive have its write cache disabled by any chance? That could
> explain why it takes so much longer to service writes.

No, camcontrol identify says it's enabled.

> Can you remove ada8 and see if your performance go back to normal?

The problem still persists.

Thank you for your help!

Nicolas
-- 
http://www.rachinsky.de/nicolas

--6TrnltStXW4iwmi0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: attachment; filename="gstat.txt"

dT: 1.001s  w: 1.000s
 L(q)  ops/s    r/s   kBps   ms/r    w/s   kBps   ms/w   %busy Name
    0      0      0      0    0.0      0      0    0.0    0.0  ad4
    0      0      0      0    0.0      0      0    0.0    0.0  ad4s1
    0      0      0      0    0.0      0      0    0.0    0.0  ad4s1a
    0      0      0      0    0.0      0      0    0.0    0.0  ad4s1b
    0      0      0      0    0.0      0      0    0.0    0.0  ad6
    0      0      0      0    0.0      0      0    0.0    0.0  ad6s1
    0      0      0      0    0.0      0      0    0.0    0.0  ada0
    0      0      0      0    0.0      0      0    0.0    0.0  ada1
    0      0      0      0    0.0      0      0    0.0    0.0  ada2
    0      0      0      0    0.0      0      0    0.0    0.0  ada3
    0      0      0      0    0.0      0      0    0.0    0.0  ada4
    0      0      0      0    0.0      0      0    0.0    0.0  ada5
    0      0      0      0    0.0      0      0    0.0    0.0  ada6
    0      0      0      0    0.0      0      0    0.0    0.0  ada7
    0      0      0      0    0.0      0      0    0.0    0.0  ada8
    0      0      0      0    0.0      0      0    0.0    0.0  ada9
    0      0      0      0    0.0      0      0    0.0    0.0  ad6s1a
    0      0      0      0    0.0      0      0    0.0    0.0  ad6s1b
    0      0      0      0    0.0      0      0    0.0    0.0  mirror/ROOT121027
    0      0      0      0    0.0      0      0    0.0    0.0  mirror/SWAP121027
    0      0      0      0    0.0      0      0    0.0    0.0  mirror/ROOT121027.journal
dT: 1.001s  w: 1.000s
 L(q)  ops/s    r/s   kBps   ms/r    w/s   kBps   ms/w   %busy Name
    0      0      0      0    0.0      0      0    0.0    0.0  ad4
    0      0      0      0    0.0      0      0    0.0    0.0  ad4s1
    0      0      0      0    0.0      0      0    0.0    0.0  ad4s1a
    0      0      0      0    0.0      0      0    0.0    0.0  ad4s1b
    0      0      0      0    0.0      0      0    0.0    0.0  ad6
    0      0      0      0    0.0      0      0    0.0    0.0  ad6s1
    0     46      0      0    0.0     46     33    0.9    0.9  ada0
    0     47      0      0    0.0     47     32    3.3    1.9  ada1
    0     47      0      0    0.0     47     32    3.3    1.9  ada2
    0     47      0      0    0.0     47     32    3.3    1.9  ada3
    0     49      0      0    0.0     49     33    0.9    0.9  ada4
    0     49      0      0    0.0     49     33    3.6    2.1  ada5
    0     46      0      0    0.0     46     33    1.0    0.9  ada6
    0     46      0      0    0.0     46     33    0.8    0.8  ada7
    0      0      0      0    0.0      0      0    0.0    0.0  ada8
    0     49      0      0    0.0     49     33    0.8    0.8  ada9
    0      0      0      0    0.0      0      0    0.0    0.0  ad6s1a
    0      0      0      0    0.0      0      0    0.0    0.0  ad6s1b
    0      0      0      0    0.0      0      0    0.0    0.0  mirror/ROOT121027
    0      0      0      0    0.0      0      0    0.0    0.0  mirror/SWAP121027
    0      0      0      0    0.0      0      0    0.0    0.0  mirror/ROOT121027.journal
dT: 1.001s  w: 1.000s
 L(q)  ops/s    r/s   kBps   ms/r    w/s   kBps   ms/w   %busy Name
    0      0      0      0    0.0      0      0    0.0    0.0  ad4
    0      0      0      0    0.0      0      0    0.0    0.0  ad4s1
    0      0      0      0    0.0      0      0    0.0    0.0  ad4s1a
    0      0      0      0    0.0      0      0    0.0    0.0  ad4s1b
    0      0      0      0    0.0      0      0    0.0    0.0  ad6
    0      0      0      0    0.0      0      0    0.0    0.0  ad6s1
   10    429      0      0    0.0    429    997    2.5   18.1  ada0
    0    458     21     27    1.2    437    994    3.1   17.7  ada1
    0    406      0      0    0.0    406    988    2.6   14.6  ada2
    0    427      0      0    0.0    427    989    2.0   12.5  ada3
   10    335      0      0    0.0    335    938    4.1   22.9  ada4
    0    419      0      0    0.0    419    990    2.1   11.9  ada5
    0    434      0      0    0.0    434   1005    2.1   13.1  ada6
    0    486     25    133    5.2    461   1006    1.4   12.1  ada7
    0      0      0      0    0.0      0      0    0.0    0.0  ada8
    0    441     20     35    6.2    421    994    1.4   13.1  ada9
    0      0      0      0    0.0      0      0    0.0    0.0  ad6s1a
    0      0      0      0    0.0      0      0    0.0    0.0  ad6s1b
    0      0      0      0    0.0      0      0    0.0    0.0  mirror/ROOT121027
    0      0      0      0    0.0      0      0    0.0    0.0  mirror/SWAP121027
    0      0      0      0    0.0      0      0    0.0    0.0  mirror/ROOT121027.journal
dT: 1.002s  w: 1.000s
 L(q)  ops/s    r/s   kBps   ms/r    w/s   kBps   ms/w   %busy Name
    0      0      0      0    0.0      0      0    0.0    0.0  ad4
    0      0      0      0    0.0      0      0    0.0    0.0  ad4s1
    0      0      0      0    0.0      0      0    0.0    0.0  ad4s1a
    0      0      0      0    0.0      0      0    0.0    0.0  ad4s1b
    0      0      0      0    0.0      0      0    0.0    0.0  ad6
    0      0      0      0    0.0      0      0    0.0    0.0  ad6s1
    0    275      0      0    0.0    274    334   12.7   56.6  ada0
    0    278     11     14    3.0    266    308    2.0   28.7  ada1
    0    305      0      0    0.0    303    315    1.5   28.7  ada2
    0    303      0      0    0.0    301    311    1.6   14.7  ada3
    0    311      0      0    0.0    309    375   15.4   69.2  ada4
    0    285      0      0    0.0    283    310    2.1   15.8  ada5
    0    282      0      0    0.0    280    306    1.7   18.6  ada6
    0    307     11     17    2.3    294    318    1.0   19.1  ada7
    0      0      0      0    0.0      0      0    0.0    0.0  ada8
    0    329      9      6    0.5    318    312    0.7   12.4  ada9
    0      0      0      0    0.0      0      0    0.0    0.0  ad6s1a
    0      0      0      0    0.0      0      0    0.0    0.0  ad6s1b
    0      0      0      0    0.0      0      0    0.0    0.0  mirror/ROOT121027
    0      0      0      0    0.0      0      0    0.0    0.0  mirror/SWAP121027
    0      0      0      0    0.0      0      0    0.0    0.0  mirror/ROOT121027.journal
dT: 1.000s  w: 1.000s
 L(q)  ops/s    r/s   kBps   ms/r    w/s   kBps   ms/w   %busy Name
    0      0      0      0    0.0      0      0    0.0    0.0  ad4
    0      0      0      0    0.0      0      0    0.0    0.0  ad4s1
    0      0      0      0    0.0      0      0    0.0    0.0  ad4s1a
    0      0      0      0    0.0      0      0    0.0    0.0  ad4s1b
    0      0      0      0    0.0      0      0    0.0    0.0  ad6
    0      0      0      0    0.0      0      0    0.0    0.0  ad6s1
    0      0      0      0    0.0      0      0    0.0    0.0  ada0
    0      0      0      0    0.0      0      0    0.0    0.0  ada1
    0      0      0      0    0.0      0      0    0.0    0.0  ada2
    0      0      0      0    0.0      0      0    0.0    0.0  ada3
    0      0      0      0    0.0      0      0    0.0    0.0  ada4
    0      0      0      0    0.0      0      0    0.0    0.0  ada5
    0      0      0      0    0.0      0      0    0.0    0.0  ada6
    0      0      0      0    0.0      0      0    0.0    0.0  ada7
    0      0      0      0    0.0      0      0    0.0    0.0  ada8
    0      0      0      0    0.0      0      0    0.0    0.0  ada9
    0      0      0      0    0.0      0      0    0.0    0.0  ad6s1a
    0      0      0      0    0.0      0      0    0.0    0.0  ad6s1b
    0      0      0      0    0.0      0      0    0.0    0.0  mirror/ROOT121027
    0      0      0      0    0.0      0      0    0.0    0.0  mirror/SWAP121027
    0      0      0      0    0.0      0      0    0.0    0.0  mirror/ROOT121027.journal
dT: 1.000s  w: 1.000s
 L(q)  ops/s    r/s   kBps   ms/r    w/s   kBps   ms/w   %busy Name
    0      0      0      0    0.0      0      0    0.0    0.0  ad4
    0      0      0      0    0.0      0      0    0.0    0.0  ad4s1
    0      0      0      0    0.0      0      0    0.0    0.0  ad4s1a
    0      0      0      0    0.0      0      0    0.0    0.0  ad4s1b
    0      0      0      0    0.0      0      0    0.0    0.0  ad6
    0      0      0      0    0.0      0      0    0.0    0.0  ad6s1
    0     23      7      7    0.2     16     21    1.3    0.6  ada0
    0     29     15     39    7.8     14     20    1.6    3.6  ada1
    0     17      2      2    0.2     15     20    1.4    0.6  ada2
    0     16      2      2    0.2     14     19    1.4    0.6  ada3
    0     19      5      5    0.2     14     19    1.2    0.5  ada4
    0     19      5      5    0.2     14     19    1.2    0.5  ada5
    0     23      7      7    0.2     16     22    1.2    0.6  ada6
    0     29     13      9    0.2     16     20    1.1    0.6  ada7
    0      0      0      0    0.0      0      0    0.0    0.0  ada8
    0     28     14     42   11.8     14     21    1.0   11.5  ada9
    0      0      0      0    0.0      0      0    0.0    0.0  ad6s1a
    0      0      0      0    0.0      0      0    0.0    0.0  ad6s1b
    0      0      0      0    0.0      0      0    0.0    0.0  mirror/ROOT121027
    0      0      0      0    0.0      0      0    0.0    0.0  mirror/SWAP121027
    0      0      0      0    0.0      0      0    0.0    0.0  mirror/ROOT121027.journal
dT: 1.001s  w: 1.000s
 L(q)  ops/s    r/s   kBps   ms/r    w/s   kBps   ms/w   %busy Name
    0      0      0      0    0.0      0      0    0.0    0.0  ad4
    0      0      0      0    0.0      0      0    0.0    0.0  ad4s1
    0      0      0      0    0.0      0      0    0.0    0.0  ad4s1a
    0      0      0      0    0.0      0      0    0.0    0.0  ad4s1b
    0      0      0      0    0.0      0      0    0.0    0.0  ad6
    0      0      0      0    0.0      0      0    0.0    0.0  ad6s1
    0      0      0      0    0.0      0      0    0.0    0.0  ada0
    0      0      0      0    0.0      0      0    0.0    0.0  ada1
    0      0      0      0    0.0      0      0    0.0    0.0  ada2
    0      0      0      0    0.0      0      0    0.0    0.0  ada3
    0      0      0      0    0.0      0      0    0.0    0.0  ada4
    0      0      0      0    0.0      0      0    0.0    0.0  ada5
    0      0      0      0    0.0      0      0    0.0    0.0  ada6
    0      0      0      0    0.0      0      0    0.0    0.0  ada7
    0      0      0      0    0.0      0      0    0.0    0.0  ada8
    0      0      0      0    0.0      0      0    0.0    0.0  ada9
    0      0      0      0    0.0      0      0    0.0    0.0  ad6s1a
    0      0      0      0    0.0      0      0    0.0    0.0  ad6s1b
    0      0      0      0    0.0      0      0    0.0    0.0  mirror/ROOT121027
    0      0      0      0    0.0      0      0    0.0    0.0  mirror/SWAP121027
    0      0      0      0    0.0      0      0    0.0    0.0  mirror/ROOT121027.journal
dT: 1.001s  w: 1.000s
 L(q)  ops/s    r/s   kBps   ms/r    w/s   kBps   ms/w   %busy Name
    0      0      0      0    0.0      0      0    0.0    0.0  ad4
    0      0      0      0    0.0      0      0    0.0    0.0  ad4s1
    0      0      0      0    0.0      0      0    0.0    0.0  ad4s1a
    0      0      0      0    0.0      0      0    0.0    0.0  ad4s1b
    0      0      0      0    0.0      0      0    0.0    0.0  ad6
    0      0      0      0    0.0      0      0    0.0    0.0  ad6s1
    0      0      0      0    0.0      0      0    0.0    0.0  ada0
    0      0      0      0    0.0      0      0    0.0    0.0  ada1
    0      0      0      0    0.0      0      0    0.0    0.0  ada2
    0      0      0      0    0.0      0      0    0.0    0.0  ada3
    0      0      0      0    0.0      0      0    0.0    0.0  ada4
    0      0      0      0    0.0      0      0    0.0    0.0  ada5
    0      0      0      0    0.0      0      0    0.0    0.0  ada6
    0      0      0      0    0.0      0      0    0.0    0.0  ada7
    0      0      0      0    0.0      0      0    0.0    0.0  ada8
    0      0      0      0    0.0      0      0    0.0    0.0  ada9
    0      0      0      0    0.0      0      0    0.0    0.0  ad6s1a
    0      0      0      0    0.0      0      0    0.0    0.0  ad6s1b
    0      0      0      0    0.0      0      0    0.0    0.0  mirror/ROOT121027
    0      0      0      0    0.0      0      0    0.0    0.0  mirror/SWAP121027
    0      0      0      0    0.0      0      0    0.0    0.0  mirror/ROOT121027.journal
dT: 1.002s  w: 1.000s
 L(q)  ops/s    r/s   kBps   ms/r    w/s   kBps   ms/w   %busy Name
    0      0      0      0    0.0      0      0    0.0    0.0  ad4
    0      0      0      0    0.0      0      0    0.0    0.0  ad4s1
    0      0      0      0    0.0      0      0    0.0    0.0  ad4s1a
    0      0      0      0    0.0      0      0    0.0    0.0  ad4s1b
    0      0      0      0    0.0      0      0    0.0    0.0  ad6
    0      0      0      0    0.0      0      0    0.0    0.0  ad6s1
    0    142      0      0    0.0    142    624    2.6    7.8  ada0
    0    148      0      0    0.0    148    628    3.4    9.2  ada1
    0    147      0      0    0.0    147    634    2.1    7.2  ada2
    0    148      0      0    0.0    148    629    2.3    7.6  ada3
    0    146      0      0    0.0    146    633    1.7    6.6  ada4
    5    140      0      0    0.0    140    623    3.2    8.6  ada5
    0    149      0      0    0.0    149    634    1.8    6.9  ada6
    0    142      0      0    0.0    142    624    1.3    6.1  ada7
    0      0      0      0    0.0      0      0    0.0    0.0  ada8
    0    146      0      0    0.0    146    627    1.4    6.2  ada9
    0      0      0      0    0.0      0      0    0.0    0.0  ad6s1a
    0      0      0      0    0.0      0      0    0.0    0.0  ad6s1b
    0      0      0      0    0.0      0      0    0.0    0.0  mirror/ROOT121027
    0      0      0      0    0.0      0      0    0.0    0.0  mirror/SWAP121027
    0      0      0      0    0.0      0      0    0.0    0.0  mirror/ROOT121027.journal
dT: 1.000s  w: 1.000s
 L(q)  ops/s    r/s   kBps   ms/r    w/s   kBps   ms/w   %busy Name
    0      0      0      0    0.0      0      0    0.0    0.0  ad4
    0      0      0      0    0.0      0      0    0.0    0.0  ad4s1
    0      0      0      0    0.0      0      0    0.0    0.0  ad4s1a
    0      0      0      0    0.0      0      0    0.0    0.0  ad4s1b
    0      0      0      0    0.0      0      0    0.0    0.0  ad6
    0      0      0      0    0.0      0      0    0.0    0.0  ad6s1
   10    842      0      0    0.0    842    714    8.1   80.0  ada0
    0    872     42     85    8.3    830    715    2.6   42.4  ada1
    0    943      0      0    0.0    943    764    1.3   18.3  ada2
    0    954      0      0    0.0    954    773    1.5   20.0  ada3
   10    815      0      0    0.0    815    700    7.8   73.2  ada4
    0    935      0      0    0.0    935    750    1.6   21.4  ada5
    0    880      0      0    0.0    880    753    3.4   40.4  ada6
    0    910     46    133    8.6    864    704    1.4   25.7  ada7
    0      0      0      0    0.0      0      0    0.0    0.0  ada8
    0    925     44     71    7.7    881    710    1.8   35.9  ada9
    0      0      0      0    0.0      0      0    0.0    0.0  ad6s1a
    0      0      0      0    0.0      0      0    0.0    0.0  ad6s1b
    0      0      0      0    0.0      0      0    0.0    0.0  mirror/ROOT121027
    0      0      0      0    0.0      0      0    0.0    0.0  mirror/SWAP121027
    0      0      0      0    0.0      0      0    0.0    0.0  mirror/ROOT121027.journal
dT: 1.001s  w: 1.000s
 L(q)  ops/s    r/s   kBps   ms/r    w/s   kBps   ms/w   %busy Name
    0      0      0      0    0.0      0      0    0.0    0.0  ad4
    0      0      0      0    0.0      0      0    0.0    0.0  ad4s1
    0      0      0      0    0.0      0      0    0.0    0.0  ad4s1a
    0      0      0      0    0.0      0      0    0.0    0.0  ad4s1b
    0      0      0      0    0.0      0      0    0.0    0.0  ad6
    0      0      0      0    0.0      0      0    0.0    0.0  ad6s1
    0     53      0      0    0.0     51     49   19.0   33.0  ada0
    0      6      0      0    0.0      4      4    0.4   30.9  ada1
    0      6      0      0    0.0      4      4    0.3   29.2  ada2
    0      6      0      0    0.0      4      4    0.3    5.1  ada3
    0     41      0      0    0.0     39     38   24.8   32.6  ada4
    0      6      0      0    0.0      4      4    0.4    7.6  ada5
    0      6      0      0    0.0      4      4    0.3   14.9  ada6
    0      6      0      0    0.0      4      4    0.2   10.2  ada7
    0      0      0      0    0.0      0      0    0.0    0.0  ada8
    0      6      0      0    0.0      4      4    0.5    9.7  ada9
    0      0      0      0    0.0      0      0    0.0    0.0  ad6s1a
    0      0      0      0    0.0      0      0    0.0    0.0  ad6s1b
    0      0      0      0    0.0      0      0    0.0    0.0  mirror/ROOT121027
    0      0      0      0    0.0      0      0    0.0    0.0  mirror/SWAP121027
    0      0      0      0    0.0      0      0    0.0    0.0  mirror/ROOT121027.journal
dT: 1.000s  w: 1.000s
 L(q)  ops/s    r/s   kBps   ms/r    w/s   kBps   ms/w   %busy Name
    0      0      0      0    0.0      0      0    0.0    0.0  ad4
    0      0      0      0    0.0      0      0    0.0    0.0  ad4s1
    0      0      0      0    0.0      0      0    0.0    0.0  ad4s1a
    0      0      0      0    0.0      0      0    0.0    0.0  ad4s1b
    0      0      0      0    0.0      0      0    0.0    0.0  ad6
    0      0      0      0    0.0      0      0    0.0    0.0  ad6s1
    0      0      0      0    0.0      0      0    0.0    0.0  ada0
    0      0      0      0    0.0      0      0    0.0    0.0  ada1
    0      0      0      0    0.0      0      0    0.0    0.0  ada2
    0      0      0      0    0.0      0      0    0.0    0.0  ada3
    0      0      0      0    0.0      0      0    0.0    0.0  ada4
    0      0      0      0    0.0      0      0    0.0    0.0  ada5
    0      0      0      0    0.0      0      0    0.0    0.0  ada6
    0      0      0      0    0.0      0      0    0.0    0.0  ada7
    0      0      0      0    0.0      0      0    0.0    0.0  ada8
    0      0      0      0    0.0      0      0    0.0    0.0  ada9
    0      0      0      0    0.0      0      0    0.0    0.0  ad6s1a
    0      0      0      0    0.0      0      0    0.0    0.0  ad6s1b
    0      0      0      0    0.0      0      0    0.0    0.0  mirror/ROOT121027
    0      0      0      0    0.0      0      0    0.0    0.0  mirror/SWAP121027
    0      0      0      0    0.0      0      0    0.0    0.0  mirror/ROOT121027.journal
dT: 1.000s  w: 1.000s
 L(q)  ops/s    r/s   kBps   ms/r    w/s   kBps   ms/w   %busy Name
    0      0      0      0    0.0      0      0    0.0    0.0  ad4
    0      0      0      0    0.0      0      0    0.0    0.0  ad4s1
    0      0      0      0    0.0      0      0    0.0    0.0  ad4s1a
    0      0      0      0    0.0      0      0    0.0    0.0  ad4s1b
    0      0      0      0    0.0      0      0    0.0    0.0  ad6
    0      0      0      0    0.0      0      0    0.0    0.0  ad6s1
    0     19      6      6    3.4     13     19    1.5    2.6  ada0
    0     28     14     41   12.1     14     21    1.7   10.8  ada1
    0     15      1      1    8.7     14     21    1.4    1.4  ada2
    0     15      1      1    0.2     14     21    1.6    0.6  ada3
    0     19      5      5    4.4     14     21    1.5    2.8  ada4
    0     19      5      5    3.7     14     21    1.7    2.4  ada5
    0     20      6      6    2.7     14     21    1.5    2.2  ada6
    0     26     13      9    7.9     13     18    1.4    4.4  ada7
    0      0      0      0    0.0      0      0    0.0    0.0  ada8
    0     27     14      9   17.8     13     20    1.4   10.3  ada9
    0      0      0      0    0.0      0      0    0.0    0.0  ad6s1a
    0      0      0      0    0.0      0      0    0.0    0.0  ad6s1b
    0      0      0      0    0.0      0      0    0.0    0.0  mirror/ROOT121027
    0      0      0      0    0.0      0      0    0.0    0.0  mirror/SWAP121027
    0      0      0      0    0.0      0      0    0.0    0.0  mirror/ROOT121027.journal
dT: 1.002s  w: 1.000s
 L(q)  ops/s    r/s   kBps   ms/r    w/s   kBps   ms/w   %busy Name
    0      0      0      0    0.0      0      0    0.0    0.0  ad4
    0      0      0      0    0.0      0      0    0.0    0.0  ad4s1
    0      0      0      0    0.0      0      0    0.0    0.0  ad4s1a
    0      0      0      0    0.0      0      0    0.0    0.0  ad4s1b
    0      0      0      0    0.0      0      0    0.0    0.0  ad6
    0      0      0      0    0.0      0      0    0.0    0.0  ad6s1
    0      0      0      0    0.0      0      0    0.0    0.0  ada0
    0      0      0      0    0.0      0      0    0.0    0.0  ada1
    0      0      0      0    0.0      0      0    0.0    0.0  ada2
    0      0      0      0    0.0      0      0    0.0    0.0  ada3
    0      0      0      0    0.0      0      0    0.0    0.0  ada4
    0      0      0      0    0.0      0      0    0.0    0.0  ada5
    0      0      0      0    0.0      0      0    0.0    0.0  ada6
    0      0      0      0    0.0      0      0    0.0    0.0  ada7
    0      0      0      0    0.0      0      0    0.0    0.0  ada8
    0      0      0      0    0.0      0      0    0.0    0.0  ada9
    0      0      0      0    0.0      0      0    0.0    0.0  ad6s1a
    0      0      0      0    0.0      0      0    0.0    0.0  ad6s1b
    0      0      0      0    0.0      0      0    0.0    0.0  mirror/ROOT121027
    0      0      0      0    0.0      0      0    0.0    0.0  mirror/SWAP121027
    0      0      0      0    0.0      0      0    0.0    0.0  mirror/ROOT121027.journal
dT: 1.001s  w: 1.000s
 L(q)  ops/s    r/s   kBps   ms/r    w/s   kBps   ms/w   %busy Name
    0      0      0      0    0.0      0      0    0.0    0.0  ad4
    0      0      0      0    0.0      0      0    0.0    0.0  ad4s1
    0      0      0      0    0.0      0      0    0.0    0.0  ad4s1a
    0      0      0      0    0.0      0      0    0.0    0.0  ad4s1b
    0      0      0      0    0.0      0      0    0.0    0.0  ad6
    0      0      0      0    0.0      0      0    0.0    0.0  ad6s1
    0      0      0      0    0.0      0      0    0.0    0.0  ada0
    0      0      0      0    0.0      0      0    0.0    0.0  ada1
    0      0      0      0    0.0      0      0    0.0    0.0  ada2
    0      0      0      0    0.0      0      0    0.0    0.0  ada3
    0      0      0      0    0.0      0      0    0.0    0.0  ada4
    0      0      0      0    0.0      0      0    0.0    0.0  ada5
    0      0      0      0    0.0      0      0    0.0    0.0  ada6
    0      0      0      0    0.0      0      0    0.0    0.0  ada7
    0      0      0      0    0.0      0      0    0.0    0.0  ada8
    0      0      0      0    0.0      0      0    0.0    0.0  ada9
    0      0      0      0    0.0      0      0    0.0    0.0  ad6s1a
    0      0      0      0    0.0      0      0    0.0    0.0  ad6s1b
    0      0      0      0    0.0      0      0    0.0    0.0  mirror/ROOT121027
    0      0      0      0    0.0      0      0    0.0    0.0  mirror/SWAP121027
    0      0      0      0    0.0      0      0    0.0    0.0  mirror/ROOT121027.journal
dT: 1.001s  w: 1.000s
 L(q)  ops/s    r/s   kBps   ms/r    w/s   kBps   ms/w   %busy Name
    0      0      0      0    0.0      0      0    0.0    0.0  ad4
    0      0      0      0    0.0      0      0    0.0    0.0  ad4s1
    0      0      0      0    0.0      0      0    0.0    0.0  ad4s1a
    0      0      0      0    0.0      0      0    0.0    0.0  ad4s1b
    0      0      0      0    0.0      0      0    0.0    0.0  ad6
    0      0      0      0    0.0      0      0    0.0    0.0  ad6s1
    0      0      0      0    0.0      0      0    0.0    0.0  ada0
    0      0      0      0    0.0      0      0    0.0    0.0  ada1
    0      0      0      0    0.0      0      0    0.0    0.0  ada2
    0      0      0      0    0.0      0      0    0.0    0.0  ada3
    0      0      0      0    0.0      0      0    0.0    0.0  ada4
    0      0      0      0    0.0      0      0    0.0    0.0  ada5
    0      0      0      0    0.0      0      0    0.0    0.0  ada6
    0      0      0      0    0.0      0      0    0.0    0.0  ada7
    0      0      0      0    0.0      0      0    0.0    0.0  ada8
    0      0      0      0    0.0      0      0    0.0    0.0  ada9
    0      0      0      0    0.0      0      0    0.0    0.0  ad6s1a
    0      0      0      0    0.0      0      0    0.0    0.0  ad6s1b
    0      0      0      0    0.0      0      0    0.0    0.0  mirror/ROOT121027
    0      0      0      0    0.0      0      0    0.0    0.0  mirror/SWAP121027
    0      0      0      0    0.0      0      0    0.0    0.0  mirror/ROOT121027.journal
dT: 1.001s  w: 1.000s
 L(q)  ops/s    r/s   kBps   ms/r    w/s   kBps   ms/w   %busy Name
    0      0      0      0    0.0      0      0    0.0    0.0  ad4
    0      0      0      0    0.0      0      0    0.0    0.0  ad4s1
    0      0      0      0    0.0      0      0    0.0    0.0  ad4s1a
    0      0      0      0    0.0      0      0    0.0    0.0  ad4s1b
    0      0      0      0    0.0      0      0    0.0    0.0  ad6
    0      0      0      0    0.0      0      0    0.0    0.0  ad6s1
    0      0      0      0    0.0      0      0    0.0    0.0  ada0
    0      0      0      0    0.0      0      0    0.0    0.0  ada1
    0      0      0      0    0.0      0      0    0.0    0.0  ada2
    0      0      0      0    0.0      0      0    0.0    0.0  ada3
    0      0      0      0    0.0      0      0    0.0    0.0  ada4
    0      0      0      0    0.0      0      0    0.0    0.0  ada5
    0      0      0      0    0.0      0      0    0.0    0.0  ada6
    0      0      0      0    0.0      0      0    0.0    0.0  ada7
    0      0      0      0    0.0      0      0    0.0    0.0  ada8
    0      0      0      0    0.0      0      0    0.0    0.0  ada9
    0      0      0      0    0.0      0      0    0.0    0.0  ad6s1a
    0      0      0      0    0.0      0      0    0.0    0.0  ad6s1b
    0      0      0      0    0.0      0      0    0.0    0.0  mirror/ROOT121027
    0      0      0      0    0.0      0      0    0.0    0.0  mirror/SWAP121027
    0      0      0      0    0.0      0      0    0.0    0.0  mirror/ROOT121027.journal
dT: 1.002s  w: 1.000s
 L(q)  ops/s    r/s   kBps   ms/r    w/s   kBps   ms/w   %busy Name
    0      0      0      0    0.0      0      0    0.0    0.0  ad4
    0      0      0      0    0.0      0      0    0.0    0.0  ad4s1
    0      0      0      0    0.0      0      0    0.0    0.0  ad4s1a
    0      0      0      0    0.0      0      0    0.0    0.0  ad4s1b
    0      0      0      0    0.0      0      0    0.0    0.0  ad6
    0      0      0      0    0.0      0      0    0.0    0.0  ad6s1
    0     87      0      0    0.0     87    481    3.0    4.2  ada0
    0     88      0      0    0.0     88    477    3.8    4.9  ada1
    0     92      0      0    0.0     92    483    2.8    4.1  ada2
    0     89      0      0    0.0     89    477    2.3    3.5  ada3
    0    100      0      0    0.0    100    480    1.8    3.4  ada4
    0    100      0      0    0.0    100    475    3.6    5.4  ada5
    0     87      0      0    0.0     87    483    1.8    3.2  ada6
    0     89      0      0    0.0     89    480    1.6    3.0  ada7
    0      0      0      0    0.0      0      0    0.0    0.0  ada8
    0    100      0      0    0.0    100    474    1.5    3.1  ada9
    0      0      0      0    0.0      0      0    0.0    0.0  ad6s1a
    0      0      0      0    0.0      0      0    0.0    0.0  ad6s1b
    0      0      0      0    0.0      0      0    0.0    0.0  mirror/ROOT121027
    0      0      0      0    0.0      0      0    0.0    0.0  mirror/SWAP121027
    0      6      0      0    0.0      6     68    0.2    0.0  mirror/ROOT121027.journal
dT: 1.001s  w: 1.000s
 L(q)  ops/s    r/s   kBps   ms/r    w/s   kBps   ms/w   %busy Name
    0      0      0      0    0.0      0      0    0.0    0.0  ad4
    0      0      0      0    0.0      0      0    0.0    0.0  ad4s1
    0      0      0      0    0.0      0      0    0.0    0.0  ad4s1a
    0      0      0      0    0.0      0      0    0.0    0.0  ad4s1b
    0      0      0      0    0.0      0      0    0.0    0.0  ad6
    0      0      0      0    0.0      0      0    0.0    0.0  ad6s1
   10    449      0      0    0.0    449    400    9.0   49.6  ada0
    0    477     26     68    4.3    451    418    2.6   21.9  ada1
    0    469      0      0    0.0    469    417    1.7   11.2  ada2
    0    466      0      0    0.0    466    412    1.7   10.9  ada3
   10    378      0      0    0.0    378    305    9.3   43.0  ada4
    0    475      0      0    0.0    475    414    1.8   12.1  ada5
    0    430      0      0    0.0    430    407    8.7   41.3  ada6
    0    511     27     35    8.0    484    423    1.2   14.2  ada7
    0      0      0      0    0.0      0      0    0.0    0.0  ada8
    0    505     26     49   10.4    479    416    0.7   17.8  ada9
    0      0      0      0    0.0      0      0    0.0    0.0  ad6s1a
    0      0      0      0    0.0      0      0    0.0    0.0  ad6s1b
    0      0      0      0    0.0      0      0    0.0    0.0  mirror/ROOT121027
    0      0      0      0    0.0      0      0    0.0    0.0  mirror/SWAP121027
    0      0      0      0    0.0      0      0    0.0    0.0  mirror/ROOT121027.journal
dT: 1.002s  w: 1.000s
 L(q)  ops/s    r/s   kBps   ms/r    w/s   kBps   ms/w   %busy Name
    0      0      0      0    0.0      0      0    0.0    0.0  ad4
    0      0      0      0    0.0      0      0    0.0    0.0  ad4s1
    0      0      0      0    0.0      0      0    0.0    0.0  ad4s1a
    0      0      0      0    0.0      0      0    0.0    0.0  ad4s1b
    0      0      0      0    0.0      0      0    0.0    0.0  ad6
    0      0      0      0    0.0      0      0    0.0    0.0  ad6s1
    0    139      0      0    0.0    137    155    6.4   24.4  ada0
    0    138      9      4    1.1    127    131    1.8   27.3  ada1
    0    153      0      0    0.0    151    137    1.7   22.6  ada2
    0    150      0      0    0.0    148    135    1.3   11.2  ada3
    0    140      0      0    0.0    138    237   15.8   41.0  ada4
    0    141      0      0    0.0    139    134    1.3   14.1  ada5
    0    129      0      0    0.0    127    128    1.4   16.2  ada6
    0    158      5      2    0.7    151    128    0.5   15.0  ada7
    0      0      0      0    0.0      0      0    0.0    0.0  ada8
    0    153      5      2   11.9    146    138    0.6   19.9  ada9
    0      0      0      0    0.0      0      0    0.0    0.0  ad6s1a
    0      0      0      0    0.0      0      0    0.0    0.0  ad6s1b
    0      0      0      0    0.0      0      0    0.0    0.0  mirror/ROOT121027
    0      0      0      0    0.0      0      0    0.0    0.0  mirror/SWAP121027
    0      0      0      0    0.0      0      0    0.0    0.0  mirror/ROOT121027.journal

--6TrnltStXW4iwmi0--



Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20130114094010.GA75529>