Date: Wed, 04 Mar 2015 16:16:12 +0100 From: Robert Schulze <rs@bytecamp.net> To: freebsd-geom@freebsd.org Subject: Re: Trim on gmirrored SSDs is slow and results in inresponsive system Message-ID: <54F721BC.5070204@bytecamp.net> In-Reply-To: <54F718D9.1080201@multiplay.co.uk> References: <54F6FE2E.60303@bytecamp.net> <54F707CC.6070105@multiplay.co.uk> <54F7147F.8070206@bytecamp.net> <54F718D9.1080201@multiplay.co.uk>
next in thread | previous in thread | raw e-mail | index | archive | help
Hi,
Am 04.03.2015 um 15:38 schrieb Steven Hartland:
>>> Do you have one disk which has really slow TRIM?
>>>
>> please note, it does not depend on the disk, it is related to gmirror.
> Yes but gmirror will wait for all disks to complete the delete before
> the top level delete is marked as complete, so if you have an issue with
> one of the underlying devices, be that cabling issue or disk on the way
> out, this could be your issue.
Both disks seem to be OK, see further down.
>>> Also how does gstat -d -p compare between gmirror and none gmirror
>>> installs on the same machine?
>> Deleting on a non-mirrored UFS does not influence the system, because
>> the BIO_DELETE calls are processed extremely fast (~ 1 sec with 1GB
>> file), with a high number of d/s (~ 100k).
> Can you get snapshot of both in action please
I've collected data via
$ iostat -w0.5 ada0 ada1
directly after unlinking a 2 GB file for all possible cases:
1) non mirrored on ada0:
tty ada0 ada1 cpu
tin tout KB/t tps MB/s KB/t tps MB/s us ni sy in id
1 154 111.45 241 26.20 14.73 12 0.17 0 0 5 0 95
0 662 32.00 122437 3826.17 0.00 0 0.00 0 0 7 0 92
0 124 0.00 0 0.00 0.00 0 0.00 0 0 0 0 100
2) non mirrored on ada1:
tty ada0 ada1 cpu
tin tout KB/t tps MB/s KB/t tps MB/s us ni sy in id
3 2876 12.78 4 0.05 108.38 190 20.14 0 0 4 0 96
0 672 0.00 0 0.00 32.00 120446 3763.94 0 0 8 0 92
0 430 0.00 0 0.00 0.00 0 0.00 0 0 0 0 100
3) gmirrored with ada0+ada1:
tty ada0 ada1 cpu
tin tout KB/t tps MB/s KB/t tps MB/s us ni sy in id
7 174 110.65 927 100.13 126.38 728 89.82 0 0 3 0 97
0 684 32.00 5629 175.90 32.00 5629 175.90 0 0 26 0 74
0 459 32.00 3587 112.08 32.00 3615 112.96 0 0 22 0 78
0 393 32.00 2192 68.49 32.00 2169 67.79 0 0 21 0 79
0 346 32.00 2102 65.67 32.00 2100 65.61 0 0 20 0 80
0 338 32.00 5391 168.47 32.00 4793 149.79 0 0 20 0 80
0 391 32.00 4461 139.39 32.00 5735 179.22 0 0 23 0 77
0 438 32.00 3008 93.99 32.00 2374 74.19 0 0 25 0 75
0 406 32.00 2141 66.90 32.00 2100 65.61 0 0 24 0 76
0 367 32.00 1825 57.04 32.00 1825 57.04 0 0 23 0 77
0 340 32.00 1835 57.33 32.00 1806 56.45 0 0 25 0 75
0 321 32.00 1517 47.40 32.00 1485 46.41 0 0 23 0 77
0 289 32.00 1376 43.00 32.00 1385 43.29 0 0 27 0 73
0 273 32.00 1271 39.71 32.00 1291 40.35 0 0 23 0 77
0 267 32.00 1220 38.12 32.00 1197 37.42 0 0 23 0 77
0 314 32.00 5665 177.04 32.00 5196 162.36 0 0 20 0 80
0 412 31.98 5568 173.88 31.98 6147 191.98 0 0 22 0 78
0 492 32.00 5530 172.81 32.00 5426 169.57 0 0 23 0 77
0 459 32.00 2505 78.30 32.00 2549 79.65 0 0 27 0 73
0 393 32.00 1940 60.62 32.00 1942 60.68 0 0 22 0 78
0 367 32.00 1882 58.80 32.00 1883 58.86 0 0 19 0 81
0 379 32.00 5076 158.63 32.00 5477 171.15 0 0 20 0 80
0 611 32.00 4432 138.51 32.00 3951 123.48 0 0 20 0 80
0 380 32.00 3062 95.69 32.00 2254 70.43 0 0 29 0 71
0 370 32.00 2365 73.90 32.00 1912 59.74 0 0 24 0 76
0 345 32.00 1940 60.62 32.00 1825 57.05 0 0 24 0 76
0 329 32.00 1229 38.40 32.00 2380 74.38 0 0 19 0 81
0 372 32.00 1692 52.89 32.00 1587 49.59 0 0 26 0 74
0 415 32.00 1588 49.64 32.00 1496 46.74 0 0 25 0 75
0 346 32.00 1456 45.50 32.00 1429 44.66 0 0 26 0 74
0 346 32.00 1355 42.33 32.00 1283 40.10 0 0 24 0 76
0 356 32.00 1155 36.09 32.00 1501 46.92 0 0 25 0 75
0 385 32.00 1177 36.78 32.00 1091 34.08 0 0 23 0 77
0 393 32.00 1805 56.39 32.00 1823 56.98 0 0 24 0 76
0 365 32.00 2021 63.15 32.00 2064 64.50 0 0 26 0 74
0 344 32.00 1868 58.39 32.00 1938 60.56 0 0 25 0 75
0 380 32.00 1897 59.27 32.00 1936 60.50 0 0 27 0 73
0 344 32.00 1774 55.45 32.00 1778 55.57 0 0 23 0 77
0 334 32.00 1674 52.31 32.00 1693 52.91 0 0 25 0 75
0 329 32.00 2254 70.43 32.00 2428 75.89 0 0 19 0 81
0 406 32.00 6331 197.85 32.00 6337 198.02 0 0 24 0 76
0 408 32.00 2966 92.69 32.00 2925 91.40 0 0 25 0 75
0 353 32.00 2280 71.25 32.00 2167 67.73 0 0 25 0 75
0 575 32.00 5051 157.84 32.00 4844 151.38 0 0 18 0 82
0 421 32.00 3801 118.79 32.00 4171 130.33 0 0 23 0 77
0 397 32.00 466 14.56 32.00 470 14.68 0 0 3 0 97
0 344 0.00 0 0.00 0.00 0 0.00 0 0 0 0 100
4) gmirrored with only one component active (ada0)
tty ada0 ada1 cpu
tin tout KB/t tps MB/s KB/t tps MB/s us ni sy in id
1 132 124.10 176 21.34 15.00 10 0.15 0 0 4 0 96
0 643 32.00 50662 1583.18 0.00 0 0.00 0 0 19 0 81
0 458 32.00 11102 346.93 0.00 0 0.00 0 0 18 0 82
0 287 32.00 4051 126.58 0.00 0 0.00 0 0 17 0 83
0 244 32.00 3500 109.38 0.00 0 0.00 0 0 18 0 82
0 256 32.00 3358 104.93 0.00 0 0.00 0 0 19 0 81
0 244 32.00 3010 94.06 0.00 0 0.00 0 0 18 0 82
0 260 32.00 2710 84.68 0.00 0 0.00 0 0 18 0 82
0 257 32.00 2524 78.88 0.00 0 0.00 0 0 18 0 82
0 234 32.00 2345 73.29 0.00 0 0.00 0 0 17 0 82
0 245 32.00 2185 68.28 0.00 0 0.00 0 0 19 0 81
0 239 32.00 2072 64.73 0.00 0 0.00 0 0 17 0 82
0 244 32.00 1970 61.56 0.00 0 0.00 0 0 18 0 81
0 243 32.00 1878 58.70 0.00 0 0.00 0 0 18 0 82
0 241 32.00 1799 56.22 0.00 0 0.00 0 0 19 0 81
0 231 32.00 1928 60.25 0.00 0 0.00 0 0 18 0 82
0 254 32.00 2442 76.30 0.00 0 0.00 0 0 15 0 85
0 252 32.00 2457 76.77 0.00 0 0.00 0 0 12 0 88
0 246 32.00 2461 76.89 0.00 0 0.00 0 0 13 0 87
0 235 32.00 2472 77.24 0.00 0 0.00 0 0 12 0 88
0 194 32.00 2491 77.83 0.00 0 0.00 0 0 13 0 87
0 201 32.00 2517 78.65 0.00 0 0.00 0 0 13 0 87
0 429 32.00 2545 79.54 0.00 0 0.00 0 0 12 0 88
0 231 31.99 9419 294.23 0.00 0 0.00 0 0 12 0 88
0 303 0.00 0 0.00 0.00 0 0.00 0 0 0 0 100
5) gmirrored with only one component active (ada1)
tty ada0 ada1 cpu
tin tout KB/t tps MB/s KB/t tps MB/s us ni sy in id
1 153 11.75 5 0.06 100.91 298 29.40 0 0 5 0 95
0 635 0.00 0 0.00 32.00 31614 987.94 0 0 19 0 81
0 372 0.00 0 0.00 32.00 20236 632.38 0 0 17 0 83
0 331 0.50 2 0.00 32.00 16922 528.74 0 0 15 0 85
0 350 0.00 0 0.00 32.00 20795 649.84 0 0 17 0 83
0 278 0.00 0 0.00 32.00 17748 554.63 0 0 15 0 85
0 276 0.00 0 0.00 32.00 14885 465.16 0 0 8 0 92
0 272 0.00 0 0.00 0.00 0 0.00 0 0 0 0 100
>> 0:40 144.87% cam
>> 0:22 77.59% g_mirror var
>> 0:14 62.79% bufdaemon
>> 0:14 0.88% intr
>> 0:13 0.59% geom
>> 0:01 0.00% rand_harvestq
>> 0:00 0.00% kernel
> 144% in cam is very high and g_mirror is high too.
>
> What OS build is this?
$ uname -a
FreeBSD XXX 10.1-RELEASE-p5 FreeBSD 10.1-RELEASE-p5 #23 r278313M: Fri
Feb 13 11:26:23 CET 2015 root@XXX:/usr/obj/usr/src/sys/BC amd64
HTH,
Robert Schulze
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?54F721BC.5070204>