From owner-freebsd-geom@FreeBSD.ORG Wed Mar 4 15:16:14 2015 Return-Path: Delivered-To: freebsd-geom@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 9095ECD0 for ; Wed, 4 Mar 2015 15:16:14 +0000 (UTC) Received: from mxout01.bytecamp.net (mxout01.bytecamp.net [212.204.60.217]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 33E9D13C for ; Wed, 4 Mar 2015 15:16:13 +0000 (UTC) Received: by mxout01.bytecamp.net (Postfix, from userid 1001) id 4CB1D30FE3D; Wed, 4 Mar 2015 16:16:12 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=bytecamp.net; h=message-id:date:from:mime-version:to:subject:references:in-reply-to:content-type:content-transfer-encoding; s=20140709; bh=M42okwipkT9gu0ivbS6RMPDfP9s=; b=yOQy0Xat7YHMOuTkplXv7oZIuC8trhNaJtxZuj2+y2L46tUsRzjK6KCelmjxW8axqaqmjYZp/NAsVNuS5nP3gkJV5IzMTFceVooa92SuclAdxB6BeSInx3/M+zcpJQ1W/JkJYu23Wwoc10iO8BevCyf63as+hxPCSrkW/fXJo3I= Received: from mail.bytecamp.net (mailstore.bytecamp.net [212.204.60.20]) by mxout01.bytecamp.net (Postfix) with ESMTP id 1FF4630FE2F for ; Wed, 4 Mar 2015 16:16:12 +0100 (CET) Received: (qmail 31551 invoked by uid 89); 4 Mar 2015 16:16:12 +0100 Received: from stella.bytecamp.net (HELO ?212.204.60.37?) (rs%bytecamp.net@212.204.60.37) by mail.bytecamp.net with AES128-SHA encrypted SMTP; 4 Mar 2015 16:16:12 +0100 Message-ID: <54F721BC.5070204@bytecamp.net> Date: Wed, 04 Mar 2015 16:16:12 +0100 From: Robert Schulze Organization: bytecamp GmbH User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:31.0) Gecko/20100101 Thunderbird/31.5.0 MIME-Version: 1.0 To: freebsd-geom@freebsd.org Subject: Re: Trim on gmirrored SSDs is slow and results in inresponsive system References: <54F6FE2E.60303@bytecamp.net> <54F707CC.6070105@multiplay.co.uk> <54F7147F.8070206@bytecamp.net> <54F718D9.1080201@multiplay.co.uk> In-Reply-To: <54F718D9.1080201@multiplay.co.uk> Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 8bit X-BeenThere: freebsd-geom@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: GEOM-specific discussions and implementations List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 04 Mar 2015 15:16:14 -0000 Hi, Am 04.03.2015 um 15:38 schrieb Steven Hartland: >>> Do you have one disk which has really slow TRIM? >>> >> please note, it does not depend on the disk, it is related to gmirror. > Yes but gmirror will wait for all disks to complete the delete before > the top level delete is marked as complete, so if you have an issue with > one of the underlying devices, be that cabling issue or disk on the way > out, this could be your issue. Both disks seem to be OK, see further down. >>> Also how does gstat -d -p compare between gmirror and none gmirror >>> installs on the same machine? >> Deleting on a non-mirrored UFS does not influence the system, because >> the BIO_DELETE calls are processed extremely fast (~ 1 sec with 1GB >> file), with a high number of d/s (~ 100k). > Can you get snapshot of both in action please I've collected data via $ iostat -w0.5 ada0 ada1 directly after unlinking a 2 GB file for all possible cases: 1) non mirrored on ada0: tty ada0 ada1 cpu tin tout KB/t tps MB/s KB/t tps MB/s us ni sy in id 1 154 111.45 241 26.20 14.73 12 0.17 0 0 5 0 95 0 662 32.00 122437 3826.17 0.00 0 0.00 0 0 7 0 92 0 124 0.00 0 0.00 0.00 0 0.00 0 0 0 0 100 2) non mirrored on ada1: tty ada0 ada1 cpu tin tout KB/t tps MB/s KB/t tps MB/s us ni sy in id 3 2876 12.78 4 0.05 108.38 190 20.14 0 0 4 0 96 0 672 0.00 0 0.00 32.00 120446 3763.94 0 0 8 0 92 0 430 0.00 0 0.00 0.00 0 0.00 0 0 0 0 100 3) gmirrored with ada0+ada1: tty ada0 ada1 cpu tin tout KB/t tps MB/s KB/t tps MB/s us ni sy in id 7 174 110.65 927 100.13 126.38 728 89.82 0 0 3 0 97 0 684 32.00 5629 175.90 32.00 5629 175.90 0 0 26 0 74 0 459 32.00 3587 112.08 32.00 3615 112.96 0 0 22 0 78 0 393 32.00 2192 68.49 32.00 2169 67.79 0 0 21 0 79 0 346 32.00 2102 65.67 32.00 2100 65.61 0 0 20 0 80 0 338 32.00 5391 168.47 32.00 4793 149.79 0 0 20 0 80 0 391 32.00 4461 139.39 32.00 5735 179.22 0 0 23 0 77 0 438 32.00 3008 93.99 32.00 2374 74.19 0 0 25 0 75 0 406 32.00 2141 66.90 32.00 2100 65.61 0 0 24 0 76 0 367 32.00 1825 57.04 32.00 1825 57.04 0 0 23 0 77 0 340 32.00 1835 57.33 32.00 1806 56.45 0 0 25 0 75 0 321 32.00 1517 47.40 32.00 1485 46.41 0 0 23 0 77 0 289 32.00 1376 43.00 32.00 1385 43.29 0 0 27 0 73 0 273 32.00 1271 39.71 32.00 1291 40.35 0 0 23 0 77 0 267 32.00 1220 38.12 32.00 1197 37.42 0 0 23 0 77 0 314 32.00 5665 177.04 32.00 5196 162.36 0 0 20 0 80 0 412 31.98 5568 173.88 31.98 6147 191.98 0 0 22 0 78 0 492 32.00 5530 172.81 32.00 5426 169.57 0 0 23 0 77 0 459 32.00 2505 78.30 32.00 2549 79.65 0 0 27 0 73 0 393 32.00 1940 60.62 32.00 1942 60.68 0 0 22 0 78 0 367 32.00 1882 58.80 32.00 1883 58.86 0 0 19 0 81 0 379 32.00 5076 158.63 32.00 5477 171.15 0 0 20 0 80 0 611 32.00 4432 138.51 32.00 3951 123.48 0 0 20 0 80 0 380 32.00 3062 95.69 32.00 2254 70.43 0 0 29 0 71 0 370 32.00 2365 73.90 32.00 1912 59.74 0 0 24 0 76 0 345 32.00 1940 60.62 32.00 1825 57.05 0 0 24 0 76 0 329 32.00 1229 38.40 32.00 2380 74.38 0 0 19 0 81 0 372 32.00 1692 52.89 32.00 1587 49.59 0 0 26 0 74 0 415 32.00 1588 49.64 32.00 1496 46.74 0 0 25 0 75 0 346 32.00 1456 45.50 32.00 1429 44.66 0 0 26 0 74 0 346 32.00 1355 42.33 32.00 1283 40.10 0 0 24 0 76 0 356 32.00 1155 36.09 32.00 1501 46.92 0 0 25 0 75 0 385 32.00 1177 36.78 32.00 1091 34.08 0 0 23 0 77 0 393 32.00 1805 56.39 32.00 1823 56.98 0 0 24 0 76 0 365 32.00 2021 63.15 32.00 2064 64.50 0 0 26 0 74 0 344 32.00 1868 58.39 32.00 1938 60.56 0 0 25 0 75 0 380 32.00 1897 59.27 32.00 1936 60.50 0 0 27 0 73 0 344 32.00 1774 55.45 32.00 1778 55.57 0 0 23 0 77 0 334 32.00 1674 52.31 32.00 1693 52.91 0 0 25 0 75 0 329 32.00 2254 70.43 32.00 2428 75.89 0 0 19 0 81 0 406 32.00 6331 197.85 32.00 6337 198.02 0 0 24 0 76 0 408 32.00 2966 92.69 32.00 2925 91.40 0 0 25 0 75 0 353 32.00 2280 71.25 32.00 2167 67.73 0 0 25 0 75 0 575 32.00 5051 157.84 32.00 4844 151.38 0 0 18 0 82 0 421 32.00 3801 118.79 32.00 4171 130.33 0 0 23 0 77 0 397 32.00 466 14.56 32.00 470 14.68 0 0 3 0 97 0 344 0.00 0 0.00 0.00 0 0.00 0 0 0 0 100 4) gmirrored with only one component active (ada0) tty ada0 ada1 cpu tin tout KB/t tps MB/s KB/t tps MB/s us ni sy in id 1 132 124.10 176 21.34 15.00 10 0.15 0 0 4 0 96 0 643 32.00 50662 1583.18 0.00 0 0.00 0 0 19 0 81 0 458 32.00 11102 346.93 0.00 0 0.00 0 0 18 0 82 0 287 32.00 4051 126.58 0.00 0 0.00 0 0 17 0 83 0 244 32.00 3500 109.38 0.00 0 0.00 0 0 18 0 82 0 256 32.00 3358 104.93 0.00 0 0.00 0 0 19 0 81 0 244 32.00 3010 94.06 0.00 0 0.00 0 0 18 0 82 0 260 32.00 2710 84.68 0.00 0 0.00 0 0 18 0 82 0 257 32.00 2524 78.88 0.00 0 0.00 0 0 18 0 82 0 234 32.00 2345 73.29 0.00 0 0.00 0 0 17 0 82 0 245 32.00 2185 68.28 0.00 0 0.00 0 0 19 0 81 0 239 32.00 2072 64.73 0.00 0 0.00 0 0 17 0 82 0 244 32.00 1970 61.56 0.00 0 0.00 0 0 18 0 81 0 243 32.00 1878 58.70 0.00 0 0.00 0 0 18 0 82 0 241 32.00 1799 56.22 0.00 0 0.00 0 0 19 0 81 0 231 32.00 1928 60.25 0.00 0 0.00 0 0 18 0 82 0 254 32.00 2442 76.30 0.00 0 0.00 0 0 15 0 85 0 252 32.00 2457 76.77 0.00 0 0.00 0 0 12 0 88 0 246 32.00 2461 76.89 0.00 0 0.00 0 0 13 0 87 0 235 32.00 2472 77.24 0.00 0 0.00 0 0 12 0 88 0 194 32.00 2491 77.83 0.00 0 0.00 0 0 13 0 87 0 201 32.00 2517 78.65 0.00 0 0.00 0 0 13 0 87 0 429 32.00 2545 79.54 0.00 0 0.00 0 0 12 0 88 0 231 31.99 9419 294.23 0.00 0 0.00 0 0 12 0 88 0 303 0.00 0 0.00 0.00 0 0.00 0 0 0 0 100 5) gmirrored with only one component active (ada1) tty ada0 ada1 cpu tin tout KB/t tps MB/s KB/t tps MB/s us ni sy in id 1 153 11.75 5 0.06 100.91 298 29.40 0 0 5 0 95 0 635 0.00 0 0.00 32.00 31614 987.94 0 0 19 0 81 0 372 0.00 0 0.00 32.00 20236 632.38 0 0 17 0 83 0 331 0.50 2 0.00 32.00 16922 528.74 0 0 15 0 85 0 350 0.00 0 0.00 32.00 20795 649.84 0 0 17 0 83 0 278 0.00 0 0.00 32.00 17748 554.63 0 0 15 0 85 0 276 0.00 0 0.00 32.00 14885 465.16 0 0 8 0 92 0 272 0.00 0 0.00 0.00 0 0.00 0 0 0 0 100 >> 0:40 144.87% cam >> 0:22 77.59% g_mirror var >> 0:14 62.79% bufdaemon >> 0:14 0.88% intr >> 0:13 0.59% geom >> 0:01 0.00% rand_harvestq >> 0:00 0.00% kernel > 144% in cam is very high and g_mirror is high too. > > What OS build is this? $ uname -a FreeBSD XXX 10.1-RELEASE-p5 FreeBSD 10.1-RELEASE-p5 #23 r278313M: Fri Feb 13 11:26:23 CET 2015 root@XXX:/usr/obj/usr/src/sys/BC amd64 HTH, Robert Schulze