Date: Tue, 13 Dec 2005 12:16:08 -0500 From: Mark Bucciarelli <mark@gaiahost.coop> To: freebsd-performance@freebsd.org Subject: [LONG] vmstat: What I/O is blocked and how to fix it? Message-ID: <20051213171608.GM2188@rabbit>
next in thread | raw e-mail | index | archive | help
This is a MIME-formatted message. If you see this text it means that your E-mail software does not support MIME-formatted messages. --=_biodiesel.gaiahost.coop-13953-1134494156-0001-2 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Content-Disposition: inline [Originally sent to freebsd-questions, then I remembered this list (to which I am not subscribed, so please CC me on any replies.] m --=_biodiesel.gaiahost.coop-13953-1134494156-0001-2 Content-Type: message/rfc822 Content-Disposition: inline Return-path: <owner-freebsd-questions@freebsd.org> Received: from solar.gaiahost.coop [64.95.78.5] by localhost with POP3 (fetchmail-6.2.5) for mark@localhost (single-drop); Tue, 13 Dec 2005 10:03:36 -0500 (EST) Received: from mx2.freebsd.org (unverified [216.136.204.119]) by solar.gaiahost.coop (Rockliffe SMTPRA 4.5.6) with ESMTP id <B0006273351@solar.gaiahost.coop> for <mark@gaiahost.coop>; Tue, 13 Dec 2005 09:55:47 -0500 Received: from hub.freebsd.org (hub.freebsd.org [216.136.204.18]) by mx2.freebsd.org (Postfix) with ESMTP id 9D55C5B259; Tue, 13 Dec 2005 14:57:46 +0000 (GMT) (envelope-from owner-freebsd-questions@freebsd.org) Received: from hub.freebsd.org (localhost [127.0.0.1]) by hub.freebsd.org (Postfix) with ESMTP id 5924E16A422; Tue, 13 Dec 2005 14:57:46 +0000 (GMT) (envelope-from owner-freebsd-questions@freebsd.org) X-Original-To: freebsd-questions@freebsd.org Delivered-To: freebsd-questions@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 46B3B16A420 for <freebsd-questions@freebsd.org>; Tue, 13 Dec 2005 14:54:20 +0000 (GMT) (envelope-from mark@gaiahost.coop) Received: from biodiesel.gaiahost.coop (biodiesel.gaiahost.coop [64.95.78.120]) by mx1.FreeBSD.org (Postfix) with ESMTP id 8073C43D8C for <freebsd-questions@freebsd.org>; Tue, 13 Dec 2005 14:53:58 +0000 (GMT) (envelope-from mark@gaiahost.coop) Received: from localhost (host-64-65-195-19.spr.choiceone.net [::ffff:64.65.195.19]) (AUTH: LOGIN mark@hubcapconsulting.com) by biodiesel.gaiahost.coop with esmtp; Tue, 13 Dec 2005 09:53:54 -0500 id 006FC064.439EE083.00007756 Received: by localhost (sSMTP sendmail emulation); Tue, 13 Dec 2005 09:54:06 -0500 Date: Tue, 13 Dec 2005 09:54:06 -0500 From: Mark Bucciarelli <mark@gaiahost.coop> To: freebsd-questions@freebsd.org Message-ID: <20051213145406.GF2188@rabbit> Mail-Followup-To: freebsd-questions@freebsd.org Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="=_biodiesel.gaiahost.coop-13953-1134494156-0001-3" Content-Disposition: inline User-Agent: Mutt/1.4.2.1i Subject: [LONG] vmstat: What I/O is blocked and how to fix it? X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: User questions <freebsd-questions.freebsd.org> List-Unsubscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-questions>, <mailto:freebsd-questions-request@freebsd.org?subject=unsubscribe> List-Archive: <http://lists.freebsd.org/pipermail/freebsd-questions>; List-Post: <mailto:freebsd-questions@freebsd.org> List-Help: <mailto:freebsd-questions-request@freebsd.org?subject=help> List-Subscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-questions>, <mailto:freebsd-questions-request@freebsd.org?subject=subscribe> Sender: owner-freebsd-questions@freebsd.org Errors-To: owner-freebsd-questions@freebsd.org This is a MIME-formatted message. If you see this text it means that your E-mail software does not support MIME-formatted messages. --=_biodiesel.gaiahost.coop-13953-1134494156-0001-3 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Content-Disposition: inline On two occasions recently, vmstat has showed me that a number of processes are blocked due to I/O. At the same time, the number of disk transactions per second reported is a small fraction of the disk's capability. I am thinking some kind of VM config will help, but based on what I have read, it is almost always best to let the default kernel behavior do its thing. Setup: Dual PIII 1.0 GHz (256MB L1 and L2 cache) 3GB RAM custom kernel (SMP + PF) 5.4-RELEASE one intel fxp0 network interface used (second unused) Case 1: (da0 36GB SCSI) The first case is simple: running spamd-setup to load the huge Composite Blacklist into spamd. The only other things running are sshd and spamd itself. When I run vmstat (vmstat -w 1 -c 10 output attached), it shows ten processes blocked due to I/O, 1 runnable and a total of four disk transactions during this particular ten-second interval. Accoring to postmark [1] da0 is capable of 250 transactions per second. Interrupts look fine (200 of the 300/sec is the clock, right?). No swapping. No network traffic to speak of. High CPU usage. Paging looks really high to me. I have plenty of RAM and swap--so why is it so high? Is there a kernel param I can tweak to improve performance? Note: While this was running, I tried running vmstat -m and got the kmem_map too small error. (I read the FAQ entry and haven't yet rebuilt the kernel using VM_KMEM_SIZE_MAX.) Should I make VM_KMEM_SIZE_MAX 400MB as mentioned in the FAQ? Any other kernel settings to switch from defaults when I build the kernel? Case 2: (mirror/gm0, da0 + da1 300GB SCSI) This case is more complex and more important. This box was heavily loaded, running SpamAssassin + ClamAv in one jail, Courier-MTA + MySql in another, and dspamd + pf in the jail host. It was processing about one incoming email per second, and had 20,000 entries in the spamd greylist and 4,000 in the whitelist. The black list was not loaded. Courier-authlib was using MySQL to figure out where to deliver the mail to for each of 20,000 virtual users. Courier-MTA was calling the ClamAV daemon via TCP/IP from a courier filter and calling SpamAssassin via TCP/IP from maildrop. Spamd (as in OpenBSD, not Apache) and courier were hitting the disk hard--I watched top in "m" mode [2] and over a five minute period these two apps had about pretty much 50% each of the WRITES. The only thing that jumps out at me is that when things finally do get written to disk, the context switches go really high. Is this a VM issue of some sort? The disk transactions per second are really low; this 300GB SCSI can do 730/sec according to postmark (well, this was testing on a single disk not with the RAID1 gmirror). Thanks for any tips. m [1] /usr/ports/benchmarks/postmark [2] Where can I find docs on what the columns mean in top's "m" mode? --=_biodiesel.gaiahost.coop-13953-1134494156-0001-3 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="vmstat.case1.out" procs memory page disks faults cpu r b w avm fre flt re pi po fr sr da0 da1 in sy cs us sy id 1 10 0 190424 1653924 1065 0 0 0 1337 0 0 0 796 2931 1971 3 8 89 1 10 0 190520 1653824 17407 0 0 0 17401 0 0 0 356 111 280 44 56 0 1 10 0 167376 1676992 23264 0 0 0 23264 0 0 0 359 119 301 40 23 37 1 10 0 190808 1659240 27728 0 0 0 29152 0 0 0 351 114 297 39 14 47 1 10 0 190936 1676864 18992 0 0 0 17544 0 4 0 352 115 286 39 45 17 1 10 0 191032 1676808 17586 0 0 0 17586 0 0 0 349 115 291 41 52 7 1 10 0 191160 1658572 28025 0 0 0 29374 0 0 0 347 114 270 42 10 47 1 10 0 191320 1676668 24949 0 0 0 23576 0 0 0 346 122 280 36 16 47 1 10 0 191384 1652968 17730 0 0 0 17724 0 0 0 350 111 278 41 59 0 1 10 0 191512 1665912 20409 0 0 0 23688 0 0 0 373 113 304 39 40 21 --=_biodiesel.gaiahost.coop-13953-1134494156-0001-3 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="vmstat.case2.out" procs memory page disks faults cpu r b w avm fre flt re pi po fr sr da0 da1 in sy cs us sy id 0 27 0 345932 2622260 333 0 0 0 318 0 0 0 504 1261 842 8 3 89 0 25 0 341644 2622880 78 0 0 0 215 0 0 0 535 195 575 0 5 95 0 25 0 341644 2622880 0 0 0 0 0 0 0 0 455 159 458 0 6 94 0 25 0 341644 2622880 5 0 0 0 5 0 5 5 488 165 572 0 4 96 1 25 0 343856 2622408 270 0 0 0 126 0 3 4 472 867 706 1 7 92 0 28 0 355672 2620588 2356 0 0 0 1701 0 44 44 749 6224 1988 24 13 63 1 29 0 357616 2620356 1961 0 0 0 1757 0 44 44 819 7174 2333 46 15 39 1 29 0 357620 2620360 809 0 0 0 735 0 25 25 700 2694 1247 65 8 27 0 29 0 350148 2621460 79 0 0 0 349 0 18 17 654 1144 896 32 6 62 1 28 0 348808 2621640 140 0 0 0 175 0 2 2 438 433 509 0 5 95 1 29 0 351016 2621284 261 0 0 0 137 0 10 10 530 781 781 0 8 92 0 29 0 351196 2620928 1790 0 0 0 1540 0 32 32 724 4914 1799 19 13 68 7 27 0 366168 2618696 3657 0 0 0 2835 0 75 75 1003 9967 2644 58 21 21 4 28 0 377836 2616740 2383 0 0 0 1719 0 44 43 898 7199 1926 84 16 0 3 28 0 348996 2621212 2004 0 0 0 2851 0 74 75 948 7730 2200 62 15 23 2 28 0 351156 2620636 396 0 0 0 335 0 119 127 928 951 1849 3 11 86 1 29 0 358544 2619816 2006 0 0 0 1702 0 104 94 1046 7823 2761 33 17 49 1 29 0 352904 2620540 791 0 0 0 920 0 21 22 582 2467 1138 23 8 69 procs memory page disks faults cpu r b w avm fre flt re pi po fr sr da0 da1 in sy cs us sy id 0 30 0 353364 2620304 1004 0 0 0 837 0 16 15 652 3595 1202 33 10 57 3 28 0 358624 2619544 2025 0 0 0 1693 0 42 42 925 5461 2103 20 13 67 1 28 0 356484 2619684 1398 0 0 0 1396 0 28 28 1034 10604 4947 47 14 39 3 27 0 362108 2618540 2623 0 0 0 2138 0 151 151 1156 6026 2356 67 24 9 1 27 0 354620 2619880 1314 0 0 0 1615 0 39 40 808 5280 1401 83 12 4 1 29 0 359088 2618976 2360 0 0 0 1960 0 47 46 888 7626 2158 41 18 41 6 28 0 364524 2617952 2360 0 0 0 2031 0 37 37 1009 10575 3258 73 15 13 2 30 0 357952 2618624 3010 0 0 0 2987 0 77 77 1123 9732 2629 64 19 17 3 28 0 363976 2617728 2358 0 0 0 1962 0 43 43 987 7696 1977 73 13 14 5 28 0 371768 2616336 2283 0 0 0 1833 0 37 37 862 6972 1760 80 17 4 3 28 0 379872 2615124 3693 0 0 0 3158 0 77 78 1070 12627 3007 73 19 8 8 29 0 374012 2616044 638 0 0 0 845 0 30 30 759 3607 1310 90 10 0 2 31 0 363880 2617056 2532 0 0 0 2586 0 56 55 1100 8357 2252 74 17 9 3 30 0 370596 2616180 1959 0 0 0 1702 0 43 43 839 8104 2062 76 14 11 3 31 0 370756 2616252 173 0 0 0 152 0 12 12 596 606 854 96 4 0 1 30 0 361572 2617508 2026 0 0 0 2190 0 44 45 838 7504 1644 63 13 25 3 30 0 354344 2618260 1301 0 0 0 1433 0 30 29 716 4942 1371 50 10 41 2 28 0 360372 2617440 2233 0 0 0 1871 0 33 33 745 6363 1656 42 13 45 procs memory page disks faults cpu r b w avm fre flt re pi po fr sr da0 da1 in sy cs us sy id 4 29 0 369572 2615616 3118 0 0 0 2442 0 44 44 1104 11499 3007 61 18 20 5 30 0 364748 2616200 2856 0 0 0 2805 0 78 77 1100 9827 2589 81 19 0 2 29 0 373248 2614944 3685 0 0 0 3174 0 75 76 1095 13451 3296 70 26 4 2 29 0 367108 2615596 1974 0 0 0 2005 0 62 63 883 6594 2055 77 21 2 1 28 0 358020 2617080 1106 0 0 0 1392 0 34 33 667 6640 1427 57 38 4 0 28 0 350552 2618204 68 0 0 0 339 0 9 9 441 983 511 21 25 54 1 29 0 360044 2616796 1556 0 0 0 1104 0 17 17 612 3628 1252 5 12 82 0 30 0 352804 2617552 199 0 0 0 551 0 375 375 2084 1935 6164 26 18 56 1 28 0 358052 2616864 2037 0 0 0 1696 0 551 551 2999 6123 10286 47 32 20 0 28 0 357948 2617000 2011 0 0 0 1931 0 41 42 754 6509 1661 39 13 48 0 27 0 348212 2618536 237 0 0 0 578 0 13 13 498 1282 664 24 7 68 1 27 0 355664 2617472 1489 0 0 0 1081 0 12 12 547 2983 1044 13 10 76 2 28 0 350432 2618136 331 0 0 0 440 0 11 10 543 1486 846 7 5 88 1 28 0 350424 2618096 896 0 0 0 831 0 38 38 765 2849 1568 9 9 81 1 29 0 350640 2617780 524 0 0 0 418 0 170 182 1210 1469 2260 10 22 68 4 27 0 365384 2616412 2592 0 0 0 2178 0 80 68 967 8095 2464 64 16 20 0 27 0 322376 2633292 730 0 0 0 4882 0 34 34 611 3569 1248 65 10 25 1 28 0 332068 2631740 1081 0 0 0 621 0 12 12 556 2683 1099 7 9 84 procs memory page disks faults cpu r b w avm fre flt re pi po fr sr da0 da1 in sy cs us sy id 3 28 0 327040 2632400 570 0 0 0 750 0 25 26 868 3020 1662 15 9 76 4 28 0 339792 2629996 2594 0 0 0 1861 0 36 36 1175 8680 2634 65 19 16 2 29 0 341052 2629700 2682 0 0 0 2441 0 65 65 1130 8717 2452 76 16 8 2 28 0 326736 2631800 130 0 0 0 627 0 17 16 641 2223 1003 14 7 80 0 29 0 326804 2631596 311 0 0 0 243 0 7 7 659 1446 1135 3 10 88 1 29 0 326828 2631324 1086 0 0 0 961 0 25 25 835 3648 1619 14 9 77 1 29 0 334092 2629960 1032 0 0 0 830 0 17 18 904 4961 1147 63 12 25 0 29 0 326776 2630852 813 0 0 0 1048 0 13 12 1037 3148 1203 28 13 58 4 28 0 334316 2630088 2358 0 0 0 2581 0 58 58 1281 55999 20971 29 38 33 2 27 0 344276 2628704 2424 0 0 0 1960 0 50 51 887 8730 2223 77 15 8 6 26 0 335252 2630092 1008 0 0 0 1275 0 33 32 701 4373 1130 85 10 6 1 26 0 329360 2631012 1615 0 0 0 1713 0 40 40 746 4744 1592 34 12 53 0 26 0 321892 2632104 69 0 0 0 323 0 10 11 614 1045 833 6 7 88 0 26 0 321892 2632104 15 0 0 0 15 0 6 6 499 265 648 0 6 94 1 25 0 321812 2632172 326 0 0 0 330 0 8 8 549 761 746 1 4 94 2 25 0 327236 2631288 927 0 0 0 635 0 15 15 516 2279 922 24 9 67 0 25 0 319764 2632388 60 0 0 0 447 0 188 188 1111 864 2635 4 11 85 1 25 0 319764 2632388 0 0 0 0 0 0 2 2 389 107 366 0 4 96 procs memory page disks faults cpu r b w avm fre flt re pi po fr sr da0 da1 in sy cs us sy id 3 26 0 329384 2630900 1229 0 0 0 743 0 14 14 638 2543 1171 3 10 88 0 26 0 321988 2631920 960 0 0 0 1156 0 30 30 691 3809 1321 53 9 38 1 26 0 322024 2631852 44 0 0 0 29 0 10 10 591 450 901 0 7 93 2 27 0 338424 2629412 1875 0 0 0 1170 0 27 27 678 5395 1730 10 12 78 1 27 0 331624 2630436 346 0 0 0 547 0 9 8 462 1473 693 61 6 33 0 28 0 326352 2631132 698 0 0 0 789 0 13 13 618 2052 1007 9 6 85 1 28 0 326476 2630648 75 0 0 0 17 0 3 3 765 1518 1191 4 6 90 1 30 0 338916 2628664 1850 0 0 0 1556 0 100 107 1458 28808 12099 18 36 45 0 30 0 330780 2629956 379 0 0 0 732 0 84 78 837 2851 1877 13 11 76 0 29 0 328640 2630260 172 0 0 0 241 0 9 9 512 850 1012 0 8 92 0 29 0 328640 2630260 0 0 0 0 4 0 2 2 389 121 369 0 4 96 1 30 0 331940 2629672 884 0 0 0 681 0 13 12 607 2761 1047 17 7 76 0 29 0 328640 2630216 40 0 0 0 168 0 9 9 508 425 641 3 6 91 0 29 0 328640 2630212 11 0 0 0 10 0 4 4 548 230 644 0 5 95 0 29 0 328640 2630212 5 0 0 0 5 0 2 2 430 219 470 0 4 96 0 29 0 328640 2630208 6 0 0 0 5 0 2 2 437 256 481 1 4 96 0 28 0 326496 2630516 277 0 0 0 348 0 8 8 526 780 788 0 6 94 0 29 0 328728 2630052 274 0 0 0 126 0 2 2 451 802 604 1 7 92 procs memory page disks faults cpu r b w avm fre flt re pi po fr sr da0 da1 in sy cs us sy id 0 29 0 328636 2630204 842 0 0 0 807 0 27 27 619 2368 1140 12 8 80 0 29 0 328636 2630200 0 0 0 0 0 0 0 0 492 113 457 0 4 96 0 30 0 330732 2629900 333 0 0 0 233 0 2 2 503 687 607 0 6 94 1 30 0 330748 2629864 15 0 0 0 1 0 1 1 400 189 401 0 6 94 2 29 0 338296 2628512 1340 0 0 0 911 0 21 21 910 4186 1712 23 11 66 1 31 0 338024 2626652 1223 0 0 0 1364 0 51 51 2972 40356 18751 38 32 30 0 31 0 333028 2628012 318 0 0 0 1159 0 32 33 757 67408 35201 7 43 50 0 31 0 338336 2627192 1044 0 0 0 909 0 86 98 923 3573 2295 24 11 65 0 31 0 333088 2627968 1367 0 0 0 1407 0 65 53 734 4235 1442 20 9 71 0 30 0 330876 2628372 29 0 0 0 130 0 0 0 405 191 401 0 5 95 0 29 0 328680 2628740 152 0 0 0 246 0 2 2 428 459 495 1 6 93 2 29 0 330924 2628308 476 0 0 0 274 0 552 552 2776 1862 14249 3 24 72 0 30 0 337484 2627268 1119 0 0 0 877 0 18 18 875 11192 5634 16 15 69 3 30 0 345588 2625052 1285 0 0 0 675 0 13 13 748 2201 1059 53 7 40 1 29 0 336604 2626380 344 0 0 0 653 0 23 23 557 1605 911 54 8 38 0 29 0 329528 2627284 159 0 0 0 407 0 15 15 581 1541 679 12 5 83 0 29 0 329528 2627284 5 0 0 0 5 0 2 2 397 159 391 0 4 96 0 29 0 329528 2627300 5 0 0 0 9 0 3 3 474 177 539 0 4 96 procs memory page disks faults cpu r b w avm fre flt re pi po fr sr da0 da1 in sy cs us sy id 2 28 0 329548 2627280 5 0 0 0 5 0 12 12 539 246 678 0 5 94 0 30 0 331624 2627356 232 0 0 0 219 0 56 56 737 552 968 2 17 82 2 30 0 331724 2627304 300 0 0 0 246 0 2 2 402 676 518 1 6 93 1 29 0 337012 2626560 1654 0 0 0 1361 0 35 35 695 4538 1521 25 11 64 0 28 0 327388 2627956 96 0 0 0 429 0 12 12 464 1038 605 14 5 81 0 28 0 327388 2627956 5 0 0 0 5 0 2 2 473 191 526 0 4 96 2 28 0 329632 2627500 280 0 0 0 130 0 4 4 518 739 751 0 5 95 0 28 0 327388 2627916 877 0 0 0 934 0 24 24 695 3220 1214 21 10 70 0 28 0 327388 2627920 110 0 0 0 113 0 0 0 411 324 406 0 6 94 1 27 0 327388 2627920 0 0 0 0 4 0 2 2 404 128 387 0 4 96 2 26 0 325268 2628176 480 0 0 0 470 0 2 2 460 958 649 1 6 93 3 26 0 332712 2627008 934 0 0 0 641 0 19 19 831 10859 4034 16 13 71 --=_biodiesel.gaiahost.coop-13953-1134494156-0001-3 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "freebsd-questions-unsubscribe@freebsd.org" --=_biodiesel.gaiahost.coop-13953-1134494156-0001-3-- --=_biodiesel.gaiahost.coop-13953-1134494156-0001-2--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20051213171608.GM2188>