From owner-freebsd-stable@FreeBSD.ORG Fri Jul 7 13:40:55 2006 Return-Path: X-Original-To: freebsd-stable@freebsd.org Delivered-To: freebsd-stable@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 71D4116A4DA for ; Fri, 7 Jul 2006 13:40:55 +0000 (UTC) (envelope-from quetzal@zone3000.net) Received: from mx1.sitevalley.com (sitevalley.com [209.67.60.43]) by mx1.FreeBSD.org (Postfix) with SMTP id F405043D46 for ; Fri, 7 Jul 2006 13:40:54 +0000 (GMT) (envelope-from quetzal@zone3000.net) Received: from zone3000.kharkov.ua (HELO localhost) (217.144.68.98) by 209.67.61.254 with SMTP; 7 Jul 2006 13:40:53 -0000 Date: Fri, 7 Jul 2006 16:40:26 +0300 From: Nikolay Pavlov To: freebsd-stable@freebsd.org Message-ID: <20060707134026.GA1750@zone3000.net> Mail-Followup-To: Nikolay Pavlov , freebsd-stable@freebsd.org Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.4.2.1i X-Operating-System: FreeBSD 6.1-RELEASE Subject: Processes in block state in vmstat. X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 07 Jul 2006 13:40:55 -0000 Hi, folks. I want to add some additional information about problem related processes in block state. I see it on my server Dual CPU: Intel(R) Xeon(TM) CPU 2.40GHz (2399.33-MHz 686-class CPU) with amrd device. FreeBSD 6.1-RELEASE-p1 I see some httpd daemons in D state: procs memory page disk faults cpu r b w avm fre flt re pi po fr sr am0 in sy cs us sy id 0 527 0 2173832 94028 49 0 0 0 2722 2658 0 9219 4140 17350 1 7 92 0 528 0 2172836 93636 66 0 0 0 218 0 6 504 512 1221 0 2 98 0 523 0 2171828 93360 56 0 0 0 188 0 6 551 298 1286 0 2 98 0 523 0 2171972 92612 29 0 0 0 189 0 6 520 395 1250 0 2 98 30 494 0 2170976 91836 92 0 0 0 333 0 11 448 250 970 0 2 98 0 518 0 2171156 90580 43 1 0 0 314 0 10 640 758 1676 0 3 97 0 510 0 2170244 89452 61 0 0 0 402 0 14 615 644 1667 0 3 97 1 512 0 2170420 89292 1 0 0 0 0 0 0 493 307 1133 0 2 98 0 517 0 2169560 86868 110 0 0 0 735 0 27 623 852 1584 0 3 96 0 514 0 2168552 87176 53 0 0 0 198 0 0 487 200 1078 0 1 99 0 517 0 2178752 84040 1523 0 0 0 1261 0 0 512 2660 1320 2 3 96 vmstat -s 3129790854 cpu context switches 1663101011 device interrupts 5504551 software interrupts 9321603 traps 746764672 system calls 54 kernel threads created 66887 fork() calls 9669 vfork() calls 0 rfork() calls 4784 swap pager pageins 5635 swap pager pages paged in 1902 swap pager pageouts 3590 swap pager pages paged out 24668 vnode pager pageins 64050 vnode pager pages paged in 0 vnode pager pageouts 0 vnode pager pages paged out 28140 page daemon wakeups 479450699 pages examined by the page daemon 75270 pages reactivated 2461680 copy-on-write faults 3805 copy-on-write optimized faults 4531914 zero fill pages zeroed 2732630 zero fill pages prezeroed 18095 intransit blocking page faults 8922821 total VM faults taken 0 pages affected by kernel thread creation 8018826 pages affected by fork() 1107169 pages affected by vfork() 0 pages affected by rfork() 491039188 pages freed 393 pages freed by daemon 5744590 pages freed by exiting processes 201969 pages active 184930 pages inactive 17190 pages in VM cache 107624 pages wired down 750 pages free 4096 bytes per page 30979017 total name lookups cache hits (82% pos + 1% neg) system 1% per-directory deletions 0%, falsehits 0%, toolong 0% vmstat -z ITEM SIZE LIMIT USED FREE REQUESTS UMA Kegs: 140, 0, 83, 13, 83 UMA Zones: 480, 0, 83, 5, 83 UMA Slabs: 64, 0, 2537, 177, 225611 UMA RCntSlabs: 104, 0, 32769, 13, 2949442 UMA Hash: 128, 0, 3, 27, 6 16 Bucket: 76, 0, 27, 23, 73 32 Bucket: 140, 0, 23, 33, 64 64 Bucket: 268, 0, 17, 39, 100 128 Bucket: 524, 0, 223, 71, 1036 VM OBJECT: 132, 0, 30655, 2318, 1833813 MAP: 192, 0, 7, 33, 7 KMAP ENTRY: 68, 65520, 285, 11251, 78210624 MAP ENTRY: 68, 0, 215821, 8011, 3882740 PV ENTRY: 24, 2155135, 1391603, 119877, 42095182 DP fakepg: 72, 0, 0, 53, 6 mt_zone: 1024, 0, 174, 126, 174 16: 16, 0, 2158, 684, 578060 32: 32, 0, 3343, 386, 87211 64: 64, 0, 7323, 819, 1449967 128: 128, 0, 3507, 333, 519528 256: 256, 0, 6226, 509, 181376 512: 512, 0, 64, 96, 77544 1024: 1024, 0, 55, 73, 38283 2048: 2048, 0, 141, 27, 25379 4096: 4096, 0, 1990, 135, 84846 Files: 72, 0, 2924, 1475, 1665330 PROC: 524, 0, 1936, 220, 76612 THREAD: 372, 0, 2156, 14, 2156 KSEGRP: 88, 0, 2156, 84, 2156 UPCALL: 44, 0, 0, 0, 0 VMSPACE: 300, 0, 1886, 337, 76512 mbuf_packet: 256, 0, 66135, 0, 1979970489 mbuf: 256, 0, 7, 1253, 3511131461 mbuf_cluster: 2048, 65536, 65538, 0, 1966671403 mbuf_jumbo_pagesize: 4096, 0, 0, 0, 0 mbuf_jumbo_9k: 9216, 0, 0, 0, 0 mbuf_jumbo_16k: 16384, 0, 0, 0, 0 ACL UMA zone: 388, 0, 0, 0, 0 g_bio: 132, 0, 0, 87, 79264695 ata_request: 204, 0, 0, 0, 0 ata_composite: 196, 0, 0, 0, 0 VNODE: 272, 0, 27508, 7072, 4868108 VNODEPOLL: 76, 0, 1, 49, 1 S VFS Cache: 68, 0, 27595, 2869, 4582987 L VFS Cache: 291, 0, 1708, 203, 314868 NAMEI: 1024, 0, 3, 129, 12912949 DIRHASH: 1024, 0, 495, 41, 4186 NFSMOUNT: 480, 0, 2, 14, 3 NFSNODE: 460, 0, 13, 27, 1525 PIPE: 408, 0, 28, 35, 31594 KNOTE: 68, 0, 0, 168, 5762 socket: 356, 16434, 2327, 1699, 277319 unpcb: 140, 65548, 42, 70, 12935 ipq: 32, 2147, 0, 0, 2235 udpcb: 180, 16434, 5, 127, 5531 inpcb: 180, 16434, 2259, 1393, 258840 tcpcb: 464, 16424, 2258, 1454, 258840 tcptw: 48, 3354, 1, 389, 38232 syncache: 100, 15366, 17, 334, 322283 hostcache: 76, 15400, 246, 304, 6938 tcpreass: 20, 4225, 1, 168, 4936 sackhole: 20, 0, 8, 499, 23975962 ripcb: 180, 16434, 0, 44, 11 rtentry: 132, 0, 5, 53, 54 pfsrctrpl: 100, 0, 0, 0, 0 pfrulepl: 604, 0, 0, 0, 0 pfstatepl: 260, 10005, 0, 0, 0 pfaltqpl: 128, 0, 0, 0, 0 pfpooladdrpl: 68, 0, 0, 0, 0 pfrktable: 1240, 0, 0, 0, 0 pfrkentry: 156, 0, 0, 0, 0 pfrkentry2: 156, 0, 0, 0, 0 pffrent: 16, 5075, 0, 0, 0 pffrag: 48, 0, 0, 0, 0 pffrcache: 48, 10062, 0, 0, 0 pffrcent: 12, 50141, 0, 0, 0 pfstatescrub: 28, 0, 0, 0, 0 pfiaddrpl: 92, 0, 0, 0, 0 pfospfen: 108, 0, 0, 0, 0 pfosfp: 28, 0, 0, 0, 0 IPFW dynamic rule zone: 108, 0, 0, 0, 0 SWAPMETA: 276, 121576, 36, 20, 1613 FFS inode: 132, 0, 27459, 1280, 4866112 FFS1 dinode: 128, 0, 0, 0, 0 FFS2 dinode: 256, 0, 27459, 1206, 4866112 Intresting thing that everytime i use "vmstat 1" problem gone and i save my uptime. Server is still under load and i see only one process in D state 85815 ?? D 0:00.00 /usr/local/sbin/megarc -ldInfo -LAll -a0 Also i have found that it's much easy to trigger this state for httpd when i use low (default) value for MaxSpareServers (I have about 300 ~ 500 connections all the time) -- ========================================================================= = Best regards, Nikolay Pavlov. <<<------------------------------------ = =========================================================================