From owner-freebsd-questions@FreeBSD.ORG Fri Dec 30 19:27:52 2011 Return-Path: Delivered-To: freebsd-questions@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 1873E106564A; Fri, 30 Dec 2011 19:27:52 +0000 (UTC) (envelope-from kes-kes@yandex.ru) Received: from forward16.mail.yandex.net (forward16.mail.yandex.net [IPv6:2a02:6b8:0:1402::1]) by mx1.freebsd.org (Postfix) with ESMTP id 00ACF8FC08; Fri, 30 Dec 2011 19:27:51 +0000 (UTC) Received: from smtp17.mail.yandex.net (smtp17.mail.yandex.net [95.108.252.17]) by forward16.mail.yandex.net (Yandex) with ESMTP id 6BAF9D24D75; Fri, 30 Dec 2011 23:27:49 +0400 (MSK) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yandex.ru; s=mail; t=1325273269; bh=C4Rkw4hy8jtUDbuZSZSFWoIhqJdZO0SYwl3FlFZZ6DY=; h=Date:From:Reply-To:Message-ID:To:CC:Subject:MIME-Version: Content-Type:Content-Transfer-Encoding; b=kVNmJHMelgJ1E3Q4eM+IsB1gR/K/09vm4S9u7d4Gjt44JTM7QXQkNcFZ+DEuepaeB lp/axqeRzPU8HcdYTH9ociU2gdv+UrzQQirtbX9HVRrZ2LHF/SZL7vd6i/970SFmnO AZf5wLQwSLIMJPrA6OYY38d/O9b2L7wATAaTEuDc= Received: from smtp17.mail.yandex.net (localhost [127.0.0.1]) by smtp17.mail.yandex.net (Yandex) with ESMTP id 45F3419000ED; Fri, 30 Dec 2011 23:27:49 +0400 (MSK) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yandex.ru; s=mail; t=1325273269; bh=C4Rkw4hy8jtUDbuZSZSFWoIhqJdZO0SYwl3FlFZZ6DY=; h=Date:From:Reply-To:Message-ID:To:CC:Subject:MIME-Version: Content-Type:Content-Transfer-Encoding; b=kVNmJHMelgJ1E3Q4eM+IsB1gR/K/09vm4S9u7d4Gjt44JTM7QXQkNcFZ+DEuepaeB lp/axqeRzPU8HcdYTH9ociU2gdv+UrzQQirtbX9HVRrZ2LHF/SZL7vd6i/970SFmnO AZf5wLQwSLIMJPrA6OYY38d/O9b2L7wATAaTEuDc= Received: from unknown (unknown [77.93.52.4]) by smtp17.mail.yandex.net (nwsmtp/Yandex) with ESMTP id RmOm4veQ-RmOahft5; Fri, 30 Dec 2011 23:27:48 +0400 X-Yandex-Spam: 1 X-Nat-Received: from [192.168.24.3]:2365 [ident-empty] by SPAM FILTER: with TPROXY id 1325280129.15066 abuse-to kes-kes@yandex.ru Date: Fri, 30 Dec 2011 21:27:46 +0200 From: =?windows-1251?B?yu7t/Oru4iDF4uPl7ejp?= X-Mailer: The Bat! (v4.0.24) Professional Organization: =?windows-1251?B?188gyu7t/Oru4iwgRnJlZUxpbmU=?= X-Priority: 3 (Normal) Message-ID: <174082910.20111230212746@yandex.ru> To: freebsd-questions@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=windows-1251 Content-Transfer-Encoding: 8bit Cc: Alexander Motin Subject: re0 + high load CPU, some tests X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: =?windows-1251?B?yu7t/Oru4iDF4uPl7ejp?= List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 30 Dec 2011 19:27:52 -0000 if this is interesting for anyone. looks strange: irq256 - WAIT and ng_queue - sleep are they wait each other? last pid: 70764; load averages: 69.14, 48.87, 26.25 up 9+03:32:40 21:01:21 197 processes: 101 running, 82 sleeping, 14 waiting CPU: 14.2% user, 0.0% nice, 44.7% system, 41.2% interrupt, 0.0% idle Mem: 192M Active, 343M Inact, 195M Wired, 112M Buf, 1259M Free Swap: 512M Total, 512M Free PID USERNAME PRI NICE SIZE RES STATE TIME WCPU COMMAND 12 root -76 - 0K 112K WAIT 19.2H 52.00% {irq256: re0} 13 root 0 - 0K 8K sleep 609:11 37.06% ng_queue 69225 root 38 0 13420K 5748K select 2:49 2.10% snmpd 70764 root 41 -10 11184K 3984K RUN 0:00 2.05% perl5.10.1 7749 bind 37 0 62980K 49004K uwait 26:40 0.73% {named} 70762 root 36 0 9952K 2168K RUN 0:00 0.10% top 3772 root 36 -10 103M 87944K wait 12:06 0.05% {mpd5} 3772 root 36 -10 103M 87944K RUN 0:00 0.05% {mpd5} # vmstat -i interrupt total rate irq14: ata0 1430344 1 irq20: hpet0 3255183935 4118 irq23: uhci0 ehci0 85 0 irq256: re0 2672832531 3381 Total 5929446895 7501 1 users Load 79.81 59.43 33.65 Dec 30 21:03 Mem:KB REAL VIRTUAL VN PAGER SWAP PAGER Tot Share Tot Share Free in out in out Act 283856 6264 721084 7948 1289560 count All 384748 8944 2879988 18292 pages Proc: Interrupts r p d s w Csw Trp Sys Int Sof Flt 54 cow 15828 total 102 33 50k 590 4377 15k 3991 242 116 zfod 3 ata0 14 9 ozfod 4120 hpet0 20 39.8%Sys 50.1%Intr 10.1%User 0.0%Nice 0.0%Idle 7%ozfod uhci0 ehci | | | | | | | | | | | daefr 11705 re0 256 ====================+++++++++++++++++++++++++>>>>> 34 prcfr 33 dtbuf 206 totfr Namei Name-cache Dir-cache 111725 desvn react Calls hits % hits % 64006 numvn pdwak 2151 2151 100 27893 frevn pdpgs intrn Disks ad1 da0 pass0 199448 wire KB/t 14.75 0.00 0.00 208820 act tps 3 0 0 338932 inact MB/s 0.04 0.00 0.00 cache %busy 1 0 0 1289560 free # uname -a FreeBSD bor 9.0-CURRENT FreeBSD 9.0-CURRENT #1: Wed Feb 23 09:39:22 UTC 2011 @:/usr/obj/usr/src/sys/KES_KERN_v9 i386 # vmstat -z ITEM SIZE LIMIT USED FREE REQ FAIL SLEEP UMA Kegs: 128, 0, 92, 28, 92, 0, 0 UMA Zones: 224, 0, 92, 10, 92, 0, 0 UMA Slabs: 284, 0, 1267, 21, 13143, 0, 0 UMA RCntSlabs: 544, 0, 727, 1, 727, 0, 0 UMA Hash: 128, 0, 2, 28, 3, 0, 0 16 Bucket: 76, 0, 38, 12, 38, 0, 0 32 Bucket: 140, 0, 53, 3, 54, 0, 0 64 Bucket: 268, 0, 89, 9, 93, 90, 0 128 Bucket: 524, 0, 861, 0, 6885, 572, 0 VM OBJECT: 136, 0, 36908, 560, 8428863, 0, 0 MAP: 140, 0, 7, 49, 7, 0, 0 KMAP ENTRY: 72, 57505, 31, 234, 105121, 0, 0 MAP ENTRY: 72, 0, 1085, 982,19434967, 0, 0 DP fakepg: 72, 0, 0, 0, 0, 0, 0 SG fakepg: 72, 0, 0, 0, 0, 0, 0 mt_zone: 2060, 0, 268, 11, 268, 0, 0 16: 16, 0, 4707, 571,57886292, 0, 0 32: 32, 0, 11879, 777,706645306, 0, 0 64: 64, 0, 6726, 767,33742858, 0, 0 128: 128, 0, 43378, 66482,61277757, 0, 0 256: 256, 0, 2808, 372,684799727, 0, 0 512: 512, 0, 652, 236, 1592927, 0, 0 1024: 1024, 0, 532, 40, 605103, 0, 0 2048: 2048, 0, 269, 201, 5976, 0, 0 4096: 4096, 0, 156, 27, 608758, 0, 0 Files: 56, 0, 193, 276,48647341, 0, 0 TURNSTILE: 72, 0, 261, 39, 261, 0, 0 umtx pi: 52, 0, 0, 0, 0, 0, 0 MAC labels: 20, 0, 0, 0, 0, 0, 0 PROC: 688, 0, 60, 30, 306557, 0, 0 THREAD: 728, 0, 136, 124, 64332, 0, 0 SLEEPQUEUE: 44, 0, 261, 93, 261, 0, 0 VMSPACE: 232, 0, 40, 45, 306544, 0, 0 cpuset: 40, 0, 2, 182, 2, 0, 0 audit_record: 816, 0, 0, 0, 0, 0, 0 mbuf_packet: 256, 0, 259, 1149,1685234486, 0, 0 mbuf: 256, 0, 5, 642,3090339409, 0, 0 mbuf_cluster: 2048, 25600, 1408, 6, 1408, 0, 0 mbuf_jumbo_page: 4096, 12800, 0, 20, 153798, 0, 0 mbuf_jumbo_9k: 9216, 6400, 0, 0, 0, 0, 0 mbuf_jumbo_16k: 16384, 3200, 0, 0, 0, 0, 0 mbuf_ext_refcnt: 4, 0, 0, 0, 0, 0, 0 NetGraph items: 36, 4130, 3, 174,540902149, 0, 0 NetGraph data items: 36, 531, 0, 531,1277775559, 0, 0 g_bio: 140, 0, 0, 3444, 5725546, 0, 0 ttyinq: 152, 0, 300, 12, 975, 0, 0 ttyoutq: 256, 0, 157, 8, 508, 0, 0 ata_request: 204, 0, 0, 855, 1431305, 0, 0 ata_composite: 180, 0, 0, 0, 0, 0, 0 VNODE: 272, 0, 64018, 46, 1519139, 0, 0 VNODEPOLL: 60, 0, 1, 125, 1, 0, 0 S VFS Cache: 72, 0, 66679, 154, 1573251, 0, 0 L VFS Cache: 292, 0, 595, 198, 20943, 0, 0 NAMEI: 1024, 0, 0, 12,38878226, 0, 0 NFSMOUNT: 532, 0, 0, 0, 0, 0, 0 NFSNODE: 460, 0, 0, 0, 0, 0, 0 DIRHASH: 1024, 0, 482, 14, 482, 0, 0 pipe: 392, 0, 11, 29, 217359, 0, 0 Mountpoints: 648, 0, 6, 6, 6, 0, 0 ksiginfo: 80, 0, 80, 976, 41491, 0, 0 itimer: 220, 0, 0, 0, 0, 0, 0 KNOTE: 72, 0, 8, 151,13777392, 0, 0 socket: 416, 25605, 92, 304,24848447, 0, 0 unpcb: 172, 25622, 27, 42, 523052, 0, 0 ipq: 32, 904, 0, 226, 1272, 0, 0 udp_inpcb: 220, 25614, 6, 138,24157973, 0, 0 udpcb: 8, 25781, 6, 400,24157973, 0, 0 tcp_inpcb: 220, 25614, 37, 935, 134055, 0, 0 tcpcb: 692, 25600, 35, 150, 134055, 0, 0 tcptw: 52, 5184, 2, 1006, 21168, 0, 0 syncache: 112, 15365, 0, 70, 87664, 0, 0 hostcache: 76, 15400, 148, 102, 1559, 0, 0 tcpreass: 20, 1690, 0, 169, 6987, 0, 0 sackhole: 20, 0, 0, 169, 351, 0, 0 sctp_ep: 880, 25600, 0, 0, 0, 0, 0 sctp_asoc: 1524, 40000, 0, 0, 0, 0, 0 sctp_laddr: 24, 80040, 0, 145, 45, 0, 0 sctp_raddr: 464, 80000, 0, 0, 0, 0, 0 sctp_chunk: 96, 400000, 0, 0, 0, 0, 0 sctp_readq: 76, 400000, 0, 0, 0, 0, 0 sctp_stream_msg_out: 72, 400044, 0, 0, 0, 0, 0 sctp_asconf: 24, 400055, 0, 0, 0, 0, 0 sctp_asconf_ack: 24, 400055, 0, 0, 0, 0, 0 ripcb: 220, 25614, 15, 129, 18472, 0, 0 rtentry: 108, 0, 9021, 231, 44539, 0, 0 IPFW dynamic rule: 108, 0, 0, 0, 0, 0, 0 divcb: 220, 25614, 0, 0, 0, 0, 0 selfd: 28, 0, 276, 232,434047205, 0, 0 ip4flow: 40, 25668, 1332, 8972,16391139, 0, 0 ip6flow: 64, 25636, 0, 174, 20, 0, 0 SWAPMETA: 276, 121576, 0, 0, 0, 0, 0 FFS inode: 116, 0, 63979, 74, 1519019, 0, 0 FFS1 dinode: 128, 0, 0, 0, 0, 0, 0 FFS2 dinode: 256, 0, 63979, 71, 1519019, 0, 0 # sysctl -a net.isr net.isr.numthreads: 1 net.isr.maxprot: 16 net.isr.defaultqlimit: 256 net.isr.maxqlimit: 10240 net.isr.bindthreads: 0 net.isr.maxthreads: 1 net.isr.direct: 1 net.isr.direct_force: 1 #sysctl -a | grep smp | less kern.timecounter.smp_tsc: 0 kern.smp.forward_signal_enabled: 1 kern.smp.topology: 0 kern.smp.cpus: 1 kern.smp.disabled: 3 kern.smp.active: 0 kern.smp.maxcpus: 32 kern.smp.maxid: 3 this system can process about 19000 interrupts from re0, just disable other CPUs to fast achive overload. renice -20 mpd5 wait 10min /usr/local/etc/mpd5 restart after 1mins ago mpd5 is restarted and everithing is stabilised to normal work PS. does "41.2% interrupt" in top output show really CPU load from 're' interrupts or in this value are some internal kernel interupts too? suggestions: when CPU can not devastate interrupts buffer re0 starts to flood by interrupts mpd5 lock interupts system when CPU is 0% idle; tuning on this machine: kern.ipc.maxsockbuf=8388608 net.graph.recvspace=524288 net.inet.ip.fw.one_pass=0 net.inet.ip.dummynet.io_fast=1 kern.smp.disabled=3 -- С уважением, Коньков mailto:kes-kes@yandex.ru