From owner-freebsd-performance@FreeBSD.ORG Sun Mar 16 14:36:30 2008 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id E9B571065700; Sun, 16 Mar 2008 14:36:30 +0000 (UTC) (envelope-from prvs=1961ac39cb=killing@multiplay.co.uk) Received: from mail1.multiplay.co.uk (core6.multiplay.co.uk [85.236.96.23]) by mx1.freebsd.org (Postfix) with ESMTP id 30FA48FC1A; Sun, 16 Mar 2008 14:36:30 +0000 (UTC) (envelope-from prvs=1961ac39cb=killing@multiplay.co.uk) DKIM-Signature: v=1; a=rsa-sha256; c=simple; d=multiplay.co.uk; s=Multiplay; t=1205677433; x=1206282233; q=dns/txt; h=Received: Message-ID:From:To:Cc:References:Subject:Date:MIME-Version: Content-Type:Content-Transfer-Encoding; bh=sdouHwdzQC11Bp4M3ukeg mHF36bZT+4JsZ41BYpFB5c=; b=Ma5SXVhr6FBw5q9eF4ADNUP/2Yvzf/piuVWoL UIAxXF9IkDwkgAEWK/eulP1JoJ3CQnL9zkUrp1XgU09WP7vTajc8/1WBNI1L8wbn Egu226oAJnok/mxNlbKf+QJDvc81UvirtpPMXdpYXvN63u4/jtxqkRpHqKWTIC3n u4BRhE= X-Spam-Checker-Version: SpamAssassin 3.1.8 (2007-02-13) on mail1.multiplay.co.uk X-Spam-Level: X-Spam-Status: No, score=-14.7 required=6.0 tests=BAYES_00, USER_IN_WHITELIST, USER_IN_WHITELIST_TO autolearn=ham version=3.1.8 Received: from r2d2 by mail1.multiplay.co.uk (MDaemon PRO v9.6.3) with ESMTP id md50005293410.msg; Sun, 16 Mar 2008 14:23:52 +0000 Message-ID: <005301c88771$5b06b6c0$b6db87d4@multiplay.co.uk> From: "Steven Hartland" To: "Kris Kennaway" References: <056601c8814c$516c0370$b6db87d4@multiplay.co.uk><20080308221441.E11432@fledge.watson.org> <006401c88181$25cf0e30$b6db87d4@multiplay.co.uk> Date: Sun, 16 Mar 2008 14:23:50 -0000 MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset="iso-8859-1"; reply-type=response Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.3138 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.3198 X-Authenticated-Sender: Killing@multiplay.co.uk X-MDRemoteIP: 212.135.219.182 X-Return-Path: prvs=1961ac39cb=killing@multiplay.co.uk X-Envelope-From: killing@multiplay.co.uk X-Spam-Processed: mail1.multiplay.co.uk, Sun, 16 Mar 2008 14:23:52 +0000 X-MDAV-Processed: mail1.multiplay.co.uk, Sun, 16 Mar 2008 14:23:53 +0000 Cc: freebsd-performance@freebsd.org, Robert Watson Subject: Re: rrdtool / mtr causing stalling on 7.0 X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 16 Mar 2008 14:36:31 -0000 Hi Kris I was wondering if you would be so kind as to take a look at the results below to see if they highlight anything that might be the cause of this performance issue. I've raised this on the rrdtool list and quite a few people seem able to run 10* the amount of updates on none FreeBSD systems without these disruptive system wide stalls. Given this and the statement on one of your papers saying you would be interested in any loads that don't run well FreeBSD I hoped you might be able to have a look at this and provide us with areas to focus on. Regards Steve ----- Original Message ----- From: "Steven Hartland" > ----- Original Message ----- > From: "Robert Watson" >> It looks like the attachment got lost on the way through the mailing list. >> >> I think the first starting point is: what sort of stall is this? Is it, for >> example, all network communication stalling, all disk I/O stalling, or the >> entire kernel and all processes stalling? The usual diagnostics are: >> >> - Does the machine stop responding to pings while stalled, and/or possibly >> "catch up" all at once when it recovers? >> >> - If you run the following loop on the machine without any network or console >> I/O, do you see gaps in time stamps: >> >> while (1) { >> sleep 1 >> date >> date.log >> } >> >> - If you write a short C program that looks a lot like the above loop, but >> logs time stamps into an in-memory buffer, and have it look for gaps in the >> sequence of >3 seconds, does it run across the stall? > > Thanks for the ideas Robert the output from the shell script > this shows significant gaps:- > Sun Mar 9 00:20:33 GMT 2008 > Sun Mar 9 00:20:34 GMT 2008 <== Stall > Sun Mar 9 00:21:09 GMT 2008 > Sun Mar 9 00:21:10 GMT 2008 > ... > Sun Mar 9 00:25:23 GMT 2008 > Sun Mar 9 00:25:24 GMT 2008 > Sun Mar 9 00:25:25 GMT 2008 > Sun Mar 9 00:25:27 GMT 2008 <== Stall > Sun Mar 9 00:25:53 GMT 2008 > Sun Mar 9 00:25:59 GMT 2008 > Sun Mar 9 00:26:00 GMT 2008 > > Running a ping along side shows no missed responses. > > Enabling lock profiling for the period changes the behaviour somewhat, > producing shorter but multiple stalls. > > Sun Mar 9 00:30:31 GMT 2008 > Sun Mar 9 00:30:32 GMT 2008 > Sun Mar 9 00:30:34 GMT 2008 > Sun Mar 9 00:30:35 GMT 2008 > Sun Mar 9 00:30:36 GMT 2008 > Sun Mar 9 00:30:37 GMT 2008 > Sun Mar 9 00:30:38 GMT 2008 > Sun Mar 9 00:30:41 GMT 2008 > Sun Mar 9 00:30:42 GMT 2008 <== Stall > Sun Mar 9 00:30:44 GMT 2008 > Sun Mar 9 00:30:45 GMT 2008 <== Stall > Sun Mar 9 00:30:47 GMT 2008 <== Stall > Sun Mar 9 00:30:49 GMT 2008 > Sun Mar 9 00:30:50 GMT 2008 <== Stall > Sun Mar 9 00:30:52 GMT 2008 <== Stall > Sun Mar 9 00:30:54 GMT 2008 > Sun Mar 9 00:30:55 GMT 2008 > Sun Mar 9 00:30:56 GMT 2008 > Sun Mar 9 00:30:57 GMT 2008 <== Stall > Sun Mar 9 00:31:03 GMT 2008 <== Stall > Sun Mar 9 00:31:05 GMT 2008 > Sun Mar 9 00:31:06 GMT 2008 <== Stall > Sun Mar 9 00:31:08 GMT 2008 > Sun Mar 9 00:31:09 GMT 2008 > Sun Mar 9 00:31:10 GMT 2008 > Sun Mar 9 00:31:11 GMT 2008 <== Stall > Sun Mar 9 00:31:14 GMT 2008 > Sun Mar 9 00:31:15 GMT 2008 > Sun Mar 9 00:31:16 GMT 2008 <== Stall > Sun Mar 9 00:31:20 GMT 2008 > Sun Mar 9 00:31:21 GMT 2008 > Sun Mar 9 00:31:22 GMT 2008 > > Using the following c code we also see stalls: > #include > #include > #include > > int main( char **argv, int argc ) > { > time_t last = time( NULL ); > while ( 1 ) > { > time_t now = time( NULL ); > time_t diff = now - last; > if ( diff >= 2 ) > { > fprintf( stderr, "stalled for %d seconds\n", diff ); > } > fprintf( stderr, ctime( &now ) ); > last = now; > sleep( 1 ); > } > > exit( 0 ); > } > > [date.log] > Sun Mar 9 00:55:40 GMT 2008 > Sun Mar 9 00:55:43 GMT 2008 <== Stall > Sun Mar 9 00:56:11 GMT 2008 > Sun Mar 9 00:56:12 GMT 2008 > Sun Mar 9 00:56:13 GMT 2008 > Sun Mar 9 00:56:14 GMT 2008 > Sun Mar 9 00:56:15 GMT 2008 > [/date.log] > > [timec output] > Sun Mar 9 00:55:40 2008 > Sun Mar 9 00:55:41 2008 > Sun Mar 9 00:55:42 2008 > stalled for 2 seconds > Sun Mar 9 00:55:44 2008 > stalled for 5 seconds > Sun Mar 9 00:55:49 2008 > stalled for 2 seconds > Sun Mar 9 00:55:51 2008 > stalled for 2 seconds > Sun Mar 9 00:55:53 2008 > Sun Mar 9 00:55:54 2008 > Sun Mar 9 00:55:55 2008 > Sun Mar 9 00:55:56 2008 > Sun Mar 9 00:55:57 2008 > Sun Mar 9 00:55:58 2008 > Sun Mar 9 00:55:59 2008 > Sun Mar 9 00:56:00 2008 > Sun Mar 9 00:56:01 2008 > Sun Mar 9 00:56:02 2008 > Sun Mar 9 00:56:03 2008 > Sun Mar 9 00:56:04 2008 > Sun Mar 9 00:56:05 2008 > Sun Mar 9 00:56:06 2008 > Sun Mar 9 00:56:07 2008 > Sun Mar 9 00:56:08 2008 > Sun Mar 9 00:56:09 2008 > Sun Mar 9 00:56:10 2008 > Sun Mar 9 00:56:11 2008 > Sun Mar 9 00:56:12 2008 > Sun Mar 9 00:56:13 2008 > Sun Mar 9 00:56:14 2008 > Sun Mar 9 00:56:15 2008 > [/timec output] > > > As the list ate the attachment, the output from the lock profile > can be found here:- > ftp://ftp1.multiplay.co.uk/pub/other/freebsd-7.0-rrdtool-stall.zip > > Regards > Steve ================================================ This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it. In the event of misdirection, illegible or incomplete transmission please telephone +44 845 868 1337 or return the E.mail to postmaster@multiplay.co.uk. From owner-freebsd-performance@FreeBSD.ORG Mon Mar 17 08:14:21 2008 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id AA9541065671 for ; Mon, 17 Mar 2008 08:14:21 +0000 (UTC) (envelope-from stefan.lambrev@moneybookers.com) Received: from blah.sun-fish.com (blah.sun-fish.com [217.18.249.150]) by mx1.freebsd.org (Postfix) with ESMTP id 112528FC16 for ; Mon, 17 Mar 2008 08:14:20 +0000 (UTC) (envelope-from stefan.lambrev@moneybookers.com) Received: by blah.sun-fish.com (Postfix, from userid 1002) id DD0C91B10EBB; Mon, 17 Mar 2008 09:14:19 +0100 (CET) X-Spam-Checker-Version: SpamAssassin 3.2.3 (2007-08-08) on blah.cmotd.com X-Spam-Level: X-Spam-Status: No, score=-8.8 required=5.0 tests=ALL_TRUSTED,BAYES_00, J_CHICKENPOX_32, J_CHICKENPOX_42, J_CHICKENPOX_66 autolearn=no version=3.2.3 Received: from [10.1.1.2] (unknown [192.168.25.10]) by blah.sun-fish.com (Postfix) with ESMTP id 7954C1B10E4E; Mon, 17 Mar 2008 09:14:16 +0100 (CET) Message-ID: <47DE2840.803@moneybookers.com> Date: Mon, 17 Mar 2008 10:13:52 +0200 From: Stefan Lambrev User-Agent: Thunderbird 2.0.0.12 (Windows/20080213) MIME-Version: 1.0 To: hugoboy@inbox.lv References: <1205491910.47da58c6ecb6a@www.inbox.lv> In-Reply-To: <1205491910.47da58c6ecb6a@www.inbox.lv> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Virus-Scanned: ClamAV 0.91.2/6273/Mon Mar 17 04:03:58 2008 on blah.cmotd.com X-Virus-Status: Clean Cc: freebsd-performance@freebsd.org Subject: Re: FreeBSD 7.0 bridge tuning X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 17 Mar 2008 08:14:21 -0000 Greetings, hugoboy@inbox.lv wrote: > Hello! > > I'm trying to tune FreeBSD 7.0 bridge. > You may want to check this thread - http://lists.freebsd.org/pipermail/freebsd-current/2008-January/082751.html > Environment: > Server - 2 x Xeon 3GHz, 2 x Gb LAN(em driver) + 1 LAN for management, > 1GB RAM. > Can you tell us the exact CPU model? Is it dual core Xeon? It's not clear how many cores you have ... > Testers -2 x Sunrise Telecom 100Mbit Ethernet testers for traffic > generation. > > What I have intended to achieve is to substitute proprietary traffic > shaper Allot with FreeBSD traffic shaper(Bridge + PF + ALTQ). > The minimum task is to make FreeBSD shaper to perform perfectly with > 100Mbit traffic in all spectrum of packet lengths (from 64 bytes to > at least 1518 bytes) > > The situation now: > with pf turned off - there is no problem, bridge throughput is > 100Mbit/s no packet loss (starting from 64 byte packets) > > With pf on I have statistics: > packet lengt -> Mbit/s without packet loss > 64 -> 46 > 100 -> 66 > 150 -> 94 > >> 200 -> 100 >> > > How many packets per second do you transmit? PF have some known limitations, hopefully they will be addressed in 8-current and back-ported someday to 7-STABLE. > Lower configuration of kernel/sysctl is displayed. > > I don't know what else can I tune? > > It seems to me that bottleneck is somewhere around pf/kernel buffers > of packet headers. I read somewhere that in bridging packet payload > does not travel through all stack - just header is evaluated. > In case of 64 byte packets in the same time unit there are more > packets for the same bandwith on interfaces and as plain layer2 > bridge performs 100Mbit/s with no problem > the problem is above layer2 :) > > btw: kern.polling.enable=1 does not help - at packetlength 64 bytes > performance is 2x worse than with interrupts. > I noticed this too - polling is not very helpful with em driver. It reduce the load but dropped packets are more with polling. In my situation increasing kern.hz to 3000 yielded best results, you can try to tune this. > kernel: > --------------------------- > > cpu I686_CPU > ident ALLOT > > # To statically compile in device wiring instead of > /boot/device.hints > #hints "GENERIC.hints" # Default places to look for > devices. > > makeoptions DEBUG=-g # Build kernel with gdb(1) > debug symbols > > options SCHED_ULE # ULE scheduler > #options SCHED_4BSD # 4BSD scheduler > options PREEMPTION # Enable kernel thread > preemption > options INET # InterNETworking > #options INET6 # IPv6 communications > protocols > #options SCTP # Stream Control Transmission > Protocol > options FFS # Berkeley Fast Filesystem > options SOFTUPDATES # Enable FFS soft updates > support > options UFS_ACL # Support for access control > lists > options UFS_DIRHASH # Improve performance on big > directories > options UFS_GJOURNAL # Enable gjournal-based UFS > journaling > options MD_ROOT # MD is a potential root > device > options NFSCLIENT # Network Filesystem Client > options NFSSERVER # Network Filesystem Server > options NFS_ROOT # NFS usable as /, requires > NFSCLIENT > options MSDOSFS # MSDOS Filesystem > options CD9660 # ISO 9660 Filesystem > options PROCFS # Process filesystem > (requires PSEUDOFS) > options PSEUDOFS # Pseudo-filesystem framework > options GEOM_PART_GPT # GUID Partition Tables. > options GEOM_LABEL # Provides labelization > options COMPAT_43TTY # BSD 4.3 TTY compat [KEEP > THIS!] > options COMPAT_FREEBSD4 # Compatible with FreeBSD4 > options COMPAT_FREEBSD5 # Compatible with FreeBSD5 > options COMPAT_FREEBSD6 # Compatible with FreeBSD6 > options SCSI_DELAY=5000 # Delay (in ms) before > probing SCSI > options KTRACE # ktrace(1) support > options SYSVSHM # SYSV-style shared memory > options SYSVMSG # SYSV-style message queues > options SYSVSEM # SYSV-style semaphores > options _KPOSIX_PRIORITY_SCHEDULING # POSIX P1003_1B > real-time extensions > options KBD_INSTALL_CDEV # install a CDEV entry in > /dev > options ADAPTIVE_GIANT # Giant mutex is adaptive. > options STOP_NMI # Stop CPUS using NMI instead > of IPI > options AUDIT # Security event auditing > > options ALTQ > options ALTQ_CBQ > options ALTQ_RED > options ALTQ_RIO > options ALTQ_HFSC > options ALTQ_CDNR > options ALTQ_PRIQ > options ALTQ_NOPCC > options HZ=1000 > options DEVICE_POLLING > options IPSTEALTH > options ZERO_COPY_SOCKETS > options MPTABLE_FORCE_HTT # Enable HTT CPUs with the MP Table > options IPI_PREEMPTION > > # To make an SMP kernel, the next two lines are needed > options SMP # Symmetric MultiProcessor > Kernel > device apic # I/O APIC > -------------------------------- > > /etc/sysctl.conf > #kern.polling.enable=1 > kern.ipc.nmbcluster=32768 > kern.ipc.maxsockbufs=2097152 > kern.ipc.somaxconn=8192 > kern.maxfiles=65536 > kern.maxfilesperproc=32768 > net.inet.tcp.delayed_ack=0 > net.inet.tcp.sendspace=65535 > net.inet.udp.recvspace=65535 > net.inet.udp.maxdgram=57344 > net.local.stream.recvspace=65535 > net.local.stream.sendspace=65535 > kern.polling.user_frac=20 > net.isr.direct=0 > net.inet.ip.forwarding=1 > ------------------------------- > > P.S. I tried pfSense, but as we have used Allot before - we need to > see queue statistics in graphs per queue, pfSense just offers > numbers.. > Seems to me that pFsense is good for many things but not for > bridge+traffic shapeing - correct me if I'm wrong. > > Best regards, > Ugis > > > _______________________________________________ > freebsd-performance@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-performance > To unsubscribe, send any mail to "freebsd-performance-unsubscribe@freebsd.org" > From owner-freebsd-performance@FreeBSD.ORG Mon Mar 17 09:24:50 2008 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 30CF01065673 for ; Mon, 17 Mar 2008 09:24:50 +0000 (UTC) (envelope-from amin.scg@gmail.com) Received: from wa-out-1112.google.com (wa-out-1112.google.com [209.85.146.181]) by mx1.freebsd.org (Postfix) with ESMTP id E73CE8FC2A for ; Mon, 17 Mar 2008 09:24:49 +0000 (UTC) (envelope-from amin.scg@gmail.com) Received: by wa-out-1112.google.com with SMTP id k17so6163269waf.3 for ; Mon, 17 Mar 2008 02:24:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:from:to:references:in-reply-to:subject:date:mime-version:content-type:content-transfer-encoding:x-mailer:thread-index:content-language:message-id; bh=/QxclEWXpmgafh1oEgMjFJFKx6OQcG2KdtSvv3y3uYs=; b=RQ6jARHr12rD3mS113EPdLQL3hQsG0u+uwRuAeWmSlRfTkSzy/KRIUD4OOQ9+XpSG6OXLbMJRYM349X5h2F7nUKG677r9ilXTqWxwQjtSGecaaRUQilxbCyrPnsjKsYRbxBaX0o3bNg+CiYmsSfnCdzzlHYPOUiN3G5+rJIcWZg= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=from:to:references:in-reply-to:subject:date:mime-version:content-type:content-transfer-encoding:x-mailer:thread-index:content-language:message-id; b=Lp6WKKR6ya/iA9taKC63zx/o3k0YGPjGyYT+qn5IkVsAZenD6uM9LsUFl5ywVvc01tYwurCabNEBOHLsRA6A2gNWA+WIfA8fkX/5emT5B2JK3W0eEfyQgks53svGQ/IqM+WLe2Kxj6FdzcDK9YvxDUR1u2krIYtJ+Kwu1/sQU7Y= Received: by 10.114.184.7 with SMTP id h7mr17132072waf.28.1205744308416; Mon, 17 Mar 2008 01:58:28 -0700 (PDT) Received: from AminuddinPC ( [219.95.177.171]) by mx.google.com with ESMTPS id l27sm34207467waf.26.2008.03.17.01.58.25 (version=TLSv1/SSLv3 cipher=OTHER); Mon, 17 Mar 2008 01:58:27 -0700 (PDT) From: "Aminuddin Abdullah" To: References: <20080210120013.4C3D116A421@hub.freebsd.org> In-Reply-To: <20080210120013.4C3D116A421@hub.freebsd.org> Date: Mon, 17 Mar 2008 16:57:59 +0800 MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit X-Mailer: Microsoft Office Outlook 12.0 Thread-Index: Achr3LVfHG7eNcdXRBSNp1SVgw7lcwcLxgFg Content-Language: en-my Message-ID: <47de32b3.1bbc720a.7cf0.ffff8ff1@mx.google.com> Subject: V7 High CPU Usage on swi5:+, what is this process? X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 17 Mar 2008 09:24:50 -0000 I have just upgraded 5 of my machines to V7 from 6.3 and then realized that all the machines has a high CPU usage. Almost all of them using 80%-90% CPU with more than 8000 connections. Using previous 6.3, it only uses 40-50% CPU with the same kind of connections. Using top -S, I can see that swi5: +, PID 17 process is using 30% of CPU time. What is this process? All the machines are Intel C2D 6300 except one which is a AMD 4000+. Is this normal for V7? How do I downgrade to 6.3 if this V7 killing the CPU? TIA From owner-freebsd-performance@FreeBSD.ORG Mon Mar 17 10:09:35 2008 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 2263B106564A for ; Mon, 17 Mar 2008 10:09:35 +0000 (UTC) (envelope-from valerio.daelli@gmail.com) Received: from el-out-1112.google.com (el-out-1112.google.com [209.85.162.177]) by mx1.freebsd.org (Postfix) with ESMTP id 8E9BA8FC18 for ; Mon, 17 Mar 2008 10:09:34 +0000 (UTC) (envelope-from valerio.daelli@gmail.com) Received: by el-out-1112.google.com with SMTP id v27so2491516ele.12 for ; Mon, 17 Mar 2008 03:09:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:to:subject:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references; bh=kreJ2QqXT0TG/xBw8tM1kz569IarKj0vFAXkq++Np6I=; b=FiGwq9SvMary+jz+ZAeOsQMDjuHh+QA2qQJ0+OPCZZlrl1R3ton8/6F5TYJEFbGROyKr9zqKZP06Xp1BhLLJZUvs8Wmftbtqgpm0zVP5H5r4ySL7c/h80tHlNLpnxWha/NgT21mIwdL078KuoArBquGTe6GvL0rtBxQogGTvLy4= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:to:subject:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references; b=IZymg2ySayRX/7dwk7UZ/ILmEqO4VsIb/0LUYHeANRwYVb/mkN3vRuVGQQ4GIRUQ6uVIWXX3duZEJ+KIf9Aa/vDR6nHyYMxzaUDSXbXmZZRwO0wSyHmS12VHSLJti5v0xW4i8+lXhJJSICWnc0AK/90qYeDsWepOonLsFWhZ7QM= Received: by 10.114.36.1 with SMTP id j1mr17311073waj.119.1205748572530; Mon, 17 Mar 2008 03:09:32 -0700 (PDT) Received: by 10.114.113.14 with HTTP; Mon, 17 Mar 2008 03:09:32 -0700 (PDT) Message-ID: <27dbfc8c0803170309p372e5904vef49b20eff2f4899@mail.gmail.com> Date: Mon, 17 Mar 2008 11:09:32 +0100 From: "Valerio Daelli" To: "Eric Anderson" , "freebsd-performance@freebsd.org" In-Reply-To: <47BEBBCF.7040907@freebsd.org> MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline References: <27dbfc8c0802190243y113d3059yd0c602850a4dbd6b@mail.gmail.com> <47BB33AD.1050005@FreeBSD.org> <27dbfc8c0802200323r13f69905l4940d0d5accd1eb1@mail.gmail.com> <47BC25C5.1000300@freebsd.org> <27dbfc8c0802200705k482152d4h1bf6e63de24edf59@mail.gmail.com> <47BC5325.8070504@freebsd.org> <27dbfc8c0802210031q3590cafbnbe31698ebdc2d1f2@mail.gmail.com> <47BEBBCF.7040907@freebsd.org> Cc: Subject: Re: Bad performance of 7.0 nfs client with Solaris nfs server X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 17 Mar 2008 10:09:35 -0000 > Just now got a chance to look at the trace. It looks like FILE_SYNC is > enabled on the write, which will cause the filer to fully commit the > block (8k in this case) to disk before replying. This will usually hurt > performance. I'm not certain where it is getting set, but you might try > some mount options, like 'async' mode. This might also be a bug in > FreeBSD that is forcing it to be enabled all the time. I'll look > through some source code and see what I can find. > > Eric > > Hi I have yes solved this issue and I have another test. Now the mount is sync (no async) and the iozone includes the -D flag. Now the write performance boosts from 3MB/s to 30MB/s. --- root@bsd7:~ iozone -D -+q 1 -i 0 -i 1 -r 2048 -n 2048 -g 2G -Raceb iozone.xls -f /mnt/nest.ifom-ieo-campus.it/iozone/file.tmp Iozone: Performance Test of File I/O Version $Revision: 3.283 $ Compiled for 32 bit mode. Build: freebsd Contributors:William Norcott, Don Capps, Isom Crawford, Kirby Collins Al Slater, Scott Rhine, Mike Wisner, Ken Goss Steve Landherr, Brad Smith, Mark Kelly, Dr. Alain CYR, Randy Dunlap, Mark Montague, Dan Million, Jean-Marc Zucconi, Jeff Blomberg, Benny Halevy, Erik Habbinga, Kris Strecker, Walter Wong. Run began: Mon Mar 17 11:06:28 2008 Using msync(MS_ASYNC) on mmap files Delay 1 seconds between tests enabled. Record Size 2048 KB Using minimum file size of 2048 kilobytes. Using maximum file size of 2097152 kilobytes. Excel chart generation enabled Auto Mode Include close in write timing Include fsync in write timing Command line used: iozone -D -+q 1 -i 0 -i 1 -r 2048 -n 2048 -g 2G -Raceb iozone.xls -f /mnt/nest.ifom-ieo-campus.it/iozone/file.tmp Output is in Kbytes/sec Time Resolution = 0.000004 seconds. Processor cache size set to 1024 Kbytes. Processor cache line size set to 32 bytes. File stride size set to 17 * record size. random random bkwd record stride KB reclen write rewrite read reread read write read rewrite read 2048 2048 49419 49755 629565 632905 4096 2048 7713 47431 625536 616224 8192 2048 28479 49564 630012 620276 16384 2048 26492 49515 631681 621500 32768 2048 13030 49572 631771 617552 65536 2048 24907 37586^C --- Notice that now we have using msync(MS_ASYNC) on mmap files (not a kernel expert so not sure if it is related to our problem). Without the -D flag we get 3MB/s with iozone. Thanks for you help! Valerio Daelli From owner-freebsd-performance@FreeBSD.ORG Mon Mar 17 10:10:48 2008 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 41A0B106566C for ; Mon, 17 Mar 2008 10:10:48 +0000 (UTC) (envelope-from valerio.daelli@gmail.com) Received: from wr-out-0506.google.com (wr-out-0506.google.com [64.233.184.235]) by mx1.freebsd.org (Postfix) with ESMTP id D6B258FC1F for ; Mon, 17 Mar 2008 10:10:47 +0000 (UTC) (envelope-from valerio.daelli@gmail.com) Received: by wr-out-0506.google.com with SMTP id 50so3697738wra.13 for ; Mon, 17 Mar 2008 03:10:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:to:subject:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references; bh=HR2p4+p5f/gy3i+YjUjFf4xGlSy8CmODco1inhAgIF4=; b=HVeKzDFhB25wTaS3dSeP+qqHln8xs0Xhhqje4meYOZCZCh3RHEsDshuMCz5+Bi/aXtvtt/yY2GrhXIqoVWJ7ZSU+xm9y4W5uatRC17z79voSmjet8kKLQlevnNiZLDorEN0eM1YOxQvnHolcpCOjDcDxPgwml9F4PwJpoAnV5Fg= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:to:subject:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references; b=kcMMlqM11kuiU9t7EslYkEeBP8DrggEQc7rziCvUHEsv0j+8WST7UuMMnc0vmhkNdlBmc32NjkEI0iKjVV8J3rRgyVN7fcTxcgvDNy1VTIofUUbW/4Cl1tZs/Mz0HlYlRO2ADjHtMBHn19ozjHptYWiD/p38MNwzBdOjPKcaC5k= Received: by 10.114.120.1 with SMTP id s1mr14425080wac.82.1205748645127; Mon, 17 Mar 2008 03:10:45 -0700 (PDT) Received: by 10.114.113.14 with HTTP; Mon, 17 Mar 2008 03:10:44 -0700 (PDT) Message-ID: <27dbfc8c0803170310x4a6ea1b6qfd6d752fc98259cf@mail.gmail.com> Date: Mon, 17 Mar 2008 11:10:44 +0100 From: "Valerio Daelli" To: "Eric Anderson" , "freebsd-performance@freebsd.org" In-Reply-To: <27dbfc8c0803170309p372e5904vef49b20eff2f4899@mail.gmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline References: <27dbfc8c0802190243y113d3059yd0c602850a4dbd6b@mail.gmail.com> <47BB33AD.1050005@FreeBSD.org> <27dbfc8c0802200323r13f69905l4940d0d5accd1eb1@mail.gmail.com> <47BC25C5.1000300@freebsd.org> <27dbfc8c0802200705k482152d4h1bf6e63de24edf59@mail.gmail.com> <47BC5325.8070504@freebsd.org> <27dbfc8c0802210031q3590cafbnbe31698ebdc2d1f2@mail.gmail.com> <47BEBBCF.7040907@freebsd.org> <27dbfc8c0803170309p372e5904vef49b20eff2f4899@mail.gmail.com> Cc: Subject: Re: Bad performance of 7.0 nfs client with Solaris nfs server X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 17 Mar 2008 10:10:48 -0000 > > I have yes solved this issue and I have another test. ^^^ I haven't yet solved this issue Sorry. > Now the mount is sync (no async) and the iozone includes > the -D flag. > Now the write performance boosts from 3MB/s to 30MB/s. > > --- > root@bsd7:~ iozone -D -+q 1 -i 0 -i 1 -r 2048 -n 2048 -g 2G -Raceb > iozone.xls -f /mnt/nest.ifom-ieo-campus.it/iozone/file.tmp > > Iozone: Performance Test of File I/O > Version $Revision: 3.283 $ > Compiled for 32 bit mode. > Build: freebsd > > Contributors:William Norcott, Don Capps, Isom Crawford, Kirby Collins > Al Slater, Scott Rhine, Mike Wisner, Ken Goss > Steve Landherr, Brad Smith, Mark Kelly, Dr. Alain CYR, > Randy Dunlap, Mark Montague, Dan Million, > Jean-Marc Zucconi, Jeff Blomberg, Benny Halevy, > Erik Habbinga, Kris Strecker, Walter Wong. > > Run began: Mon Mar 17 11:06:28 2008 > > Using msync(MS_ASYNC) on mmap files > > Delay 1 seconds between tests enabled. > Record Size 2048 KB > > Using minimum file size of 2048 kilobytes. > Using maximum file size of 2097152 kilobytes. > > Excel chart generation enabled > Auto Mode > Include close in write timing > Include fsync in write timing > Command line used: iozone -D -+q 1 -i 0 -i 1 -r 2048 -n 2048 -g 2G > -Raceb iozone.xls -f /mnt/nest.ifom-ieo-campus.it/iozone/file.tmp > > Output is in Kbytes/sec > Time Resolution = 0.000004 seconds. > > Processor cache size set to 1024 Kbytes. > Processor cache line size set to 32 bytes. > > File stride size set to 17 * record size. > random > random bkwd record stride > KB reclen write rewrite read reread read > write read rewrite read > 2048 2048 49419 49755 629565 632905 > 4096 2048 7713 47431 625536 616224 > 8192 2048 28479 49564 630012 620276 > 16384 2048 26492 49515 631681 621500 > 32768 2048 13030 49572 631771 617552 > 65536 2048 24907 37586^C > --- > > Notice that now we have using msync(MS_ASYNC) on mmap files > (not a kernel expert so not sure if it is related to our problem). > Without the -D flag we get 3MB/s with iozone. > Thanks for you help! > > Valerio Daelli > From owner-freebsd-performance@FreeBSD.ORG Mon Mar 17 18:44:31 2008 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 949D71065670 for ; Mon, 17 Mar 2008 18:44:31 +0000 (UTC) (envelope-from gofp-freebsd-performance@m.gmane.org) Received: from ciao.gmane.org (main.gmane.org [80.91.229.2]) by mx1.freebsd.org (Postfix) with ESMTP id 4D0E28FC1D for ; Mon, 17 Mar 2008 18:44:31 +0000 (UTC) (envelope-from gofp-freebsd-performance@m.gmane.org) Received: from list by ciao.gmane.org with local (Exim 4.43) id 1JbKJv-00070t-TQ for freebsd-performance@freebsd.org; Mon, 17 Mar 2008 18:44:27 +0000 Received: from 89-172-59-221.adsl.net.t-com.hr ([89.172.59.221]) by main.gmane.org with esmtp (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Mon, 17 Mar 2008 18:44:27 +0000 Received: from ivoras by 89-172-59-221.adsl.net.t-com.hr with local (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Mon, 17 Mar 2008 18:44:27 +0000 X-Injected-Via-Gmane: http://gmane.org/ To: freebsd-performance@freebsd.org From: Ivan Voras Date: Mon, 17 Mar 2008 19:44:12 +0100 Lines: 40 Message-ID: References: <20080210120013.4C3D116A421@hub.freebsd.org> <47de32b3.1bbc720a.7cf0.ffff8ff1@mx.google.com> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="------------enig1D5BAE7BF2643B2161389CA9" X-Complaints-To: usenet@ger.gmane.org X-Gmane-NNTP-Posting-Host: 89-172-59-221.adsl.net.t-com.hr User-Agent: Thunderbird 2.0.0.12 (Windows/20080213) In-Reply-To: <47de32b3.1bbc720a.7cf0.ffff8ff1@mx.google.com> X-Enigmail-Version: 0.95.6 Sender: news Subject: Re: V7 High CPU Usage on swi5:+, what is this process? X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 17 Mar 2008 18:44:31 -0000 This is an OpenPGP/MIME signed message (RFC 2440 and 3156) --------------enig1D5BAE7BF2643B2161389CA9 Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: quoted-printable Aminuddin Abdullah wrote: > I have just upgraded 5 of my machines to V7 from 6.3 and then realized = that > all the machines has a high CPU usage. Almost all of them using 80%-90%= CPU > with more than 8000 connections. Using previous 6.3, it only uses 40-50= % CPU > with the same kind of connections. >=20 > Using top -S, I can see that swi5: +, PID 17 process is using 30% of CP= U > time. What is this process? =20 "swi" stands for "software interrupt", but to find out which one you'll=20 need to give more information about the systems. What are they running?=20 Any routing, firewall? Check with "vmstat -ia" to see if you have an=20 interrupt storm. --------------enig1D5BAE7BF2643B2161389CA9 Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.5 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD4DBQFH3rv8ldnAQVacBcgRAscSAKDr2ydwD2wrBttA6y+WHhjTSlQVmACYoj2z 9U/VpLgvhs1Hb+N8RbSLpg== =6SzW -----END PGP SIGNATURE----- --------------enig1D5BAE7BF2643B2161389CA9-- From owner-freebsd-performance@FreeBSD.ORG Tue Mar 18 11:21:24 2008 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 1BC691065674; Tue, 18 Mar 2008 11:21:24 +0000 (UTC) (envelope-from rwatson@FreeBSD.org) Received: from cyrus.watson.org (cyrus.watson.org [209.31.154.42]) by mx1.freebsd.org (Postfix) with ESMTP id D24FD8FC39; Tue, 18 Mar 2008 11:21:23 +0000 (UTC) (envelope-from rwatson@FreeBSD.org) Received: from fledge.watson.org (fledge.watson.org [209.31.154.41]) by cyrus.watson.org (Postfix) with ESMTP id 59E7046C06; Tue, 18 Mar 2008 07:21:23 -0400 (EDT) Date: Tue, 18 Mar 2008 11:21:23 +0000 (GMT) From: Robert Watson X-X-Sender: robert@fledge.watson.org To: Aminuddin Abdullah In-Reply-To: <47de32b3.1bbc720a.7cf0.ffff8ff1@mx.google.com> Message-ID: <20080318111805.W17188@fledge.watson.org> References: <20080210120013.4C3D116A421@hub.freebsd.org> <47de32b3.1bbc720a.7cf0.ffff8ff1@mx.google.com> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: freebsd-performance@freebsd.org, jhb@FreeBSD.org Subject: Re: V7 High CPU Usage on swi5:+, what is this process? X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 18 Mar 2008 11:21:24 -0000 On Mon, 17 Mar 2008, Aminuddin Abdullah wrote: > I have just upgraded 5 of my machines to V7 from 6.3 and then realized that > all the machines has a high CPU usage. Almost all of them using 80%-90% CPU > with more than 8000 connections. Using previous 6.3, it only uses 40-50% CPU > with the same kind of connections. > > Using top -S, I can see that swi5: +, PID 17 process is using 30% of CPU > time. What is this process? > > All the machines are Intel C2D 6300 except one which is a AMD 4000+. > > Is this normal for V7? How do I downgrade to 6.3 if this V7 killing the CPU? '+' is used in a swi name to indicate that the names of the interrupts to put in the thread name are too long, and the code looks like it was written under the assumption that at least one name would fit. It sounds like in this case, none fit. We should fix this code, but in the mean time, what you might consider doing is hacking intr_event_update() in kern_intr.c to print out overflowing names to the console using printf(9) so you can at least see what they are. This is the somewhat suspect bit of code: 212 /* 213 * If the handler names were too long, add +'s to indicate missing 214 * names. If we run out of room and still have +'s to add, change 215 * the last character from a + to a *. 216 */ 217 last = &ie->ie_fullname[sizeof(ie->ie_fullname) - 2]; 218 while (missed-- > 0) { 219 if (strlen(ie->ie_fullname) + 1 == sizeof(ie->ie_fullname)) { 220 if (*last == '+') { 221 *last = '*'; 222 break; 223 } else 224 *last = '+'; 225 } else if (space) { 226 strcat(ie->ie_fullname, " +"); 227 space = 0; 228 } else 229 strcat(ie->ie_fullname, "+"); 230 } I've CC'd John, who might have views on what we should do about this. It would be nice if we had a way to export information on all the interrupt event sources, including soft ones, and their mappings to ithreads, including swis, using sysctl. Or maybe we do already and he'll point us at it. :-) Robert N M Watson Computer Laboratory University of Cambridge From owner-freebsd-performance@FreeBSD.ORG Tue Mar 18 13:04:07 2008 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id B66051065675; Tue, 18 Mar 2008 13:04:07 +0000 (UTC) (envelope-from rwatson@FreeBSD.org) Received: from cyrus.watson.org (cyrus.watson.org [209.31.154.42]) by mx1.freebsd.org (Postfix) with ESMTP id 806AC8FC31; Tue, 18 Mar 2008 13:04:07 +0000 (UTC) (envelope-from rwatson@FreeBSD.org) Received: from fledge.watson.org (fledge.watson.org [209.31.154.41]) by cyrus.watson.org (Postfix) with ESMTP id B74F046CA2; Tue, 18 Mar 2008 09:04:05 -0400 (EDT) Date: Tue, 18 Mar 2008 13:04:05 +0000 (GMT) From: Robert Watson X-X-Sender: robert@fledge.watson.org To: John Baldwin In-Reply-To: <200803180845.28959.jhb@freebsd.org> Message-ID: <20080318130241.J17188@fledge.watson.org> References: <20080210120013.4C3D116A421@hub.freebsd.org> <47de32b3.1bbc720a.7cf0.ffff8ff1@mx.google.com> <20080318111805.W17188@fledge.watson.org> <200803180845.28959.jhb@freebsd.org> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: freebsd-performance@freebsd.org, Aminuddin Abdullah Subject: Re: V7 High CPU Usage on swi5:+, what is this process? X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 18 Mar 2008 13:04:07 -0000 On Tue, 18 Mar 2008, John Baldwin wrote: >> '+' is used in a swi name to indicate that the names of the interrupts to >> put in the thread name are too long, and the code looks like it was written >> under the assumption that at least one name would fit. It sounds like in >> this case, none fit. We should fix this code, but in the mean time, what >> you might consider doing is hacking intr_event_update() in kern_intr.c to >> print out overflowing names to the console using printf(9) so you can at >> least see what they are. This is the somewhat suspect bit of code: > > The code is not suspect as p_comm is of fixed length. Someone just used too > long of a name for a swi handler. I was wondering whether we might not do better to put as much in as we can but truncate with a '*', so you at least get a fractional swi name. Under what situations do we use a single ithread for multiple swi's? Robert N M Watson Computer Laboratory University of Cambridge From owner-freebsd-performance@FreeBSD.ORG Tue Mar 18 13:06:56 2008 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 838161065670 for ; Tue, 18 Mar 2008 13:06:56 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from elvis.mu.org (elvis.mu.org [192.203.228.196]) by mx1.freebsd.org (Postfix) with ESMTP id 7F9ED8FC27 for ; Tue, 18 Mar 2008 13:06:56 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from zion.baldwin.cx (66-23-211-162.clients.speedfactory.net [66.23.211.162]) by elvis.mu.org (Postfix) with ESMTP id BFB5E1A4D7E; Tue, 18 Mar 2008 05:48:06 -0700 (PDT) From: John Baldwin To: Robert Watson Date: Tue, 18 Mar 2008 08:45:28 -0400 User-Agent: KMail/1.9.7 References: <20080210120013.4C3D116A421@hub.freebsd.org> <47de32b3.1bbc720a.7cf0.ffff8ff1@mx.google.com> <20080318111805.W17188@fledge.watson.org> In-Reply-To: <20080318111805.W17188@fledge.watson.org> MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit Content-Disposition: inline Message-Id: <200803180845.28959.jhb@freebsd.org> X-Mailman-Approved-At: Tue, 18 Mar 2008 13:11:54 +0000 Cc: freebsd-performance@freebsd.org, Aminuddin Abdullah Subject: Re: V7 High CPU Usage on swi5:+, what is this process? X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 18 Mar 2008 13:06:56 -0000 On Tuesday 18 March 2008 07:21:23 am Robert Watson wrote: > On Mon, 17 Mar 2008, Aminuddin Abdullah wrote: > > I have just upgraded 5 of my machines to V7 from 6.3 and then realized > > that all the machines has a high CPU usage. Almost all of them using > > 80%-90% CPU with more than 8000 connections. Using previous 6.3, it only > > uses 40-50% CPU with the same kind of connections. > > > > Using top -S, I can see that swi5: +, PID 17 process is using 30% of CPU > > time. What is this process? > > > > All the machines are Intel C2D 6300 except one which is a AMD 4000+. > > > > Is this normal for V7? How do I downgrade to 6.3 if this V7 killing the > > CPU? > > '+' is used in a swi name to indicate that the names of the interrupts to > put in the thread name are too long, and the code looks like it was written > under the assumption that at least one name would fit. It sounds like in > this case, none fit. We should fix this code, but in the mean time, what > you might consider doing is hacking intr_event_update() in kern_intr.c to > print out overflowing names to the console using printf(9) so you can at > least see what they are. This is the somewhat suspect bit of code: The code is not suspect as p_comm is of fixed length. Someone just used too long of a name for a swi handler. > I've CC'd John, who might have views on what we should do about this. It > would be nice if we had a way to export information on all the interrupt > event sources, including soft ones, and their mappings to ithreads, > including swis, using sysctl. Or maybe we do already and he'll point us at > it. :-) We don't and that is what we need for a userland interrupt binding interface to make sense. -- John Baldwin From owner-freebsd-performance@FreeBSD.ORG Tue Mar 18 13:59:16 2008 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 93E3F1065670; Tue, 18 Mar 2008 13:59:16 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from elvis.mu.org (elvis.mu.org [192.203.228.196]) by mx1.freebsd.org (Postfix) with ESMTP id 9142A8FC26; Tue, 18 Mar 2008 13:59:16 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from zion.baldwin.cx (66-23-211-162.clients.speedfactory.net [66.23.211.162]) by elvis.mu.org (Postfix) with ESMTP id D8CCA1A4D7E; Tue, 18 Mar 2008 06:57:57 -0700 (PDT) From: John Baldwin To: Robert Watson Date: Tue, 18 Mar 2008 09:23:12 -0400 User-Agent: KMail/1.9.7 References: <20080210120013.4C3D116A421@hub.freebsd.org> <200803180845.28959.jhb@freebsd.org> <20080318130241.J17188@fledge.watson.org> In-Reply-To: <20080318130241.J17188@fledge.watson.org> MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit Content-Disposition: inline Message-Id: <200803180923.13032.jhb@freebsd.org> X-Mailman-Approved-At: Tue, 18 Mar 2008 14:16:06 +0000 Cc: freebsd-performance@freebsd.org, Aminuddin Abdullah Subject: Re: V7 High CPU Usage on swi5:+, what is this process? X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 18 Mar 2008 13:59:16 -0000 On Tuesday 18 March 2008 09:04:05 am Robert Watson wrote: > On Tue, 18 Mar 2008, John Baldwin wrote: > >> '+' is used in a swi name to indicate that the names of the interrupts > >> to put in the thread name are too long, and the code looks like it was > >> written under the assumption that at least one name would fit. It > >> sounds like in this case, none fit. We should fix this code, but in the > >> mean time, what you might consider doing is hacking intr_event_update() > >> in kern_intr.c to print out overflowing names to the console using > >> printf(9) so you can at least see what they are. This is the somewhat > >> suspect bit of code: > > > > The code is not suspect as p_comm is of fixed length. Someone just used > > too long of a name for a swi handler. > > I was wondering whether we might not do better to put as much in as we can > but truncate with a '*', so you at least get a fractional swi name. Under > what situations do we use a single ithread for multiple swi's? The softclock one gets overloaded with some tty handlers. This code is also just generic ithread code common to swi's and hardware interrupts. -- John Baldwin From owner-freebsd-performance@FreeBSD.ORG Thu Mar 20 17:11:25 2008 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 4189B1065671 for ; Thu, 20 Mar 2008 17:11:25 +0000 (UTC) (envelope-from gofp-freebsd-performance@m.gmane.org) Received: from ciao.gmane.org (main.gmane.org [80.91.229.2]) by mx1.freebsd.org (Postfix) with ESMTP id 036C38FC20 for ; Thu, 20 Mar 2008 17:11:24 +0000 (UTC) (envelope-from gofp-freebsd-performance@m.gmane.org) Received: from list by ciao.gmane.org with local (Exim 4.43) id 1JcOIU-0000LC-Q5 for freebsd-performance@freebsd.org; Thu, 20 Mar 2008 17:11:22 +0000 Received: from 85.114.53.250 ([85.114.53.250]) by main.gmane.org with esmtp (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Thu, 20 Mar 2008 17:11:22 +0000 Received: from ivoras by 85.114.53.250 with local (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Thu, 20 Mar 2008 17:11:22 +0000 X-Injected-Via-Gmane: http://gmane.org/ To: freebsd-performance@freebsd.org From: Ivan Voras Date: Thu, 20 Mar 2008 18:11:16 +0100 Lines: 10 Message-ID: References: <20080210120013.4C3D116A421@hub.freebsd.org> <47de32b3.1bbc720a.7cf0.ffff8ff1@mx.google.com> <20080318111805.W17188@fledge.watson.org> Mime-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-Complaints-To: usenet@ger.gmane.org X-Gmane-NNTP-Posting-Host: 85.114.53.250 User-Agent: Thunderbird 1.5.0.14 (Windows/20071210) In-Reply-To: <20080318111805.W17188@fledge.watson.org> X-Enigmail-Version: 0.94.1.0 Sender: news Subject: Re: V7 High CPU Usage on swi5:+, what is this process? X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 20 Mar 2008 17:11:25 -0000 Robert Watson wrote: > I've CC'd John, who might have views on what we should do about this. > It would be nice if we had a way to export information on all the > interrupt event sources, including soft ones, and their mappings to > ithreads, including swis, using sysctl. Or maybe we do already and > he'll point us at it. :-) How about, for starts, the truncating loop gets overriden / replaced for ithread process names?