From owner-freebsd-performance@FreeBSD.ORG Tue Feb 1 12:07:55 2011 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id A65BC106564A; Tue, 1 Feb 2011 12:07:55 +0000 (UTC) (envelope-from stefan.lambrev@moneybookers.com) Received: from g1.moneybookers.com (g1.moneybookers.com [217.18.249.148]) by mx1.freebsd.org (Postfix) with ESMTP id D43A48FC15; Tue, 1 Feb 2011 12:07:54 +0000 (UTC) Received: from g1.moneybookers.com (localhost [127.0.0.1]) by g1.moneybookers.com (Postfix) with ESMTPS id 42F3C272C73; Tue, 1 Feb 2011 13:07:53 +0100 (CET) Received: from jailbay5-inferno.sf.moneybookers.net (jailbay5-inferno.sf.moneybookers.net [10.128.2.69]) by g1.moneybookers.com (Postfix) with ESMTP id 0135E272C68; Tue, 1 Feb 2011 13:07:52 +0100 (CET) Received: from hater.sf.moneybookers.net (hater.sf.moneybookers.net [10.129.23.125]) by jailbay5-inferno.sf.moneybookers.net (Postfix) with ESMTP id D802B3612387; Tue, 1 Feb 2011 13:07:51 +0100 (CET) Mime-Version: 1.0 (Apple Message framework v1082) From: Stefan Lambrev In-Reply-To: <20110201113724.GS18170@zxy.spb.ru> Date: Tue, 1 Feb 2011 14:07:51 +0200 Message-Id: <8979148D-8F2E-49E3-86EE-41CE6F57CDA4@moneybookers.com> References: <20110128143355.GD18170@zxy.spb.ru> <22E77EED-6455-4164-9115-BBD359EC8CA6@moneybookers.com> <20110128161035.GF18170@zxy.spb.ru> <4D42F87C.7020909@freebsd.org> <20110128172516.GG18170@zxy.spb.ru> <20110129070205.Q7034@besplex.bde.org> <20110201113724.GS18170@zxy.spb.ru> To: Slawa Olhovchenkov X-Mailer: Apple Mail (2.1082) X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00,HTML_MESSAGE, T_FRT_STOCK2,UNPARSEABLE_RELAY autolearn=ham version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on g1.sf.moneybookers.net Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: freebsd-performance@freebsd.org, Julian Elischer , Bruce Evans Subject: Re: Interrupt performance X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 01 Feb 2011 12:07:55 -0000 Hi, On Feb 1, 2011, at 1:37 PM, Slawa Olhovchenkov wrote: > On Sat, Jan 29, 2011 at 07:52:11AM +1100, Bruce Evans wrote: >=20 >>>> there are profiling tools that you may decide to run. >>>=20 >>> What tools I can use on amd64? >>>=20 >>> I boot kernel configured with 'config -p'. >>> Most time in spinlock_exit and acpi_cpu_c1. >>=20 >> Normal profiling works poorly (I see you found my old mail about high >> resolution profiling). Linux might be misreporting the overhead for >> exactly the same reasons that normal profiling works poorly: >> - the profiling clock frequency of ~1 KHz was adequate for 5 MHz = machines >> in 1998, but is now too slow. Statistics clocks are even slower = (128 >> Hz in FreeBSD, and possibly 100 Hz (?) jiffies in Linux). >> - the statistics clock might be too synchronized with other = interrupts. >> The above spinlock_exit and acpi_cpu_c1 times indicate that the >> statistics clock almost always fires on exit from another spinlock >> and/or inside ACPI, for waking up from idle for the latter. Seeing >> lots of exits from spinlocks may indicated that spinlocks are being >> used too much. >> But FreeBSD will report interrupt times and system for = non-fast-interrupts >> to an accuracy of about 1 microsecond, since it doesn't use the >> statistics clock much for this. OTOH, for fast interrupts it is = typical >> behaviour in FreeBSD and Linux to not see them at all from the = statistics >> clock interrupt, since they mask all interrupts so they mask the >> statistics clock interrupt in particular. In FreeBSD, lots of time >> apparently spent in spinlock_exit is a typical result of this, or at >> least similar things, since spinlock_enter masks all interrupts = (except >> in my version of course). Linux doesn't have fast interrupts in the >> same way that FreeBSD does, but at least in old versions almost all = of >> its interrupts masked other interrupts a lot. >=20 > I do some more test and build kernel with KTR. > Now I don't think that inetrrupt overhead on FreeBSD weight: I try > polling and don't see any difference. >=20 > I see many reported by netperf send errors. I found this > http://docs.freebsd.org/cgi/mid.cgi?E1Aice9-0002by-00. >=20 > After insert into src/nettest_bsd.c usleep(1000) if ENOBUF I see 53% > idle and ./loop 2000000000 "Elapsed 15188006 us" -- this near to linux > (Elapsed 14107670 us). >=20 > 10% of difference may be by more weight network stack (only 32104 > ticks from 126136 in interrupt handler and task switching, and 94032 > -- UDP processing in network stack and passing datagram to driver). > May be weight SOCKBUF_LOCK/SOCKBUF_UNLOCK and/or > INP_INFO_RUNLOCK/INP_RUNLOCK. Try to run with the same network buffers on FreeBSD and Linux. I think, the default values in freebsd are much, much lower. Also in the past ENOBUF was not handled properly in linux. http://wiki.freebsd.org/AvoidingLinuxisms - Do not rely on = Linux-specific socket behaviour. In particular, default socket buffer = sizes are different (call setsockopt() with SO_SNDBUF and SO_RCVBUF), = and while Linux's send() blocks when the socket buffer is full, = FreeBSD's will fail and set ENOBUFS in errno. -- Best Wishes, Stefan Lambrev ICQ# 24134177