From owner-freebsd-performance@FreeBSD.ORG Sat Jan 29 10:24:23 2011 Return-Path: Delivered-To: freebsd-performance@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 2D19B106566B; Sat, 29 Jan 2011 10:24:23 +0000 (UTC) (envelope-from slw@zxy.spb.ru) Received: from zxy.spb.ru (zxy.spb.ru [195.70.199.98]) by mx1.freebsd.org (Postfix) with ESMTP id 77E5E8FC13; Sat, 29 Jan 2011 10:24:22 +0000 (UTC) Received: from slw by zxy.spb.ru with local (Exim 4.69 (FreeBSD)) (envelope-from ) id 1Pj7ye-000FSq-3w; Sat, 29 Jan 2011 13:24:20 +0300 Date: Sat, 29 Jan 2011 13:24:20 +0300 From: Slawa Olhovchenkov To: Bruce Evans Message-ID: <20110129102420.GK18170@zxy.spb.ru> References: <20110128143355.GD18170@zxy.spb.ru> <22E77EED-6455-4164-9115-BBD359EC8CA6@moneybookers.com> <20110128161035.GF18170@zxy.spb.ru> <4D42F87C.7020909@freebsd.org> <20110128172516.GG18170@zxy.spb.ru> <20110129070205.Q7034@besplex.bde.org> <20110128215215.GJ18170@zxy.spb.ru> <20110129133859.O967@besplex.bde.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20110129133859.O967@besplex.bde.org> User-Agent: Mutt/1.5.20 (2009-06-14) X-SA-Exim-Connect-IP: X-SA-Exim-Mail-From: slw@zxy.spb.ru X-SA-Exim-Scanned: No (on zxy.spb.ru); SAEximRunCond expanded to false Cc: freebsd-performance@FreeBSD.org, Julian Elischer , Stefan Lambrev Subject: Re: Interrupt performance X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 29 Jan 2011 10:24:23 -0000 On Sat, Jan 29, 2011 at 02:43:07PM +1100, Bruce Evans wrote: > On Sat, 29 Jan 2011, Slawa Olhovchenkov wrote: > > > On Sat, Jan 29, 2011 at 07:52:11AM +1100, Bruce Evans wrote: > >> > >> To see how much CPU is actually available, run something else and see how > >> fast it runs. A simple counting loops works well on UP systems. > > > > === > > #include > > #include > > > > int Dummy; > > > > int > > main(int argc, char *argv[]) > > { > > long int count,i,dt; > > struct timeval st,et; > > > > count = atol(argv[1]); > > > > gettimeofday(&st, NULL); > > for(i=count;i;i--) Dummy++; > > gettimeofday(&et, NULL); > > dt = (et.tv_sec-st.tv_sec)*1000000 + et.tv_usec-st.tv_usec; > > printf("Elapsed %d us\n",dt); > > } > > === > > > > This is ok? > > It's better not to compete with the interrupt handler in the kernel by > spinning making syscalls, but that will do for a start. In this programm inner loop don't contain any syscall. Better varian -- loop with syscalls? > > ./loop 2000000000 > > > > FreeBSD > > 1 process: Elapsed 7554193 us > > 2 process: Elapsed 14493692 us > > netperf + 1 process: Elapsed 21403644 us > > This shows about 35% user 65% network. > > > Linux > > 1 process: Elapsed 7524843 us > > 2 process: Elapsed 14995866 us > > netperf + 1 process: Elapsed 14107670 us > > This shows about 53% user 47% network. > > So FreeBSD has about 18% more network overhead (absolute: 65-47), or > about 38% more network overhead (relative: (65-47)/47). Not too > surprising -- the context switches alone might cost that. For only 14K vs 56K interrupt. 152% more network overhead per one interrupt. And I see drammaticaly less number of context switches in linux stats (by dstat). > BTW, even -current vs my version of FreeBSD-5.2 has 10-20% more network > overhead (relative) for tx, apparently due to bloat in the network > stack. This apparently has nothing to do with hardware. The slowdown > is much the same with bge (heavily modified in my version) and em > (barely modified). One thing that I modify in both drivers is increase > the tx ifq length by a massive amount (from about 512 to about 20000). > This must be bad for overhead because such a large queue cannot fit > in the L2 cache. A large amount and perhaps more than half of NIC > overhead consists of waiting for cache misses. The slowdown in -current > might be caused by minor bloat crossing a threshold and thus causing > just one more cache miss every packet or 2. I read disccursion about support DCA in FreeBSD, sound nice. > >> Normal profiling works poorly (I see you found my old mail about high > >> resolution profiling). Linux might be misreporting the overhead for > > > > I think next server will be support PMC. > > Report from PMC still poorly? > > I should be adequate, but I prefer my version of perfmon which can > count cache misses precisely for every function. But without patches, > perfmon is even more broken than high resolution kernel profiling. Can I use your version of perfmon? How? I don't have expirence with any kernel profiling. > >> ... > >> generate too many interrupts and don't have much or any way to control > >> this. Linux will certainly be about to handle 56K int/S better than > >> FreeBSD since it doesn't have heavyweight interrupt threads AFAIK. > >> FreeBSD also has "fast" interrupts, which are much like normal interrupts > >> used to be in FreeBSD. I don't know if your NIC driver uses these. I > > > > re0: [FILTER] > > > > I think this is answer ([FILTER]), but I don't understand this answer :). > > [FILTER] means "fast". re used to unconditionally use "fast" interrupts > and a task queue, which IMO is a bad way to program an interrupt > handler, but yongari@ recently overhauled re (again :-) so that it now > doesn't use fast interrupts by default for the MSI/MSIX case . (BTW, Ineresting, but I don't think this help for this case -- old PCI chip, old CPU, old RAM, old matherboard -- all old. I don't try to get wirespeed gigabit performance from this old box, I try to compare relative performance FreeBSD vs Linux (in last time I got many feedback about poor network performance FreeBSD vs Linux). > it still bogusly uses INTR_MPSAFE for the fast interrupt bus_setup_intr().) > The overhaul probably also reduces interrupt overhead if it works on > your hardware, just be reducing the interrupt frequency. I don't > understand what moderates the interrupt frequency in the MSI case. > > >> I don't really know if this is low-end, but guess all RealTeks are :-). > > > > FreeBSD support interrupt moderation on this chip, and chip support > > TOE :) > > The support was poor according to yongari@'s long messages about > improving it. With working interrupt moderation, you just don't get > an interrupt rate of even 14KHz, except transiently. 4KHz would be > all I would be happy with on a 1-core 1.6GHz CPU. Since re is so > primitive, yongari@ only managed to limit the rate to 20KHz. (Is this > only for the non-MSI case, with the MSI case better even before?) > Linux with its reduced interrupt latency can handled your observed > 56KHz without losing by so much. My 2-core Athlon (Turion) with its > low-end bge 5705, whose brokenness consists mainly of: > completely broken interrupt moderation > can't handle full 1Gbps -- saturates at about 300Mbps > CPU and/or DMA resources used for 300Mbps are about the same as for > a bge 5701 at a full 1Mbps > saturates at about 100 KHz bge interrupts. This many interrupts takes > about all of 1 CPU to handle the hardware part and 20% of another to > generate packets. The system saturates in much the same way under > WinXP. I only recently got a version of Linux to boot on this system > and haven't tried network performance tests under Linux on it. > > Bruce > _______________________________________________ > freebsd-performance@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-performance > To unsubscribe, send any mail to "freebsd-performance-unsubscribe@freebsd.org"