From owner-freebsd-performance@FreeBSD.ORG Fri Jan 28 15:15:32 2011 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 01FAE106566B for ; Fri, 28 Jan 2011 15:15:32 +0000 (UTC) (envelope-from slw@zxy.spb.ru) Received: from zxy.spb.ru (zxy.spb.ru [195.70.199.98]) by mx1.freebsd.org (Postfix) with ESMTP id 705948FC13 for ; Fri, 28 Jan 2011 15:15:31 +0000 (UTC) Received: from slw by zxy.spb.ru with local (Exim 4.69 (FreeBSD)) (envelope-from ) id 1PipOd-0003YK-7I for freebsd-performance@freebsd.org; Fri, 28 Jan 2011 17:33:55 +0300 Date: Fri, 28 Jan 2011 17:33:55 +0300 From: Slawa Olhovchenkov To: freebsd-performance@freebsd.org Message-ID: <20110128143355.GD18170@zxy.spb.ru> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.5.20 (2009-06-14) X-SA-Exim-Connect-IP: X-SA-Exim-Mail-From: slw@zxy.spb.ru X-SA-Exim-Scanned: No (on zxy.spb.ru); SAEximRunCond expanded to false Subject: Interrupt performance X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 28 Jan 2011 15:15:32 -0000 I test network performance and found some strange result -- on the same hardware Linux more then 10x used CPU resources for interrupt processing. FreeBSD system utilise 70% CPU (32% idle, 59% interrupt, 9% sys) and network card generate 14K-18K interrupt per second. Linux system utilise 20% CPU (80% idle, 13% system, 3% hiq, 4% siq) and network card generate 56K interrupt per second. I used 'netperf -H host -t UDP_STREAM -l 60 -C -c -- -m 8972 -s 128K -S 128K' for generate network traffic. NIC: re0: port 0x4000-0x40ff mem 0xf0100000-0xf01000ff irq 19 at device 4.0 on pci11 re0: Chip rev. 0x18000000 re0: MAC rev. 0x00000000 miibus0: on re0 rgephy0: PHY 1 on miibus0 CPU: CPU: Intel(R) Celeron(R) CPU 420 @ 1.60GHz (1596.05-MHz K8-class CPU) Origin = "GenuineIntel" Id = 0x10661 Family = 6 Model = 16 Stepping = 1 Features=0xafebfbff Features2=0xe31d AMD Features=0x20100800 AMD Features2=0x1 TSC: P-state invariant RAM: one DDR2-667 DIMM. OS: 8.2-RC2, amd64 From owner-freebsd-performance@FreeBSD.ORG Fri Jan 28 16:03:18 2011 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 479951065670 for ; Fri, 28 Jan 2011 16:03:18 +0000 (UTC) (envelope-from stefan.lambrev@moneybookers.com) Received: from g1.moneybookers.com (g1.moneybookers.com [217.18.249.148]) by mx1.freebsd.org (Postfix) with ESMTP id 005EC8FC0C for ; Fri, 28 Jan 2011 16:03:17 +0000 (UTC) Received: from g1.moneybookers.com (localhost [127.0.0.1]) by g1.moneybookers.com (Postfix) with ESMTPS id 7AF38272C68; Fri, 28 Jan 2011 17:03:16 +0100 (CET) Received: from jailbay5-inferno.sf.moneybookers.net (jailbay5-inferno.sf.moneybookers.net [10.128.2.69]) by g1.moneybookers.com (Postfix) with ESMTP id B67DE272C51; Fri, 28 Jan 2011 17:03:15 +0100 (CET) Received: from hater.sf.moneybookers.net (hater.sf.moneybookers.net [10.129.23.125]) by jailbay5-inferno.sf.moneybookers.net (Postfix) with ESMTP id 9A08F3612158; Fri, 28 Jan 2011 17:03:15 +0100 (CET) Mime-Version: 1.0 (Apple Message framework v1082) Content-Type: text/plain; charset=us-ascii From: Stefan Lambrev In-Reply-To: <20110128143355.GD18170@zxy.spb.ru> Date: Fri, 28 Jan 2011 18:03:15 +0200 Content-Transfer-Encoding: quoted-printable Message-Id: <22E77EED-6455-4164-9115-BBD359EC8CA6@moneybookers.com> References: <20110128143355.GD18170@zxy.spb.ru> To: Slawa Olhovchenkov X-Mailer: Apple Mail (2.1082) X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00,UNPARSEABLE_RELAY autolearn=ham version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on g1.sf.moneybookers.net Cc: freebsd-performance@freebsd.org Subject: Re: Interrupt performance X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 28 Jan 2011 16:03:18 -0000 Hi, Do the test with netblast ;) Most perf tools are written badly and for Linux. In our internal test netblast running on freebsd outperform everything = else. P.S. - /usr/src/tools/tools/netrate/netblast - we have tested little = more expensive card - em/igb and bce. On Jan 28, 2011, at 4:33 PM, Slawa Olhovchenkov wrote: > I test network performance and found some strange result -- on the > same hardware Linux more then 10x used CPU resources for interrupt > processing. >=20 > FreeBSD system utilise 70% CPU (32% idle, 59% interrupt, 9% sys) and > network card generate 14K-18K interrupt per second. >=20 > Linux system utilise 20% CPU (80% idle, 13% system, 3% hiq, 4% siq) > and network card generate 56K interrupt per second. >=20 > I used 'netperf -H host -t UDP_STREAM -l 60 -C -c -- -m 8972 -s > 128K -S 128K' for generate network traffic. >=20 > NIC: >=20 > re0: port = 0x4000-0x40ff mem 0xf0100000-0xf01000ff irq 19 at device 4.0 on pci11 > re0: Chip rev. 0x18000000 > re0: MAC rev. 0x00000000 > miibus0: on re0 > rgephy0: PHY 1 on miibus0 >=20 >=20 > CPU: >=20 > CPU: Intel(R) Celeron(R) CPU 420 @ 1.60GHz (1596.05-MHz = K8-class CPU) > Origin =3D "GenuineIntel" Id =3D 0x10661 Family =3D 6 Model =3D 16 > Stepping =3D 1 > = Features=3D0xafebfbff > Features2=3D0xe31d > AMD Features=3D0x20100800 > AMD Features2=3D0x1 > TSC: P-state invariant >=20 > RAM: one DDR2-667 DIMM. >=20 > OS: 8.2-RC2, amd64 >=20 > _______________________________________________ > freebsd-performance@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-performance > To unsubscribe, send any mail to = "freebsd-performance-unsubscribe@freebsd.org" -- Best Wishes, Stefan Lambrev ICQ# 24134177 From owner-freebsd-performance@FreeBSD.ORG Fri Jan 28 16:10:37 2011 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 342EA10656CA for ; Fri, 28 Jan 2011 16:10:37 +0000 (UTC) (envelope-from slw@zxy.spb.ru) Received: from zxy.spb.ru (zxy.spb.ru [195.70.199.98]) by mx1.freebsd.org (Postfix) with ESMTP id 9FBDF8FC36 for ; Fri, 28 Jan 2011 16:10:36 +0000 (UTC) Received: from slw by zxy.spb.ru with local (Exim 4.69 (FreeBSD)) (envelope-from ) id 1PiquB-0004eL-S9; Fri, 28 Jan 2011 19:10:35 +0300 Date: Fri, 28 Jan 2011 19:10:35 +0300 From: Slawa Olhovchenkov To: Stefan Lambrev Message-ID: <20110128161035.GF18170@zxy.spb.ru> References: <20110128143355.GD18170@zxy.spb.ru> <22E77EED-6455-4164-9115-BBD359EC8CA6@moneybookers.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <22E77EED-6455-4164-9115-BBD359EC8CA6@moneybookers.com> User-Agent: Mutt/1.5.20 (2009-06-14) X-SA-Exim-Connect-IP: X-SA-Exim-Mail-From: slw@zxy.spb.ru X-SA-Exim-Scanned: No (on zxy.spb.ru); SAEximRunCond expanded to false Cc: freebsd-performance@freebsd.org Subject: Re: Interrupt performance X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 28 Jan 2011 16:10:37 -0000 On Fri, Jan 28, 2011 at 06:03:15PM +0200, Stefan Lambrev wrote: > Do the test with netblast ;) > Most perf tools are written badly and for Linux. > In our internal test netblast running on freebsd outperform everything else. I don't speak about bad performance. I speak about overhead. Linux: overhead 7% for 56K int/s FreeBSD: overhead 59% for 14K int/s For processing 1/4 interrupts FreeBSD need 8x CPU. > P.S. - /usr/src/tools/tools/netrate/netblast - we have tested little more expensive card - em/igb and bce. > > On Jan 28, 2011, at 4:33 PM, Slawa Olhovchenkov wrote: > > > I test network performance and found some strange result -- on the > > same hardware Linux more then 10x used CPU resources for interrupt > > processing. > > > > FreeBSD system utilise 70% CPU (32% idle, 59% interrupt, 9% sys) and > > network card generate 14K-18K interrupt per second. > > > > Linux system utilise 20% CPU (80% idle, 13% system, 3% hiq, 4% siq) > > and network card generate 56K interrupt per second. > > > > I used 'netperf -H host -t UDP_STREAM -l 60 -C -c -- -m 8972 -s > > 128K -S 128K' for generate network traffic. > > > > NIC: > > > > re0: port 0x4000-0x40ff mem 0xf0100000-0xf01000ff irq 19 at device 4.0 on pci11 > > re0: Chip rev. 0x18000000 > > re0: MAC rev. 0x00000000 > > miibus0: on re0 > > rgephy0: PHY 1 on miibus0 > > > > > > CPU: > > > > CPU: Intel(R) Celeron(R) CPU 420 @ 1.60GHz (1596.05-MHz K8-class CPU) > > Origin = "GenuineIntel" Id = 0x10661 Family = 6 Model = 16 > > Stepping = 1 > > Features=0xafebfbff > > Features2=0xe31d > > AMD Features=0x20100800 > > AMD Features2=0x1 > > TSC: P-state invariant > > > > RAM: one DDR2-667 DIMM. > > > > OS: 8.2-RC2, amd64 > > > > _______________________________________________ > > freebsd-performance@freebsd.org mailing list > > http://lists.freebsd.org/mailman/listinfo/freebsd-performance > > To unsubscribe, send any mail to "freebsd-performance-unsubscribe@freebsd.org" > > -- > Best Wishes, > Stefan Lambrev > ICQ# 24134177 > > > > > > _______________________________________________ > freebsd-performance@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-performance > To unsubscribe, send any mail to "freebsd-performance-unsubscribe@freebsd.org" From owner-freebsd-performance@FreeBSD.ORG Fri Jan 28 16:15:56 2011 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 5FA71106566B for ; Fri, 28 Jan 2011 16:15:56 +0000 (UTC) (envelope-from stefan.lambrev@moneybookers.com) Received: from g1.moneybookers.com (g1.moneybookers.com [217.18.249.148]) by mx1.freebsd.org (Postfix) with ESMTP id D96B48FC08 for ; Fri, 28 Jan 2011 16:15:55 +0000 (UTC) Received: from g1.moneybookers.com (localhost [127.0.0.1]) by g1.moneybookers.com (Postfix) with ESMTPS id CB156272C6D; Fri, 28 Jan 2011 17:15:54 +0100 (CET) Received: from jailbay5-inferno.sf.moneybookers.net (jailbay5-inferno.sf.moneybookers.net [10.128.2.69]) by g1.moneybookers.com (Postfix) with ESMTP id 04911272C6C; Fri, 28 Jan 2011 17:15:54 +0100 (CET) Received: from hater.sf.moneybookers.net (hater.sf.moneybookers.net [10.129.23.125]) by jailbay5-inferno.sf.moneybookers.net (Postfix) with ESMTP id DC6E9361218D; Fri, 28 Jan 2011 17:15:53 +0100 (CET) Mime-Version: 1.0 (Apple Message framework v1082) Content-Type: text/plain; charset=us-ascii From: Stefan Lambrev In-Reply-To: <20110128161035.GF18170@zxy.spb.ru> Date: Fri, 28 Jan 2011 18:15:53 +0200 Content-Transfer-Encoding: quoted-printable Message-Id: References: <20110128143355.GD18170@zxy.spb.ru> <22E77EED-6455-4164-9115-BBD359EC8CA6@moneybookers.com> <20110128161035.GF18170@zxy.spb.ru> To: Slawa Olhovchenkov X-Mailer: Apple Mail (2.1082) X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00,UNPARSEABLE_RELAY autolearn=ham version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on g1.sf.moneybookers.net Cc: freebsd-performance@freebsd.org Subject: Re: Interrupt performance X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 28 Jan 2011 16:15:56 -0000 The overhead comes from badly written software. This software is optimized for linux and you have to optimize it for = freebsd, then you will have the same overhead. All those *popular* benchmarks like hping, iperf, netperf have some = strange optimizations for linux - we call them linuxism. Just search the archives - I'm pretty sure patches are flying around. On Jan 28, 2011, at 6:10 PM, Slawa Olhovchenkov wrote: > On Fri, Jan 28, 2011 at 06:03:15PM +0200, Stefan Lambrev wrote: >=20 >> Do the test with netblast ;) >> Most perf tools are written badly and for Linux. >> In our internal test netblast running on freebsd outperform = everything else. >=20 > I don't speak about bad performance. > I speak about overhead. >=20 > Linux: overhead 7% for 56K int/s > FreeBSD: overhead 59% for 14K int/s >=20 > For processing 1/4 interrupts FreeBSD need 8x CPU. >=20 >> P.S. - /usr/src/tools/tools/netrate/netblast - we have tested little = more expensive card - em/igb and bce. >>=20 >> On Jan 28, 2011, at 4:33 PM, Slawa Olhovchenkov wrote: >>=20 >>> I test network performance and found some strange result -- on the >>> same hardware Linux more then 10x used CPU resources for interrupt >>> processing. >>>=20 >>> FreeBSD system utilise 70% CPU (32% idle, 59% interrupt, 9% sys) and >>> network card generate 14K-18K interrupt per second. >>>=20 >>> Linux system utilise 20% CPU (80% idle, 13% system, 3% hiq, 4% siq) >>> and network card generate 56K interrupt per second. >>>=20 >>> I used 'netperf -H host -t UDP_STREAM -l 60 -C -c -- -m 8972 -s >>> 128K -S 128K' for generate network traffic. >>>=20 >>> NIC: >>>=20 >>> re0: port = 0x4000-0x40ff mem 0xf0100000-0xf01000ff irq 19 at device 4.0 on pci11 >>> re0: Chip rev. 0x18000000 >>> re0: MAC rev. 0x00000000 >>> miibus0: on re0 >>> rgephy0: PHY 1 on miibus0 >>>=20 >>>=20 >>> CPU: >>>=20 >>> CPU: Intel(R) Celeron(R) CPU 420 @ 1.60GHz (1596.05-MHz = K8-class CPU) >>> Origin =3D "GenuineIntel" Id =3D 0x10661 Family =3D 6 Model =3D = 16 >>> Stepping =3D 1 >>> = Features=3D0xafebfbff >>> Features2=3D0xe31d >>> AMD Features=3D0x20100800 >>> AMD Features2=3D0x1 >>> TSC: P-state invariant >>>=20 >>> RAM: one DDR2-667 DIMM. >>>=20 >>> OS: 8.2-RC2, amd64 >>>=20 >>> _______________________________________________ >>> freebsd-performance@freebsd.org mailing list >>> http://lists.freebsd.org/mailman/listinfo/freebsd-performance >>> To unsubscribe, send any mail to = "freebsd-performance-unsubscribe@freebsd.org" >>=20 >> -- >> Best Wishes, >> Stefan Lambrev >> ICQ# 24134177 >>=20 >>=20 >>=20 >>=20 >>=20 >> _______________________________________________ >> freebsd-performance@freebsd.org mailing list >> http://lists.freebsd.org/mailman/listinfo/freebsd-performance >> To unsubscribe, send any mail to = "freebsd-performance-unsubscribe@freebsd.org" -- Best Wishes, Stefan Lambrev ICQ# 24134177 From owner-freebsd-performance@FreeBSD.ORG Fri Jan 28 17:25:20 2011 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 23B0F106564A for ; Fri, 28 Jan 2011 17:25:20 +0000 (UTC) (envelope-from slw@zxy.spb.ru) Received: from zxy.spb.ru (zxy.spb.ru [195.70.199.98]) by mx1.freebsd.org (Postfix) with ESMTP id 6E8AE8FC19 for ; Fri, 28 Jan 2011 17:25:19 +0000 (UTC) Received: from slw by zxy.spb.ru with local (Exim 4.69 (FreeBSD)) (envelope-from ) id 1Pis4T-0005s3-0N; Fri, 28 Jan 2011 20:25:17 +0300 Date: Fri, 28 Jan 2011 20:25:16 +0300 From: Slawa Olhovchenkov To: Julian Elischer Message-ID: <20110128172516.GG18170@zxy.spb.ru> References: <20110128143355.GD18170@zxy.spb.ru> <22E77EED-6455-4164-9115-BBD359EC8CA6@moneybookers.com> <20110128161035.GF18170@zxy.spb.ru> <4D42F87C.7020909@freebsd.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <4D42F87C.7020909@freebsd.org> User-Agent: Mutt/1.5.20 (2009-06-14) X-SA-Exim-Connect-IP: X-SA-Exim-Mail-From: slw@zxy.spb.ru X-SA-Exim-Scanned: No (on zxy.spb.ru); SAEximRunCond expanded to false Cc: freebsd-performance@freebsd.org, Stefan Lambrev Subject: Re: Interrupt performance X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 28 Jan 2011 17:25:20 -0000 On Fri, Jan 28, 2011 at 09:10:20AM -0800, Julian Elischer wrote: > On 1/28/11 8:15 AM, Stefan Lambrev wrote: > > The overhead comes from badly written software. > > This software is optimized for linux and you have to optimize it for freebsd, then you will have the same overhead. > > All those *popular* benchmarks like hping, iperf, netperf have some strange optimizations for linux - we call them linuxism. > > Just search the archives - I'm pretty sure patches are flying around. > > > He wants to know why the freeBSD driver spends 8 x as much time on > each interrupt. Yes! > there are of course several possible answers, including: > > 1/ Sometimes BSD and Linux report things differently. Linux may or may not > account for the lowest level interrupt tie the same as BSD But I see only 20% idle on FreeBSD and 80% idle on Linux. > 2/ the BSD driver for that chip may be badly written, or may > be doing more or different work for some reason > 3/ the FreeBSD interrupt code may be misconfigured for that driver. > > or maybe combinations... > > there are profiling tools that you may decide to run. What tools I can use on amd64? I boot kernel configured with 'config -p'. Most time in spinlock_exit and acpi_cpu_c1. > > On Jan 28, 2011, at 6:10 PM, Slawa Olhovchenkov wrote: > > > >> On Fri, Jan 28, 2011 at 06:03:15PM +0200, Stefan Lambrev wrote: > >> > >>> Do the test with netblast ;) > >>> Most perf tools are written badly and for Linux. > >>> In our internal test netblast running on freebsd outperform everything else. > >> I don't speak about bad performance. > >> I speak about overhead. > >> > >> Linux: overhead 7% for 56K int/s > >> FreeBSD: overhead 59% for 14K int/s > >> > >> For processing 1/4 interrupts FreeBSD need 8x CPU. > >> > >>> P.S. - /usr/src/tools/tools/netrate/netblast - we have tested little more expensive card - em/igb and bce. > >>> > >>> On Jan 28, 2011, at 4:33 PM, Slawa Olhovchenkov wrote: > >>> > >>>> I test network performance and found some strange result -- on the > >>>> same hardware Linux more then 10x used CPU resources for interrupt > >>>> processing. > >>>> > >>>> FreeBSD system utilise 70% CPU (32% idle, 59% interrupt, 9% sys) and > >>>> network card generate 14K-18K interrupt per second. > >>>> > >>>> Linux system utilise 20% CPU (80% idle, 13% system, 3% hiq, 4% siq) > >>>> and network card generate 56K interrupt per second. > >>>> > >>>> I used 'netperf -H host -t UDP_STREAM -l 60 -C -c -- -m 8972 -s > >>>> 128K -S 128K' for generate network traffic. > >>>> > >>>> NIC: > >>>> > >>>> re0: port 0x4000-0x40ff mem 0xf0100000-0xf01000ff irq 19 at device 4.0 on pci11 > >>>> re0: Chip rev. 0x18000000 > >>>> re0: MAC rev. 0x00000000 > >>>> miibus0: on re0 > >>>> rgephy0: PHY 1 on miibus0 > >>>> > >>>> > >>>> CPU: > >>>> > >>>> CPU: Intel(R) Celeron(R) CPU 420 @ 1.60GHz (1596.05-MHz K8-class CPU) > >>>> Origin = "GenuineIntel" Id = 0x10661 Family = 6 Model = 16 > >>>> Stepping = 1 > >>>> Features=0xafebfbff > >>>> Features2=0xe31d > >>>> AMD Features=0x20100800 > >>>> AMD Features2=0x1 > >>>> TSC: P-state invariant > >>>> > >>>> RAM: one DDR2-667 DIMM. > >>>> > >>>> OS: 8.2-RC2, amd64 > >>>> > >>>> _______________________________________________ > >>>> freebsd-performance@freebsd.org mailing list > >>>> http://lists.freebsd.org/mailman/listinfo/freebsd-performance > >>>> To unsubscribe, send any mail to "freebsd-performance-unsubscribe@freebsd.org" > >>> -- > >>> Best Wishes, > >>> Stefan Lambrev > >>> ICQ# 24134177 > >>> > >>> > >>> > >>> > >>> > >>> _______________________________________________ > >>> freebsd-performance@freebsd.org mailing list > >>> http://lists.freebsd.org/mailman/listinfo/freebsd-performance > >>> To unsubscribe, send any mail to "freebsd-performance-unsubscribe@freebsd.org" > > -- > > Best Wishes, > > Stefan Lambrev > > ICQ# 24134177 > > > > > > > > > > > > _______________________________________________ > > freebsd-performance@freebsd.org mailing list > > http://lists.freebsd.org/mailman/listinfo/freebsd-performance > > To unsubscribe, send any mail to "freebsd-performance-unsubscribe@freebsd.org" > > > From owner-freebsd-performance@FreeBSD.ORG Fri Jan 28 17:42:46 2011 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 7FBF21065679 for ; Fri, 28 Jan 2011 17:42:46 +0000 (UTC) (envelope-from julian@freebsd.org) Received: from vps1.elischer.org (vps1.elischer.org [204.109.63.16]) by mx1.freebsd.org (Postfix) with ESMTP id CCE488FC15 for ; Fri, 28 Jan 2011 17:42:45 +0000 (UTC) Received: from julian-mac.elischer.org (h-67-100-89-137.snfccasy.static.covad.net [67.100.89.137]) (authenticated bits=0) by vps1.elischer.org (8.14.4/8.14.4) with ESMTP id p0SH1F4f071459 (version=TLSv1/SSLv3 cipher=DHE-RSA-CAMELLIA256-SHA bits=256 verify=NO); Fri, 28 Jan 2011 17:01:27 GMT (envelope-from julian@freebsd.org) Message-ID: <4D42F87C.7020909@freebsd.org> Date: Fri, 28 Jan 2011 09:10:20 -0800 From: Julian Elischer User-Agent: Mozilla/5.0 (Macintosh; U; PPC Mac OS X 10.4; en-US; rv:1.9.2.13) Gecko/20101207 Thunderbird/3.1.7 MIME-Version: 1.0 To: Stefan Lambrev References: <20110128143355.GD18170@zxy.spb.ru> <22E77EED-6455-4164-9115-BBD359EC8CA6@moneybookers.com> <20110128161035.GF18170@zxy.spb.ru> In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-performance@freebsd.org, Slawa Olhovchenkov Subject: Re: Interrupt performance X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 28 Jan 2011 17:42:46 -0000 On 1/28/11 8:15 AM, Stefan Lambrev wrote: > The overhead comes from badly written software. > This software is optimized for linux and you have to optimize it for freebsd, then you will have the same overhead. > All those *popular* benchmarks like hping, iperf, netperf have some strange optimizations for linux - we call them linuxism. > Just search the archives - I'm pretty sure patches are flying around. He wants to know why the freeBSD driver spends 8 x as much time on each interrupt. there are of course several possible answers, including: 1/ Sometimes BSD and Linux report things differently. Linux may or may not account for the lowest level interrupt tie the same as BSD 2/ the BSD driver for that chip may be badly written, or may be doing more or different work for some reason 3/ the FreeBSD interrupt code may be misconfigured for that driver. or maybe combinations... there are profiling tools that you may decide to run. Julian > On Jan 28, 2011, at 6:10 PM, Slawa Olhovchenkov wrote: > >> On Fri, Jan 28, 2011 at 06:03:15PM +0200, Stefan Lambrev wrote: >> >>> Do the test with netblast ;) >>> Most perf tools are written badly and for Linux. >>> In our internal test netblast running on freebsd outperform everything else. >> I don't speak about bad performance. >> I speak about overhead. >> >> Linux: overhead 7% for 56K int/s >> FreeBSD: overhead 59% for 14K int/s >> >> For processing 1/4 interrupts FreeBSD need 8x CPU. >> >>> P.S. - /usr/src/tools/tools/netrate/netblast - we have tested little more expensive card - em/igb and bce. >>> >>> On Jan 28, 2011, at 4:33 PM, Slawa Olhovchenkov wrote: >>> >>>> I test network performance and found some strange result -- on the >>>> same hardware Linux more then 10x used CPU resources for interrupt >>>> processing. >>>> >>>> FreeBSD system utilise 70% CPU (32% idle, 59% interrupt, 9% sys) and >>>> network card generate 14K-18K interrupt per second. >>>> >>>> Linux system utilise 20% CPU (80% idle, 13% system, 3% hiq, 4% siq) >>>> and network card generate 56K interrupt per second. >>>> >>>> I used 'netperf -H host -t UDP_STREAM -l 60 -C -c -- -m 8972 -s >>>> 128K -S 128K' for generate network traffic. >>>> >>>> NIC: >>>> >>>> re0: port 0x4000-0x40ff mem 0xf0100000-0xf01000ff irq 19 at device 4.0 on pci11 >>>> re0: Chip rev. 0x18000000 >>>> re0: MAC rev. 0x00000000 >>>> miibus0: on re0 >>>> rgephy0: PHY 1 on miibus0 >>>> >>>> >>>> CPU: >>>> >>>> CPU: Intel(R) Celeron(R) CPU 420 @ 1.60GHz (1596.05-MHz K8-class CPU) >>>> Origin = "GenuineIntel" Id = 0x10661 Family = 6 Model = 16 >>>> Stepping = 1 >>>> Features=0xafebfbff >>>> Features2=0xe31d >>>> AMD Features=0x20100800 >>>> AMD Features2=0x1 >>>> TSC: P-state invariant >>>> >>>> RAM: one DDR2-667 DIMM. >>>> >>>> OS: 8.2-RC2, amd64 >>>> >>>> _______________________________________________ >>>> freebsd-performance@freebsd.org mailing list >>>> http://lists.freebsd.org/mailman/listinfo/freebsd-performance >>>> To unsubscribe, send any mail to "freebsd-performance-unsubscribe@freebsd.org" >>> -- >>> Best Wishes, >>> Stefan Lambrev >>> ICQ# 24134177 >>> >>> >>> >>> >>> >>> _______________________________________________ >>> freebsd-performance@freebsd.org mailing list >>> http://lists.freebsd.org/mailman/listinfo/freebsd-performance >>> To unsubscribe, send any mail to "freebsd-performance-unsubscribe@freebsd.org" > -- > Best Wishes, > Stefan Lambrev > ICQ# 24134177 > > > > > > _______________________________________________ > freebsd-performance@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-performance > To unsubscribe, send any mail to "freebsd-performance-unsubscribe@freebsd.org" > From owner-freebsd-performance@FreeBSD.ORG Fri Jan 28 17:45:01 2011 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id E18B91065670 for ; Fri, 28 Jan 2011 17:45:01 +0000 (UTC) (envelope-from stefan.lambrev@moneybookers.com) Received: from g1.moneybookers.com (g1.moneybookers.com [217.18.249.148]) by mx1.freebsd.org (Postfix) with ESMTP id 16B888FC13 for ; Fri, 28 Jan 2011 17:45:00 +0000 (UTC) Received: from g1.moneybookers.com (localhost [127.0.0.1]) by g1.moneybookers.com (Postfix) with ESMTPS id 37277272C6C; Fri, 28 Jan 2011 18:44:59 +0100 (CET) Received: from jailbay5-inferno.sf.moneybookers.net (jailbay5-inferno.sf.moneybookers.net [10.128.2.69]) by g1.moneybookers.com (Postfix) with ESMTP id 5C561272C2F; Fri, 28 Jan 2011 18:44:57 +0100 (CET) Received: from hater.sf.moneybookers.net (hater.sf.moneybookers.net [10.129.23.125]) by jailbay5-inferno.sf.moneybookers.net (Postfix) with ESMTP id 3F0B43612153; Fri, 28 Jan 2011 18:44:57 +0100 (CET) Mime-Version: 1.0 (Apple Message framework v1082) From: Stefan Lambrev In-Reply-To: <20110128172516.GG18170@zxy.spb.ru> Date: Fri, 28 Jan 2011 19:44:57 +0200 Message-Id: References: <20110128143355.GD18170@zxy.spb.ru> <22E77EED-6455-4164-9115-BBD359EC8CA6@moneybookers.com> <20110128161035.GF18170@zxy.spb.ru> <4D42F87C.7020909@freebsd.org> <20110128172516.GG18170@zxy.spb.ru> To: Slawa Olhovchenkov X-Mailer: Apple Mail (2.1082) X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00,HTML_MESSAGE, UNPARSEABLE_RELAY autolearn=ham version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on g1.sf.moneybookers.net Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: freebsd-performance@freebsd.org, Julian Elischer Subject: Re: Interrupt performance X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 28 Jan 2011 17:45:02 -0000 Hi, On Jan 28, 2011, at 7:25 PM, Slawa Olhovchenkov wrote: > On Fri, Jan 28, 2011 at 09:10:20AM -0800, Julian Elischer wrote: >=20 >> On 1/28/11 8:15 AM, Stefan Lambrev wrote: >>> The overhead comes from badly written software. >>> This software is optimized for linux and you have to optimize it for = freebsd, then you will have the same overhead. >>> All those *popular* benchmarks like hping, iperf, netperf have some = strange optimizations for linux - we call them linuxism. >>> Just search the archives - I'm pretty sure patches are flying = around. >>=20 >>=20 >> He wants to know why the freeBSD driver spends 8 x as much time on=20 >> each interrupt. >=20 > Yes! >=20 >> there are of course several possible answers, including: >>=20 >> 1/ Sometimes BSD and Linux report things differently. Linux may or = may not >> account for the lowest level interrupt tie the same as BSD >=20 > But I see only 20% idle on FreeBSD and 80% idle on Linux. >=20 >> 2/ the BSD driver for that chip may be badly written, or may >> be doing more or different work for some reason >> 3/ the FreeBSD interrupt code may be misconfigured for that driver. >>=20 >> or maybe combinations... >>=20 >> there are profiling tools that you may decide to run. >=20 > What tools I can use on amd64? Look at this document - = http://software.intel.com/sites/oss/pdfs/profiling_debugging_freebsd_kerne= l_321772.pdf It contains brief information for all useful profiling tools, or just = google for "freebsd kernel profiling" I'm not sure what the situation with RealTek driver, but in the past = when I have done profiling, I saw that much of the CPU time was spent on expensive (for FreeBSD) calls, which where = very cheap in linux. >=20 > I boot kernel configured with 'config -p'. > Most time in spinlock_exit and acpi_cpu_c1. >=20 -- Best Wishes, Stefan Lambrev ICQ# 24134177 From owner-freebsd-performance@FreeBSD.ORG Fri Jan 28 17:48:23 2011 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 0CC0B1065673 for ; Fri, 28 Jan 2011 17:48:23 +0000 (UTC) (envelope-from luke@foolishgames.com) Received: from qmta15.westchester.pa.mail.comcast.net (qmta15.westchester.pa.mail.comcast.net [76.96.59.228]) by mx1.freebsd.org (Postfix) with ESMTP id B40908FC19 for ; Fri, 28 Jan 2011 17:48:22 +0000 (UTC) Received: from omta18.westchester.pa.mail.comcast.net ([76.96.62.90]) by qmta15.westchester.pa.mail.comcast.net with comcast id 15Xq1g0091wpRvQ5F5b7qd; Fri, 28 Jan 2011 17:35:07 +0000 Received: from stargazer.midnightbsd.org ([70.91.226.201]) by omta18.westchester.pa.mail.comcast.net with comcast id 15b61g00v4MLobJ3e5b71d; Fri, 28 Jan 2011 17:35:07 +0000 Received: from [10.1.10.166] ([10.1.10.166]) (authenticated bits=0) by stargazer.midnightbsd.org (8.14.4/8.14.4) with ESMTP id p0SHZ3RV090601 (version=TLSv1/SSLv3 cipher=AES128-SHA bits=128 verify=NOT); Fri, 28 Jan 2011 12:35:05 -0500 (EST) (envelope-from luke@foolishgames.com) X-Habeas-Swe-2: brightly anticipated In-Reply-To: <20110128172516.GG18170@zxy.spb.ru> X-Habeas-Swe-7: warrant mark warrants that this is a Habeas Compliant X-Habeas-Swe-4: Copyright 2002 Habeas (tm) X-Habeas-Swe-1: winter into spring Mime-Version: 1.0 (Apple Message framework v1082) X-Habeas-Swe-9: mark in spam to . Date: Fri, 28 Jan 2011 12:35:03 -0500 X-Habeas-Swe-6: email in exchange for a license for this Habeas Message-Id: To: Slawa Olhovchenkov X-Habeas-Swe-3: like Habeas SWE (tm) Content-Transfer-Encoding: quoted-printable From: Lucas Holt References: <20110128143355.GD18170@zxy.spb.ru> <22E77EED-6455-4164-9115-BBD359EC8CA6@moneybookers.com> <20110128161035.GF18170@zxy.spb.ru> <4D42F87C.7020909@freebsd.org> <20110128172516.GG18170@zxy.spb.ru> X-Habeas-Swe-8: Message (HCM) and not spam. Please report use of this Content-Type: text/plain; charset=us-ascii X-Habeas-Swe-5: Sender Warranted Email (SWE) (tm). The sender of this X-Mailer: Apple Mail (2.1082) Cc: freebsd-performance@freebsd.org, Julian Elischer , Stefan Lambrev Subject: Re: Interrupt performance X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 28 Jan 2011 17:48:23 -0000 Assuming your problems are related to the driver, maybe you could try = the driver from Realtek's website for FreeBSD. If there is a bug in the = FreeBSD driver and the Realtek driver works, that would narrow down = things quite a bit. =20 http://www.realtek.com.tw/downloads/ From owner-freebsd-performance@FreeBSD.ORG Fri Jan 28 17:57:09 2011 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 7544E1065670; Fri, 28 Jan 2011 17:57:09 +0000 (UTC) (envelope-from slw@zxy.spb.ru) Received: from zxy.spb.ru (zxy.spb.ru [195.70.199.98]) by mx1.freebsd.org (Postfix) with ESMTP id DCBC58FC26; Fri, 28 Jan 2011 17:57:08 +0000 (UTC) Received: from slw by zxy.spb.ru with local (Exim 4.69 (FreeBSD)) (envelope-from ) id 1PisZI-0006N5-06; Fri, 28 Jan 2011 20:57:08 +0300 Date: Fri, 28 Jan 2011 20:57:07 +0300 From: Slawa Olhovchenkov To: Stefan Lambrev Message-ID: <20110128175707.GH18170@zxy.spb.ru> References: <20110128143355.GD18170@zxy.spb.ru> <22E77EED-6455-4164-9115-BBD359EC8CA6@moneybookers.com> <20110128161035.GF18170@zxy.spb.ru> <4D42F87C.7020909@freebsd.org> <20110128172516.GG18170@zxy.spb.ru> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.20 (2009-06-14) X-SA-Exim-Connect-IP: X-SA-Exim-Mail-From: slw@zxy.spb.ru X-SA-Exim-Scanned: No (on zxy.spb.ru); SAEximRunCond expanded to false Cc: freebsd-performance@freebsd.org, Julian Elischer Subject: Re: Interrupt performance X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 28 Jan 2011 17:57:09 -0000 On Fri, Jan 28, 2011 at 07:44:57PM +0200, Stefan Lambrev wrote: > >> there are profiling tools that you may decide to run. > > > > What tools I can use on amd64? > > Look at this document - http://software.intel.com/sites/oss/pdfs/profiling_debugging_freebsd_kernel_321772.pdf > It contains brief information for all useful profiling tools, or just google for "freebsd kernel profiling" > I'm not sure what the situation with RealTek driver, but in the past when I have done profiling, I saw that much of > the CPU time was spent on expensive (for FreeBSD) calls, which where very cheap in linux. kgmon badly work on amd64. PmcStat don't supported on this CPU. Also, http://lists.freebsd.org/pipermail/freebsd-questions/2008-October/183977.html === This is one of many defects that are not present in high resolution kernel profiling (kgmon -B instead of kgmon -b; availaible on amd64 and i386). However, high resolution kernel profiling doesn't work right with SMP, and was completely broken by gcc-4. Ordinary profiling was less completely broken by gcc-4, and you can recover the old behaviour by turning off new optimizations (mainly -funit-at-a-time and/or -finline-functions-called-once and or all of -O2). === > > > > I boot kernel configured with 'config -p'. > > Most time in spinlock_exit and acpi_cpu_c1. From owner-freebsd-performance@FreeBSD.ORG Fri Jan 28 18:45:02 2011 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 8BCBB106566B for ; Fri, 28 Jan 2011 18:45:02 +0000 (UTC) (envelope-from slw@zxy.spb.ru) Received: from zxy.spb.ru (zxy.spb.ru [195.70.199.98]) by mx1.freebsd.org (Postfix) with ESMTP id D4D9E8FC15 for ; Fri, 28 Jan 2011 18:45:01 +0000 (UTC) Received: from slw by zxy.spb.ru with local (Exim 4.69 (FreeBSD)) (envelope-from ) id 1PitJb-00070j-FI; Fri, 28 Jan 2011 21:44:59 +0300 Date: Fri, 28 Jan 2011 21:44:59 +0300 From: Slawa Olhovchenkov To: Stefan Lambrev Message-ID: <20110128184459.GI18170@zxy.spb.ru> References: <20110128143355.GD18170@zxy.spb.ru> <22E77EED-6455-4164-9115-BBD359EC8CA6@moneybookers.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <22E77EED-6455-4164-9115-BBD359EC8CA6@moneybookers.com> User-Agent: Mutt/1.5.20 (2009-06-14) X-SA-Exim-Connect-IP: X-SA-Exim-Mail-From: slw@zxy.spb.ru X-SA-Exim-Scanned: No (on zxy.spb.ru); SAEximRunCond expanded to false Cc: freebsd-performance@freebsd.org Subject: Re: Interrupt performance X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 28 Jan 2011 18:45:02 -0000 On Fri, Jan 28, 2011 at 06:03:15PM +0200, Stefan Lambrev wrote: > Do the test with netblast ;) > Most perf tools are written badly and for Linux. > In our internal test netblast running on freebsd outperform everything else. > > P.S. - /usr/src/tools/tools/netrate/netblast - we have tested little more expensive card - em/igb and bce. Now I install 8.2-RC2/i386 and use netblast: # /usr/obj/usr/src/tools/tools/netrate/netblast/netblast 10.200.0.1 1122 1400 10 start: 1296239273.106911353 finish: 1296239283.107222845 send calls: 2703219 send errors: 2090049 approx send rate: 61317 approx error rate: 0 CPU load (No difference with netperf): iostat: tty ad0 cpu tin tout KB/t tps MB/s us ni sy in id 1 48 0.00 0 0.00 4 0 47 20 29 0 44 0.00 0 0.00 2 0 68 30 0 0 44 0.00 0 0.00 12 0 65 23 0 0 44 0.00 0 0.00 6 0 73 21 0 0 44 0.00 0 0.00 8 0 54 38 0 0 44 0.00 0 0.00 5 0 66 29 0 0 44 0.00 0 0.00 7 0 65 29 0 0 44 0.00 0 0.00 7 0 64 29 0 0 44 0.00 0 0.00 8 0 64 28 0 0 44 0.00 0 0.00 7 0 57 35 0 0 235 0.00 0 0.00 4 0 13 12 71 vmstat: procs memory page disk faults cpu r b w avm fre flt re pi po fr sr ad0 in sy cs us sy id 1 0 0 97748K 431M 0 0 0 0 0 0 0 15331 269981 30697 6 94 0 1 0 0 97748K 431M 0 0 0 0 0 0 0 15333 269072 30716 5 95 0 1 0 0 97748K 431M 0 0 0 0 0 0 0 15332 269512 30690 9 91 0 1 0 0 97748K 431M 0 0 0 0 0 0 0 15331 269633 30698 8 92 0 1 0 0 97748K 431M 0 0 0 0 0 0 0 15334 269613 30693 7 93 0 1 0 0 97748K 431M 0 0 0 0 0 0 0 15332 269438 30713 13 87 0 1 0 0 97748K 431M 0 0 0 0 0 0 0 15337 269460 30705 7 93 0 1 0 0 97748K 431M 0 0 0 0 0 0 0 15299 266321 30617 6 94 0 1 0 0 97748K 431M 0 0 0 0 0 0 0 15300 266147 30625 11 89 0 (Linux, for compare, only 1279 cs) systat -vmstat 1 3 users Load 0.29 0.09 0.07 Jan 28 21:31 Mem:KB REAL VIRTUAL VN PAGER SWAP PAGER Tot Share Tot Share Free in out in out Act 16068 4496 99772 5208 440312 count All 37664 6644 2227916 10384 pages Proc: Interrupts r p d s w Csw Trp Sys Int Sof Flt cow 17372 total 1 23 30k 12 270k 15k 30k zfod atkbd0 1 ozfod ata0 irq14 63.2%Sys 29.3%Intr 7.5%User 0.0%Nice 0.0%Idle %ozfod ata1 irq15 | | | | | | | | | | | daefr 15366 re0 irq19 ================================++++++++++++++>>>> prcfr 2006 cpu0: time 1 dtbuf totfr Namei Name-cache Dir-cache 34738 desvn react Calls hits % hits % 595 numvn pdwak 34 frevn pdpgs intrn Disks ad0 26572 wire KB/t 0.00 12800 act tps 0 12732 inact MB/s 0.00 104 cache %busy 0 440208 free 11552 buf > On Jan 28, 2011, at 4:33 PM, Slawa Olhovchenkov wrote: > > > I test network performance and found some strange result -- on the > > same hardware Linux more then 10x used CPU resources for interrupt > > processing. > > > > FreeBSD system utilise 70% CPU (32% idle, 59% interrupt, 9% sys) and > > network card generate 14K-18K interrupt per second. > > > > Linux system utilise 20% CPU (80% idle, 13% system, 3% hiq, 4% siq) > > and network card generate 56K interrupt per second. > > > > I used 'netperf -H host -t UDP_STREAM -l 60 -C -c -- -m 8972 -s > > 128K -S 128K' for generate network traffic. > > > > NIC: > > > > re0: port 0x4000-0x40ff mem 0xf0100000-0xf01000ff irq 19 at device 4.0 on pci11 > > re0: Chip rev. 0x18000000 > > re0: MAC rev. 0x00000000 > > miibus0: on re0 > > rgephy0: PHY 1 on miibus0 > > > > > > CPU: > > > > CPU: Intel(R) Celeron(R) CPU 420 @ 1.60GHz (1596.05-MHz K8-class CPU) > > Origin = "GenuineIntel" Id = 0x10661 Family = 6 Model = 16 > > Stepping = 1 > > Features=0xafebfbff > > Features2=0xe31d > > AMD Features=0x20100800 > > AMD Features2=0x1 > > TSC: P-state invariant > > > > RAM: one DDR2-667 DIMM. > > > > OS: 8.2-RC2, amd64 > > > > _______________________________________________ > > freebsd-performance@freebsd.org mailing list > > http://lists.freebsd.org/mailman/listinfo/freebsd-performance > > To unsubscribe, send any mail to "freebsd-performance-unsubscribe@freebsd.org" > > -- > Best Wishes, > Stefan Lambrev > ICQ# 24134177 > > > > > From owner-freebsd-performance@FreeBSD.ORG Fri Jan 28 21:52:18 2011 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 2F55B106566B; Fri, 28 Jan 2011 21:52:18 +0000 (UTC) (envelope-from slw@zxy.spb.ru) Received: from zxy.spb.ru (zxy.spb.ru [195.70.199.98]) by mx1.freebsd.org (Postfix) with ESMTP id 7C8FE8FC0C; Fri, 28 Jan 2011 21:52:17 +0000 (UTC) Received: from slw by zxy.spb.ru with local (Exim 4.69 (FreeBSD)) (envelope-from ) id 1PiwEp-0009oP-FT; Sat, 29 Jan 2011 00:52:15 +0300 Date: Sat, 29 Jan 2011 00:52:15 +0300 From: Slawa Olhovchenkov To: Bruce Evans Message-ID: <20110128215215.GJ18170@zxy.spb.ru> References: <20110128143355.GD18170@zxy.spb.ru> <22E77EED-6455-4164-9115-BBD359EC8CA6@moneybookers.com> <20110128161035.GF18170@zxy.spb.ru> <4D42F87C.7020909@freebsd.org> <20110128172516.GG18170@zxy.spb.ru> <20110129070205.Q7034@besplex.bde.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20110129070205.Q7034@besplex.bde.org> User-Agent: Mutt/1.5.20 (2009-06-14) X-SA-Exim-Connect-IP: X-SA-Exim-Mail-From: slw@zxy.spb.ru X-SA-Exim-Scanned: No (on zxy.spb.ru); SAEximRunCond expanded to false Cc: freebsd-performance@freebsd.org, Julian Elischer , Stefan Lambrev Subject: Re: Interrupt performance X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 28 Jan 2011 21:52:18 -0000 On Sat, Jan 29, 2011 at 07:52:11AM +1100, Bruce Evans wrote: > >> there are of course several possible answers, including: > >> > >> 1/ Sometimes BSD and Linux report things differently. Linux may or may not > >> account for the lowest level interrupt tie the same as BSD > > > > But I see only 20% idle on FreeBSD and 80% idle on Linux. > > The time must be counted somewhere, so when it is not properly accounted > to packet handling, and nothing much else is running, it is accounted to > idle. > > To see how much CPU is actually available, run something else and see how > fast it runs. A simple counting loops works well on UP systems. === #include #include int Dummy; int main(int argc, char *argv[]) { long int count,i,dt; struct timeval st,et; count = atol(argv[1]); gettimeofday(&st, NULL); for(i=count;i;i--) Dummy++; gettimeofday(&et, NULL); dt = (et.tv_sec-st.tv_sec)*1000000 + et.tv_usec-st.tv_usec; printf("Elapsed %d us\n",dt); } === This is ok? ./loop 2000000000 FreeBSD 1 process: Elapsed 7554193 us 2 process: Elapsed 14493692 us netperf + 1 process: Elapsed 21403644 us Linux 1 process: Elapsed 7524843 us 2 process: Elapsed 14995866 us netperf + 1 process: Elapsed 14107670 us > >> 2/ the BSD driver for that chip may be badly written, or may > >> be doing more or different work for some reason > >> 3/ the FreeBSD interrupt code may be misconfigured for that driver. > >> > >> or maybe combinations... > > Possibly, but it's a low-end NIC and those normally take a lot of CPU. > 128 kpps might take 20% of 1 3GHz CPU for even a high-end NIC on FreeBSD. > Linux has generally lower overheads and should be expected to reduce this > a bit, to perhaps as low as 15%, depending on how much of the overhead is > due to the NIC. > > >> there are profiling tools that you may decide to run. > > > > What tools I can use on amd64? > > > > I boot kernel configured with 'config -p'. > > Most time in spinlock_exit and acpi_cpu_c1. > > Normal profiling works poorly (I see you found my old mail about high > resolution profiling). Linux might be misreporting the overhead for I think next server will be support PMC. Report from PMC still poorly? > exactly the same reasons that normal profiling works poorly: > - the profiling clock frequency of ~1 KHz was adequate for 5 MHz machines > in 1998, but is now too slow. Statistics clocks are even slower (128 > Hz in FreeBSD, and possibly 100 Hz (?) jiffies in Linux). > - the statistics clock might be too synchronized with other interrupts. > The above spinlock_exit and acpi_cpu_c1 times indicate that the > statistics clock almost always fires on exit from another spinlock > and/or inside ACPI, for waking up from idle for the latter. Seeing > lots of exits from spinlocks may indicated that spinlocks are being > used too much. > But FreeBSD will report interrupt times and system for non-fast-interrupts > to an accuracy of about 1 microsecond, since it doesn't use the > statistics clock much for this. OTOH, for fast interrupts it is typical > behaviour in FreeBSD and Linux to not see them at all from the statistics > clock interrupt, since they mask all interrupts so they mask the > statistics clock interrupt in particular. In FreeBSD, lots of time > apparently spent in spinlock_exit is a typical result of this, or at > least similar things, since spinlock_enter masks all interrupts (except > in my version of course). Linux doesn't have fast interrupts in the > same way that FreeBSD does, but at least in old versions almost all of > its interrupts masked other interrupts a lot. Linux kernel 2.6.26. > >>> On Jan 28, 2011, at 6:10 PM, Slawa Olhovchenkov wrote: > >>> > >>>> On Fri, Jan 28, 2011 at 06:03:15PM +0200, Stefan Lambrev wrote: > >>>> > >>>>> Do the test with netblast ;) > >>>>> Most perf tools are written badly and for Linux. > >>>>> In our internal test netblast running on freebsd outperform everything else. > >>>> I don't speak about bad performance. > >>>> I speak about overhead. > >>>> > >>>> Linux: overhead 7% for 56K int/s > >>>> FreeBSD: overhead 59% for 14K int/s > >>>> > >>>> For processing 1/4 interrupts FreeBSD need 8x CPU. > > You showed context switches in another reply. 56k interrupts on FreeBSD > would give at least 112k context switches taking several uSec each to do > nothing except switch. This would give an overhead in the 59% range. > 14K is not so bad, but still too high unless you have a spare CPU or 32 > to handle it. Part of the lowness of low-end NICs is that they tend to > generate too many interrupts and don't have much or any way to control > this. Linux will certainly be about to handle 56K int/S better than > FreeBSD since it doesn't have heavyweight interrupt threads AFAIK. > FreeBSD also has "fast" interrupts, which are much like normal interrupts > used to be in FreeBSD. I don't know if your NIC driver uses these. I re0: [FILTER] I think this is answer ([FILTER]), but I don't understand this answer :). > guess not, since if it did then it should move the "interrupt" processing > to a task queue, where it would show up under another label and be reduced > insignificantly. > > >>>>> P.S. - /usr/src/tools/tools/netrate/netblast - we have tested little more expensive card - em/igb and bce. > > netblast should be able to saturate a low-end NIC, but may take 100% of 1 > CPU to do so (it has to busy-wait, since there is no way to select() on > the NIC ring being unfull, and timeouts don't work either since their > granularity is too large). If the NIC activity alone saturates 1 CPU, > then you might see the 100% CPU being shown for Linux too. > >>>>>> re0: port 0x4000-0x40ff mem 0xf0100000-0xf01000ff irq 19 at device 4.0 on pci11 > >>>>>> re0: Chip rev. 0x18000000 > >>>>>> re0: MAC rev. 0x00000000 > >>>>>> miibus0: on re0 > >>>>>> rgephy0: PHY 1 on miibus0 > > I don't really know if this is low-end, but guess all RealTeks are :-). FreeBSD support interrupt moderation on this chip, and chip support TOE :) > >>>>>> CPU: Intel(R) Celeron(R) CPU 420 @ 1.60GHz (1596.05-MHz K8-class CPU) > > This is low end :-). > > I mostly use old AthlonXP and Athlon64 2GHz systems for network testing, > These are a bit faster than the above. A single medium end bge (5701) > on a PCI33 bus takes 100% CPU at about 512 kpps. A single low end bge > (5705+) on a PC1333 takes 120% CPU at about 240 kpps on a 2-core system. > Linux-2.6.10 saturates well below 512 kpps on the same hardware. > > Bruce From owner-freebsd-performance@FreeBSD.ORG Fri Jan 28 22:19:46 2011 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id A148E106566C for ; Fri, 28 Jan 2011 22:19:46 +0000 (UTC) (envelope-from brde@optusnet.com.au) Received: from fallbackmx07.syd.optusnet.com.au (fallbackmx07.syd.optusnet.com.au [211.29.132.9]) by mx1.freebsd.org (Postfix) with ESMTP id 288EC8FC16 for ; Fri, 28 Jan 2011 22:19:45 +0000 (UTC) Received: from mail04.syd.optusnet.com.au (mail04.syd.optusnet.com.au [211.29.132.185]) by fallbackmx07.syd.optusnet.com.au (8.13.1/8.13.1) with ESMTP id p0SKqHLB020030 for ; Sat, 29 Jan 2011 07:52:17 +1100 Received: from c122-106-165-206.carlnfd1.nsw.optusnet.com.au (c122-106-165-206.carlnfd1.nsw.optusnet.com.au [122.106.165.206]) by mail04.syd.optusnet.com.au (8.13.1/8.13.1) with ESMTP id p0SKqB63024693 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Sat, 29 Jan 2011 07:52:13 +1100 Date: Sat, 29 Jan 2011 07:52:11 +1100 (EST) From: Bruce Evans X-X-Sender: bde@besplex.bde.org To: Slawa Olhovchenkov In-Reply-To: <20110128172516.GG18170@zxy.spb.ru> Message-ID: <20110129070205.Q7034@besplex.bde.org> References: <20110128143355.GD18170@zxy.spb.ru> <22E77EED-6455-4164-9115-BBD359EC8CA6@moneybookers.com> <20110128161035.GF18170@zxy.spb.ru> <4D42F87C.7020909@freebsd.org> <20110128172516.GG18170@zxy.spb.ru> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: freebsd-performance@freebsd.org, Julian Elischer , Stefan Lambrev Subject: Re: Interrupt performance X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 28 Jan 2011 22:19:46 -0000 On Fri, 28 Jan 2011, Slawa Olhovchenkov wrote: > On Fri, Jan 28, 2011 at 09:10:20AM -0800, Julian Elischer wrote: > >> On 1/28/11 8:15 AM, Stefan Lambrev wrote: >>> The overhead comes from badly written software. >>> This software is optimized for linux and you have to optimize it for freebsd, then you will have the same overhead. >>> All those *popular* benchmarks like hping, iperf, netperf have some strange optimizations for linux - we call them linuxism. >>> Just search the archives - I'm pretty sure patches are flying around. >> >> He wants to know why the freeBSD driver spends 8 x as much time on >> each interrupt. 32 x (1/4 as many interrupts reported to be taking 8 x longer altogether). This must be because the reporting is more broken in Linux :-). > Yes! > >> there are of course several possible answers, including: >> >> 1/ Sometimes BSD and Linux report things differently. Linux may or may not >> account for the lowest level interrupt tie the same as BSD > > But I see only 20% idle on FreeBSD and 80% idle on Linux. The time must be counted somewhere, so when it is not properly accounted to packet handling, and nothing much else is running, it is accounted to idle. To see how much CPU is actually available, run something else and see how fast it runs. A simple counting loops works well on UP systems. >> 2/ the BSD driver for that chip may be badly written, or may >> be doing more or different work for some reason >> 3/ the FreeBSD interrupt code may be misconfigured for that driver. >> >> or maybe combinations... Possibly, but it's a low-end NIC and those normally take a lot of CPU. 128 kpps might take 20% of 1 3GHz CPU for even a high-end NIC on FreeBSD. Linux has generally lower overheads and should be expected to reduce this a bit, to perhaps as low as 15%, depending on how much of the overhead is due to the NIC. >> there are profiling tools that you may decide to run. > > What tools I can use on amd64? > > I boot kernel configured with 'config -p'. > Most time in spinlock_exit and acpi_cpu_c1. Normal profiling works poorly (I see you found my old mail about high resolution profiling). Linux might be misreporting the overhead for exactly the same reasons that normal profiling works poorly: - the profiling clock frequency of ~1 KHz was adequate for 5 MHz machines in 1998, but is now too slow. Statistics clocks are even slower (128 Hz in FreeBSD, and possibly 100 Hz (?) jiffies in Linux). - the statistics clock might be too synchronized with other interrupts. The above spinlock_exit and acpi_cpu_c1 times indicate that the statistics clock almost always fires on exit from another spinlock and/or inside ACPI, for waking up from idle for the latter. Seeing lots of exits from spinlocks may indicated that spinlocks are being used too much. But FreeBSD will report interrupt times and system for non-fast-interrupts to an accuracy of about 1 microsecond, since it doesn't use the statistics clock much for this. OTOH, for fast interrupts it is typical behaviour in FreeBSD and Linux to not see them at all from the statistics clock interrupt, since they mask all interrupts so they mask the statistics clock interrupt in particular. In FreeBSD, lots of time apparently spent in spinlock_exit is a typical result of this, or at least similar things, since spinlock_enter masks all interrupts (except in my version of course). Linux doesn't have fast interrupts in the same way that FreeBSD does, but at least in old versions almost all of its interrupts masked other interrupts a lot. >>> On Jan 28, 2011, at 6:10 PM, Slawa Olhovchenkov wrote: >>> >>>> On Fri, Jan 28, 2011 at 06:03:15PM +0200, Stefan Lambrev wrote: >>>> >>>>> Do the test with netblast ;) >>>>> Most perf tools are written badly and for Linux. >>>>> In our internal test netblast running on freebsd outperform everything else. >>>> I don't speak about bad performance. >>>> I speak about overhead. >>>> >>>> Linux: overhead 7% for 56K int/s >>>> FreeBSD: overhead 59% for 14K int/s >>>> >>>> For processing 1/4 interrupts FreeBSD need 8x CPU. You showed context switches in another reply. 56k interrupts on FreeBSD would give at least 112k context switches taking several uSec each to do nothing except switch. This would give an overhead in the 59% range. 14K is not so bad, but still too high unless you have a spare CPU or 32 to handle it. Part of the lowness of low-end NICs is that they tend to generate too many interrupts and don't have much or any way to control this. Linux will certainly be about to handle 56K int/S better than FreeBSD since it doesn't have heavyweight interrupt threads AFAIK. FreeBSD also has "fast" interrupts, which are much like normal interrupts used to be in FreeBSD. I don't know if your NIC driver uses these. I guess not, since if it did then it should move the "interrupt" processing to a task queue, where it would show up under another label and be reduced insignificantly. >>>>> P.S. - /usr/src/tools/tools/netrate/netblast - we have tested little more expensive card - em/igb and bce. netblast should be able to saturate a low-end NIC, but may take 100% of 1 CPU to do so (it has to busy-wait, since there is no way to select() on the NIC ring being unfull, and timeouts don't work either since their granularity is too large). If the NIC activity alone saturates 1 CPU, then you might see the 100% CPU being shown for Linux too. >>>>>> re0: port 0x4000-0x40ff mem 0xf0100000-0xf01000ff irq 19 at device 4.0 on pci11 >>>>>> re0: Chip rev. 0x18000000 >>>>>> re0: MAC rev. 0x00000000 >>>>>> miibus0: on re0 >>>>>> rgephy0: PHY 1 on miibus0 I don't really know if this is low-end, but guess all RealTeks are :-). >>>>>> CPU: Intel(R) Celeron(R) CPU 420 @ 1.60GHz (1596.05-MHz K8-class CPU) This is low end :-). I mostly use old AthlonXP and Athlon64 2GHz systems for network testing, These are a bit faster than the above. A single medium end bge (5701) on a PCI33 bus takes 100% CPU at about 512 kpps. A single low end bge (5705+) on a PC1333 takes 120% CPU at about 240 kpps on a 2-core system. Linux-2.6.10 saturates well below 512 kpps on the same hardware. Bruce From owner-freebsd-performance@FreeBSD.ORG Sat Jan 29 03:03:54 2011 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id A5FD7106564A for ; Sat, 29 Jan 2011 03:03:54 +0000 (UTC) (envelope-from pierre@userid.org) Received: from mail.storm.ca (unknown [IPv6:2607:f0b0:0:6:209:87:239:66]) by mx1.freebsd.org (Postfix) with ESMTP id 4EFE28FC0C for ; Sat, 29 Jan 2011 03:03:54 +0000 (UTC) Received: from mail.userid.org (pandora.userid.org [216.106.102.33]) by mail.storm.ca (8.14.2+Sun/8.14.2) with ESMTP id p0T0d099005343 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Fri, 28 Jan 2011 19:39:06 -0500 (EST) Received: from [IPv6:2607:f0b0:1:3800:68b7:d0f5:5955:9e2] (unknown [IPv6:2607:f0b0:1:3800:68b7:d0f5:5955:9e2]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) (Authenticated sender: pierre) by mail.userid.org (Postfix) with ESMTP id 751DA2C7521; Fri, 28 Jan 2011 19:38:53 -0500 (EST) Message-ID: <4D4361AF.9090602@userid.org> Date: Fri, 28 Jan 2011 19:39:11 -0500 From: Pierre Lamy User-Agent: Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.2.13) Gecko/20101207 Thunderbird/3.1.7 MIME-Version: 1.0 To: Slawa Olhovchenkov , freebsd-performance@freebsd.org References: <20110128143355.GD18170@zxy.spb.ru> <22E77EED-6455-4164-9115-BBD359EC8CA6@moneybookers.com> <20110128184459.GI18170@zxy.spb.ru> In-Reply-To: <20110128184459.GI18170@zxy.spb.ru> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-userid-MailScanner-Information: Please contact the ISP for more information X-userid-MailScanner-ID: 751DA2C7521.AC159 X-userid-MailScanner: Found to be clean X-userid-MailScanner-SpamCheck: not spam, SpamAssassin (not cached, score=0.599, required 6, J_CHICKENPOX_32 0.60, NO_RELAYS -0.00) X-userid-MailScanner-From: pierre@userid.org X-Spam-Status: No Cc: Subject: Re: Interrupt performance X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 29 Jan 2011 03:03:54 -0000 On a Lanner 7535 atom d510 system DUT, using a single gig port, running -CURRENT from Jan 6. input (Total) output packets errs idrops bytes packets errs bytes colls 410015 0 0 43461452 204 0 15592 0 410341 0 0 43496128 203 0 14546 0 408855 0 0 43338492 204 0 15400 0 408812 0 0 43333980 201 0 14278 0 408802 0 0 43332874 203 0 15170 0 408827 0 0 43335570 201 0 14278 0 procs memory page disk faults cpu r b w avm fre flt re pi po fr sr ad5 in sy cs us sy id 0 0 0 383M 3669M 64 0 0 0 4 0 42 2074 260 4247 0 25 75 0 0 0 383M 3669M 0 0 0 0 0 0 2 1820 168 3622 0 25 75 0 0 0 383M 3669M 65 0 0 0 0 0 2 2112 279 4240 0 23 77 Sending box: [/var/preserve/root] # ./netblast 5.5.5.243 80 64 60 start: 1296261208.502195530 finish: 1296261268.503127492 send calls: 33880997 send errors: 9389311 approx send rate: 408194 approx error rate: 0 On 1/28/2011 1:44 PM, Slawa Olhovchenkov wrote: > On Fri, Jan 28, 2011 at 06:03:15PM +0200, Stefan Lambrev wrote: > >> Do the test with netblast ;) >> Most perf tools are written badly and for Linux. >> In our internal test netblast running on freebsd outperform everything else. >> >> P.S. - /usr/src/tools/tools/netrate/netblast - we have tested little more expensive card - em/igb and bce. > Now I install 8.2-RC2/i386 and use netblast: > > # /usr/obj/usr/src/tools/tools/netrate/netblast/netblast 10.200.0.1 1122 1400 10 > > start: 1296239273.106911353 > finish: 1296239283.107222845 > send calls: 2703219 > send errors: 2090049 > approx send rate: 61317 > approx error rate: 0 > > CPU load (No difference with netperf): > > iostat: > tty ad0 cpu > tin tout KB/t tps MB/s us ni sy in id > 1 48 0.00 0 0.00 4 0 47 20 29 > 0 44 0.00 0 0.00 2 0 68 30 0 > 0 44 0.00 0 0.00 12 0 65 23 0 > 0 44 0.00 0 0.00 6 0 73 21 0 > 0 44 0.00 0 0.00 8 0 54 38 0 > 0 44 0.00 0 0.00 5 0 66 29 0 > 0 44 0.00 0 0.00 7 0 65 29 0 > 0 44 0.00 0 0.00 7 0 64 29 0 > 0 44 0.00 0 0.00 8 0 64 28 0 > 0 44 0.00 0 0.00 7 0 57 35 0 > 0 235 0.00 0 0.00 4 0 13 12 71 > > vmstat: > procs memory page disk faults cpu > r b w avm fre flt re pi po fr sr ad0 in sy cs us sy id > 1 0 0 97748K 431M 0 0 0 0 0 0 0 15331 269981 30697 6 94 0 > 1 0 0 97748K 431M 0 0 0 0 0 0 0 15333 269072 30716 5 95 0 > 1 0 0 97748K 431M 0 0 0 0 0 0 0 15332 269512 30690 9 91 0 > 1 0 0 97748K 431M 0 0 0 0 0 0 0 15331 269633 30698 8 92 0 > 1 0 0 97748K 431M 0 0 0 0 0 0 0 15334 269613 30693 7 93 0 > 1 0 0 97748K 431M 0 0 0 0 0 0 0 15332 269438 30713 13 87 0 > 1 0 0 97748K 431M 0 0 0 0 0 0 0 15337 269460 30705 7 93 0 > 1 0 0 97748K 431M 0 0 0 0 0 0 0 15299 266321 30617 6 94 0 > 1 0 0 97748K 431M 0 0 0 0 0 0 0 15300 266147 30625 11 89 0 > > > (Linux, for compare, only 1279 cs) > > systat -vmstat 1 > 3 users Load 0.29 0.09 0.07 Jan 28 21:31 > > Mem:KB REAL VIRTUAL VN PAGER SWAP PAGER > Tot Share Tot Share Free in out in out > Act 16068 4496 99772 5208 440312 count > All 37664 6644 2227916 10384 pages > Proc: Interrupts > r p d s w Csw Trp Sys Int Sof Flt cow 17372 total > 1 23 30k 12 270k 15k 30k zfod atkbd0 1 > ozfod ata0 irq14 > 63.2%Sys 29.3%Intr 7.5%User 0.0%Nice 0.0%Idle %ozfod ata1 irq15 > | | | | | | | | | | | daefr 15366 re0 irq19 > ================================++++++++++++++>>>> prcfr 2006 cpu0: time > 1 dtbuf totfr > Namei Name-cache Dir-cache 34738 desvn react > Calls hits % hits % 595 numvn pdwak > 34 frevn pdpgs > intrn > Disks ad0 26572 wire > KB/t 0.00 12800 act > tps 0 12732 inact > MB/s 0.00 104 cache > %busy 0 440208 free > 11552 buf > > >> On Jan 28, 2011, at 4:33 PM, Slawa Olhovchenkov wrote: >> >>> I test network performance and found some strange result -- on the >>> same hardware Linux more then 10x used CPU resources for interrupt >>> processing. >>> >>> FreeBSD system utilise 70% CPU (32% idle, 59% interrupt, 9% sys) and >>> network card generate 14K-18K interrupt per second. >>> >>> Linux system utilise 20% CPU (80% idle, 13% system, 3% hiq, 4% siq) >>> and network card generate 56K interrupt per second. >>> >>> I used 'netperf -H host -t UDP_STREAM -l 60 -C -c -- -m 8972 -s >>> 128K -S 128K' for generate network traffic. >>> >>> NIC: >>> >>> re0: port 0x4000-0x40ff mem 0xf0100000-0xf01000ff irq 19 at device 4.0 on pci11 >>> re0: Chip rev. 0x18000000 >>> re0: MAC rev. 0x00000000 >>> miibus0: on re0 >>> rgephy0: PHY 1 on miibus0 >>> >>> >>> CPU: >>> >>> CPU: Intel(R) Celeron(R) CPU 420 @ 1.60GHz (1596.05-MHz K8-class CPU) >>> Origin = "GenuineIntel" Id = 0x10661 Family = 6 Model = 16 >>> Stepping = 1 >>> Features=0xafebfbff >>> Features2=0xe31d >>> AMD Features=0x20100800 >>> AMD Features2=0x1 >>> TSC: P-state invariant >>> >>> RAM: one DDR2-667 DIMM. >>> >>> OS: 8.2-RC2, amd64 >>> >>> _______________________________________________ >>> freebsd-performance@freebsd.org mailing list >>> http://lists.freebsd.org/mailman/listinfo/freebsd-performance >>> To unsubscribe, send any mail to "freebsd-performance-unsubscribe@freebsd.org" >> -- >> Best Wishes, >> Stefan Lambrev >> ICQ# 24134177 >> >> >> >> >> > _______________________________________________ > freebsd-performance@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-performance > To unsubscribe, send any mail to "freebsd-performance-unsubscribe@freebsd.org" > From owner-freebsd-performance@FreeBSD.ORG Sat Jan 29 03:43:15 2011 Return-Path: Delivered-To: freebsd-performance@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 011861065672; Sat, 29 Jan 2011 03:43:15 +0000 (UTC) (envelope-from brde@optusnet.com.au) Received: from mail03.syd.optusnet.com.au (mail03.syd.optusnet.com.au [211.29.132.184]) by mx1.freebsd.org (Postfix) with ESMTP id 7750F8FC13; Sat, 29 Jan 2011 03:43:14 +0000 (UTC) Received: from c122-106-165-206.carlnfd1.nsw.optusnet.com.au (c122-106-165-206.carlnfd1.nsw.optusnet.com.au [122.106.165.206]) by mail03.syd.optusnet.com.au (8.13.1/8.13.1) with ESMTP id p0T3h7gV016272 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Sat, 29 Jan 2011 14:43:10 +1100 Date: Sat, 29 Jan 2011 14:43:07 +1100 (EST) From: Bruce Evans X-X-Sender: bde@besplex.bde.org To: Slawa Olhovchenkov In-Reply-To: <20110128215215.GJ18170@zxy.spb.ru> Message-ID: <20110129133859.O967@besplex.bde.org> References: <20110128143355.GD18170@zxy.spb.ru> <22E77EED-6455-4164-9115-BBD359EC8CA6@moneybookers.com> <20110128161035.GF18170@zxy.spb.ru> <4D42F87C.7020909@freebsd.org> <20110128172516.GG18170@zxy.spb.ru> <20110129070205.Q7034@besplex.bde.org> <20110128215215.GJ18170@zxy.spb.ru> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: freebsd-performance@FreeBSD.org, Julian Elischer , Bruce Evans , Stefan Lambrev Subject: Re: Interrupt performance X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 29 Jan 2011 03:43:15 -0000 On Sat, 29 Jan 2011, Slawa Olhovchenkov wrote: > On Sat, Jan 29, 2011 at 07:52:11AM +1100, Bruce Evans wrote: >> >> To see how much CPU is actually available, run something else and see how >> fast it runs. A simple counting loops works well on UP systems. > > === > #include > #include > > int Dummy; > > int > main(int argc, char *argv[]) > { > long int count,i,dt; > struct timeval st,et; > > count = atol(argv[1]); > > gettimeofday(&st, NULL); > for(i=count;i;i--) Dummy++; > gettimeofday(&et, NULL); > dt = (et.tv_sec-st.tv_sec)*1000000 + et.tv_usec-st.tv_usec; > printf("Elapsed %d us\n",dt); > } > === > > This is ok? It's better not to compete with the interrupt handler in the kernel by spinning making syscalls, but that will do for a start. > ./loop 2000000000 > > FreeBSD > 1 process: Elapsed 7554193 us > 2 process: Elapsed 14493692 us > netperf + 1 process: Elapsed 21403644 us This shows about 35% user 65% network. > Linux > 1 process: Elapsed 7524843 us > 2 process: Elapsed 14995866 us > netperf + 1 process: Elapsed 14107670 us This shows about 53% user 47% network. So FreeBSD has about 18% more network overhead (absolute: 65-47), or about 38% more network overhead (relative: (65-47)/47). Not too surprising -- the context switches alone might cost that. BTW, even -current vs my version of FreeBSD-5.2 has 10-20% more network overhead (relative) for tx, apparently due to bloat in the network stack. This apparently has nothing to do with hardware. The slowdown is much the same with bge (heavily modified in my version) and em (barely modified). One thing that I modify in both drivers is increase the tx ifq length by a massive amount (from about 512 to about 20000). This must be bad for overhead because such a large queue cannot fit in the L2 cache. A large amount and perhaps more than half of NIC overhead consists of waiting for cache misses. The slowdown in -current might be caused by minor bloat crossing a threshold and thus causing just one more cache miss every packet or 2. >> Normal profiling works poorly (I see you found my old mail about high >> resolution profiling). Linux might be misreporting the overhead for > > I think next server will be support PMC. > Report from PMC still poorly? I should be adequate, but I prefer my version of perfmon which can count cache misses precisely for every function. But without patches, perfmon is even more broken than high resolution kernel profiling. >> ... >> generate too many interrupts and don't have much or any way to control >> this. Linux will certainly be about to handle 56K int/S better than >> FreeBSD since it doesn't have heavyweight interrupt threads AFAIK. >> FreeBSD also has "fast" interrupts, which are much like normal interrupts >> used to be in FreeBSD. I don't know if your NIC driver uses these. I > > re0: [FILTER] > > I think this is answer ([FILTER]), but I don't understand this answer :). [FILTER] means "fast". re used to unconditionally use "fast" interrupts and a task queue, which IMO is a bad way to program an interrupt handler, but yongari@ recently overhauled re (again :-) so that it now doesn't use fast interrupts by default for the MSI/MSIX case . (BTW, it still bogusly uses INTR_MPSAFE for the fast interrupt bus_setup_intr().) The overhaul probably also reduces interrupt overhead if it works on your hardware, just be reducing the interrupt frequency. I don't understand what moderates the interrupt frequency in the MSI case. >> I don't really know if this is low-end, but guess all RealTeks are :-). > > FreeBSD support interrupt moderation on this chip, and chip support > TOE :) The support was poor according to yongari@'s long messages about improving it. With working interrupt moderation, you just don't get an interrupt rate of even 14KHz, except transiently. 4KHz would be all I would be happy with on a 1-core 1.6GHz CPU. Since re is so primitive, yongari@ only managed to limit the rate to 20KHz. (Is this only for the non-MSI case, with the MSI case better even before?) Linux with its reduced interrupt latency can handled your observed 56KHz without losing by so much. My 2-core Athlon (Turion) with its low-end bge 5705, whose brokenness consists mainly of: completely broken interrupt moderation can't handle full 1Gbps -- saturates at about 300Mbps CPU and/or DMA resources used for 300Mbps are about the same as for a bge 5701 at a full 1Mbps saturates at about 100 KHz bge interrupts. This many interrupts takes about all of 1 CPU to handle the hardware part and 20% of another to generate packets. The system saturates in much the same way under WinXP. I only recently got a version of Linux to boot on this system and haven't tried network performance tests under Linux on it. Bruce From owner-freebsd-performance@FreeBSD.ORG Sat Jan 29 10:24:23 2011 Return-Path: Delivered-To: freebsd-performance@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 2D19B106566B; Sat, 29 Jan 2011 10:24:23 +0000 (UTC) (envelope-from slw@zxy.spb.ru) Received: from zxy.spb.ru (zxy.spb.ru [195.70.199.98]) by mx1.freebsd.org (Postfix) with ESMTP id 77E5E8FC13; Sat, 29 Jan 2011 10:24:22 +0000 (UTC) Received: from slw by zxy.spb.ru with local (Exim 4.69 (FreeBSD)) (envelope-from ) id 1Pj7ye-000FSq-3w; Sat, 29 Jan 2011 13:24:20 +0300 Date: Sat, 29 Jan 2011 13:24:20 +0300 From: Slawa Olhovchenkov To: Bruce Evans Message-ID: <20110129102420.GK18170@zxy.spb.ru> References: <20110128143355.GD18170@zxy.spb.ru> <22E77EED-6455-4164-9115-BBD359EC8CA6@moneybookers.com> <20110128161035.GF18170@zxy.spb.ru> <4D42F87C.7020909@freebsd.org> <20110128172516.GG18170@zxy.spb.ru> <20110129070205.Q7034@besplex.bde.org> <20110128215215.GJ18170@zxy.spb.ru> <20110129133859.O967@besplex.bde.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20110129133859.O967@besplex.bde.org> User-Agent: Mutt/1.5.20 (2009-06-14) X-SA-Exim-Connect-IP: X-SA-Exim-Mail-From: slw@zxy.spb.ru X-SA-Exim-Scanned: No (on zxy.spb.ru); SAEximRunCond expanded to false Cc: freebsd-performance@FreeBSD.org, Julian Elischer , Stefan Lambrev Subject: Re: Interrupt performance X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 29 Jan 2011 10:24:23 -0000 On Sat, Jan 29, 2011 at 02:43:07PM +1100, Bruce Evans wrote: > On Sat, 29 Jan 2011, Slawa Olhovchenkov wrote: > > > On Sat, Jan 29, 2011 at 07:52:11AM +1100, Bruce Evans wrote: > >> > >> To see how much CPU is actually available, run something else and see how > >> fast it runs. A simple counting loops works well on UP systems. > > > > === > > #include > > #include > > > > int Dummy; > > > > int > > main(int argc, char *argv[]) > > { > > long int count,i,dt; > > struct timeval st,et; > > > > count = atol(argv[1]); > > > > gettimeofday(&st, NULL); > > for(i=count;i;i--) Dummy++; > > gettimeofday(&et, NULL); > > dt = (et.tv_sec-st.tv_sec)*1000000 + et.tv_usec-st.tv_usec; > > printf("Elapsed %d us\n",dt); > > } > > === > > > > This is ok? > > It's better not to compete with the interrupt handler in the kernel by > spinning making syscalls, but that will do for a start. In this programm inner loop don't contain any syscall. Better varian -- loop with syscalls? > > ./loop 2000000000 > > > > FreeBSD > > 1 process: Elapsed 7554193 us > > 2 process: Elapsed 14493692 us > > netperf + 1 process: Elapsed 21403644 us > > This shows about 35% user 65% network. > > > Linux > > 1 process: Elapsed 7524843 us > > 2 process: Elapsed 14995866 us > > netperf + 1 process: Elapsed 14107670 us > > This shows about 53% user 47% network. > > So FreeBSD has about 18% more network overhead (absolute: 65-47), or > about 38% more network overhead (relative: (65-47)/47). Not too > surprising -- the context switches alone might cost that. For only 14K vs 56K interrupt. 152% more network overhead per one interrupt. And I see drammaticaly less number of context switches in linux stats (by dstat). > BTW, even -current vs my version of FreeBSD-5.2 has 10-20% more network > overhead (relative) for tx, apparently due to bloat in the network > stack. This apparently has nothing to do with hardware. The slowdown > is much the same with bge (heavily modified in my version) and em > (barely modified). One thing that I modify in both drivers is increase > the tx ifq length by a massive amount (from about 512 to about 20000). > This must be bad for overhead because such a large queue cannot fit > in the L2 cache. A large amount and perhaps more than half of NIC > overhead consists of waiting for cache misses. The slowdown in -current > might be caused by minor bloat crossing a threshold and thus causing > just one more cache miss every packet or 2. I read disccursion about support DCA in FreeBSD, sound nice. > >> Normal profiling works poorly (I see you found my old mail about high > >> resolution profiling). Linux might be misreporting the overhead for > > > > I think next server will be support PMC. > > Report from PMC still poorly? > > I should be adequate, but I prefer my version of perfmon which can > count cache misses precisely for every function. But without patches, > perfmon is even more broken than high resolution kernel profiling. Can I use your version of perfmon? How? I don't have expirence with any kernel profiling. > >> ... > >> generate too many interrupts and don't have much or any way to control > >> this. Linux will certainly be about to handle 56K int/S better than > >> FreeBSD since it doesn't have heavyweight interrupt threads AFAIK. > >> FreeBSD also has "fast" interrupts, which are much like normal interrupts > >> used to be in FreeBSD. I don't know if your NIC driver uses these. I > > > > re0: [FILTER] > > > > I think this is answer ([FILTER]), but I don't understand this answer :). > > [FILTER] means "fast". re used to unconditionally use "fast" interrupts > and a task queue, which IMO is a bad way to program an interrupt > handler, but yongari@ recently overhauled re (again :-) so that it now > doesn't use fast interrupts by default for the MSI/MSIX case . (BTW, Ineresting, but I don't think this help for this case -- old PCI chip, old CPU, old RAM, old matherboard -- all old. I don't try to get wirespeed gigabit performance from this old box, I try to compare relative performance FreeBSD vs Linux (in last time I got many feedback about poor network performance FreeBSD vs Linux). > it still bogusly uses INTR_MPSAFE for the fast interrupt bus_setup_intr().) > The overhaul probably also reduces interrupt overhead if it works on > your hardware, just be reducing the interrupt frequency. I don't > understand what moderates the interrupt frequency in the MSI case. > > >> I don't really know if this is low-end, but guess all RealTeks are :-). > > > > FreeBSD support interrupt moderation on this chip, and chip support > > TOE :) > > The support was poor according to yongari@'s long messages about > improving it. With working interrupt moderation, you just don't get > an interrupt rate of even 14KHz, except transiently. 4KHz would be > all I would be happy with on a 1-core 1.6GHz CPU. Since re is so > primitive, yongari@ only managed to limit the rate to 20KHz. (Is this > only for the non-MSI case, with the MSI case better even before?) > Linux with its reduced interrupt latency can handled your observed > 56KHz without losing by so much. My 2-core Athlon (Turion) with its > low-end bge 5705, whose brokenness consists mainly of: > completely broken interrupt moderation > can't handle full 1Gbps -- saturates at about 300Mbps > CPU and/or DMA resources used for 300Mbps are about the same as for > a bge 5701 at a full 1Mbps > saturates at about 100 KHz bge interrupts. This many interrupts takes > about all of 1 CPU to handle the hardware part and 20% of another to > generate packets. The system saturates in much the same way under > WinXP. I only recently got a version of Linux to boot on this system > and haven't tried network performance tests under Linux on it. > > Bruce > _______________________________________________ > freebsd-performance@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-performance > To unsubscribe, send any mail to "freebsd-performance-unsubscribe@freebsd.org" From owner-freebsd-performance@FreeBSD.ORG Sat Jan 29 12:54:17 2011 Return-Path: Delivered-To: freebsd-performance@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 38200106566B; Sat, 29 Jan 2011 12:54:17 +0000 (UTC) (envelope-from brde@optusnet.com.au) Received: from mail09.syd.optusnet.com.au (mail09.syd.optusnet.com.au [211.29.132.190]) by mx1.freebsd.org (Postfix) with ESMTP id C5BE28FC13; Sat, 29 Jan 2011 12:54:16 +0000 (UTC) Received: from c122-106-165-206.carlnfd1.nsw.optusnet.com.au (c122-106-165-206.carlnfd1.nsw.optusnet.com.au [122.106.165.206]) by mail09.syd.optusnet.com.au (8.13.1/8.13.1) with ESMTP id p0TCsBP5014051 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Sat, 29 Jan 2011 23:54:13 +1100 Date: Sat, 29 Jan 2011 23:54:11 +1100 (EST) From: Bruce Evans X-X-Sender: bde@besplex.bde.org To: Slawa Olhovchenkov In-Reply-To: <20110129102420.GK18170@zxy.spb.ru> Message-ID: <20110129233542.O20731@besplex.bde.org> References: <20110128143355.GD18170@zxy.spb.ru> <22E77EED-6455-4164-9115-BBD359EC8CA6@moneybookers.com> <20110128161035.GF18170@zxy.spb.ru> <4D42F87C.7020909@freebsd.org> <20110128172516.GG18170@zxy.spb.ru> <20110129070205.Q7034@besplex.bde.org> <20110128215215.GJ18170@zxy.spb.ru> <20110129133859.O967@besplex.bde.org> <20110129102420.GK18170@zxy.spb.ru> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: freebsd-performance@FreeBSD.org, Julian Elischer , Bruce Evans , Stefan Lambrev Subject: Re: Interrupt performance X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 29 Jan 2011 12:54:17 -0000 On Sat, 29 Jan 2011, Slawa Olhovchenkov wrote: > On Sat, Jan 29, 2011 at 02:43:07PM +1100, Bruce Evans wrote: > >> On Sat, 29 Jan 2011, Slawa Olhovchenkov wrote: >> >>> On Sat, Jan 29, 2011 at 07:52:11AM +1100, Bruce Evans wrote: >>>> >>>> To see how much CPU is actually available, run something else and see how >>>> fast it runs. A simple counting loops works well on UP systems. >>> >>> === >>> #include >>> #include >>> >>> int Dummy; >>> >>> int >>> main(int argc, char *argv[]) >>> { >>> long int count,i,dt; >>> struct timeval st,et; >>> >>> count = atol(argv[1]); >>> >>> gettimeofday(&st, NULL); >>> for(i=count;i;i--) Dummy++; >>> gettimeofday(&et, NULL); >>> dt = (et.tv_sec-st.tv_sec)*1000000 + et.tv_usec-st.tv_usec; >>> printf("Elapsed %d us\n",dt); >>> } >>> === >>> >>> This is ok? >> >> It's better not to compete with the interrupt handler in the kernel by >> spinning making syscalls, but that will do for a start. > > In this programm inner loop don't contain any syscall. > Better varian -- loop with syscalls? Oops. It is like I meant already. You could try it with syscalls and/or heavy memory accesses to see if there is a problem with memory resource contention (maybe more cache misses). >>> ./loop 2000000000 >>> >>> FreeBSD >>> 1 process: Elapsed 7554193 us >>> 2 process: Elapsed 14493692 us >>> netperf + 1 process: Elapsed 21403644 us >> >> This shows about 35% user 65% network. >> >>> Linux >>> 1 process: Elapsed 7524843 us >>> 2 process: Elapsed 14995866 us >>> netperf + 1 process: Elapsed 14107670 us >> >> This shows about 53% user 47% network. >> >> So FreeBSD has about 18% more network overhead (absolute: 65-47), or >> about 38% more network overhead (relative: (65-47)/47). Not too >> surprising -- the context switches alone might cost that. > > For only 14K vs 56K interrupt. 152% more network overhead per one interrupt. No, FreeBSD does 4 times as much work per interrupt. 4 times as much (300%) "overhead" per interrupt is to be expected, since most (hopefully more than half :-) of the "overhead" is actual work. > And I see drammaticaly less number of context switches in linux stats > (by dstat). FreeBSD uses ithreds for most interrupts, so of course it does many more context switches (at least 2 per interrupt). This doesn't make much difference provided there are not too many. I think the version of re that you are using actually uses "fast" interrupts and a task queue. This also seems to be making little difference. You get a relatively lightweight "fast" interrupt following by followed by a context switch to and from the task. IIRC, your statistics showed about twice as many context switches as interrupts, so the task queue isn't doing much to reduce the "interrupt overhead" -- it just gives context switches to the task instead of to an ithread. >>> I think next server will be support PMC. >>> Report from PMC still poorly? >> >> I should be adequate, but I prefer my version of perfmon which can >> count cache misses precisely for every function. But without patches, >> perfmon is even more broken than high resolution kernel profiling. > > Can I use your version of perfmon? How? I don't have expirence with > any kernel profiling. Not the place to start. >> [FILTER] means "fast". re used to unconditionally use "fast" interrupts >> and a task queue, which IMO is a bad way to program an interrupt >> handler, but yongari@ recently overhauled re (again :-) so that it now >> doesn't use fast interrupts by default for the MSI/MSIX case . (BTW, > > Ineresting, but I don't think this help for this case -- old PCI > chip, old CPU, old RAM, old matherboard -- all old. I don't try to get > wirespeed gigabit performance from this old box, I try to compare > relative performance FreeBSD vs Linux (in last time I got many > feedback about poor network performance FreeBSD vs Linux). Old hardware will certainly amplify any overheads. 50% overhead becomes 100% if the system is 2 times slower... Bruce From owner-freebsd-performance@FreeBSD.ORG Sat Jan 29 14:17:42 2011 Return-Path: Delivered-To: freebsd-performance@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id AA21C106564A; Sat, 29 Jan 2011 14:17:42 +0000 (UTC) (envelope-from slw@zxy.spb.ru) Received: from zxy.spb.ru (zxy.spb.ru [195.70.199.98]) by mx1.freebsd.org (Postfix) with ESMTP id 640F78FC0C; Sat, 29 Jan 2011 14:17:41 +0000 (UTC) Received: from slw by zxy.spb.ru with local (Exim 4.69 (FreeBSD)) (envelope-from ) id 1PjBcR-000I1w-4y; Sat, 29 Jan 2011 17:17:39 +0300 Date: Sat, 29 Jan 2011 17:17:39 +0300 From: Slawa Olhovchenkov To: Bruce Evans Message-ID: <20110129141739.GL18170@zxy.spb.ru> References: <22E77EED-6455-4164-9115-BBD359EC8CA6@moneybookers.com> <20110128161035.GF18170@zxy.spb.ru> <4D42F87C.7020909@freebsd.org> <20110128172516.GG18170@zxy.spb.ru> <20110129070205.Q7034@besplex.bde.org> <20110128215215.GJ18170@zxy.spb.ru> <20110129133859.O967@besplex.bde.org> <20110129102420.GK18170@zxy.spb.ru> <20110129233542.O20731@besplex.bde.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20110129233542.O20731@besplex.bde.org> User-Agent: Mutt/1.5.20 (2009-06-14) X-SA-Exim-Connect-IP: X-SA-Exim-Mail-From: slw@zxy.spb.ru X-SA-Exim-Scanned: No (on zxy.spb.ru); SAEximRunCond expanded to false Cc: freebsd-performance@FreeBSD.org, Julian Elischer , Stefan Lambrev Subject: Re: Interrupt performance X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 29 Jan 2011 14:17:42 -0000 On Sat, Jan 29, 2011 at 11:54:11PM +1100, Bruce Evans wrote: > >> So FreeBSD has about 18% more network overhead (absolute: 65-47), or > >> about 38% more network overhead (relative: (65-47)/47). Not too > >> surprising -- the context switches alone might cost that. > > > > For only 14K vs 56K interrupt. 152% more network overhead per one interrupt. > > No, FreeBSD does 4 times as much work per interrupt. 4 times as much > (300%) "overhead" per interrupt is to be expected, since most (hopefully > more than half :-) of the "overhead" is actual work. > > > And I see drammaticaly less number of context switches in linux stats > > (by dstat). > > FreeBSD uses ithreds for most interrupts, so of course it does many > more context switches (at least 2 per interrupt). This doesn't make > much difference provided there are not too many. I think the version > of re that you are using actually uses "fast" interrupts and a task > queue. This also seems to be making little difference. You get a > relatively lightweight "fast" interrupt following by followed by a > context switch to and from the task. IIRC, your statistics showed > about twice as many context switches as interrupts, so the task queue > isn't doing much to reduce the "interrupt overhead" -- it just gives > context switches to the task instead of to an ithread. It seems as root of difference with linux systems. I found this link http://www.freebsd.org/doc/en/books/arch-handbook/smp-design.html. No light weight context switches in the near future?