Date: Thu, 25 Dec 2003 10:25:49 -0800 From: Luigi Rizzo <rizzo@icir.org> To: Michael Sierchio <kudzu@tenebras.com> Cc: soekris-tech@lists.soekris.com Subject: Re: bridge with access on both interfaces Message-ID: <20031225102549.A79161@xorpc.icir.org> In-Reply-To: <3FEA27A0.7030902@tenebras.com>; from kudzu@tenebras.com on Wed, Dec 24, 2003 at 03:56:16PM -0800 References: <Pine.NEB.3.96L.1031224184144.66152G-100000@fledge.watson.org> <3FEA27A0.7030902@tenebras.com>
next in thread | previous in thread | raw e-mail | index | archive | help
[slightly rearranging the Cc list...]
i have read Soren's report, and I think he is probably right
in pointing to the interrupt handling code rhather than the bridging
code. If the 4801 has the "sis" driver, the following lines in
/sys/pci/if_sis.c: sis_intr() might be the cause of the problem:
/* Supress unwanted interrupts */
if (!(ifp->if_flags & IFF_UP)) {
sis_stop(sc);
return;
}
If you have shared interrupts, and one of the interfaces is not up, you
end up doing a lot of useless calls to sis_stop(), which is terribly
expensive (it even includes a DELAY(1000) call).
At the very least, one should add a 'stopped' flag so that
sis_stop() is only called when necessary -- or possibly even
never at all.
The second (minor) cause of performance loss are the calls
(not shown above)
to disable and re-enable the interrupts around the driver. These are
completely useless, yet cause some extra PCI transactions.
If my diagnosis is correct, you should be able to fix the performance
problem by either doing an 'ifconfig sisX up' on the interface
that is idle, or by putting '#ifdef 0 ... #endif' around the block
above.
Another way to test is to use a kernel with DEVICE_POLLING and
enable it with sysctl kern.polling.enable=1
I believe there are several drivers which share the same structure
hence have the same potential problem.
cheers
luigi
On Wed, Dec 24, 2003 at 03:56:16PM -0800, Michael Sierchio wrote:
...
> Robert -
>
> when digging into the bridging code, you may want to look
> at an apparent performance bug. This was discovered by
> Soekris users -- when two of three interfaces were configured
> in the bridge, performance was significantly less than when
> all interfaces were, even if one interface was unused/uncabled.
> From: Soren Kristensen <soren@soekris.com>
> Organization: Soekris Engineering
> To: timg@tpi.com
> CC: Soekris-tech <soekris-tech@lists.soekris.com>
> Subject: Re: [Soekris] Slow net speed on Net4801
>
>
> I just did a little testing on the net4801 ethernet performance, as I
> wanted to make sure there wasn't any hardware problems.
>
> I wanted to be sure as the Geode used to have some serious PCI bus
> performance issues, but those problems should be fixed on the SC1100 if
> you program it up correctly. Also, one difference between the net4501
> and net4801 is that the 3 ethernet controllers now share one interrupt.
>
> As I had limited interrupts available on the SC1100, I decided that I
> would prefer that chips using the same drivers are sharing rather than
> random chips used on the expansion slots.... Sharing interrupts will
> reduce performance a little, but not very much on a correct implemented
> shared interrupt system.
>
> So I set up a full FreeBSD 4.9 Release (on one of those nice new
> 2.2Gbyte Microdrives....) and enabled bridging.
>
> I quickly found performance problems, but after testing I now believe I
> instead found a bug in FreeBSD interrupt code....
>
> At first I got about 35 Mbit/sec with 99% interrupt time.
> I then tried on a net4501 and got 50 Mbit/sec with 60% interrupt time.
>
> But after some testing I found out that when I on the net4801 configured
> all 3 ethernet controllers for bridging instead of the first 2 as I
> started with, the net4801 got 50 Mbit/sec with 40% interrupt time, much
> better and beating the net4501 as you would expect. You should also
> expect higher raw speed, but my test setup was limited to 50 Mbit/sec.
> (hand timing filecopy in msdos window on win2000....).
>
> As soon as I set net.link.ether.bridge_cfg=sis0,sis1,sis2, performance
> was much better on the net4801, but didn't affect the net4501.
>
> So the conclusion so far is that you should be able to get very good
> performance, but the ethernet drivers and operating systems need to
> handle shared interrupts correctly.
>
> Apperently there is a problem with FreeBSD when you have 3 ethernet
> controllers sharing one interrupt and only bridging 2 of them. When
> doing that, the processor spend a lot of time in interrupt....
>
> Can one of the FreeBSD gurus (Poul-Henning ?) look into the FreeBSD
> interrupt and/or bridging code ?
>
>
> Regards,
>
>
> Soren Kristensen
>
> _____________________________________________________________________
> Soekris Engineering, technical discussion mailing list
> [un]subscribe: http://lists.soekris.com/mailman/listinfo/soekris-tech
>
>
>
> _______________________________________________
> freebsd-net@freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-net
> To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org"
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20031225102549.A79161>