Date: Sun, 2 Mar 2008 03:12:08 +0100 From: Erik Trulsson <ertr1013@student.uu.se> To: Barney Cordoba <barney_cordoba@yahoo.com> Cc: net@freebsd.org Subject: Re: FBSD 1GBit router? Message-ID: <20080302021208.GA87031@owl.midgard.homeip.net> In-Reply-To: <638970.20021.qm@web63906.mail.re1.yahoo.com> References: <20080301225727.GA85851@owl.midgard.homeip.net> <638970.20021.qm@web63906.mail.re1.yahoo.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On Sat, Mar 01, 2008 at 04:39:57PM -0800, Barney Cordoba wrote: > > --- Erik Trulsson <ertr1013@student.uu.se> wrote: > > > On Sat, Mar 01, 2008 at 01:27:46PM -0800, Barney > > Cordoba wrote: > > > > > > --- Ingo Flaschberger <if@xip.at> wrote: > > > > > > > Dear Barney, > > > > > > > > > It seems absolutely ridiculous to buy such > > > > hardware > > > > > and not install a PCIx or 4x PCIe card for > > another > > > > > $100. or less. Saying a 1x is "fast enough" is > > > > like > > > > > saying a Celeron is "fast enough". > > > > > > > > The box is a small 1HE appliance and can boot > > from a > > > > CF-Card. > > > > I trust them more than a "al cheapo" pc. > > > > 1x axiomtek NA-820 > > > > 1x P4 3Ghz cpu > > > > 1x 1gb ddr2 > > > > --- > > > > 850eur without taxes. > > > > > > > > A good chipset, good cpu, good ram, good > > harddisk, > > > > god powersupply has > > > > same price. > > > > And don't forget that in exchanges you pay for > > each > > > > HE. > > > > > > > > And back to 1x is not fast enough: > > > > There are no 1gbit single port network cards > > that > > > > support more than 1 > > > > lane, even if you plug it into a 16 lane slot. > > > > (and I'm not talking about 10gbit cards; if you > > have > > > > 10gbit upstream you > > > > have enough $$ to buy good gear) > > > > > > Ok, well I've never seen a router with 1 port. I > > > thought we were talking about building a router? > > > > He did not say anything about a single port router. > > He talked about single port network cards. You can > > use more than one of them when building a router. > > His argument is that there are only 1x PCIe cards that > have 1 port. Since he needs 2 ports, and there are 2 > port > PCIe cards, then his argument makes no sense. But the > point is that PCIe NICs are implemented with 1 port > per lane in the chip. So a 2 port card will use 2 > lanes No, the argument was that all 1-port PCI-E cards are only x1, ergo a x1 card must be fast enough for 1 port, else there would be 1-port cards manufactured that used more than one lane. > > > > > > > > > The lack of PCIe cards is a good reason to > > consider a > > > PCIX machine. > > > > What lack of PCI-E cards? These days there are > > quite a > > few to choose between. > > Yes, but they are all 1x, while there are many 1 and 2 > port PCIx cards which are twice as fast. There are dual- and quad-port PCI-E cards available too, and they are generally x4 lane models. Today there really is very little reason to use PCI-X instead of PCI-E when one is putting together a brand new system. > > > > > > On the systems that we have, the 1x PCIe > > > ports are a lot slower than a PCI-X card in the > > slot. > > > > > > You need 4Gb/s of throughput to handle a gigablt > > > router. (1 GB/s full duplex times 2). 1x is 4Gb/s > > > maximum. In my view, you always need twice the > > > bandwidth on the bus to avoid contention issues. > > > > What contention issues? With PCI-E each device is > > essentially on its own > > bus and does not need to contend with other devices > > for bandwidth on that > > bus. > > > > A PCI-E 1x connection provides more bandwidth than > > one gigabit ethernet > > connection can use. > > Does each PCIe slot have its own dedicated memory > controller? The concept that there is some sort of > mutually exclusive, independent path for each > controller is simply not the case in practice. You're > accessing the same memory; you're going through > shared hubs and bridges. You're doing I/O on the same > bus. North bridges typically have a 512 Byte payload > maximum, so you can't even burst a full packet. You > have transaction overhead for each transfer. There are > many factors that will chip away at your realizable > bandwidth. Its not like a hose gushing a continuos > stream of water. Sure, but those limitations apply equally regardless of what kind of slots you use. A x8 lane PCI-E card will suffer just as much of contention on the FSB between the CPU and the chipset as a x1 lane card will. Having a wider channel between the NIC and the chipset will not help if the bottleneck is elswehere. > > Another factor is that "server" chipsets do PCIe > better than "desktop" chipsets. Server chipsets are > optimized in the chipset to handle multiple devices > more efficiently. So don't expect your "desktop" > chipset to be as efficient as a server chipset at the > same task. There's a reason that intel has desktop and > server chipsets. Intel uses the same southbridge chips for all their single-CPU chipsets, regardless of whether the nortbridge is labeled as a server or desktop part. Most I/O-devices are normally connected to the southbridge. > > I've tested dual port cards based on the 82546 and > 82571 parts on the same system that has both PCIX and > PCIe slots. The parts are essentially the same, and > the driver is essentially the same, with the bus being > the only real difference. The PCIe card is a 2x card. PCI-E cards and slots only come in 4 physical sizes: x1, x4, x8, and x16. It is not uncommon for a slot to have fewer lanes attached to it than the physical size allows, so there do exist some 'x2' slots. I have however never seen any PCI-E cards being sold as x2, only as one of the standard widths. > so its the equivalent of 2 PCIe cards with dedicated > 1x lanes. The results are the that PCIx card is simply > more efficient, in that is uses less CPU with the same > load (an indication of less contention for I/O), and > the PCIX card has a higher capacity (that is, the > point in which the cpu is saturated is higher by about > 10%) On that particular system, with those particular cards, it might well be the case that the PCI-X card performed better than the PCI-E card. That says essentially nothing about the relative performance of PCI-X vs PCI-E in general, and I doubt that there was any lack of bandwidth between either of the cards and the chipset anyway. How the different buses and chips are connected on the motherboard can cause quite a bit of difference. If, for example, the PCI-X bus was connected to the Northbridge of the chipset, while the PCI-E x1 slots were connected to the Southbridge, and there was a fairly narrow connection (equivalent to a x4 lane PCI-E bus) between the north- and south-bridge, then you could well get the results you saw. On another motherboard, with a different topology, you might get the opposite results from those same cards. > > In practice your little box is never going to be > routing bi-direction, full gigabit traffic, but if you > only have "just enough" bandwidth, your CPU is going > to get less and less efficient as you get higher and > higher usage. You'll need more cpu to do the same task > with less bus. But a x1 lane PCI-E connection does not provide "just enough" bandwidth for a gigabit ethernet connection. It provides "more than enough" bandwidth. (The theoretical bandwidth of such a x1 lane is twice what is needed for the ethernet connection, which is quite enough even taking into account various overheads.) > > I'm not claiming that PCIX is faster than pcie at the > same "speed", but that the limitation of 1x per NIC > causes PCIX to be a better choice for a 2 port system > in most cases. There is no inherent "limitation of 1x per NIC", but even if there was this is not a problem as long as you do not put more than 1 port per NIC. -- <Insert your favourite quote here.> Erik Trulsson ertr1013@student.uu.se
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20080302021208.GA87031>