Date: Sat, 29 Sep 2001 21:27:46 -0700 From: Terry Lambert <tlambert2@mindspring.com> To: "Kenneth D. Merry" <ken@kdm.org> Cc: Jonathan Lemon <jlemon@flugsvamp.com>, the_srinivas@hotmail.com, hackers@FreeBSD.ORG Subject: Re: TCP&IP cksum offload on FreeBSD 4.2 Message-ID: <3BB69F42.5C1ED705@mindspring.com> References: <200109270157.f8R1vZ546863@prism.flugsvamp.com> <3BB42E50.B32F5E95@mindspring.com> <20010928074347.A36960@panzer.kdm.org> <3BB61A6B.A97E955A@mindspring.com> <20010929153932.A50715@panzer.kdm.org>
next in thread | previous in thread | raw e-mail | index | archive | help
"Kenneth D. Merry" wrote: > > Unfortunately, it can not correctly interoperate with a > > number of cards in jumbogram mode, so unless you know the > > card on the other end and manually configure it (it can't > > negotiate properly), you can't really use jumbograms. Or > > you could rewrite the firmware (of course). > > Huh? Care to explain? Why wouldn't it interoperate with jumbo frames? > Why are jumbo frames something that need be negotiated by the card? > They're not negotiated by the card, but rather by the TCP stack. Because at the time the Tigon II was released, the jumbogram wire format had not solidified. Therefore cards built during that time used different wire data for the jumbogram framing. > > I have never seen the board with the stock FreeBSD driver > > get better than about half wire rate with normal size > > packets and window depth. > > Yep, it doesn't handle large numbers of small packets very well. You can > easily get wire rate with jumbo frames, though. Again, that's not useful, since it requires hand negotiation; you can't automatically set the MTU up as a result of doing a driver level negotiation with the other end of the link (unless the other end is also a Tigon II). > > On the other hand, the Tigon III > > is capable of 960 megabits -- about the wire rate limit -- > > with normal size packets, if you implement software interrupt > > coelescing (which doesn't help, unless you crank the load up > > to close to wire speed and/or do more of the stack processing > > at interrupt time). > > > > The Tigon II also has the unfortunate need to download its > > firmware, and a FreeBSD bug -- the lack of a distinct state > > for firmware download -- means that the firmware get sent > > to the card each time the thing is config'ed up, or an IP > > alias is added to the thing. Plus the damn things are more > > expensive than the Tigon III based cards. > > That's partially because Bill wanted to have a process context > when he was downloading the firmware, and didn't want to do a > kernel thread to do it. Bill and I discussed this at length, in person. The problem is that FreeBSD resets the ethernet cards, for fear that it is the user's intent to recover a hung card tis way. The firmware download issue is because there is not a seperate entry point for firmware download, combined with the initial gratuitous ARP request being placed on the transmit queue for the card before it is really up, and the only other alternative place to hook the firmware download meant that you would have to hang the ifconfig command -- nad therefore the /etc/rc, and therefore the boot process -- waiting for th download to complete. The upshot of this is that adding a virtual IP address to the interface is assumed to be a very infrequent event. This assumption is incorrect for a large number of IP takeover protocols (e.g. VRRP), whose timing granularity is small enough that the firmware download exceeds the interval by a factor of 10 or more. By the time the alias address assignment (and the concommitant firmware download) are completed, the takeover window has been greatly exceeded. For all three of the takeover protocols for which there is published documentation available, this results in the address being configured on, the firmware downloaded, the takeover window being missed, losing the contention race for the takeover, and the address being configured off -- and the firmware being downloaded yet again. Another "amusing" consequence is that, if there is traffic at the time of the event, the driver goes into a perpetual "watchdog timeout: resetting" loop, which can oly be resolved by a system reboot. > So it gets done when the card is ifconfiged up. I wrote some code for my > former employer to wait up to 10 seconds for link to show up when the board > is configured. The alternative is a 'sleep 5' (for fiber boards) or 'sleep > 8' (for copper boards) after it is ifconfiged up. The kernel code is trivial, but it's not in there by default, and wouldn't need to be there, if there were a firmware download phase for the card which could be triggered to occur asynchronously (e.g. so that the driver did not hand user space in the attach). The "sleep" workaround doesn't really work, unless you're willing to hack up your rc files to deal correctly with a mix of cards; even so, then it adds an additional ~10 seconds per ti interface to the boot process, and to any reconfiguration process that can occur later (and later reconfiguration processes risk going into the "watchdog timeout: resetting" infinite loop, if there are any proceses with network links active at the time -- something which is almost inevitable). > Having downloadable firmware is actually a huge advantage. You can do > things with the Tigon II that just aren't possible with any other chip, > either because they don't have downloadable firmware, or because the vendor > won't release firmware source. I agree; I'm well aware of a number of people who have done things to the firmware. The problem is the FreeBSD ethernet driver architecture doesn't provide for an entry point hook for asynchronous firmware download. > This is a problem with the Broadcom Tigon III boards, and to some extent > with the Tigon II. Basically it looks like the firmware for the Tigon II > is very hard to get now that 3Com has control of it, and I don't think > Broadcom will release the Tigon III firmware. (Assuming it is a > firmware-based chip.) The Tigon II firmware is available from their web site. You can license the Tigon III firmware, but I agree, that it is definitely a step backward, compared to general availability. > > To put a final nail in the coffin, the card can't offload > > TCP checksum with VLANs enabled (it breaks). > > Probably something that could be fixed in the firmware. Too bad > the vendor isn't really supporting it very much now. The card has been end of lifed. > > All in all, the Tigon II is a bad board. > > That's your opinion, but I disagree. I can more than double my connections per second, and triple or quadruple my overall throughput by switching from a Tigon II to a Tigon III, with no other changes. The Tigon III is the first board to be able to do Gigabit wire rate with normal sized packets. -- Terry To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-hackers" in the body of the message
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?3BB69F42.5C1ED705>